Agent task learning method and device

Abstract

The present invention provides an agent task learning method and device, the method comprising: based on the natural policy gradient algorithm, taking the state information of the agent under each historical task as a sample, taking the action information of the agent under each historical task as a sample label, iteratively updating the first policy parameter of the action prediction model, obtaining the optimal first policy parameter of the action prediction model corresponding to each historical task; taking the key influencing factors corresponding to all historical tasks and the optimal first policy parameters corresponding to all historical tasks as a training sample set, iteratively training the task learning model; inputting the key influencing factors corresponding to the new task into the trained task learning model, obtaining the optimal second policy parameter of the action prediction model corresponding to the new task output by the task learning model. The present invention enables the agent to have both the general knowledge extraction capability of the domain task and the zero-sample strategy generation capability for the new task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent agent task learning method and device. Background Art

[0002] The core of general artificial intelligence algorithm research is to add intelligent algorithms to the intelligent body so that the intelligent body has human-like intelligence. After the intelligent body has human-like intelligence, it will be able to complete the production work that humans are engaged in, thereby liberating social productivity. Therefore, general artificial intelligence algorithm research has great application potential and value.

[0003] The strategic knowledge extraction algorithm in the lifelong learning framework is a typical general artificial intelligence algorithm, which includes an algorithm framework for cognitive functions of learning and understanding information. It consists of two parts: a learning module and a knowledge module. The function of the learning module is to enable the intelligent agent to have the ability to learn tasks, and the function of the knowledge module is to enable the intelligent agent to have the ability to extract general knowledge, so as to quickly learn and master new tasks.

[0004] The existing policy knowledge extraction algorithm uses PGELLA (Policy Gradient Efficient Life-Long Learning Algorithm) in online multi-task learning based on the policy gradient method. It extracts shared knowledge between different task strategies to form a shared base; then, it transfers knowledge from the shared base and learns small samples to achieve rapid learning of new tasks, so as to achieve rapid adaptability to different tasks in the same field. However, there is usually a shortage of samples for new tasks, and it is difficult to quickly train and optimize effective learning models that meet the new tasks, resulting in poor performance of the intelligent agent in extracting general knowledge. Therefore, how to enable the intelligent agent to have good performance in extracting general knowledge in the case of zero samples is an urgent problem to be solved in this field. Summary of the invention

[0005] The present invention provides an intelligent agent task learning method and device to solve the problem that there is a shortage of samples of new tasks in the prior art, and it is difficult to quickly train and optimize effective learning models that meet the new tasks, resulting in poor performance of the intelligent agent in extracting general knowledge. The method and device can achieve the goal of enabling the intelligent agent to have good performance in extracting general knowledge under zero sample conditions.

[0006] The present invention provides an agent task learning method, comprising:

[0007] Based on the natural policy gradient algorithm, the state information of the agent under each historical task is used as a sample, and the action information of the agent under each historical task is used as a sample label, and the first policy parameter of the action prediction model is iteratively updated to obtain the optimal first policy parameter of the action prediction model corresponding to each historical task;

[0008] The key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks are used as training sample sets to iteratively train the task learning model; wherein the key influencing factors are factors that affect the output results of the action prediction model;

[0009] The key influencing factors corresponding to the new task are input into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0010] According to an agent task learning method provided by the present invention, the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks are used as training sample sets to iteratively train the task learning model, including:

[0011] The historical tasks whose reward values ​​corresponding to the optimal first strategy parameters are greater than a set threshold are used as target historical tasks;

[0012] The key influencing factors corresponding to all target historical tasks and the optimal first strategy parameters corresponding to all target historical tasks are used as the training sample set to iteratively train the task learning model.

[0013] According to an agent task learning method provided by the present invention, the task learning model includes a plurality of subtask learning models;

[0014] Each subtask learning model is used to learn the mapping relationship between the key influencing factors and each optimal first strategy parameter;

[0015] The loss function of the task learning model is the joint result of the loss functions of multiple subtask learning models.

[0016] According to an agent task learning method provided by the present invention, the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks are used as training sample sets to iteratively train the task learning model, including:

[0017] Vectorize the optimal first strategy parameters corresponding to each historical task;

[0018] The key influencing factors corresponding to all historical tasks and the optimal first strategy parameters after vectorization processing corresponding to all historical tasks are used as the training sample set to iteratively train the task learning model.

[0019] According to an agent task learning method provided by the present invention, the iterative updating of the first strategy parameter of the action prediction model includes:

[0020] Iteratively updating the first strategy parameter according to the attenuation coefficient and Fisher information corresponding to the first strategy parameter, and the gradient of the first strategy parameter and the objective function of the action prediction model;

[0021] Among them, the attenuation coefficient corresponding to the first strategy parameter is determined according to the Fisher information corresponding to the first strategy parameter, the gradient of the objective function of the action prediction model and the preset constraint factor.

[0022] According to an agent task learning method provided by the present invention, the action prediction model includes a linear matrix or a multi-layer neural network.

[0023] According to an agent task learning method provided by the present invention, after inputting the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model, the method further includes:

[0024] Optimizing the action prediction model according to the optimal second strategy parameter to obtain an optimized action prediction model;

[0025] The state information of the agent under the new task is input into the optimized action prediction model to obtain the action information of the agent under the new task output by the optimized action prediction model.

[0026] The present invention also provides an intelligent agent task learning device, comprising:

[0027] A learning module, which is used to iteratively update the first policy parameter of the action prediction model based on the natural policy gradient algorithm, using the state information of the agent under each historical task as a sample and the action information of the agent under each historical task as a sample label, so as to obtain the optimal first policy parameter of the action prediction model corresponding to each historical task;

[0028] The first knowledge module is used to iteratively train the task learning model by using the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as a training sample set; wherein the key influencing factors are factors that have an impact on the output result of the action prediction model;

[0029] The second knowledge module is used to input the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described intelligent agent task learning methods are implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described intelligent agent task learning methods.

[0032] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned intelligent agent task learning methods.

[0033] The intelligent agent task learning method and device provided by the present invention use the natural policy gradient algorithm to take the state information and action information of the intelligent agent under various historical tasks as samples, and iteratively train the action prediction model in the learning module in the policy knowledge extraction algorithm framework to obtain the optimal first policy parameters of the action prediction model under various historical tasks; then, take the key influencing factors and the optimal first policy parameters under various historical tasks as sample pairs, and iteratively train the task learning model in the knowledge module in the policy knowledge extraction algorithm framework to obtain the optimal task learning model with general knowledge under different tasks; and only need to input the key influencing factors under the new task into the optimal task learning model to obtain the optimal action prediction model under the new task, so as to quickly adapt to the new task without learning the samples under the new task, and the learning efficiency of the new task is higher, and it can effectively realize that the knowledge module has both the general knowledge extraction ability of the domain task and the zero-sample policy generation ability for the new task, and the refined general knowledge has stronger versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is one of the flow charts of the agent task learning method provided by the present invention;

[0036] Figure 2 It is a structural diagram of a knowledge module and a learning module in the agent task learning method provided by the present invention;

[0037] Figure 3It is one of the structural schematic diagrams of the simulation results in the agent task learning method provided by the present invention;

[0038] Figure 4 This is the second structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0039] Figure 5 This is the third structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0040] Figure 6 This is the fourth structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0041] Figure 7 This is the fifth structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0042] Figure 8 This is the sixth structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0043] Fig. 9 This is the seventh structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0044] Fig.10 This is the eighth structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0045] Fig.11 It is the ninth structural diagram of the simulation result in the agent task learning method provided by the present invention;

[0046] Fig.12 This is the second flow chart of the agent task learning method provided by the present invention;

[0047] Fig.13 It is a structural schematic diagram of the intelligent agent task learning device provided by the present invention;

[0048] Fig.14 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The agent task learning method in this embodiment is an improved strategy knowledge extraction algorithm, which includes two parts: a learning module and a knowledge module. The knowledge module can extract common knowledge of different tasks in the same task domain. On this basis, the strategy parameters of the new task can be adjusted according to the key influencing factors that affect the action strategy, that is, the factors that affect the output results of the action prediction model, so as to realize the knowledge module's zero-sample generation and application of strategies for new tasks.

[0051] Combine the following Figure 1 The agent task learning method of the present invention is described, and the method includes: step 101, based on the natural policy gradient algorithm, using the state information of the agent under each historical task as a sample, using the action information of the agent under each historical task as a sample label, iteratively updating the first policy parameter of the action prediction model, and obtaining the optimal first policy parameter of the action prediction model corresponding to each historical task;

[0052] The execution subject in this embodiment is the processor inside the intelligent body;

[0053] The intelligent agent may be a robot, a ground unmanned platform, a drone, etc., which is not specifically limited in this embodiment.

[0054] The status information includes, but is not limited to, position status information, direction status information, body status information of each component of the intelligent body, and action status information, such as joint status information of the robot, etc. This embodiment does not make specific limitations on this.

[0055] Action information is information that controls the robot to perform corresponding actions.

[0056] like Figure 2 As shown, the strategy knowledge extraction algorithm framework consists of two parts: learning module and knowledge module;

[0057] Among them, the action prediction model is an algorithm in the learning module, which is used to learn the mapping relationship between the state information and action information of the intelligent agent under different historical tasks.

[0058] The task learning model is an algorithm in the knowledge module, which is used to learn the mapping relationship between the key influencing factors corresponding to different historical tasks and the optimal first strategy parameters corresponding to different historical tasks.

[0059] It should be noted that the training processes of the learning module and the knowledge module are carried out independently, and the general knowledge extracted by the knowledge module can be applied to the learning module that is trained and learned using different reinforcement learning algorithms, that is, to regulate the strategy of the action prediction model in the learning module.

[0060] The historical tasks are a variety of different learning tasks in the same field. The number and type of historical tasks can be set according to actual needs, and this embodiment does not specifically limit this.

[0061] The NPG (Natural Policy Gradient) algorithm is a reinforcement learning method that learns based on strategies and is used to obtain the optimal strategy parameters.

[0062] Optionally, the action prediction model, i.e., the task policy π, is trained using the natural policy gradient method in the reinforcement learning algorithm. t (s|a) for training and learning, where t is the tth historical task and the task strategy π t It can be a linear matrix or a multi-layer neural network, etc., which is not specifically limited in this embodiment; s and a are the state information and action information of the agent under the tth historical task respectively.

[0063] Among them, the task strategy π t The corresponding first strategy parameter can be expressed as θ t =(θ t1 ,θ t2 ,…,θ tm ) T .

[0064] Based on the natural policy gradient method, the state information of the agent under each historical task is used as a sample, and the action information of the agent under each historical task is used as a sample label. The first policy parameter of the action prediction model is iteratively updated until the preset termination condition of the action prediction model is met, so as to obtain the optimal first policy parameter of the action prediction model corresponding to each historical task.

[0065] The preset termination condition may be reaching a maximum number of iterations and / or convergence of the action prediction model;

[0066] The maximum number of iterations can be set according to actual needs, and this embodiment does not specifically limit this, such as 200 rounds.

[0067] Step 102, using the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as a training sample set, and iteratively training the task learning model; wherein the key influencing factors are factors that have an impact on the output result of the action prediction model;

[0068] Among them, the task learning model can learn to refine the common knowledge of different tasks in the same task domain, and realize zero-sample generation and application of strategies for new tasks based on the common knowledge.

[0069] It should be noted that since the task learning model can extract shared knowledge between different task strategies, and after extracting the shared knowledge, it can apply knowledge to other similar situations and directly adapt to new tasks in this field without any learning; therefore, this embodiment calls the task learning model an artificial metacognitive neural network.

[0070] The task learning model may include one or more subtask learning models, which is not specifically limited in this embodiment.

[0071] Optionally, when the task learning model includes only one subtask learning model, a plurality of optimal first strategy parameters are fully connected for learning and training through the subtask learning model, which is referred to as a fully connected training task learning model in this example;

[0072] When the task learning model includes multiple subtask learning models, multiple optimal first strategy parameters are trained by single-point learning through the multiple subtask learning models. This example refers to the single-point training task learning model.

[0073] This embodiment may adopt any of the above methods to train and obtain the optimal task learning model, and this embodiment does not specifically limit this.

[0074] Optionally, after obtaining the optimal first strategy parameters of the action prediction model corresponding to each historical task, the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks can be directly used as a training sample set; or the optimal first strategy parameters corresponding to the historical tasks can be screened, and the key influencing factors and optimal first strategy parameters corresponding to the screened historical tasks can be used as a training sample set. This embodiment does not specifically limit this.

[0075] Then, the task learning model is iteratively trained based on the training sample set;

[0076] Optionally, the task learning model may be iteratively trained using the training sample set directly, or the task learning model may be iteratively trained after preprocessing the training sample set, which is not specifically limited in this embodiment.

[0077] Optionally, the key influencing factors and the optimal first strategy parameters corresponding to each historical task are used as input-output training sample pairs of the task learning model to extract common knowledge of different task strategies in the same field, and the task learning network is trained using the back propagation algorithm, and the mean square error is used as the loss function to obtain the optimal task learning network.

[0078] Step 103: input the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0079] Optionally, when a new task needs to be learned, the key influencing factors corresponding to the new task are directly input into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task;

[0080] By using the optimal second strategy parameter to adjust the strategy parameters of the action prediction model, the optimal action prediction model applicable to the new task can be obtained. Based on the optimal action prediction model, the action information prediction of the intelligent agent under the new task can be realized.

[0081] The following describes the agent task learning method in this embodiment by taking 100 different historical tasks and 1 new task of the robot in the jumper task domain as an example.

[0082] Among them, the state information is the joint state information of the robot; the action information is the execution action corresponding to the joint state information;

[0083] Firstly, the joint state information of the robot in 100 different historical tasks is used as samples, and the action information corresponding to the joint state is used as the sample label. The first strategy parameter of the action prediction model is iteratively updated to obtain the optimal first strategy parameter of the action prediction model in 100 different historical tasks.

[0084] Then, the task learning model is iteratively trained based on the optimal first strategy parameters and key influencing factors of the action prediction model under 100 different historical tasks as training samples to obtain the optimal task learning model;

[0085] The key influencing factors corresponding to the new task are input into the optimal task learning model to obtain the optimal second strategy parameters under the new task. The action prediction model is adjusted according to the optimal second strategy parameters to obtain the optimal action prediction model under the new task.

[0086] When it is necessary to predict the motion information of the robot under a new task, the joint state information of the robot under the new task is input into the optimal motion prediction model under the new task to obtain the motion information of the robot under the new task.

[0087] The existing technology uses the PGELLA algorithm to extract shared knowledge between different task strategies, and then uses small samples to quickly learn new tasks; however, the knowledge modules in this strategy knowledge extraction algorithm do not have the human-like ability to apply knowledge to other similar tasks, that is, they cannot directly adapt to new tasks in this field without any learning.

[0088] The intelligent agent task learning method of this embodiment adapts to and masters new tasks by regulating the action strategy through the key influencing factors of the environment, so that the knowledge module has both the ability to extract general knowledge of domain tasks and the ability to generate zero-sample strategies, and the extracted general knowledge has stronger versatility; and there is no need to store specific knowledge of different tasks, so the increase in learning tasks will not increase the space storage cost of the knowledge base.

[0089] This embodiment uses the natural policy gradient algorithm to take the state information and action information of the intelligent agent under various historical tasks as samples, and iteratively trains the action prediction model in the learning module in the policy knowledge extraction algorithm framework to obtain the optimal first policy parameters of the action prediction model under various historical tasks; then, the key influencing factors and the optimal first policy parameters under various historical tasks are used as sample pairs to iteratively train the task learning model in the knowledge module in the policy knowledge extraction algorithm framework to obtain the optimal task learning model with general knowledge under different tasks; and the optimal action prediction model under the new task can be obtained by simply inputting the key influencing factors under the new task into the optimal task learning model, so as to quickly adapt to the new task without learning the samples under the new task, and the learning efficiency of the new task is higher, and it can effectively realize that the knowledge module has both the general knowledge extraction ability of the domain task and the zero-sample strategy generation ability for the new task, and the refined general knowledge has stronger versatility.

[0090] On the basis of the above embodiments, the present embodiment uses the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as training sample sets to iteratively train the task learning model, including: using the historical tasks whose reward values ​​corresponding to the optimal first strategy parameters are greater than a set threshold as the target historical tasks; using the key influencing factors corresponding to all target historical tasks and the optimal first strategy parameters corresponding to all target historical tasks as the training sample sets to iteratively train the task learning model.

[0091] The reward value corresponding to the optimal first strategy parameter is used to characterize the performance of the action prediction model corresponding to the optimal first strategy parameter. The higher the reward value, the better the performance of the action prediction model.

[0092] Action prediction models include linear matrices or multi-layer neural networks;

[0093] The threshold value may be set according to actual needs. For example, the threshold value of the reward value set by the linear matrix task strategy is 2500, and the threshold value of the reward value set by the neural network task strategy is 3000. This embodiment does not specifically limit this.

[0094] In order to ensure that the performance of the action prediction model corresponding to each optimal first strategy parameter is good, thereby avoiding affecting the training of the task learning model, in the process of training the task learning model, this embodiment first screens the optimal first strategy parameters of the action prediction model corresponding to each historical task, and selects the historical tasks whose reward values ​​corresponding to the optimal first strategy parameters are greater than the set threshold, and uses them as the target historical tasks;

[0095] Then, the key influencing factor set X = {x1, x2, …, x k} and the optimal first strategy parameter set corresponding to all target historical tasks The task learning model is iteratively trained using the training sample set to obtain an optimal task learning model.

[0096] In the task learning model training process of this embodiment, the optimal first strategy parameters of the action prediction model corresponding to each historical task are first screened, and the historical tasks whose reward values ​​corresponding to the optimal first strategy parameters are greater than the set threshold are screened out to ensure that the performance of the action prediction model corresponding to each screened out optimal first strategy parameter is good; then, the task learning model is iteratively trained according to the screened out optimal first strategy parameters, so that the trained task learning model has better performance in extracting general knowledge for tasks in the same field.

[0097] Based on the above embodiments, the task learning model described in this embodiment includes multiple sub-task learning models; each sub-task learning model is used to learn the mapping relationship between the key influencing factors and each optimal first strategy parameter; the loss function of the task learning model is the joint result of the loss functions of multiple sub-task learning models.

[0098] Among them, the network structure of the task learning model can be expressed as:

[0099]

[0100] Where θ is the optimal first strategy parameter corresponding to the historical task, F(·) is the network structure of the task learning model, and is the mapping function from the key influencing factor q to the optimal first strategy parameter θ; q is the key influencing factor corresponding to the historical task, including the agent ontology environment parameters and interaction environment parameters used to regulate the action strategy, which have an impact on the output results of the action prediction model.

[0101] Compared with the commonly used fully connected network connection mode, the task learning model in this embodiment includes multiple sub-task learning models. During the training process, each sub-task learning model is trained to learn the nonlinear relationship between the key influencing factors and each optimal first strategy parameter. This training method is called single-point training.

[0102] Among them, the task learning model under single-point training contains multiple sub-task learning models f i (·), respectively for different optimal first strategy parameters θ i Conduct joint training;

[0103] Among them, the network structure of the task learning model under single-point training can be expressed as:

[0104]

[0105] During the training process, the knowledge module uses the back propagation algorithm to train the task learning model, and combines the mean square error of all subtask learning models as the loss function of the task learning model. The specific calculation formula is:

[0106]

[0107] Among them, MSE is the loss function of the task learning model, and m is the number of subtask learning models, that is, the number of optimal first strategy parameters.

[0108] This example uses multiple subtask learning models and trains different subtask learning models separately to accurately obtain the nonlinear relationship between the key influencing factors and each optimal first strategy parameter. This single-point training method can accurately learn the nonlinear relationship between the key influencing factors and each optimal first strategy parameter, which can not only make the task learning model more accurate and reliable, but also reduce the number of training sample sets. Since the time cost of training effective task strategies in the learning module is high, the reduction in the number of strategy samples required will greatly reduce the time spent on collecting strategy samples, thereby reducing training costs.

[0109] On the basis of the above embodiments, the present embodiment uses the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as training sample sets to iteratively train the task learning model, including: vectorizing the optimal first strategy parameters corresponding to each historical task; using the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters after vectorization processing corresponding to all historical tasks as the training sample set to iteratively train the task learning model.

[0110] Optionally, before iteratively training the task learning model, it is necessary to vectorize the optimal first strategy parameters corresponding to each historical task to obtain a one-dimensional vector;

[0111] That is, the optimal first strategy parameter of the action prediction model is vectorized into a vector y of mx1 dimensions t , the sample label set in the training sample set is obtained as Y = {y1,y2,…,y k}; where m is the number of optimal first strategy parameters. where k is the number of historical tasks.

[0112] For example, if the action prediction model is a 5x10x10x3 multi-layer neural network, the number of optimal first strategy parameters m is 5x10+10x10+10x3=180, and the vectorized y t is a 180x1 dimensional vector.

[0113] Then, the key influencing factors corresponding to each historical task and the optimal first strategy parameters after vectorization processing corresponding to each historical task are used as sample pairs to construct a training sample set;

[0114] The task learning model is iteratively trained using the training sample set to extract common knowledge of different task strategies in the same field and obtain the optimal task learning model.

[0115] This embodiment vectorizes the optimal first strategy parameters corresponding to each historical task and then iteratively trains the task learning model as a training sample set to avoid the impact of the dimension of the optimal first strategy parameters on the training of the task learning model, so as to quickly and accurately obtain the optimal task learning model.

[0116] On the basis of the above embodiments, the iterative update of the first strategy parameter of the action prediction model described in this embodiment includes: iteratively updating the first strategy parameter according to the attenuation coefficient and Fisher information corresponding to the first strategy parameter, and the first strategy parameter and the gradient of the objective function of the action prediction model; wherein the attenuation coefficient corresponding to the first strategy parameter is determined according to the Fisher information corresponding to the first strategy parameter, the gradient of the objective function of the action prediction model and a preset constraint factor.

[0117] Optionally, the policy knowledge extraction algorithm framework includes two parts: a learning module and a knowledge module, such as Figure 2 As shown, the learning module uses NPG to learn the optimal first strategy parameters corresponding to different tasks. The basis of the natural policy gradient algorithm is the policy gradient algorithm, which defines the first policy parameter vector To represent the first strategy π, that is, the set of all θ is the first strategy π;

[0118] The maximum average reward value obtained from the trajectory is used as the objective function to optimize the first strategy parameter of the action prediction model, where the calculation formula of the objective function is:

[0119]

[0120] in, is the average reward value of the action trajectory τ, represents the set of all possible action trajectories, p θ (τ) is the probability of the trajectory τ appearing when the first strategy parameter θ is executed; r n+1 is the reward value of the nth action executed under the state information s.

[0121] Optionally, in order to make the first policy parameter after gradient update closer to the policy parameter before update, that is, to avoid the update amplitude being too large and the inability to accurately obtain the optimal first policy parameter; the natural gradient algorithm updates the first policy parameter each time, that is Adding quadratic constraints To accurately obtain the optimal first strategy parameters; where δ is the preset constraint factor, g θ is the gradient of the objective function of the action prediction model, i.e., the policy gradient; θ is the first policy parameter; θ new is the updated first strategy parameter; is the Fisher information corresponding to the first strategy parameter; where, is the score function corresponding to the first strategy parameter, s is the state information, a is the action information, and π(s,a) is the first strategy.

[0122] Therefore, according to the attenuation coefficient and Fisher information corresponding to the first strategy parameter, and the gradient of the objective function of the first strategy parameter and the action prediction model, the calculation formula for iteratively updating the first strategy parameter is:

[0123]

[0124] Among them, θ new is the first strategy parameter after update, θ is the first strategy parameter before update, η θ is the attenuation coefficient corresponding to the first strategy parameter, is the inverse matrix of the Fisher information matrix corresponding to the first strategy parameter; g θ is the gradient of the objective function of the action prediction model.

[0125] The attenuation coefficient corresponding to the first strategy parameter is calculated by the following formula based on the Fisher information corresponding to the first strategy parameter, the gradient of the objective function of the action prediction model, and the preset constraint factor:

[0126]

[0127] Among them, η θ is the attenuation coefficient corresponding to the first strategy parameter, δ is the preset constraint factor, g θ is the gradient of the objective function of the action prediction model, is the inverse matrix of the Fisher information matrix corresponding to the first strategy parameter.

[0128] This embodiment automatically iteratively updates the first strategy parameter according to the attenuation coefficient and Fisher information corresponding to the first strategy parameter, as well as the gradient of the objective function of the first strategy parameter and the action prediction model, and adaptively adjusts the learning rate to improve the training accuracy while ensuring the training speed of the action prediction model.

[0129] On the basis of the above embodiments, the action prediction model in this embodiment includes a linear matrix or a multi-layer neural network.

[0130] Among them, the structure or number of layers of the multi-layer neural network can be set according to actual needs, such as a 3-layer BP neural network, or a deep convolutional neural network, etc., which is not specifically limited in this embodiment.

[0131] The dimension of the linear matrix can also be set according to actual needs, and this embodiment does not specifically limit this.

[0132] The action prediction model in this embodiment includes a linear matrix or a multi-layer neural network, so that the action prediction model has strong applicability and can quickly and accurately learn linear or nonlinear tasks of the intelligent agent.

[0133] On the basis of the above embodiments, in this embodiment, after inputting the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model, it also includes: optimizing the action prediction model according to the optimal second strategy parameters to obtain an optimized action prediction model; inputting the state information of the intelligent agent under the new task into the optimized action prediction model to obtain the action information of the intelligent agent under the new task output by the optimized action prediction model.

[0134] Optionally, after obtaining the optimal second strategy parameters of the action prediction model corresponding to the new task, the state information of the agent under the new task can be input into the action prediction model corresponding to the optimal second strategy parameters to quickly and accurately obtain the action information of the agent under the new task.

[0135] In order to further verify the effectiveness of the intelligent agent task learning method of this embodiment, the intelligent agent task learning method of this embodiment is evaluated in combination with specific examples.

[0136] The simulation platform and simulation environment of the intelligent agent task learning method in this embodiment can be set according to actual needs. For example, simulation verification can be carried out on the Mujoco150 experimental platform; the experimental platform software support environment is python3.7 on Ubuntu16.04 LTS, and pytorch1.3.0 based on GeForce GTX 1660TI graphics card and CUDA10.1 version for neural network training.

[0137] In order to effectively verify that the intelligent agent has good performance in extracting general knowledge, two task domains, Hopper and Walker, were selected for experimental learning, training and testing verification.

[0138] Among them, the verification experiments include three groups. The first group of experiments is used to verify the general performance of the intelligent agent task learning method in different task domains, such as jumper and walker task domains; the second group of experiments is used to verify the general performance of the intelligent agent task learning method in different strategy parameter spaces; the third group of experiments is used to verify the general performance of the intelligent agent task learning method on task learning models with different training methods.

[0139] In addition, in order to verify the performance of the agent task learning method in this embodiment, a variety of methods were used for comparative experiments, including the policy gradient effective lifelong learning algorithm (PGELLA for short), and the exemplary algorithms in this embodiment, including the environment adaptive strategy generator algorithm (EAPG for short), the advanced environment adaptive strategy generator algorithm (AEAPG for short), the fully connected training task learning model (AMNN-F), and the single point training task learning model (AMNN-S).

[0140] Among them, the environment adaptability strategy generator algorithm, the advanced environment adaptability strategy generator algorithm and the fully connected training agent task learning method use the same network structure. The difference is that all activation functions of the environment adaptability strategy generator algorithm are linear functions, the activation functions of all hidden layers of the advanced environment adaptability strategy generator algorithm are nonlinear functions, the output layer activation function is a linear function, and all activation functions of the single-point training task learning model and the fully connected training task learning model are nonlinear functions.

[0141] like Figure 3-6 As shown in Table 1, the general performance of the agent task learning method in this embodiment on different task domains;

[0142] Figure 3 and Figure 4The application performance of various comparison methods on 100 historical tasks in the Hopper-learned tasks and Walker-learned tasks. Figure 5 and Figure 6 The application performance of various comparison methods in 10 new tasks in the jumper task domain and the walker task domain.

[0143] Table 1 Application performance of various comparison methods

[0144]

[0145] Table 1 shows the average application performance of various comparison methods on 100 historical tasks and the average application performance of 10 new tasks in the Hopper-learned tasks and Walker-learned tasks.

[0146] Among them, the higher the reward value (Reward), the better the application strategy corresponding to the task performs.

[0147] Depend on Figure 3-Figure 6 As can be seen from Table 1, compared with the existing PGELLA algorithm, the EAPG algorithm and the AEAPG algorithm in this embodiment, AMNN-F and AMNN-S have higher reward values ​​in new tasks or historical tasks in different fields;

[0148] In addition, compared with the existing PGELLA algorithm, as well as the EAPG algorithm and AEAPG algorithm in this embodiment, AMNN-F and AMNN-S have the highest reward values ​​in new tasks or historical tasks, and have better application performance and adaptability; therefore, compared with the linear activation function, the nonlinear activation function has a stronger learning ability and a stronger knowledge extraction ability; therefore, in this embodiment, the nonlinear activation function is preferably used as the activation function of the prediction model.

[0149] At the same time, the agent task learning method in this embodiment can generate zero-sample strategies for new tasks; compared with PGELLA in the prior art, which requires small samples to learn the strategy parameters of new tasks and needs to store new task strategies, the general knowledge extracted by the agent task learning method in this embodiment has stronger versatility and reduces the storage of specific knowledge for different tasks; therefore, even if the learning tasks increase, the space storage cost of the knowledge base will not increase, and the time consumption and space storage consumption corresponding to mastering new tasks will be lower. At the same time, the task learning model can automatically adjust the action strategy through the key influencing factors of the environment to adapt to and master new tasks, so that the optimal second strategy parameters of the action prediction model corresponding to the new task are more accurate and reasonable.

[0150] In addition, for the two training modes, it can be seen from Table 1 that when the number of training samples is the same, the average application performance of the single-point training task learning model (AMNN-S) is better than that of the fully connected training task learning model (AMNN-F).

[0151] Figure 7-10 Table 2 and Table 3 show the general performance of the agent task learning method in this embodiment in different strategy parameter spaces.

[0152] Table 2 Application performance of multiple comparison methods 2

[0153]

[0154] Figure 7 and Figure 8 The application performance of various comparison methods on 100 historical tasks in the linear policy parameter space (Linear Policy Network) and nonlinear policy parameter space (Deep Policy Network) in the Hopper task domain. Fig. 9 and Fig.10 The application performance of various comparison methods on 10 new tasks in the linear policy parameter space (LinearPolicy Network) and nonlinear policy parameter space (Deep Policy Network) of the Jumper task domain are shown respectively.

[0155] Table 2 shows the average application performance of the comparative experimental algorithms on 100 different historical tasks in the linear strategy parameter space and the nonlinear strategy parameter space, and the average application performance of 10 different new tasks.

[0156] Among them, the higher the reward value, the better the application strategy corresponding to the task performs.

[0157] from Figure 7-10 As can be seen from Table 2, compared with the existing PGELLA algorithm, the EAPG algorithm and the AEAPG algorithm in this embodiment, the reward values ​​of AMNN-F and AMNN-S in new tasks or historical tasks in different strategy parameter spaces are higher;

[0158] Among them, the newly generated strategy of the single-point training task learning model still has the best application performance and adaptability in different strategy parameter spaces. And Table 2 shows that the fully connected training task learning model is better than the environment adaptability strategy generator algorithm and the advanced environment adaptability strategy generator algorithm, that is, the nonlinear activation function still has stronger knowledge extraction ability in different strategy parameter spaces.

[0159] However, since the number of parameters in the nonlinear strategy parameter space is much greater than the number of parameters in the linear strategy parameter space, when the number of training samples of the single-point training task learning model (AMNN-S) and the fully connected training task learning model (AMNN-F) is the same, the average application performance of the fully connected training task learning model is much lower than that of the single-point training task learning model. Therefore, the single-point training task learning model requires a lower number of strategy samples than the fully connected training task learning model, and can obtain the optimal second strategy parameters more quickly and accurately. Therefore, in this embodiment, the single-point training method is preferably used to train the task learning model.

[0160] The third set of experiments is used to verify the general performance of the agent task learning method on task learning models with different training methods. It is used to illustrate that the single-point training task learning model requires fewer strategy samples than the fully connected training task learning model, and to further illustrate that the average application performance of the single-point training task learning model is better than the fully connected training task learning model.

[0161] The third set of experiments trained the single-point training task learning model using different numbers of policy training samples in the Jumper nonlinear policy parameter space task. Fig.11 The average application effect of the new task strategy when the number of strategy samples is [50, 100, 200, 400, 600, 800, 1000, 1124] is shown.

[0162] The experimental results show that when the single-point training task learning model uses a strategy sample range of [100,300], the average application performance of the strategy of the new task is comparable to the average application performance of the strategy of the new task when the fully connected training task learning model uses a strategy sample number of 1124. The experiment shows that compared with the fully connected training task learning model, the single-point training task learning model has a significantly lower strategy sample requirement and can obtain the optimal second strategy parameters more quickly and accurately. Therefore, in this embodiment, the single-point training method is preferably used to train the task learning model.

[0163] In summary, the agent task learning method of this embodiment is a highly versatile strategy knowledge extraction algorithm that solves the universality of the learning strategy field without extra time consumption.

[0164] like Fig.12 As shown in FIG. 1 , it is a complete flow chart of the agent task learning method in this embodiment, and the main steps include:

[0165] Step 1: Use the reinforcement learning algorithm to train and learn and collect the optimal first strategy parameters corresponding to different historical tasks;

[0166] Optionally, use the natural policy gradient method in reinforcement learning algorithm to perform task policy π t(s|a) training learning, where t is the tth historical learning task, the first task strategy π t It can be a linear matrix or a multi-layer neural network. The first task strategy π t The corresponding policy parameter can be expressed as θ t =(θ t1 ,θ t2 ,…,θ tm ) T . Use the natural policy gradient method to update the first policy parameters for 200 rounds to obtain the optimal first policy corresponding to different historical tasks

[0167] Step 2, screening available and effective optimal first strategy parameters;

[0168] The best first strategy collected using the natural policy gradient method Screening is performed. When the reward value corresponding to the best first strategy is higher than the set threshold, the best first strategy is retained. Filter the best first strategy set to be retained Will serve as training samples for knowledge modules;

[0169] Step 3: vectorize each optimal first strategy in the optimal first strategy set and use it as the sample label y in the training sample set of the knowledge module t , the sample label set is Y = {y1,y2,…,y k}; Take the key influencing factors corresponding to each optimal first strategy as the sample y in the training sample set of the knowledge module t , the sample set is X={x1,x2,…,x k}; Use X and Y as the input and output training samples of the task learning model to train the task learning model in the knowledge module to extract the common knowledge of different task strategies in the same field.

[0170] Step 4: Use the mean square error training task to learn the model until convergence;

[0171] Optionally, the key influencing factors corresponding to each optimal first strategy and each optimal first strategy after vectorization are used as input and output training sample pairs of the task learning model to extract common knowledge of different task strategies in the same field, use the back propagation algorithm to perform learning and training of the task learning module, and use the mean square error as the loss function until the task learning model converges;

[0172] Step 5: Get the trained task learning model.

[0173] In summary, the agent task learning method of this embodiment enables the knowledge module to have both the ability to extract general knowledge of domain tasks and the ability to generate zero-sample strategies. On this basis, a single-point training mode is proposed to greatly reduce the demand for strategy samples required for the task learning module to extract general knowledge. Simulations also show that the strategy parameters for new tasks generated by this method can be directly applied to new tasks without additional time consumption, and have good average application performance.

[0174] The intelligent agent task learning device provided by the present invention is described below. The intelligent agent task learning device described below and the intelligent agent task learning method described above can be referenced to each other.

[0175] like Fig.13 As shown, an agent task learning device provided by this embodiment includes a first training module 1301, a second training module 1302 and a learning module 1303, wherein:

[0176] The first training module 1301 is used to iteratively update the first strategy parameter of the action prediction model based on the natural policy gradient algorithm, using the state information of the agent under each historical task as a sample and the action information of the agent under each historical task as a sample label, so as to obtain the optimal first strategy parameter of the action prediction model corresponding to each historical task;

[0177] Optionally, the action prediction model, i.e., the task policy π, is trained using the natural policy gradient method in the reinforcement learning algorithm. t (s|a) for training and learning, where t is the tth historical task and the task strategy π t It can be a linear matrix or a multi-layer neural network, etc., which is not specifically limited in this embodiment; s and a are the state information and action information of the agent under the tth historical task respectively.

[0178] Among them, the task strategy π t The corresponding first strategy parameter can be expressed as θ t =(θ t1 ,θ t2 ,…,θ tm ) T .

[0179] Based on the natural policy gradient method, the state information of the agent under each historical task is used as a sample, and the action information of the agent under each historical task is used as a sample label. The first policy parameter of the action prediction model is iteratively updated until the preset termination condition of the action prediction model is met, so as to obtain the optimal first policy parameter of the action prediction model corresponding to each historical task. The preset termination condition may be reaching a maximum number of iterations and / or convergence of the action prediction model.

[0180] The second training module 1302 is used to use the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as training sample sets to iteratively train the task learning model; wherein the key influencing factors are factors that have an impact on the output results of the action prediction model;

[0181] Optionally, after obtaining the optimal first strategy parameters of the action prediction model corresponding to each historical task, the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks can be directly used as a training sample set; or the optimal first strategy parameters corresponding to the historical tasks can be screened, and the key influencing factors and optimal first strategy parameters corresponding to the screened historical tasks can be used as a training sample set. This embodiment does not specifically limit this.

[0182] Then, the task learning model is iteratively trained based on the training sample set;

[0183] Optionally, the task learning model may be iteratively trained using the training sample set directly, or the task learning model may be iteratively trained after preprocessing the training sample set, which is not specifically limited in this embodiment.

[0184] Optionally, the key influencing factors and the optimal first strategy parameters corresponding to each historical task are used as input-output training sample pairs of the task learning model to extract common knowledge of different task strategies in the same field, and the task learning network is trained using the back propagation algorithm, and the mean square error is used as the loss function to obtain the optimal task learning network.

[0185] The learning module 1303 is used to input the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0186] Optionally, when a new task needs to be learned, the key influencing factors corresponding to the new task are directly input into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task;

[0187] By using the optimal second strategy parameter to adjust the strategy parameters of the action prediction model, the optimal action prediction model applicable to the new task can be obtained. Based on the optimal action prediction model, the action information prediction of the intelligent agent under the new task can be realized.

[0188] This embodiment uses the natural policy gradient algorithm to take the state information and action information of the intelligent agent under various historical tasks as samples, and iteratively trains the action prediction model in the learning module in the policy knowledge extraction algorithm framework to obtain the optimal first policy parameters of the action prediction model under various historical tasks; then, the key influencing factors and the optimal first policy parameters under various historical tasks are used as sample pairs to iteratively train the task learning model in the knowledge module in the policy knowledge extraction algorithm framework to obtain the optimal task learning model with general knowledge under different tasks; and the optimal action prediction model under the new task can be obtained by simply inputting the key influencing factors under the new task into the optimal task learning model, so as to quickly adapt to the new task without learning the samples under the new task, which can effectively realize that the knowledge module has both the general knowledge extraction capability and the zero-sample policy generation capability of the domain task, and the refined general knowledge has stronger versatility and higher learning efficiency for new tasks.

[0189] Fig.14 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.14 As shown, the electronic device may include: a processor (processor) 1401, a communication interface (Communications Interface) 1402, a memory (memory) 1403 and a communication bus 1404, wherein the processor 1401, the communication interface 1402, and the memory 1403 communicate with each other through the communication bus 1404. The processor 1401 can call the logic instructions in the memory 1403 to execute the agent task learning method, which includes: based on the natural policy gradient algorithm, taking the state information of the agent under each historical task as a sample, and taking the action information of the agent under each historical task as a sample label, iteratively updating the first strategy parameters of the action prediction model to obtain the optimal first strategy parameters of the action prediction model corresponding to each historical task; taking the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as training sample sets, and iteratively training the task learning model; wherein the key influencing factors are factors that affect the output results of the action prediction model; inputting the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0190] In addition, the logic instructions in the above-mentioned memory 1403 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent agent task learning method provided by the above methods, which method includes: based on the natural policy gradient algorithm, taking the state information of the intelligent agent under each historical task as a sample, and taking the action information of the intelligent agent under each historical task as a sample label, iteratively updating the first strategy parameters of the action prediction model to obtain the optimal first strategy parameters of the action prediction model corresponding to each historical task; taking the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as a training sample set, and iteratively training the task learning model; wherein the key influencing factors are factors that affect the output results of the action prediction model; inputting the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0192] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the intelligent agent task learning method provided by the above-mentioned methods, the method comprising: based on a natural policy gradient algorithm, taking the state information of the intelligent agent under each historical task as a sample, taking the action information of the intelligent agent under the each historical task as a sample label, iteratively updating the first strategy parameters of the action prediction model, and obtaining the optimal first strategy parameters of the action prediction model corresponding to each historical task; taking the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as a training sample set, and iteratively training the task learning model; wherein the key influencing factors are factors that affect the output results of the action prediction model; inputting the key influencing factors corresponding to the new task into the trained task learning model, and obtaining the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

[0193] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An agent task learning method, characterized in that: include: Based on the natural policy gradient algorithm, the state information of the agent under each historical task is used as a sample, and the action information of the agent under each historical task is used as a sample label, and the first policy parameter of the action prediction model is iteratively updated to obtain the optimal first policy parameter of the action prediction model corresponding to each historical task; The key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks are used as training sample sets to iteratively train the task learning model; wherein the key influencing factors are factors that affect the output results of the action prediction model; The key influencing factors corresponding to the new task are input into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

2. The agent task learning method according to claim 1, characterized in that: The key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks are used as training sample sets to iteratively train the task learning model, including: The historical tasks whose reward values ​​corresponding to the optimal first strategy parameters are greater than a set threshold are used as target historical tasks; The key influencing factors corresponding to all target historical tasks and the optimal first strategy parameters corresponding to all target historical tasks are used as the training sample set to iteratively train the task learning model.

3. The agent task learning method according to claim 1, characterized in that: The task learning model includes multiple subtask learning models; Each subtask learning model is used to learn the mapping relationship between the key influencing factors and each optimal first strategy parameter; The loss function of the task learning model is the joint result of the loss functions of multiple subtask learning models.

4. The agent task learning method according to any one of claims 1 to 3, characterized in that: The key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks are used as training sample sets to iteratively train the task learning model, including: Vectorize the optimal first strategy parameters corresponding to each historical task; The key influencing factors corresponding to all historical tasks and the optimal first strategy parameters after vectorization processing corresponding to all historical tasks are used as the training sample set to iteratively train the task learning model.

5. The agent task learning method according to any one of claims 1 to 3, characterized in that: The iterative updating of the first strategy parameter of the action prediction model comprises: Iteratively updating the first strategy parameter according to the attenuation coefficient and Fisher information corresponding to the first strategy parameter, and the gradient of the first strategy parameter and the objective function of the action prediction model; Among them, the attenuation coefficient corresponding to the first strategy parameter is determined according to the Fisher information corresponding to the first strategy parameter, the gradient of the objective function of the action prediction model and the preset constraint factor.

6. The agent task learning method according to any one of claims 1 to 3, characterized in that: The action prediction model includes a linear matrix or a multi-layer neural network.

7. The agent task learning method according to any one of claims 1 to 3, characterized in that: After inputting the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model, the method further includes: Optimizing the action prediction model according to the optimal second strategy parameter to obtain an optimized action prediction model; The state information of the agent under the new task is input into the optimized action prediction model to obtain the action information of the agent under the new task output by the optimized action prediction model.

8. An agent task learning device, characterized in that: include: A learning module, which is used to iteratively update the first policy parameter of the action prediction model based on the natural policy gradient algorithm, using the state information of the agent under each historical task as a sample and the action information of the agent under each historical task as a sample label, so as to obtain the optimal first policy parameter of the action prediction model corresponding to each historical task; The first knowledge module is used to iteratively train the task learning model by using the key influencing factors corresponding to all historical tasks and the optimal first strategy parameters corresponding to all historical tasks as a training sample set; wherein the key influencing factors are factors that have an impact on the output result of the action prediction model; The second knowledge module is used to input the key influencing factors corresponding to the new task into the trained task learning model to obtain the optimal second strategy parameters of the action prediction model corresponding to the new task output by the task learning model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the agent task learning method as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the agent task learning method as described in any one of claims 1 to 7 are implemented.

Images