Training methods and devices for reinforcement learning

Abstract

This application provides a training method and apparatus for reinforcement learning, which can improve the exploration efficiency of meta-reinforcement learning to achieve rapid adaptation to new tasks. The method includes: initializing a context task encoder, an exploration policy, and an exploitation policy; executing a first action obtained based on first environment information, the context task encoder, and the exploration policy to obtain a first sample based on the first action, the first sample including first feedback from the environment in response to the first action; executing a second action obtained based on second environment information, the context task encoder, and the exploitation policy to obtain a second sample based on the second action, the second sample including second feedback from the environment in response to the second action; and optimizing the exploration policy and the exploitation policy based on the first feedback and the second feedback.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a training method and device for reinforcement learning. Background Art

[0002] Reinforcement learning is a general method for achieving sequential decision-making. The agent learns by trial and error, and the reward obtained by interacting with the environment through actions guides the behavior. The goal is to maximize the reward for the agent. At present, reinforcement learning has received great attention from academia and industry. However, the learning efficiency of reinforcement learning agents trained from scratch is usually low for new tasks, especially complex tasks. The main problem is that they cannot effectively call on prior knowledge to solve the current problem.

[0003] In order to improve the learning ability of reinforcement learning for new tasks, one of the current mainstream methods is to combine meta-learning. Meta-learning provides an agent with a method to quickly adapt to new tasks using a small number of samples. Meta-reinforcement learning uses prior knowledge to achieve rapid learning of new tasks based on small samples. At present, meta-reinforcement learning is mainly divided into gradient-based meta-reinforcement learning and context-based meta-reinforcement learning. Among them, gradient-based meta-reinforcement learning achieves rapid adaptation to new tasks through gradient descent with a small number of steps, while context-based meta-reinforcement learning obtains more task-related information through reasoning on task encoding to improve the exploration efficiency of new tasks, thereby achieving rapid adaptation to new tasks. In terms of adaptation efficiency to new tasks, context-based meta-reinforcement learning has more advantages than gradient-based meta-reinforcement learning.

[0004] In order to achieve efficient adaptation to new tasks, exploration efficiency is an important factor affecting the rapid adaptation of meta-reinforcement learning to new tasks. Therefore, a meta-reinforcement learning method with high exploration efficiency is particularly important to achieve rapid adaptation to new tasks. Summary of the invention

[0005] The present application provides a reinforcement learning training method and device, which can improve the exploration efficiency of meta-reinforcement learning to achieve rapid adaptation to new tasks.

[0006] In a first aspect, a reinforcement learning training method is provided, comprising: initializing a context task encoder, an exploration strategy, and an exploitation strategy; executing a first behavior acquired according to first environmental information, the context task encoder, and the exploration strategy to obtain a first sample based on the first behavior, the first sample comprising a first feedback of the environment in response to the first behavior; executing a second behavior acquired according to second environmental information, the context task encoder, and the exploitation strategy to obtain a second sample based on the second behavior, the second sample comprising a second feedback of the environment in response to the second behavior; and optimizing the exploration strategy and the exploitation strategy according to the first feedback and the second feedback.

[0007] In the existing meta-reinforcement learning method, the exploration strategy and the utilization strategy are integrated into one, the exploration strategy is conservative, and there is a lack of immediate feedback during the exploration process, which makes it difficult to achieve rapid adaptation to the task and the exploration efficiency is low. The training method of meta-reinforcement learning in the embodiment of the present application has independent exploration strategies and utilization strategies, which avoids the conservative exploration strategy caused by the mutual constraints between the exploration strategy and the utilization strategy, and improves the exploration efficiency. At the same time, immediate feedback from the environment is obtained to provide a continuous exploration signal for the exploration strategy, which can also achieve efficient adaptation for small sample tasks with sparse feedback.

[0008] In combination with the first aspect, in a possible implementation of the first aspect, the method also includes: before executing a first behavior acquired according to first environmental information, a context task encoder, and an exploration strategy, determining that the number of trajectories of a first sample is less than a first threshold; before executing a second behavior acquired according to second environmental information, a context task encoder, and a utilization strategy, determining that the number of trajectories of a second sample is less than a second threshold.

[0009] In the training method of meta-reinforcement learning in the embodiment of the present application, the number of executions of the exploration strategy and the number of executions of the utilization strategy can be preset. Before the number of sample trajectories obtained by executing the exploration strategy reaches a preset value, the exploration strategy is executed. After the number of sample trajectories obtained by executing the exploration strategy reaches a preset value, and before the number of sample trajectories obtained by executing the utilization strategy reaches a preset value, the utilization strategy is executed. After the number of sample trajectories obtained by executing the utilization strategy reaches a preset value, the exploration strategy and the utilization strategy are optimized.

[0010] In combination with the first aspect, in a possible implementation of the first aspect, a first behavior based on an exploration strategy is performed according to the context task encoder and the first environmental information to obtain a first sample based on the first behavior, including: obtaining the first environmental information as an input of the context task encoder to obtain a first output; using the first output as an input of the exploration strategy to obtain the first behavior; and executing the first behavior to obtain a first sample.

[0011] In combination with the first aspect, in a possible implementation of the first aspect, a second behavior based on the utilization strategy is performed according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: obtaining the second environment information as an input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; and executing the second behavior to obtain a second sample.

[0012] In the training method of meta-reinforcement learning in the embodiment of the present application, the environment is sampled according to the context task encoder, and the sampling results are used as the input of the exploration strategy or the utilization strategy. The output of the exploration strategy or the utilization strategy is the behavior that should be taken on the environment, and then immediate feedback from the environment is obtained.

[0013] In combination with the first aspect, in a possible implementation of the first aspect, the exploration strategy and the utilization strategy are optimized according to the first feedback and the second feedback, including: obtaining a first intrinsic incentive corresponding to the first sample and a second intrinsic incentive corresponding to the second sample; obtaining a first reward according to the first intrinsic incentive and the first feedback; optimizing the exploration strategy according to the first reward; obtaining a second reward according to the second intrinsic incentive and the second feedback; and optimizing the utilization strategy according to the second reward.

[0014] In the training method of meta-reinforcement learning in the embodiment of the present application, the reward of the environment after executing the behavior includes two aspects: immediate feedback from the environment and intrinsic motivation of the sample.

[0015] In combination with the first aspect, in a possible implementation of the first aspect, obtaining a first intrinsic excitation corresponding to a first sample and a second intrinsic excitation corresponding to a second sample includes: obtaining first mutual information between first environmental information and an encoding of the first environmental information; obtaining the first excitation based on the first mutual information; obtaining second mutual information between the second environmental information and the encoding of the second environmental information; and obtaining the second excitation based on the second mutual information.

[0016] In combination with the first aspect, in a possible implementation of the first aspect, obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: obtaining a first true value of a first feedback and a first environmental state; obtaining a first predicted value of an encoding of the first environmental information; obtaining the first intrinsic excitation based on a gap between the first true value and the first predicted value; obtaining a second true value of the second feedback and the second environmental state; obtaining a second predicted value of the encoding of the second environmental information; and obtaining the second intrinsic excitation based on a gap between the second true value and the second predicted value.

[0017] In the training method of meta-reinforcement learning in the embodiment of the present application, two methods for obtaining intrinsic motivation are provided. One is to obtain intrinsic motivation by measuring the mutual information between environmental information and the encoding of environmental information; the other is to obtain intrinsic motivation by measuring the gap between feedback and the true value of the state and the predicted value based on the encoding of environmental information.

[0018] In a second aspect, a method for adapting tasks is provided, which is characterized by comprising: loading a trained context task encoder, an exploration strategy and an utilization strategy; executing a first behavior based on first environmental information, the context task encoder and the exploration strategy to obtain a first sample based on the first behavior, the first sample including a first feedback of the environment in response to the first behavior; executing a second behavior based on second environmental information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, the second sample including a second feedback of the environment in response to the second behavior.

[0019] In addition to providing a training method for meta-reinforcement learning, the embodiments of the present application also provide a method for adapting tasks, which uses the context task encoder, exploration strategy and utilization strategy trained according to the training method of meta-reinforcement learning to achieve rapid adaptation to tasks.

[0020] In combination with the second aspect, in a possible implementation of the second aspect, the method also includes: before executing a first behavior acquired according to first environmental information, a context task encoder, and an exploration strategy, determining that the number of trajectories of a first sample is less than a first threshold; before executing a second behavior acquired according to second environmental information, a context task encoder, and a utilization strategy, determining that the number of trajectories of a second sample is less than a second threshold.

[0021] In combination with the second aspect, in a possible implementation of the second aspect, a first behavior based on an exploration strategy is performed according to the context task encoder and the first environmental information to obtain a first sample based on the first behavior, including: obtaining the first environmental information as an input of the context task encoder to obtain a first output; using the first output as an input of the exploration strategy to obtain the first behavior; and executing the first behavior to obtain a first sample.

[0022] In combination with the second aspect, in a possible implementation of the second aspect, a second behavior based on the utilization strategy is performed according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: obtaining the second environment information as an input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; and executing the second behavior to obtain a second sample.

[0023] In a third aspect, a reinforcement learning training device is provided, characterized in that it includes: an initialization unit for initializing a context task encoder, an exploration strategy and an utilization strategy; an interaction unit for executing a first behavior according to first environmental information, a context task encoder and an exploration strategy to obtain a first sample based on the first behavior, the first sample including a first feedback of the environment in response to the first behavior; the interaction unit is also used to execute a second behavior according to second environmental information, a context task encoder and an utilization strategy to obtain a second sample based on the second behavior, the second sample including a second feedback of the environment in response to the second behavior; an optimization unit for optimizing the exploration strategy and the utilization strategy according to the first feedback and the second feedback.

[0024] In combination with the third aspect, in a possible implementation of the third aspect, the interaction unit is also used to: before executing a first behavior obtained according to the first environmental information, the context task encoder, and the exploration strategy, determine that the number of trajectories of the first sample is less than a first threshold; before executing a second behavior obtained according to the second environmental information, the context task encoder, and the utilization strategy, determine that the number of trajectories of the second sample is less than a second threshold.

[0025] In combination with the third aspect, in a possible implementation of the third aspect, the interaction unit performs a first behavior based on the exploration strategy according to the context task encoder and the first environmental information to obtain a first sample based on the first behavior, including: obtaining the first environmental information as an input of the context task encoder to obtain a first output; using the first output as an input of the exploration strategy to obtain the first behavior; and executing the first behavior to obtain a first sample.

[0026] In combination with the third aspect, in a possible implementation of the third aspect, the interaction unit performs a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: obtaining the second environment information as an input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; and executing the second behavior to obtain a second sample.

[0027] In combination with the third aspect, in a possible implementation of the third aspect, the optimization unit optimizes the exploration strategy and the utilization strategy based on the first feedback and the second feedback, including: obtaining a first intrinsic incentive corresponding to the first sample and a second intrinsic incentive corresponding to the second sample; obtaining a first reward based on the first intrinsic incentive and the first feedback; optimizing the exploration strategy based on the first reward; obtaining a second reward based on the second intrinsic incentive and the second feedback; and optimizing the utilization strategy based on the second reward.

[0028] In combination with the third aspect, in a possible implementation of the third aspect, obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: obtaining first mutual information between the first environmental information and the encoding of the first environmental information; obtaining the first excitation based on the first mutual information; obtaining second mutual information between the second environmental information and the encoding of the second environmental information; and obtaining the second excitation based on the second mutual information.

[0029] In combination with the third aspect, in a possible implementation of the third aspect, obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: obtaining a first true value of a first feedback and a first environmental state; obtaining a first predicted value of an encoding of the first environmental information; obtaining the first intrinsic excitation based on a gap between the first true value and the first predicted value; obtaining a second true value of the second feedback and the second environmental state; obtaining a second predicted value of the encoding of the second environmental information; and obtaining the second intrinsic excitation based on a gap between the second true value and the second predicted value.

[0030] In a fourth aspect, a device for adapting tasks is provided, characterized in that it includes: a loading unit for loading a trained context task encoder, an exploration strategy and an utilization strategy; an interaction unit for executing a first behavior based on first environmental information, the context task encoder and the exploration strategy to obtain a first sample based on the first behavior, the first sample including a first feedback of the environment in response to the first behavior; the interaction unit is also used to execute a second behavior based on second environmental information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, the second sample including a second feedback of the environment in response to the second behavior.

[0031] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the interaction unit is also used to: before executing a first behavior obtained according to the first environmental information, the context task encoder, and the exploration strategy, determine that the number of trajectories of the first sample is less than a first threshold; before executing a second behavior obtained according to the second environmental information, the context task encoder, and the utilization strategy, determine that the number of trajectories of the second sample is less than a second threshold.

[0032] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the interaction unit performs a first behavior based on the exploration strategy according to the context task encoder and the first environmental information to obtain a first sample based on the first behavior, including: obtaining the first environmental information as an input of the context task encoder to obtain a first output; using the first output as an input of the exploration strategy to obtain the first behavior; and executing the first behavior to obtain a first sample.

[0033] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the interaction unit performs a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: obtaining the second environment information as an input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; and executing the second behavior to obtain a second sample.

[0034] In a fifth aspect, a computer-readable storage medium is provided, in which program instructions are stored. When the program instructions are executed by a processor, a method of implementing the above-mentioned first aspect and any possible implementation method of the first aspect or the above-mentioned second aspect and any possible implementation method of the second aspect is implemented.

[0035] In a sixth aspect, a computer program product is provided, which includes a computer program code. When the computer program code runs on a computer, it implements the method of the above-mentioned first aspect and any possible implementation method of the first aspect or the above-mentioned second aspect and any possible implementation method of the second aspect.

[0036] In the seventh aspect, an intelligent robot is provided, characterized in that it includes a device according to the third aspect and any possible implementation of the third aspect or the fourth aspect and any possible implementation of the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of a system architecture for applying a training method for meta-reinforcement learning in an embodiment of the present application;

[0038] Figure 2 is a schematic flow chart of a training method for meta-reinforcement learning according to an embodiment of the present application;

[0039] Figure 3 is a schematic flow chart of a method for adapting a task according to an embodiment of the present application;

[0040] Figure 4 is a schematic block diagram of the model training of meta-reinforcement learning in an embodiment of the present application;

[0041] Figure 5 is a schematic block diagram of a model application of meta-reinforcement learning in an embodiment of the present application;

[0042] Figure 6 is a schematic block diagram of a training device for reinforcement learning according to an embodiment of the present application;

[0043] Figure 7 is a schematic structural diagram of a training device for reinforcement learning according to an embodiment of the present application;

[0044] Figure 8 is a schematic block diagram of an apparatus for adapting a task according to an embodiment of the present application;

[0045] Fig. 9 It is a schematic structural diagram of a device for adapting tasks according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0047] There are two main existing context-based meta-reinforcement learning methods. The first method is to achieve a balance between exploration and exploitation by maximizing the average performance of the entire adaptation process, which is called the "automatic balancer"; the second method is to encode the task experience during the meta-training process to obtain a task encoder, and then use the encoder to encode the key information of the new task during the adaptation process of the new task, learn the new task by continuously updating the judgment of the task, and achieve a balance between exploration and exploitation by sampling the encoder's posterior.

[0048] The first existing context-based meta-reinforcement learning method proposes a general meta-reinforcement learning framework. In meta-training, a recurrent neural network is used to learn a series of tasks to obtain the weights of the neural network. Then, in the adaptation process, the hidden state of the recurrent neural network is adjusted based on the data continuously generated by the new task, and the constantly changing hidden state is used as the reinforcement learning strategy for the current task.

[0049] The specific process includes: 1. Define a task set based on a certain problem; 2. Construct an agent based on a recursive neural network, where the agent input is the feedback, action, whether the segment is finished and the current state at the previous moment; 3. Extract tasks from the task set, fix the number of exploration segments (such as Figure 1 , the number of exploration segments is 2); 4. New tasks are continuously extracted for continuous learning. The hidden state of the recurrent neural network is not reset after each segment. The agent uses historical input to infer the current task information and adjust the hidden state and strategy; 5. The optimization goal is to maximize the total reward return of all segments under each task.

[0050] The first existing context-based meta-reinforcement learning method maximizes the sum of rewards for all segments of each task, obtains a description of the properties of a class of tasks based on a recursive neural network, infers the task characteristics based on the current task data, and adjusts the reinforcement learning strategy to achieve adaptation to new tasks. Based on this type of optimization goal, the balance between exploration and utilization of the reinforcement learning strategy is actually achieved by maximizing the average performance of the entire adaptation process. Because the exploration cost is directly considered, it is easy to lead to a relatively conservative exploration strategy. When the task is complex and the feedback is sparse, adaptation to the task is usually not achieved, and the performance is even worse than that of ordinary reinforcement learning algorithms, resulting in the loss of the advantages of meta-reinforcement learning.

[0051] The second existing context-based meta-reinforcement learning method proposes a general meta-reinforcement learning framework, which encodes the accumulated task experience through the meta-training process to obtain a task encoder. Then, in the adaptation process, the encoder is used to encode the key information of the new task, and the new task is learned by continuously updating the judgment of the task. The strategy is explored based on the encoder's posterior sampling.

[0052] The specific process includes: 1. Define a task set based on a certain problem; 2. Randomly sample tasks from it to obtain context-based experience data; 3. Use the encoder network that takes contextual experience as input to infer the variational approximation of the posterior belief state; 4. Sample from the belief state and use it as the input of the reinforcement learning algorithm to achieve exploration; 5. Optimize the encoder by minimizing the mutual information between the contextual experience and the sampled belief state.

[0053] The second existing context-based meta-reinforcement learning method uses a reinforcement learning strategy based on the encoder's posterior sampling for exploration. The agent maintains a data-based distribution for the task, iteratively samples new tasks from the distribution and uses them as policy input. It determines what actions should be taken based on the current cognition of the task, and uses the collected data to update the distribution. As more data is collected, the variance of the posterior distribution decreases, thereby achieving a transition between the exploration strategy and the exploitation strategy. However, this exploration method ignores the immediate feedback during the exploration process. In addition to the indirect uncertainty inferred from the current task as input to guide the exploration, there are no other effective exploration signals, especially in complex tasks with sparse feedback, resulting in low exploration efficiency.

[0054] Therefore, the embodiments of the present application provide a context-based meta-reinforcement learning method and device, which can improve the exploration efficiency of meta-reinforcement learning to achieve rapid adaptation to new tasks.

[0055] Figure 1 A schematic diagram of the system architecture of the training method for meta-reinforcement learning in the embodiment of the present application is shown. Figure 1As shown, the system mainly includes a task / environment information acquisition module, a training module of a context-based meta-reinforcement learning method, a decision module of a context-based meta-reinforcement learning method, and a control module.

[0056] The task / environment information acquisition module is used to integrate the information transmitted by each sensor, determine the environmental information of the task, and pass this information to the training module or decision module. The training module or decision module is the core module of the embodiment of the present application. After receiving the task or environmental information, according to the context-based meta-reinforcement learning method provided by the embodiment of the present application, the decision module outputs the decision instruction to the control module, and the training module outputs the decision instruction information to the control module and simultaneously trains and optimizes the exploration strategy and utilization strategy of meta-reinforcement learning. The control module controls the carrier to complete the task according to the received decision instruction.

[0057] The training method and device of meta-reinforcement learning of the embodiment of the present application can be applied to an intelligent carrier equipped with a computer system. The executable code of task / environment information, decision-making, and control is on the storage component of the computer system, and the CPU / GPU and the storage system need to be coordinated during operation. At the same time, the communication between modules also requires the use of the network communication component of the computer system, and the neural network model is stored on the storage component of the computer system.

[0058] Intelligent carriers mainly include fixed robots, such as robotic arms used in industrial production; and mobile robots, which use wheels, legs or similar machines to move around in the environment. Common ones include freight robots (such as warehouse robots), aerial robots (such as drones) and automatic vehicles (such as self-driving vehicles).

[0059] It should be understood that the above is only an example of the application scenario of the training method and device of meta-reinforcement learning of the embodiment of the present application, and does not constitute a limitation of the embodiment of the present application. In addition to the above application scenarios, the embodiment of the present application can also be applied to other artificial intelligence scenarios.

[0060] Figure 2 A schematic flow chart of a training method for meta-reinforcement learning in an embodiment of the present application is shown, as Figure 2 As shown, it includes steps 201 to 204, and these steps are respectively introduced in detail below.

[0061] S201, initialize the context task encoder, exploration strategy and exploitation strategy.

[0062] Specifically, the context task encoder, exploration strategy, and exploitation strategy are three neural network models.

[0063] S202, executing a first behavior acquired according to first environmental information, a context task encoder, and an exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior.

[0064] Specifically, the training method of meta-reinforcement learning of the embodiment of the present application is executed in a pre-set environment. First, the first environmental information is obtained as the input of the context task encoder to obtain the output of the context task encoder, which is named as the first output; then the first output is used as the input of the exploration strategy to obtain the output of the exploration strategy, wherein the output of the exploration strategy includes the behavior that needs to be performed on the environment, which is named as the first behavior. For example, when the intelligent carrier is a mobile robot, the first behavior can be forward, backward, etc. When the intelligent carrier is an automatic vehicle, the first behavior can be left turn, right turn, straight, etc.; finally, the first behavior is executed in the environment to obtain a first sample based on the first behavior, wherein the first sample includes the immediate feedback of the environment in response to the first behavior, which is named as the first feedback. In addition, the first sample can also include the first environmental information, the first behavior, and the next moment environmental state information.

[0065] Optionally, before executing the first behavior acquired according to the first environmental information, the context task encoder, and the exploration strategy, the meta-reinforcement learning training method of the embodiment of the present application further includes determining that the number of trajectories of the first sample is less than a first threshold. When the number of trajectories of the first sample is less than the first threshold, the method in S202 is executed; when the number of trajectories of the first sample is greater than or equal to the first threshold, the method in S203 is executed.

[0066] Optionally, before determining that the number of trajectories of the first sample is less than the first threshold, the meta-reinforcement learning training method of the embodiment of the present application also includes determining that the trajectory length of the first sample is less than H, where H is a preset value, that is, when the trajectory length of the first sample is less than H, executing the method in S202, and when the trajectory length of the first sample is greater than or equal to H, executing the judgment of the number of trajectories of the first sample.

[0067] S203, executing a second behavior acquired according to the second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, wherein the second sample includes a second feedback of the environment in response to the second behavior.

[0068] Specifically, the training method of meta-reinforcement learning of the embodiment of the present application is executed in a pre-set environment. First, the second environment information is obtained as the input of the context task encoder to obtain the output of the context task encoder, which is named as the second output; then the second output is used as the input of the utilization strategy to obtain the output of the utilization strategy, wherein the output of the utilization strategy includes the behavior that needs to be performed on the environment, which is named as the second behavior. For example, when the intelligent carrier is a mobile robot, the second behavior can be forward, backward, etc. When the intelligent carrier is an automatic vehicle, the second behavior can be left turn, right turn, straight, etc.; finally, the second behavior is executed in the environment to obtain a second sample based on the second behavior, wherein the second sample includes the immediate feedback of the environment in response to the second behavior, which is named as the second feedback. In addition, the second sample can also include the second environment information, the second behavior, and the next moment environment state information.

[0069] Optionally, before executing the second behavior acquired according to the second environment information, the context task encoder, and the exploration strategy, the meta-reinforcement learning training method of the embodiment of the present application further includes determining that the number of trajectories of the second sample is less than a second threshold. When the number of trajectories of the second sample is less than the second threshold, the method in S203 is executed; when the number of trajectories of the second sample is greater than or equal to the second threshold, the method in S204 is executed.

[0070] Optionally, before determining that the number of trajectories of the second sample is less than the second threshold, the meta-reinforcement learning training method of the embodiment of the present application further includes determining that the trajectory length of the second sample is less than H, where H is a preset value, that is, when the trajectory length of the second sample is less than H, executing the method in S203, and when the trajectory length of the second sample is greater than or equal to H, executing the determination of the number of trajectories of the second sample. S204, optimizing the exploration strategy and the utilization strategy according to the first feedback and the second feedback.

[0071] Specifically, first obtain the first intrinsic incentive corresponding to the first sample, and obtain the second intrinsic incentive corresponding to the second sample. Then obtain the first reward according to the first intrinsic incentive and the first feedback, for example, the first reward is the sum of the first intrinsic incentive and the first feedback; obtain the second reward according to the second intrinsic incentive and the second feedback, for example, the second reward is the sum of the second intrinsic incentive and the second feedback. Finally, optimize the exploration strategy according to the first reward, and optimize the utilization strategy according to the second reward.

[0072] Among them, obtaining the first intrinsic excitation corresponding to the first sample or obtaining the second intrinsic excitation corresponding to the second sample can be obtained according to two existing methods. One is to obtain the intrinsic excitation by measuring the mutual information between the environmental information and the encoding of the environmental information; the other is to obtain the intrinsic excitation by measuring the gap between the feedback and the state true value and the predicted value based on the encoding of the environmental information, that is, obtaining the first true value of the first feedback and the first environmental state, obtaining the first predicted value of the encoding of the first environmental information, and obtaining the first intrinsic excitation according to the gap between the first true value and the first predicted value, or obtaining the second true value of the second feedback and the second environmental state, obtaining the second predicted value of the encoding of the second environmental information, and obtaining the second intrinsic excitation according to the gap between the second true value and the second predicted value.

[0073] The training method of meta-reinforcement learning in the embodiment of the present application has independent exploration strategies and utilization strategies, which avoids the conservative exploration strategy caused by the mutual constraints between the exploration strategy and the utilization strategy, and improves the exploration efficiency. At the same time, the training method of meta-reinforcement learning in the embodiment of the present application obtains the reward of the behavior through the immediate feedback of the environment and the intrinsic motivation of the sample, provides a continuous exploration signal for the exploration strategy, and can also achieve efficient adaptation for small sample tasks with sparse feedback.

[0074] Figure 3 A schematic flow chart of a method for adapting a task according to an embodiment of the present application is shown, Figure 3 As shown, it includes step 301 to step 303, and these steps are introduced respectively below.

[0075] The so-called task is to perform corresponding behaviors in a specific environment to achieve a preset goal. For example, the task of controlling the walking of a humanoid robot, the task of a household mobile robot finding a specific target object in a home scene, and the task of an autonomous vehicle driving at an intersection.

[0076] S301, load the trained context task encoder, exploration strategy and exploitation strategy.

[0077] Specifically, the context task encoder, exploration strategy and exploitation strategy used to adapt to the task are three pre-trained neural network models, and their training methods can refer to the above Figure 2 For the sake of brevity, the embodiments of the present application will not be described in detail here.

[0078] S302, executing a first behavior according to first environmental information, a context task encoder, and an exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior.

[0079] Specifically, the method of adapting tasks in the embodiment of the present application is performed in a pre-set environment. First, the first environmental information is obtained as the input of the trained context task encoder to obtain the output of the trained context task encoder, which is named as the first output; then the first output is used as the input of the trained exploration strategy to obtain the output of the trained exploration strategy, wherein the output of the trained exploration strategy includes the behavior that needs to be performed on the environment, which is named as the first behavior. For example, when the intelligent carrier is a mobile robot, the first behavior can be forward, backward, etc. When the intelligent carrier is an automatic vehicle, the first behavior can be left turn, right turn, straight, etc.; finally, the first behavior is executed in the environment to obtain a first sample based on the first behavior, wherein the first sample includes the immediate feedback of the environment in response to the first behavior, which is named as the first feedback. In addition, the first sample can also include the first environmental information, the first behavior, and the next moment environmental state information.

[0080] Optionally, before executing the first behavior acquired according to the first environment information, the context task encoder, and the exploration strategy, the meta-reinforcement learning training method of the embodiment of the present application further includes determining that the number of trajectories of the first sample is less than a first threshold. When the number of trajectories of the first sample is less than the first threshold, the method in S302 is executed; when the number of trajectories of the first sample is greater than or equal to the first threshold, the method in S303 is executed.

[0081] Optionally, before determining that the number of trajectories of the first sample is less than the first threshold, the meta-reinforcement learning training method of the embodiment of the present application also includes determining that the trajectory length of the first sample is less than H, where H is a preset value, that is, when the trajectory length of the first sample is less than H, executing the method in S302, and when the trajectory length of the first sample is greater than or equal to H, executing the judgment of the number of trajectories of the first sample.

[0082] S303, executing a second behavior according to the second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, wherein the second sample includes a second feedback of the environment in response to the second behavior.

[0083] Specifically, the method of adapting tasks in the embodiment of the present application is performed in a pre-set environment. First, the second environment information is obtained as the input of the trained context task encoder to obtain the output of the trained context task encoder, which is named as the second output; then the second output is used as the input of the trained utilization strategy to obtain the output of the trained utilization strategy, wherein the output of the trained utilization strategy includes the behavior that needs to be performed on the environment, which is named as the second behavior. For example, when the intelligent carrier is a mobile robot, the second behavior can be forward, backward, etc. When the intelligent carrier is an automatic vehicle, the second behavior can be left turn, right turn, straight, etc.; finally, the second behavior is executed in the environment to obtain a second sample based on the second behavior, wherein the second sample includes the immediate feedback of the environment in response to the second behavior, which is named as the second feedback. In addition, the second sample can also include the second environment information, the second behavior, and the next moment environment state information.

[0084] Optionally, before executing the second behavior acquired according to the second environment information, the context task encoder and the exploration strategy, the meta-reinforcement learning training method of the embodiment of the present application further includes determining that the number of trajectories of the second sample is less than a second threshold. When the number of trajectories of the second sample is less than the second threshold, the method in S303 is executed; when the number of trajectories of the second sample is greater than or equal to the second threshold, the trajectory execution is completed and the program ends.

[0085] Optionally, before determining that the number of trajectories of the second sample is less than the second threshold, the meta-reinforcement learning training method of the embodiment of the present application also includes determining that the trajectory length of the second sample is less than H, where H is a preset value, that is, when the trajectory length of the second sample is less than H, executing the method in S303, and when the trajectory length of the second sample is greater than or equal to H, executing the judgment of the number of trajectories of the second sample.

[0086] Figure 3 The method of adapting tasks in the Figure 2 The context task encoder, exploration strategy, and utilization strategy trained in the CNN do not need to use data to update the neural network model.

[0087] The following is a detailed introduction to the context-based meta-reinforcement learning method of the embodiment of the present application in combination with specific application scenarios.

[0088] The first application scenario is to apply the context-based meta-reinforcement learning method of the embodiment of the present application to robot control. The control task is a Mujoco standard task. Mujoco standard tasks are a series of existing stable, well-tested, easy-to-use and modify continuous control tasks. The standardized actions, observations and reward structures of these tasks make the benchmark simple and the learning curve easy to explain. Mujoco standard tasks mainly achieve different goals by multi-joint control of different robots, including reaching the target moving speed or reaching the target position. The complexity of the task is mainly related to the number of joints, various physical environment parameters and whether the target feedback is sparse, so a series of task variants can be obtained according to Mujoco standard tasks. For example, the humanoid walker task in the Mujoco standard task has the goal of reaching the target moving speed. By setting different target speeds, the feedback function can be changed, thereby realizing the design of a series of tasks; or it is also a humanoid walker task, but by changing the system parameters (physical model) to change the state transition function, thereby realizing the design of a series of tasks.

[0089] The embodiment of the present application takes the task of a humanoid walker in the Mujoco standard task to reach the target movement speed as an example. The humanoid walker is a planar bipedal robot with 7 limbs (including two legs and a torso) and 6 movable joints. Completing the task requires coordinating the joints to avoid falling while achieving the expected movement speed. First, define: 1. Task set; 2. State information; 3. Action information; 4. Feedback function. The details are as follows:

[0090] 1. Task set {T i} i＝1,2,...,M : preset the sampling interval of the target speed, from which a series of task variants are sampled;

[0091] 2. State information s: 21-dimensional observations, including the angles, velocities, and center of gravity positions of all joints of the humanoid walker;

[0092] 3. Action information a: 6-dimensional control action, corresponding to 6 joints;

[0093] 4. Feedback function r: rewards or penalties are given based on whether the target speed is reached, and the trajectory length is limited.

[0094] The above definition can realize the training and application of the corresponding reinforcement learning model according to the context-based meta-reinforcement learning method of the embodiment of the present application. Figure 4 A schematic block diagram of the model training of meta-reinforcement learning in an embodiment of the present application is shown. Figure 5 A schematic block diagram of the model application of meta-reinforcement learning in an embodiment of the present application is shown below. Figure 4 and Figure 5This paper introduces the training and application of the model.

[0095] Model training part:

[0096] 1. From the task set {T i} i＝1,2,...,M Get a task T i , for task T i Initialize three experience replay pools c i , B i ,

[0097] 2. Initialize the context task encoder q φ , explore strategies Using Strategy π θ Among them, the context task encoder q φ , explore strategies Using Strategy π θ Both are neural network models.

[0098] 3. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is less than K1, perform the following steps, where K1 can be a preset value.

[0099] (1) According to task T i Get the corresponding environment state information s, based on the experience replay pool c i Calculate q φ (z|c i ), the context information c to be collected i As the context task encoder q φ The input of the context encoder is the posterior distribution q φ (z|c i ), the posterior distribution represents the confidence of the inference of the current task based on the current context information. Then q φ (z|c i ) as an exploration strategy The input is , and the output is Get and execute action a.

[0100] (2) After executing action a, you can obtain experience samples<s,a,r,s'> , respectively stored in the experience replay pool c i , B i , In the equation, r is the immediate feedback of the environment obtained by performing action a and interacting with the environment once, and s' is the state information of the environment at the next moment obtained by performing action a and interacting with the environment once.

[0101] (3) Determine the trajectory length. If the trajectory length is less than H, return to step 3 (1) and continue to execute steps 3 (1) to (3). If the trajectory length is greater than or equal to H, re-determine the number of sample trajectories that have been collected. H can be a preset value.

[0102] 4. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is greater than or equal to K1 and less than K2, perform the following steps, where K2 can be a preset value.

[0103] (1) According to task T i Get the corresponding environment state information s, based on the experience replay pool c i Calculate q φ (z|c i ), which will be c i As the context task encoder q φ The input is the posterior distribution q φ (z|c i ), sample z from it, z is the encoding of the task, and then use z as the utilization strategy π θ The input is π, and the output is θ (a|s,z), get and execute action a.

[0104] (2) After executing action a, you can obtain experience samples<s,a,r,s'> , respectively stored in the experience replay pool c i , B i middle.

[0105] (3) Determine the trajectory length. If the trajectory length is less than H, return to step 4 (1) and continue to execute steps 4 (1) to (3). If the trajectory length is greater than or equal to H, re-determine the number of sample trajectories that have been collected. H can be a preset value.

[0106] 5. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is greater than or equal to K2, determine whether the number of optimization times is greater than or equal to N. When the number of optimization times is less than N, perform the following steps, where N can be a preset value.

[0107] (1) From the experience replay pool Randomly sample batch samples From Experience Replay Pool B i Randomly sample batch samples b i .

[0108] (2) Calculate b i Each sample in each trajectory k ,a k ,r k ,s​k+1 >The corresponding intrinsic motivation r' MI , where the subscript k represents the kth sample, so that the reward obtained after executing action a can be obtained

[0109] The embodiments of the present application provide two methods for calculating intrinsic incentives.

[0110] The first method for calculating intrinsic motivation is to measure the mutual information between the encoding z of the task and the context information c:

[0111]

[0112] where c -1:k-1 is the context information, p(z|c -1:k ) represents the confidence of z inference based on context information, D KL is the KL divergence, which is used to measure the difference between two distributions.

[0113] The second method for calculating intrinsic motivation is to measure the gap between the feedback and the true value of the state and the predicted value based on the task encoding z:

[0114]

[0115] where c -1:k-1 is the context information, p(r k ,s k+1 |c -1:k+1 ,a k ,z) is the predicted value based on z, p(r k ,s k+1 |c -1:k-1 ,a k ) is the feedback and state truth value, and Dist() is implemented through KL divergence.

[0116] (3) Based on the sample and rewards Optimizing exploration strategies and using strategy π θ .

[0117] Specifically, off-policy reinforcement learning algorithms can be used, such as a soft actor critic (SAC) algorithm based on maximum entropy reinforcement learning, an asynchronous advantage actor critic (A3C) algorithm, a deep deterministic policy gradient (DDPG) algorithm, etc. The embodiments of the present application are not specifically limited here.

[0118] (4) Based on sample b i and samples Optimizing the context task encoder q φ Specifically, the context task encoder q φ The optimization goal is to reconstruct the value network Q value used to evaluate the strategy through the decoder. The Q value represents the expected benefit that can be obtained by taking action a in state s. θ The value network Q serves as the context task encoder q φ The corresponding decoder, sample As the context task encoder q φ Input, sample b i and As the input of the value network Q, it is optimized based on the following objective function:

[0119]

[0120] Among them, β is a hyperparameter and can be a manually preset value.

[0121] 6. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is greater than or equal to K2, determine whether the number of optimizations is greater than or equal to N. When the number of optimizations is greater than or equal to N, return to step 1 and re-acquire a task T. i , then follow steps 1 to 5.

[0122] When all neural network model context task encoders q φ , explore strategies Using Strategy π θ After convergence, the program ends and the trained context task encoder q is obtained. φ , explore strategies Using Strategy π θ .

[0123] After the neural network model is trained, it can be applied to specific scenarios. Here we still use the task of the humanoid walker reaching the target moving speed as an example.

[0124] Model application part:

[0125] 1. From the task set {T i} i＝1，2，...，M Get a task T i , for task T i Initialize the experience replay pool c.

[0126] 2. Load the trained context task encoder q φ , explore strategies Using Strategy πθ .

[0127] 3. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is less than K1, perform the following steps.

[0128] (1) According to task T i Get the corresponding environment state information s, and calculate q based on the experience replay pool c φ (z|c), that is, c is used as the context task encoder q φ The input is the posterior distribution q φ (z|c), then q φ (z|c) as an exploration strategy The input is , and the output is Get and execute action a.

[0129] (2) After executing action a, you can obtain experience samples<s,a,r,s'> , stored in the experience replay pool c, where r is the immediate feedback of the environment obtained by performing action a and interacting with the environment once, and s' is the environmental state information at the next moment obtained by performing action a and interacting with the environment once.

[0130] (3) Determine the trajectory length. If the trajectory length is less than H, return to step 3 (1) and continue to execute steps 3 (1) to (3). If the trajectory length is greater than or equal to H, re-determine the number of sample trajectories that have been collected. H can be a preset value.

[0131] 4. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is greater than or equal to K1 and less than K2, perform the following steps, where K2 can be a preset value.

[0132] (1) According to task T i Get the corresponding environment state information s, and calculate q based on the experience replay pool c φ (z|c), that is, c is used as the context task encoder q φ The input is the posterior distribution q φ (z|c), from which z is sampled, z is the encoding of the task, and then z is used as the utilization strategy π θ The input is π, and the output is θ (a|s,z), get and execute action a.

[0133] (2) After executing action a, you can obtain experience samples<s,a,r,s'> , stored in the experience replay pool c.

[0134] (3) Determine the trajectory length. If the trajectory length is less than H, return to step 4 (1) and continue to execute steps 4 (1) to (3). If the trajectory length is greater than or equal to H, re-determine the number of sample trajectories that have been collected. H can be a preset value.

[0135] 5. Determine the number of sample trajectories that have been collected. When the number of sample trajectories that have been collected is greater than or equal to K2, the trajectory execution is completed and the program ends.

[0136] During the application process, the neural network model no longer needs to use data to update the model. It only needs to replay the pool c based on the experience gained from the current task to update the execution degree of the current task inference.

[0137] The training method of meta-reinforcement learning in the embodiment of the present application samples tasks from the task set, completes sampling according to the exploration strategy and the utilization strategy for each task, and optimizes the exploration strategy, the utilization strategy, and the context task encoder after calculating the intrinsic incentive until the task set is fully covered and the neural network model converges. Then, tasks are randomly sampled based on the same task distribution, and the trained exploration strategy, the utilization strategy, and the context task encoder are used to adapt to the new task.

[0138] The training method of the meta-reinforcement learning of the embodiment of the present application is based on contextual meta-reinforcement learning and intrinsic incentive mechanism, and robust control is realized for multiple robots. Wherein the confidence in task judgment is obtained based on the contextual information related to the task and the intrinsic incentive is calculated, and independent exploration strategy and utilization strategy are obtained by maximizing the intrinsic incentive. In this way, while providing effective feedback for complex tasks, the mutual influence between the exploration strategy and the utilization strategy is reduced, so as to achieve efficient exploration of tasks, and finally it is possible to achieve rapid identification and adaptation to complex new tasks. According to multiple standard tasks in the Mujoco simulator, the effect of the meta-reinforcement learning method based on the context of the embodiment of the present application is verified. Under the premise of sparse feedback and limiting the number and length of trajectories, the existing meta-reinforcement learning methods such as MAML, RL^2, ProMP and PEARL are compared. The meta-reinforcement learning method based on the meta-reinforcement learning method of the embodiment of the present application can solve all standard tasks in the Mujoco simulator, and the exploration is efficient, and the adaptability to new tasks in the application stage has significant advantages.

[0139] The second application scenario is to apply the training method of meta-reinforcement learning of the embodiment of the present application to the adaptive visual cruise of indoor household robots. This task mainly controls the movable household robot in the indoor environment so that it can find different targets in different scenes. For example, in the living room scene, the robot is controlled to find the location of the TV based on visual information; or in the bedroom scene, the robot is controlled to find the table lamp based on visual information. Based on the above scenario, a series of task variants for indoor adaptive cruise tasks can be obtained, and the specific description dimensions of the task include scenes (such as kitchens, living rooms, bedrooms, etc.), targets (bowls, TVs, table lamps, etc.) and the starting position of the robot. A series of task variants are obtained by random combination of three dimensions, such as randomly extracting the kitchen from the scene list, randomly extracting the target microwave oven from the list of items in the kitchen, and finally randomly setting the robot to the starting position of the cruise in the kitchen.

[0140] Similar to the first application scenario, the embodiment of the present application uses the SAC reinforcement learning algorithm to implement the exploration strategy and utilization strategy learning of controlling the robot's cruise, and uses the variational autoencoder (VAE) to learn the context task encoder. 1. Task set; 2. State information; 3. Action information; 4. Feedback function. The details are as follows:

[0141] 1. Task set {T i} i＝1,2,...,M : A series of task variations can be obtained based on the preset scene list, the corresponding target list and the starting position;

[0142] 2. State information s: the robot’s position and the perception information of the environment based on its own perspective;

[0143] 3. Action information a: the waypoint information at the next moment, including the position and posture of the robot;

[0144] 4. Feedback function r: rewards or penalties are given based on whether the target speed is reached, and the trajectory length is limited.

[0145] The above definition can realize the training and application of the corresponding reinforcement learning model according to the context-based meta-reinforcement learning method of the embodiment of the present application. The specific process of model training and application can refer to the description in the first application scenario. For the sake of brevity, the embodiment of the present application will not be repeated here.

[0146] The training method of meta-reinforcement learning in the embodiment of the present application is based on contextual meta-reinforcement learning and intrinsic incentive mechanism to achieve adaptive cruising of indoor household robots. The confidence in task judgment is obtained based on task-related contextual information, and the intrinsic incentive is calculated, and an independent exploration strategy is obtained by maximizing the intrinsic incentive. In this way, effective feedback can be provided for complex tasks while reducing the mutual influence between exploration and strategy, thereby achieving efficient exploration of tasks, and ultimately achieving comprehensive coverage of cruising scenes and tasks, and using task-specific encoders to continuously update the inference of the current task to achieve rapid adaptation to small samples of new tasks.

[0147] The second application scenario is to apply the training method of meta-reinforcement learning of the embodiment of the present application to intersection cruising in autonomous driving. Similar to the first and second application scenarios, the embodiment of the present application uses the SAC reinforcement learning algorithm to implement the exploration strategy and utilization strategy learning of controlling the robot cruising, and uses the variational autoencoder (VAE) to learn the context task encoder. 1. Task set; 2. State information; 3. Action information; 4. Feedback function. The details are as follows:

[0148] 1. Task set {T i} i＝1,2,...,M : A series of mission variations can be obtained based on the preset scenario list (including different numbers of vehicles, different driving styles, and different numbers of lanes, etc.) and the corresponding driving target list (including left turn, right turn, straight, etc.) and the starting intersection;

[0149] 2. State information s: the location of the autonomous vehicle and the perception information of the environment based on its own perspective;

[0150] 3. Action information a: the waypoint information at the next moment, including the position and posture of the autonomous driving vehicle;

[0151] 4. Feedback function r: rewards or penalties are given based on whether the target speed is reached, and the trajectory length is limited.

[0152] The above definition can realize the training and application of the corresponding reinforcement learning model according to the context-based meta-reinforcement learning method of the embodiment of the present application. The specific process of model training and application can refer to the description in the first application scenario. For the sake of brevity, the embodiment of the present application will not be repeated here.

[0153] Figure 6600 is a schematic block diagram of a training device 600 for reinforcement learning in an embodiment of the present application. The device 600 can be used to execute the training method for reinforcement learning provided in the above embodiment, and for the sake of brevity, it is not repeated here. The device 600 can be a computer system, or a chip or circuit in a computer system, or can also be called an AI module. Figure 6 As shown, the device 600 includes:

[0154] The acquisition unit 610 is used to initialize the context task encoder, the exploration strategy and the exploitation strategy.

[0155] The interaction unit 620 is used to execute a first behavior according to the first environment information, the context task encoder and the exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior.

[0156] The interaction unit 620 is further configured to execute a second behavior according to the second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, wherein the second sample includes a second feedback of the environment in response to the second behavior.

[0157] The optimization unit 630 is used to optimize the exploration strategy and the utilization strategy according to the first feedback and the second feedback.

[0158] Optionally, the interaction unit 620 is also used to: before executing a first behavior obtained according to first environmental information, a context task encoder, and an exploration strategy, determine that the number of trajectories of a first sample is less than a first threshold; before executing a second behavior obtained according to second environmental information, a context task encoder, and an exploration strategy, determine that the number of trajectories of a second sample is less than a second threshold.

[0159] Optionally, the interaction unit 620 performs a first behavior based on the exploration strategy according to the context task encoder and the first environmental information to obtain a first sample based on the first behavior, including: obtaining the first environmental information as an input of the context task encoder to obtain a first output; using the first output as an input of the exploration strategy to obtain the first behavior; and executing the first behavior to obtain a first sample.

[0160] Optionally, the interaction unit performs a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: obtaining the second environment information as an input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; and executing the second behavior to obtain a second sample.

[0161] Optionally, the optimization unit 620 optimizes the exploration strategy and the utilization strategy according to the first feedback and the second feedback, including: obtaining a first intrinsic motivation corresponding to the first sample and a second intrinsic motivation corresponding to the second sample; obtaining a first reward according to the first intrinsic motivation and the first feedback; optimizing the exploration strategy according to the first reward; obtaining a second reward according to the second intrinsic motivation and the second feedback; and optimizing the utilization strategy according to the second reward.

[0162] Optionally, obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: obtaining first mutual information between the first environmental information and encoding of the first environmental information;

[0163] A first stimulus is obtained according to the first mutual information; second mutual information between the second environment information and the encoding of the second environment information is obtained; and a second stimulus is obtained according to the second mutual information.

[0164] Optionally, obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: obtaining a first true value of a first feedback and a first environmental state; obtaining a first predicted value of an encoding of the first environmental information; obtaining the first intrinsic excitation based on a gap between the first true value and the first predicted value; obtaining a second true value of the second feedback and the second environmental state; obtaining a second predicted value of the encoding of the second environmental information; and obtaining the second intrinsic excitation based on a gap between the second true value and the second predicted value.

[0165] Figure 7 : is a schematic structural diagram of a training device 700 for reinforcement learning in an embodiment of the present application. The device 700 can be used to execute the training method for reinforcement learning provided in the above embodiment, and for the sake of brevity, it is not repeated here. The device 700 includes: a processor 710, the processor 710 is coupled to a memory 720, the memory 720 is used to store computer programs or instructions, and the processor 710 is used to execute the computer program or instructions stored in the memory 720, so that the method in the above method embodiment is executed.

[0166] Figure 8 800 is a schematic block diagram of an apparatus 800 for adapting a task according to an embodiment of the present application. The apparatus 800 can be used to execute the method for adapting a task provided in the above embodiment, and for the sake of brevity, it is not described here in detail. The apparatus 800 can be a computer system, or a chip or circuit in a computer system, or can also be called an AI module. Figure 8 As shown, the device 800 includes:

[0167] The loading unit 810 is used to load the trained context task encoder, exploration strategy and utilization strategy.

[0168] The interaction unit 820 is used to execute a first behavior according to the first environment information, the context task encoder and the exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior.

[0169] The interaction unit 820 is further configured to execute a second behavior according to the second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, wherein the second sample includes a second feedback of the environment in response to the second behavior.

[0170] Optionally, the interaction unit 820 is also used to: before executing a first behavior obtained according to first environmental information, a context task encoder, and an exploration strategy, determine that the number of trajectories of a first sample is less than a first threshold; before executing a second behavior obtained according to second environmental information, a context task encoder, and an exploration strategy, determine that the number of trajectories of a second sample is less than a second threshold.

[0171] Optionally, the interaction unit 820 performs a first behavior based on the exploration strategy according to the context task encoder and the first environmental information to obtain a first sample based on the first behavior, including: obtaining the first environmental information as an input of the context task encoder to obtain a first output; using the first output as an input of the exploration strategy to obtain the first behavior; and executing the first behavior to obtain a first sample.

[0172] The interaction unit 820 performs a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: obtaining the second environment information as the input of the context task encoder to obtain a second output; using the second output as the input of the utilization strategy to obtain the second behavior; and executing the second behavior to obtain a second sample.

[0173] Fig. 9 : is a schematic structural diagram of an apparatus 900 for adapting a task according to an embodiment of the present application. The apparatus 900 can be used to execute the method for adapting a task provided in the above embodiment, and for the sake of brevity, it is not described here in detail. The apparatus 900 includes: a processor 910, the processor 910 is coupled to a memory 920, the memory 920 is used to store a computer program or instruction, and the processor 910 is used to execute the computer program or instruction stored in the memory 920, so that the method in the above method embodiment is executed.

[0174] The embodiment of the present application also provides a computer-readable storage medium on which computer instructions for implementing the method in the above method embodiment are stored.

[0175] For example, when the computer program is executed by a computer, the computer can implement the method in the above method embodiment.

[0176] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to implement the method in the above method embodiment.

[0177] An embodiment of the present application also provides an intelligent robot, which includes the device in the above embodiment and can execute the method in the above embodiment.

[0178] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method for reinforcement learning, characterized in that: include: Initializing a context task encoder, an exploration strategy, and an exploitation strategy, wherein the context task encoder, the exploration strategy, and the exploitation strategy are three neural network models; Execute a first behavior acquired according to first environment information, the context task encoder and the exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior; Execute a second behavior acquired according to second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, wherein the second sample includes a second feedback of the environment in response to the second behavior, and the first behavior and the second behavior are both behaviors of an intelligent carrier, and the intelligent carrier includes a fixed robot and a mobile robot; Optimizing the exploration strategy and the utilization strategy according to the first feedback and the second feedback to adapt to a task, wherein the task is to perform corresponding actions in a specific environment to achieve a preset goal; The performing a first behavior based on the exploration strategy according to the context task encoder and the first environment information to obtain a first sample based on the first behavior includes: Obtaining the first environment information as an input of the context task encoder to obtain a first output; Using the first output as an input of the exploration strategy to obtain the first behavior; Execute the first behavior to obtain the first sample; The performing a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior includes: Acquire the second environment information as input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; The second behavior is executed to obtain the second sample.

2. The method according to claim 1, characterized in that The method further comprises: Before executing a first behavior acquired according to the first environmental information, the context task encoder, and the exploration strategy, determining that the number of trajectories of the first sample is less than a first threshold; Before executing a second behavior acquired according to second environment information, the context task encoder, and the utilization strategy, it is determined that the number of trajectories of the second sample is less than a second threshold.

3. The method according to claim 1 or 2, characterized in that: The optimizing the exploration strategy and the utilization strategy according to the first feedback and the second feedback includes: Obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample; Obtaining a first reward according to the first intrinsic motivation and the first feedback; optimizing the exploration strategy according to the first reward; Obtaining a second reward according to the second intrinsic motivation and the second feedback; The utilization strategy is optimized according to the second reward.

4. The method according to claim 3, characterized in that The obtaining of a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: Acquire first mutual information between the first environment information and encoding of the first environment information; Acquire the first intrinsic excitation according to the first mutual information; Acquire second mutual information between the second environment information and encoding of the second environment information; The second intrinsic excitation is obtained according to the second mutual information.

5. The method according to claim 3, characterized in that: The obtaining of a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: Acquire a first truth value of the first feedback and a first environmental state; Obtaining a first prediction value of an encoding of first environmental information; Acquire the first intrinsic incentive according to the difference between the first true value and the first predicted value; Acquire a second truth value of the second feedback and a second environmental state; Obtaining a second prediction value for encoding the second environmental information; The second intrinsic motivation is obtained according to the gap between the second true value and the second predicted value.

6. A method for adapting a task, characterized in that The task is to perform corresponding actions in a specific environment to achieve a preset goal, and the method includes: Loading the trained context task encoder, exploration strategy and utilization strategy, wherein the context task encoder, the exploration strategy and the utilization strategy are three neural network models; Execute a first behavior according to first environment information, the context task encoder and the exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior; executing a second behavior according to second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, wherein the second sample includes a second feedback of the environment in response to the second behavior, wherein the first behavior and the second behavior are both behaviors of an intelligent carrier, and the intelligent carrier includes a fixed robot and a mobile robot; The performing a first behavior based on the exploration strategy according to the context task encoder and the first environment information to obtain a first sample based on the first behavior includes: Obtaining the first environment information as an input of the context task encoder to obtain a first output; Using the first output as an input of the exploration strategy to obtain the first behavior; Execute the first behavior to obtain the first sample; The performing a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior includes: Acquire the second environment information as input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; The second behavior is executed to obtain the second sample.

7. The method according to claim 6, characterized in that The method further comprises: Before executing a first behavior acquired according to the first environmental information, the context task encoder, and the exploration strategy, determining that the number of trajectories of the first sample is less than a first threshold; Before executing a second behavior acquired according to second environment information, the context task encoder, and the utilization strategy, it is determined that the number of trajectories of the second sample is less than a second threshold.

8. A training device for reinforcement learning, characterized in that: include: An initialization unit, used to initialize a context task encoder, an exploration strategy, and an utilization strategy, wherein the context task encoder, the exploration strategy, and the utilization strategy are three neural network models; an interaction unit, configured to execute a first behavior according to first environmental information, the contextual task encoder, and the exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior; The interaction unit is further used to execute a second behavior according to second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, the second sample includes a second feedback of the environment in response to the second behavior, the first behavior and the second behavior are both behaviors of an intelligent carrier, and the intelligent carrier includes a fixed robot and a mobile robot; an optimization unit, configured to optimize the exploration strategy and the utilization strategy according to the first feedback and the second feedback to adapt to a task, wherein the task is to perform corresponding actions in a specific environment to achieve a preset goal; The interaction unit performs a first behavior based on the exploration strategy according to the context task encoder and the first environment information to obtain a first sample based on the first behavior, including: Obtaining the first environment information as an input of the context task encoder to obtain a first output; Using the first output as an input of the exploration strategy to obtain the first behavior; Execute the first behavior to obtain the first sample; The interaction unit performs a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: Acquire the second environment information as input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; The second behavior is executed to obtain the second sample.

9. The device according to claim 8, characterized in that The interaction unit is also used for: Before executing a first behavior acquired according to the first environmental information, the context task encoder, and the exploration strategy, determining that the number of trajectories of the first sample is less than a first threshold; Before executing a second behavior acquired according to second environment information, the context task encoder, and the utilization strategy, it is determined that the number of trajectories of the second sample is less than a second threshold.

10. The device according to claim 8 or 9, characterized in that The optimization unit optimizes the exploration strategy and the utilization strategy according to the first feedback and the second feedback, including: Obtaining a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample; Obtaining a first reward according to the first intrinsic motivation and the first feedback; optimizing the exploration strategy according to the first reward; Obtaining a second reward according to the second intrinsic motivation and the second feedback; The utilization strategy is optimized according to the second reward.

11. The device according to claim 10, characterized in that The obtaining of a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: Acquire first mutual information between the first environment information and encoding of the first environment information; Acquire the first intrinsic excitation according to the first mutual information; Acquire second mutual information between the second environment information and encoding of the second environment information; The second intrinsic excitation is obtained according to the second mutual information.

12. The device according to claim 10, characterized in that The obtaining of a first intrinsic excitation corresponding to the first sample and a second intrinsic excitation corresponding to the second sample includes: Acquire a first truth value of the first feedback and a first environmental state; Obtaining a first prediction value of an encoding of first environmental information; Acquire the first intrinsic incentive according to the difference between the first true value and the first predicted value; Acquire a second truth value of the second feedback and a second environmental state; Obtaining a second prediction value for encoding the second environmental information; The second intrinsic motivation is obtained according to the gap between the second true value and the second predicted value.

13. A device for adapting tasks, characterized in that The task is to perform corresponding actions in a specific environment to achieve a preset goal, and the device includes: A loading unit, used to load the trained context task encoder, exploration strategy and utilization strategy, wherein the context task encoder, the exploration strategy and the utilization strategy are three neural network models; an interaction unit, configured to execute a first behavior according to first environmental information, the contextual task encoder, and the exploration strategy to obtain a first sample based on the first behavior, wherein the first sample includes a first feedback of the environment in response to the first behavior; The interaction unit is further used to execute a second behavior according to second environment information, the context task encoder and the utilization strategy to obtain a second sample based on the second behavior, the second sample includes a second feedback of the environment in response to the second behavior, the first behavior and the second behavior are both behaviors of an intelligent carrier, and the intelligent carrier includes a fixed robot and a mobile robot; The interaction unit performs a first behavior based on the exploration strategy according to the context task encoder and the first environment information to obtain a first sample based on the first behavior, including: Obtaining the first environment information as an input of the context task encoder to obtain a first output; Using the first output as an input of the exploration strategy to obtain the first behavior; Execute the first behavior to obtain the first sample; The interaction unit performs a second behavior based on the utilization strategy according to the context task encoder and the second environment information to obtain a second sample based on the second behavior, including: Acquire the second environment information as input of the context task encoder to obtain a second output; using the second output as an input of the utilization strategy to obtain the second behavior; The second behavior is executed to obtain the second sample.

14. The device according to claim 13, characterized in that The interaction unit is also used for: Before executing a first behavior acquired according to the first environmental information, the context task encoder, and the exploration strategy, determining that the number of trajectories of the first sample is less than a first threshold; Before executing a second behavior acquired according to second environment information, the context task encoder, and the utilization strategy, it is determined that the number of trajectories of the second sample is less than a second threshold.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by a processor, the method of any one of claims 1 to 5 or 6 to 7 is implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the method of any one of claims 1 to 5 or 6 to 7 is implemented.

17. An intelligent robot, characterized in that: A device comprising any one of claims 8 to 12 or 13 to 14.

Images