Reinforcement learning method based on meta-target learning

Abstract

The invention provides a reinforcement learning method based on meta-target learning, and relates to the technical field of machine learning, and the method comprises the steps: controlling a robot to act based on a state signal observed by the robot; constructing a reinforcement learning model by using the meta-objective function and the double-layer optimization framework; after a tetrad is obtained through interaction with the environment, firstly, reviewer parameters in the reinforcement learning model are updated, and secondly, actor parameters in the reinforcement learning model are updated in combination with a meta-objective function and a reinforcement learning loss function calculated by a reviewer; alternately training the reinforcement learning and the meta-objective function until convergence, determining a final meta-objective function, reviewer parameters and actor parameters, obtaining a trained reinforcement learning model, and completing reinforcement learning based on meta-objective learning; wherein the reinforcement learning model is used for analyzing the current state data of the robot to obtain updated robot motion control data. According to the method, the constraint problem of a manual objective function on a reinforcement learning strategy is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a reinforcement learning method based on meta-goal learning. Background Art

[0002] Reinforcement Learning (RL) is one of the most basic and important fields in machine learning research. It aims to maximize environmental rewards through trial and error, that is, to learn strategies that can maximize rewards in the cycle of "receiving state, taking action, obtaining rewards and next state, and improving strategy". Reinforcement learning has achieved great success in various tasks with dense rewards, such as Go, games, and large language models. However, reinforcement learning has difficulty learning optimal strategies from real-world tasks with sparse rewards. For example, there is a reward signal only when the navigation robot reaches the end point, and there is no reward feedback for many intermediate steps. Effective reinforcement learning in sparse reward environments is of great benefit to many applications, such as autonomous driving, product recommendations, and robot motion control.

[0003] For reinforcement learning in sparse reward environments, researchers have proposed many methods, such as reward reshaping, hierarchical reinforcement learning, and imitation learning (IL). Existing work is mainly devoted to using imitation learning to help reinforcement learning agents obtain reward signals, thereby improving online reinforcement learning in sparse reward environments. Specifically, by imitating actions in specific states in offline demonstration data, the agent's strategy is made close to the (suboptimal) demonstration strategy; based on the (suboptimal) demonstration strategy, the agent is sufficient to obtain the necessary reward signals in a sparse reward environment, thereby gradually improving to the optimal strategy. However, these methods find it difficult to strike a balance between imitation learning and reinforcement learning (such as Figure 1 As shown in Figure 2, the intelligent agent often falls into the suboptimal strategy brought by imitation learning.

[0004] The difficulty in achieving a balance between imitation learning and reinforcement learning is that it is difficult to achieve both versatility and high performance. Existing methods mainly balance imitation learning and reinforcement learning in two ways. The first method manually adjusts hyperparameters on specific tasks to achieve high performance, but when encountering new tasks, a lot of time needs to be spent on re-adjusting parameters, which seriously reduces versatility. The second method automatically balances dynamic parameters to ensure high performance while improving scenario versatility, but dynamic parameters are only compatible with a single reinforcement learning algorithm and cannot enable other reinforcement learning algorithms to achieve high performance. Therefore, it is necessary to design a method that can automatically balance imitation learning and reinforcement learning to ensure high performance, and can be combined with multiple reinforcement learning algorithms in different scenarios. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a reinforcement learning method based on meta-goal learning to solve the constraint problem of manual objective function on reinforcement learning strategy.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a reinforcement learning method based on meta-goal learning, comprising:

[0007] S1: Based on the robot motion control data, obtain the robot state quaternary;

[0008] S2: constructing a reinforcement learning model using a meta-objective function and a two-layer optimization framework; wherein the reinforcement learning model includes a meta-training layer constructed using a meta-objective function and an actor-critic algorithm and a meta-optimization layer constructed using reinforcement learning-imitation learning scoring;

[0009] S3: inputting the robot state quadruple into the reinforcement learning model, updating the critic parameters in the reinforcement learning model by minimizing the critic parameters and the meta-objective function, and updating the actor parameters in the reinforcement learning model by combining the meta-objective function and the reinforcement learning loss function calculated by the critic, to obtain updated critic parameters and updated actor parameters;

[0010] S4: inputting the four-tuple into the upper meta-optimization layer, training to obtain a better meta-objective function, using the small batch training data and the updated critic parameters, calculating the meta-loss, updating the meta-objective function, and obtaining an updated meta-objective function;

[0011] S5: Iteratively execute S3 and S4, alternately train reinforcement learning and meta-objective function until convergence, determine the final meta-objective function, critic parameters and actor parameters, obtain the trained reinforcement learning model, and complete the reinforcement learning based on meta-objective learning; wherein the reinforcement learning model is used to analyze the current state data of the robot to obtain updated robot motion control data.

[0012] The beneficial effects of the present invention are as follows: the processor uses the four-tuple collected during the interaction between the robot and the environment to train the reinforcement learning model, and the strategy generated by the model can analyze the environmental state signal and control the robot's motion. The meta-objective function proposed by the present invention extracts knowledge from offline demonstration data to improve reinforcement learning, avoiding the constraints of the manual objective function on the reinforcement learning strategy in traditional imitation learning, and achieving a balance between imitation learning and reinforcement learning under a suboptimal demonstration strategy.

[0013] Further, the S1 includes:

[0014] The robot observes the environment through sensors and obtains current status data;

[0015] Based on the current state data, the robot motion control data is used to control the robot to make movements and interact with the environment to obtain movement data;

[0016] When the robot's action changes the environment, the environment shifts to the next state, and the next state data and the robot's reward data are obtained; wherein the current state data, the action data, the next state data and the reward data belong to the robot state quadruple.

[0017] Furthermore, the expression of the two-layer optimization framework is:

[0018]

[0019] Among them, ω represents the parameter of the meta-loss function, φ * represents the optimal actor parameter, θ * represents the optimal critic parameters, represents the loss of yuan, Minimize To obtain the optimal parameters of the meta-loss function, represents the reinforcement learning loss function output by the critic, represents the meta-objective function, Minimize and To obtain the parameters φ, L MSBE (θ) represents the mean square Bellman loss function, Minimize L MSBE (θ) to obtain the parameter θ * .

[0020] Unlike mainstream methods that are limited to a single reinforcement learning algorithm, the meta-objective function proposed in the present invention can be organically combined with multiple reinforcement learning algorithms, greatly improving the versatility under different reinforcement learning algorithms. The inventors combined the meta-objective function with three mainstream reinforcement learning benchmark algorithms, and each algorithm was experimented on four different robot motion tasks. The experimental results show that the meta-objective function proposed in the present invention can be organically combined with multiple reinforcement learning algorithms to improve the robot's motion ability in a sparse reward environment.

[0021] Further, the S3 includes:

[0022] S310: storing the robot state quadruple in an experience replay buffer, and randomly sampling small batches of training data therefrom as initial input of a reinforcement learning model;

[0023] S320: using the small batch training data and minimizing the mean square Bellman error loss function, updating the critic parameters to obtain updated critic parameters;

[0024] S330: Using the small batch training data, the combined meta-objective function and the reinforcement learning loss function calculated by the updated critic parameters, the actor parameters are updated to obtain updated actor parameters.

[0025] Unlike mainstream methods that require a lot of time to adjust hyperparameters, the meta-objective function proposed in the present invention can automatically adjust the internal parameters of the module when dealing with different scenarios. There is no need to manually adjust the module hyperparameters, which greatly improves the versatility in different scenarios.

[0026] Furthermore, the expression of the updated critic parameter is:

[0027]

[0028] Among them, θ (k) represents the critic parameter at step k, θ (k+1) represents the critic parameter of the k+1th step, α represents the model learning rate, represents the derivative of the parameter θ, N represents the batch size of random sampling,<s,a,r,s'> is a four-tuple randomly sampled from the experience replay buffer, s represents the current state, a represents the action, r represents the reward, s′ represents the next state, D represents the experience replay buffer, Q θ (s,a) represents the state-action value function calculated by the critic based on the current state and action, γ represents the depreciation rate, Q θ (s′, φ(s′)) represents the state-action value function calculated by the critic based on the next state and the action taken by the actor in the next state, φ (k) represents the actor parameters at step k;

[0029] The expression of the updated actor parameter is:

[0030]

[0031] Among them, φ (k+1) represents the actor parameter at step k+1, φ (k) represents the actor parameters at step k, represents the derivative of parameter φ, s d Indicates the demonstration state, a d Indicates demonstration action, Q θ (s d ,φ(s d )) represents the state-action value function calculated by the critic based on the demonstration state and the action taken by the actor in the demonstration state, represents the meta-objective function, ω (k) represents the parameter of the meta-objective function at step k; s d and a d Belongs to small batch training data, Qθ (s d ,φ(s d )) belongs to the updated critic parameters.

[0032] The meta-loss compares the performance of reinforcement learning agents trained based on the meta-objective function with the performance of reinforcement learning agents trained based on traditional imitation learning, aiming to make the meta-objective function look critically at suboptimal demonstration data, thereby surpassing traditional imitation learning. The meta-loss designed based on the above idea has stronger interpretability and can cope with different robot motion tasks and offline demonstration data of different quality.

[0033] Further, the S4 includes:

[0034] Using the small batch training data, the parameters of the actor replica are updated through traditional imitation learning and reinforcement learning to obtain updated actor replica parameters:

[0035]

[0036] in, Parameters representing the actor's replica, represents the parameters of the actor replica at step k+1, represents the actor replica parameters at step k, Indicates the parameters Find the derivative, D dem represents offline demonstration data, (s d ,a d ) is a state-action pair randomly sampled from offline demonstration data, s d Indicates the demonstration state, a d Indicates demonstration action, w rl and w il represents the hyperparameters that balance reinforcement learning and imitation learning, represents the state-action value function calculated by the critic based on the demonstration state and the actions taken by the actor replica in the demonstration state, Represents the objective function in traditional imitation learning;

[0037] Using the updated actor replica parameters and the updated critic parameters, according to the mini-batch training data, the meta-loss is obtained by calculation:

[0038]

[0039] in, represents the meta-loss, φ (k+1) represents the actor replica parameter at step k+1, s val represents the demonstration state for meta-loss computation, θ (k+1)represents the critic parameter at step k+1, represents the parameters of the actor replica, ω (k) represents the parameters of the meta-objective function at the k-th step;

[0040] Based on the meta-loss, calculate the derivative of the meta-loss with respect to the meta-objective function:

[0041]

[0042] in, represents, ω represents the parameters of the meta-objective function, φ represents the actor parameters, represents the derivative of φ with respect to ω Abbreviation of φ (k+1) represents the actor parameters at step k+1, θ (k+1) represents the critic parameter at step k+1, ω (k) represents the parameters of the meta-objective function at the k-th step;

[0043] Based on the derivative of the meta-loss with respect to the meta-objective function, the meta-objective function is updated to obtain an updated meta-objective function:

[0044]

[0045] Among them, ω (k+1) represents the parameter of the meta-objective function at the k+1th step, α represents the model learning rate, represents the meta-objective function, φ (k) represents the actor parameter at step k, φ (k+1) represents the actor parameters at the k+1th step.

[0046] Thanks to the core idea of ​​extracting information from data, the meta-objective function proposed in this invention converges after quickly extracting all valid information. This provides a way to maintain performance and accelerate training: use the meta-objective function and the actor-critic framework to train the reinforcement learning agent before the meta-objective function converges, and only use the actor-critic framework to train the reinforcement learning agent after the meta-objective function converges. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0048] Figure 1 This is an exemplary flowchart of a reinforcement learning method based on meta-goal learning according to some embodiments of this specification. DETAILED DESCRIPTION

[0049] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0050] Example

[0051] Figure 1 is an exemplary flow chart of a reinforcement learning method based on meta-goal learning according to some embodiments of this specification. Figure 1 As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0052] S1: Based on the robot motion control data, obtain the robot state quaternary.

[0053] Robot motion control data is data used to control the motion of the robot. For example, the robot motion control data may include data such as environment images, robot positions, joint angles, motion speeds, etc.

[0054] In some embodiments, the processor may acquire digital signals based on robot sensors as robot motion control data.

[0055] The robot state quadruple is a data set that reflects the interaction between the robot and the environment. For example, the robot state quadruple may include current state data, action data, next state data, and reward data.

[0056] In some embodiments, the robot can continuously interact with the environment and collect multiple <current state, action, reward, next state> quadruple as the robot state quadruple.

[0057] In some embodiments, the processor can implement S1 based on the following steps: the robot observes the environment through sensors and obtains current state data; based on the current state data, the robot motion control data is used to control the robot to make actions to interact with the environment and obtain action data; when the robot's actions change the environment, the environment shifts to the next state, and the next state data and the robot's reward data are obtained.

[0058] The current state data is data reflecting the current motion state of the robot.

[0059] In some embodiments, the processor can obtain digital signals such as images, robot positions, joint angles, movement speeds, etc. based on robot sensors as current state data.

[0060] Motion data is data used to control the movement of the robot.

[0061] In some embodiments, the processor can control the torque applied to various joints of the robot, such as ankles, knees, and wrists, so as to adjust the direction and speed of movement of the robot as the motion data of the robot.

[0062] The next state data is data reflecting the robot's next stage of motion state.

[0063] Reward data is the data used to motivate the robot to interact with the environment.

[0064] S2: Constructing a reinforcement learning model using a meta-objective function and a two-layer optimization framework; wherein the reinforcement learning model includes a meta-training layer constructed using a meta-objective function and an actor-critic algorithm and a meta-optimization layer constructed using reinforcement learning-imitation learning scoring.

[0065] The meta-objective function is the learning objective of the upper meta-optimization layer, a function produced by the general demonstration-based imitation learning module.

[0066] The two-layer optimization framework is an optimization framework built to train and optimize reinforcement learning models in sparse reward environments, including a lower meta-training layer and an upper meta-optimization layer.

[0067] In some embodiments, the expression of the two-layer optimization framework is:

[0068]

[0069] Among them, ω represents the parameter of the meta-loss function, φ * represents the optimal actor parameter, θ * represents the optimal critic parameters, represents the loss of yuan, Minimize To obtain the optimal parameters of the meta-loss function, represents the reinforcement learning loss function output by the critic, represents the meta-objective function, Minimize and To obtain the parameters φ, L MSBE (θ) represents the mean square Bellman loss function, Minimize L MSBE (θ) to obtain the parameter θ * .

[0070] The reinforcement learning model is a neural network model used to analyze the robot's state and obtain the robot's motion control data. Reinforcement learning can use a variety of neural network models, such as convolutional neural network models, generative adversarial neural network models, etc.

[0071] In some embodiments, the input of the reinforcement learning model may be a robot state quadruple, and the output of the reinforcement learning model may be robot motion control data.

[0072] In some embodiments, the structure of the reinforcement learning model is as follows:

[0073] The reinforcement learning model includes a meta-training layer and a meta-optimization layer. The output of the meta-optimization layer is used as the input of the meta-training layer, and the output of the meta-training layer is used as the final output of the reinforcement learning model.

[0074] The meta-optimization layer is used to train the meta-objective function using the updated actor parameters and the updated critic parameters to obtain an updated meta-objective function. The input of the meta-optimization layer may include the updated actor parameters and the updated critic parameters, and the output may include the updated meta-objective function.

[0075] The meta-training layer is used to update the actor parameters and the critic parameters. The input of the meta-training layer may include the robot state quadruple and the updated meta-objective function, and the output may include the updated actor parameters and the updated critic parameters.

[0076] In some embodiments, the processor can combine the meta-objective function and the critic to train the actor, and finally the actor analyzes the state of the robot, controls the action of the robot, and obtains the robot motion control data.

[0077] Robot motion control data is data used to adjust the robot's motion direction and speed. For example, robot motion control data may include joint angles and applied torques, etc.

[0078] In some embodiments, the processor can adjust the robot's movements (adjust the joint angles and the applied torque to control the robot's forward direction and speed) based on the robot's motion control data. After the robot moves to a new position in the environment, it re-obtains status and environmental reward data from the sensor, and then re-acts according to the strategy, interacting with the environment over and over again.

[0079] S3: Input the robot state quadruple into the reinforcement learning model, update the critic parameters in the reinforcement learning model by minimizing the critic parameters and the meta-objective function, and update the actor parameters in the reinforcement learning model by combining the meta-objective function and the reinforcement learning loss function calculated by the critic to obtain updated critic parameters and updated actor parameters.

[0080] Actor parameters are the policy parameters used to control the robot's motion.

[0081] Critic parameters are the parameters used to evaluate how good an actor's strategy is.

[0082] In some embodiments, the processor may implement S3 based on the following steps.

[0083] S310: Storing the robot state quadruple in an experience replay buffer, and randomly sampling small batches of training data therefrom as initial input of a reinforcement learning model.

[0084] The experience replay buffer is a storage unit for storing and reusing the quadruplets.

[0085] In some embodiments, the memory may construct an experience replay buffer based on the quadruple obtained by the processor interacting with the environment.

[0086] The mini-batch training data is a mini-batch of robot state quadruple data used to train the reinforcement learning model.

[0087] S320: Using the small batch training data and minimizing the mean square Bellman error loss function, updating the critic parameters to obtain updated critic parameters.

[0088] In some embodiments, the expression of the updated critic parameter may be:

[0089]

[0090] Among them, θ (k) represents the critic parameter at step k, θ (k+1) represents the critic parameter of the k+1th step, α represents the model learning rate, represents the derivative of the parameter θ, N represents the batch size of random sampling, s, a, r, s'> is a four-tuple randomly sampled from the experience replay buffer, s represents the current state, a represents the action, r represents the reward, s' represents the next state, D represents the experience replay buffer, Q θ (s,a) represents the state-action value function calculated by the critic based on the current state and action, γ represents the depreciation rate, Q θ (s′, φ(s′)) represents the state-action value function calculated by the critic based on the next state and the action taken by the actor in the next state, φ (k) represents the actor parameters at the k-th step.

[0091] S330: Using the small batch training data, the combined meta-objective function and the reinforcement learning loss function calculated by the updated critic parameters, the actor parameters are updated to obtain updated actor parameters.

[0092] In some embodiments, the expression of the updated actor parameter may be:

[0093]

[0094] Among them, φ (k+1) represents the actor parameter at step k+1, φ (k) represents the actor parameters at step k, represents the derivative of parameter φ, s d Indicates the demonstration state, a d Indicates demonstration action, Q θ (s d ,φ(s d )) represents the state-action value function calculated by the critic based on the demonstration state and the action taken by the actor in the demonstration state, represents the meta-objective function, ω (k) represents the parameter of the meta-objective function at step k; s d and a d Belongs to small batch training data, Q θ (s d ,φ(s d )) belongs to the updated critic parameters.

[0095] S4: Input the four-tuple into the upper meta-optimization layer, train to obtain a better meta-objective function, use the updated critic parameters and the updated actor parameters to calculate the meta-loss, update the meta-objective function, and obtain an updated meta-objective function.

[0096] In some embodiments, the reinforcement learning model can be obtained by training using quads as training samples. For example, it is obtained by continuously interacting with the environment and collecting quads in the interaction process. A loss function is constructed around the rewards in the quads, and the parameters of the initial reinforcement learning model are iteratively updated by gradient descent or other methods based on the loss function. When the preset conditions are met, the model training is completed and a trained reinforcement learning model is obtained. Among them, the preset conditions can be that the loss function converges, the number of iterations reaches a threshold, etc.

[0097] In some embodiments, the training sample may include a 4-tuple of <current state, action, reward, next state> recorded during the robot's interaction with the environment. The state may be a digital signal such as an image obtained from the robot's sensor, the robot's position, joint angle, and movement speed. The action may be a torque applied to each joint of the robot such as the ankle, knee, and wrist as the robot's movement.

[0098] In some embodiments, the processor may implement S4 based on the following steps: using the mini-batch training data, through traditional imitation learning and reinforcement learning, updating the parameters of the actor replica to obtain updated actor replica parameters; using the updated actor replica parameters and the updated critic parameters, according to the mini-batch training data, through calculation, obtaining a meta-loss; based on the meta-loss, calculating the derivative of the meta-loss with respect to the meta-objective function; based on the derivative of the meta-loss with respect to the meta-objective function, updating the meta-objective function to obtain an updated meta-objective function.

[0099] The actor parameter replica is an independent replica of the actor parameters before the update.

[0100] In some embodiments, the expression of the actor parameter replica may be:

[0101]

[0102] in, Parameters representing the actor's replica, represents the parameters of the actor replica at step k+1, represents the actor replica parameters at step k, Indicates the parameters Find the derivative, D dem represents offline demonstration data, (s d ,a d ) is a state-action pair randomly sampled from offline demonstration data, s d Indicates the demonstration state, a d Indicates demonstration action, w rl and w il represents the hyperparameters that balance reinforcement learning and imitation learning, represents the state-action value function calculated by the critic based on the demonstration state and the actions taken by the actor replica in the demonstration state, Represents the objective function in traditional imitation learning.

[0103] Meta-loss is a parameter that reflects the loss of the meta-objective function.

[0104] In some embodiments, the expression of the meta-loss may be:

[0105]

[0106] in, represents the meta-loss, φ (k+1) represents the actor replica parameter at step k+1, s val surface

[0107] Demonstration state of the loss calculation, θ (k+1)represents the critic parameter at step k+1, represents the parameters of the actor replica, ω (k) represents the parameters of the meta-objective function at the k-th step.

[0108] In some embodiments, the expression of the derivative of the meta-objective function may be:

[0109]

[0110] in, represents, ω represents the parameters of the meta-objective function, φ represents the actor parameters, represents the derivative of φ with respect to ω Abbreviation of φ (k+1) represents the actor parameters at step k+1, θ (k+1) represents the critic parameter at step k+1, ω (k) represents the parameters of the meta-objective function at the k-th step.

[0111] The updated meta-objective function is the meta-objective function used to supervise the reinforcement learning model for training.

[0112] In some embodiments, the expression of the updated meta-objective function may be:

[0113]

[0114] Among them, ω (k+1) represents the parameter of the meta-objective function at the k+1th step, α represents the model learning rate, represents the meta-objective function, φ (k) represents the actor parameter at step k, φ (k+1) represents the actor parameters at the k+1th step.

[0115] S5: Iteratively execute S3 and S4, alternately train reinforcement learning and meta-objective function until convergence, determine the final meta-objective function, critic parameters and actor parameters, obtain the trained reinforcement learning model, and complete the reinforcement learning based on meta-objective learning; wherein the reinforcement learning model is used to analyze the current state data of the robot to obtain updated robot motion control data.

[0116] In some embodiments of this specification, the processor uses the four-tuple collected during the interaction between the robot and the environment to train the reinforcement learning model, and the strategy generated by the model can analyze the environmental state signal and control the robot's motion. The meta-objective function proposed by the present invention extracts knowledge from offline demonstration data to improve reinforcement learning, avoiding the constraints of the manual objective function on the reinforcement learning strategy in traditional imitation learning, and achieving a balance between imitation learning and reinforcement learning under a suboptimal demonstration strategy.

Claims

1. A reinforcement learning method based on meta-goal learning, characterized in that: include: S1: Based on the robot motion control data, obtain the robot state quaternary; S2: constructing a reinforcement learning model using a meta-objective function and a two-layer optimization framework; wherein the reinforcement learning model includes a meta-training layer constructed using a meta-objective function and an actor-critic algorithm and a meta-optimization layer constructed using reinforcement learning-imitation learning scoring; S3: inputting the robot state quadruple into the reinforcement learning model, updating the critic parameters in the reinforcement learning model by minimizing the critic parameters and the meta-objective function, and updating the actor parameters in the reinforcement learning model by combining the meta-objective function and the reinforcement learning loss function calculated by the critic, to obtain updated critic parameters and updated actor parameters; S4: inputting the four-tuple into the upper meta-optimization layer, training to obtain a better meta-objective function, using the small batch training data and the updated critic parameters, calculating the meta-loss, updating the meta-objective function, and obtaining an updated meta-objective function; S5: Iteratively execute S3 and S4, alternately train reinforcement learning and meta-objective function until convergence, determine the final meta-objective function, critic parameters and actor parameters, obtain the trained reinforcement learning model, and complete the reinforcement learning based on meta-objective learning; wherein the reinforcement learning model is used to analyze the current state data of the robot to obtain updated robot motion control data.

2. The reinforcement learning method based on meta-goal learning according to claim 1, characterized in that: The S1 includes: The robot observes the environment through sensors and obtains current status data; Based on the current state data, the robot motion control data is used to control the robot to make movements and interact with the environment to obtain movement data; When the robot's action changes the environment, the environment shifts to the next state, and the next state data and the robot's reward data are obtained; wherein the current state data, the action data, the next state data and the reward data belong to the robot state quadruple.

3. The reinforcement learning method based on meta-goal learning according to claim 1, characterized in that: The expression of the two-layer optimization framework is: Among them, ω represents the parameter of the meta-loss function, φ * represents the optimal actor parameter, θ * represents the optimal critic parameters, represents the loss of yuan, Minimize To obtain the optimal parameters of the meta-loss function, represents the reinforcement learning loss function output by the critic, represents the meta-objective function, Minimize and To obtain the parameters φ, L MSBE (θ) represents the mean square Bellman loss function, Minimize L MSBE (θ) to obtain the parameter θ * .

4. The reinforcement learning method based on meta-goal learning according to claim 1, characterized in that: The S3 includes: S310: storing the robot state quadruple in an experience replay buffer, and randomly sampling small batches of training data therefrom as initial input of a reinforcement learning model; S320: using the small batch training data and minimizing the mean square Bellman error loss function, updating the critic parameters to obtain updated critic parameters; S330: Using the small batch training data, the combined meta-objective function and the reinforcement learning loss function calculated by the updated critic parameters, the actor parameters are updated to obtain updated actor parameters.

5. The reinforcement learning method based on meta-goal learning according to claim 4, characterized in that: The expression of the updated critic parameter is: Among them, θ (k) represents the critic parameter at step k, θ (k+1) represents the critic parameter of the k+1th step, α represents the model learning rate, represents the derivative of the parameter θ, N represents the batch size of random sampling,<s,a,r,s'> is a four-tuple randomly sampled from the experience replay buffer, s represents the current state, a represents the action, r represents the reward, s′ represents the next state, D represents the experience replay buffer, Q θ (s,a) represents the state-action value function calculated by the critic based on the current state and action, γ represents the depreciation rate, Q θ (s′, φ(s′)) represents the state-action value function calculated by the critic based on the next state and the action taken by the actor in the next state, φ (k) represents the actor parameters at step k; The expression of the updated actor parameter is: Among them, φ (k+1) represents the actor parameter at step k+1, φ (k) represents the actor parameters at step k, represents the derivative of parameter φ, s d Indicates the demonstration state, a d Indicates demonstration action, Q θ (s d ,φ(s d )) represents the state-action value function calculated by the critic based on the demonstration state and the action taken by the actor in the demonstration state, represents the meta-objective function, ω (k) represents the parameter of the meta-objective function at step k; s d and a d Belongs to small batch training data, Q θ (s d ,φ(s d )) belongs to the updated critic parameters.

6. The reinforcement learning method based on meta-goal learning according to claim 1, characterized in that: The S4 includes: Using the small batch training data, the parameters of the actor replica are updated through traditional imitation learning and reinforcement learning to obtain updated actor replica parameters: in, Parameters representing the actor's replica, represents the parameters of the actor replica at step k+1, represents the actor replica parameters at step k, Indicates the parameters Find the derivative, D dem represents offline demonstration data, (s d ,a d ) is a state-action pair randomly sampled from offline demonstration data, s d Indicates the demonstration state, a d Indicates demonstration action, w rl and w il represents the hyperparameters that balance reinforcement learning and imitation learning, represents the state-action value function calculated by the critic based on the demonstration state and the actions taken by the actor replica in the demonstration state, Represents the objective function in traditional imitation learning; Using the updated actor replica parameters and the updated critic parameters, according to the mini-batch training data, the meta-loss is obtained by calculation: in, represents the meta-loss, φ (k+1) represents the actor replica parameter at step k+1, s val represents the demonstration state for meta-loss computation, θ (k+1) represents the critic parameter at step k+1, represents the parameters of the actor replica, ω (k) represents the parameters of the meta-objective function at the k-th step; Based on the meta-loss, calculate the derivative of the meta-loss with respect to the meta-objective function: in, represents, ω represents the parameters of the meta-objective function, φ represents the actor parameters, represents the derivative of φ with respect to ω Abbreviation of φ (k+1) represents the actor parameters at step k+1, θ (k+1) represents the critic parameter at step k+1, ω (k) represents the parameters of the meta-objective function at the k-th step; Based on the derivative of the meta-loss with respect to the meta-objective function, the meta-objective function is updated to obtain an updated meta-objective function: Among them, ω (k+1) represents the parameter of the meta-objective function at the k+1th step, α represents the model learning rate, represents the meta-objective function, φ (k) represents the actor parameter at step k, φ (k+1) represents the actor parameters at the k+1th step.

Images