A reinforcement learning method based on meta-goal learning

Abstract

This invention provides a reinforcement learning method based on meta-objective learning, belonging to the field of machine learning technology. This method controls the robot to perform actions based on observed state signals. A reinforcement learning model is constructed using a meta-objective function and a two-layer optimization framework. After interacting with the environment to obtain a four-tuple, the critic parameters in the reinforcement learning model are first updated. Then, the actor parameters in the reinforcement learning model are updated by combining the meta-objective function and the reinforcement learning loss function calculated by the critic. The reinforcement learning and meta-objective function are trained alternately until convergence, and the final meta-objective function, critic parameters, and actor parameters are determined, resulting in a trained reinforcement learning model, thus completing the reinforcement learning based on meta-objective learning. The reinforcement learning model is used to analyze the robot's current state data to obtain updated robot motion control data. This invention solves the problem of constraints imposed by manually crafted objective functions on reinforcement learning strategies.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a reinforcement learning method based on meta-objective learning. Background Technology

[0002] Reinforcement learning (RL) is one of the most fundamental and important areas of machine learning research. It aims to maximize environmental rewards through trial and error, learning a policy that maximizes rewards within a loop of "receiving a state, taking an action, obtaining a reward and the next state, and improving the policy." Reinforcement learning has achieved great success in various tasks with dense rewards, such as Go, games, and large language models. However, it struggles to learn optimal policies from real-world tasks with sparse rewards. For example, a navigation robot might only receive a reward signal when it reaches its destination, while many intermediate steps offer no reward feedback. Achieving effective reinforcement learning in sparse reward environments is highly beneficial for many applications, such as autonomous driving, product recommendation, and robot motion control.

[0003] For reinforcement learning in sparse reward environments, researchers have proposed many methods, such as reward reshaping, hierarchical reinforcement learning, and imitation learning (IL). Existing work mainly focuses on using imitation learning to help reinforcement learning agents acquire reward signals, thereby improving online reinforcement learning in sparse reward environments. Specifically, by imitating actions in specific states in offline demonstration data, the agent's policy is made to approach the (suboptimal) demonstration policy; based on the (suboptimal) demonstration policy, the agent is able to acquire the necessary reward signals in sparse reward environments, thus gradually improving to the optimal policy. However, these methods struggle to achieve a balance between imitation learning and reinforcement learning (e.g., Figure 1 As shown in the figure, this often leads to agents falling into suboptimal strategies resulting from imitation learning.

[0004] The challenge in striking a balance between imitation learning and reinforcement learning lies in the difficulty of simultaneously achieving generality and high performance. Existing methods primarily employ two approaches to balance these two aspects. The first involves manually adjusting hyperparameters for specific tasks to achieve high performance, but this requires significant time to readjust hyperparameters for new tasks, severely reducing generality. The second approach uses dynamic parameter balancing to ensure high performance while improving scenario-specific generality, but dynamic parameters only adapt to a single reinforcement learning algorithm and cannot enable high performance in other reinforcement learning algorithms. Therefore, it is necessary to design a method that can automatically balance imitation learning and reinforcement learning to ensure high performance, while also being compatible with various reinforcement learning algorithms in different scenarios. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a reinforcement learning method based on meta-objective learning, which solves the problem of constraints imposed by handcrafted objective functions on reinforcement learning strategies.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: a reinforcement learning method based on meta-objective learning, comprising:

[0007] S1: Obtain the robot state quadruple based on robot motion control data;

[0008] S2: Construct a reinforcement learning model using a meta-objective function and a two-layer optimization framework; wherein, the reinforcement learning model includes a meta-training layer constructed using a meta-objective function and an actor-critic algorithm and a meta-optimization layer constructed using reinforcement learning-imitation learning scoring;

[0009] S3: Input the robot state quadruple into the reinforcement learning model, update the critic parameters in the reinforcement learning model by minimizing the critic parameters and the meta-objective function, and update the actor parameters in the reinforcement learning model by combining the meta-objective function and the reinforcement learning loss function calculated by the critic, to obtain the updated critic parameters and the updated actor parameters;

[0010] S4: Input the quadruple into the upper meta-optimization layer to train and obtain a better meta-objective function. Calculate the meta-loss using the mini-batch training data and the updated commentator parameters, and update the meta-objective function to obtain the updated meta-objective function.

[0011] S5: Iteratively execute S3 and S4, alternately training reinforcement learning and the meta-objective function until convergence, determine the final meta-objective function, critic parameters, and actor parameters, and obtain the trained reinforcement learning model, completing the reinforcement learning based on meta-objective learning; wherein, the reinforcement learning model is used to analyze the robot's current state data to obtain updated robot motion control data.

[0012] The beneficial effects of this invention are as follows: the processor uses the quadruples collected during the robot's interaction with the environment to train the reinforcement learning model, and the policy generated by the model can analyze environmental state signals and perform motion control on the robot. The meta-objective function proposed in this invention extracts knowledge from offline demonstration data to improve reinforcement learning, avoiding the constraints of manual objective functions on reinforcement learning policies in traditional imitation learning, and achieving a balance between imitation learning and reinforcement learning under suboptimal demonstration policies.

[0013] Further, S1 includes:

[0014] The robot observes its environment through sensors and obtains current state data;

[0015] Based on the current state data, the robot motion control data is used to control the robot to perform actions and interact with the environment, thereby obtaining motion data.

[0016] When the robot's actions change the environment, the environment transitions to the next state, and the next state data and the robot's reward data are obtained; wherein, the current state data, the action data, the next state data, and the reward data belong to the robot state quadruple.

[0017] Furthermore, the expression for the two-layer optimization framework is:

[0018]

[0019] Where ω represents the parameters of the meta-loss function, φ * θ represents the optimal actor parameter. * Indicates the best critic parameter. Indicates the original loss. Minimize To obtain the optimal parameters for the meta-loss function, This represents the reinforcement learning loss function output by the critic. Describe the meta-objective function. Indicates minimizing simultaneously and To obtain the parameters φ, L MSBE (θ) represents the mean square Bellman loss function. Minimize L MSBE (θ) to obtain the parameter θ * .

[0020] Unlike mainstream methods that are limited to a single reinforcement learning algorithm, the meta-objective function proposed in this invention can be organically combined with multiple reinforcement learning algorithms, greatly improving its versatility across different algorithms. The inventors combined the meta-objective function with three mainstream reinforcement learning benchmark algorithms, conducting experiments on four different robot motion tasks for each algorithm. Experimental results show that the proposed meta-objective function can be organically combined with multiple reinforcement learning algorithms, improving the robot's motion capabilities in sparse reward environments.

[0021] Further, S3 includes:

[0022] S310: Store the robot state quadruple in the experience replay buffer, and randomly sample small batches of training data from it as the initial input of the reinforcement learning model;

[0023] S320: Using the mini-batch training data and minimizing the mean square Bellman error loss function, update the critic parameters to obtain the updated critic parameters;

[0024] S330: Using the mini-batch training data, the reinforcement learning loss function calculated by combining the meta-objective function and the updated critic parameters is used to update the actor parameters, thus obtaining the updated actor parameters.

[0025] Unlike mainstream methods that require a lot of time to adjust hyperparameters, the meta-objective function proposed in this invention can automatically adjust the internal parameters of the module when dealing with different scenarios, without the need for manual adjustment of the module hyperparameters, which greatly improves the versatility in different scenarios.

[0026] Furthermore, the expression for the updated critic parameter is:

[0027]

[0028] Where, θ (k) Let θ represent the critic parameter at step k. (k+1) Let α represent the commentator parameters at step k+1, and α represent the model learning rate. This represents taking the derivative with respect to the parameter θ, where N represents the batch size of the random sampling.<s,a,r,s'> It is a quadruple randomly sampled from the experience replay buffer, where s represents the current state, a represents the action, r represents the reward, s′ represents the next state, D represents the experience replay buffer, and Q represents the next state. θ (s,a) represents the state-action value function calculated by the commentator based on the current state and actions, γ represents the depreciation rate, and Q... θ (s′,φ(s′)) represents the state-action value function calculated by the critic based on the next state and the actor's actions in the next state, φ (k) This represents the actor parameters at step k;

[0029] The expression for the updated actor parameters is:

[0030]

[0031] Where, φ (k+1) φ represents the actor parameters at step k+1. (k) This represents the actor parameters at step k. This represents the derivative with respect to the parameter φ, s d Indicates the demonstration state, a d Indicates the demonstration action, Q θ (s d ,φ(s d ) represents the state-action value function calculated by the critic based on the demonstration state and the actions performed by the actor in that state. Let ω represent the meta-objective function. (k) The parameters of the meta-objective function at step k; s d and a d This belongs to mini-batch training data, Qθ (s d ,φ(s d This belongs to the updated commenter parameters.

[0032] The meta-loss compares the performance of reinforcement learning agents trained based on a meta-objective function with that trained based on traditional imitation learning. Its aim is to enable the meta-objective function to critically consider suboptimal demonstration data, thereby surpassing traditional imitation learning. The meta-loss designed based on this idea possesses stronger interpretability and can handle different robot motion tasks and offline demonstration data of varying quality.

[0033] Further, S4 includes:

[0034] Using the aforementioned mini-batch training data, the parameters of the actor clone are updated through traditional imitation learning and reinforcement learning, resulting in the updated actor clone parameters:

[0035]

[0036] in, Parameters representing actor clones, The parameters represent the actor clone at step k+1. This represents the parameters of the actor clone at step k. Indicates the parameter Find the derivative, D dem This represents offline demonstration data, (s d ,a d ) is a state-action pair randomly sampled from offline demo data, s d Indicates the demonstration state, a d To demonstrate an action, w rl and w il This represents the hyperparameters that balance reinforcement learning and imitation learning. This represents the state-action value function calculated by the critic based on the demonstration state and the actions performed by the actor's clone in the demonstration state. This represents the objective function in traditional imitation learning;

[0037] Using the updated actor replica parameters and the updated critic parameters, and based on the mini-batch training data, the meta-loss is calculated as follows:

[0038]

[0039] in, φ represents the elemental loss. (k+1) Let s represent the parameters of the actor clone at step k+1. val This represents the demonstration state of the meta-loss calculation, θ (k+1)This represents the commentator parameter at step k+1. The parameter ω represents the actor clone. (k) The parameters of the meta-objective function at step k are represented.

[0040] Based on the aforementioned meta-loss, calculate the derivative of the meta-loss with respect to the meta-objective function:

[0041]

[0042] in, Let ω represent the parameters of the meta-objective function, and φ represent the actor parameters. The derivative of φ with respect to ω abbreviation, φ (k+1) Let θ represent the actor parameters at step k+1. (k+1) ω represents the critic parameter at step k+1. (k) The parameters of the meta-objective function at step k are represented.

[0043] Based on the derivative of the meta-loss with respect to the meta-objective function, the meta-objective function is updated to obtain the updated meta-objective function:

[0044]

[0045] Where, ω (k+1) Let represent the parameters of the meta-objective function at step k+1, and α represent the model learning rate. Denotes the meta-objective function, φ (k) φ represents the actor parameters at step k. (k+1) This represents the actor parameters at step k+1.

[0046] Thanks to the core idea of ​​extracting information from data, the meta-objective function proposed in this invention converges rapidly after extracting all effective information. This provides a way to maintain performance while accelerating training: the reinforcement learning agent is trained using the meta-objective function and the actor-critic framework before the meta-objective function converges, and the reinforcement learning agent is trained using only the actor-critic framework after the meta-objective function converges. Attached Figure Description

[0047] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0048] Figure 1 This is an exemplary flowchart illustrating a reinforcement learning method based on meta-objective learning according to some embodiments of this specification. Detailed Implementation

[0049] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0050] Example

[0051] Figure 1 This is an exemplary flowchart illustrating a reinforcement learning method based on meta-objective learning, according to some embodiments of this specification. Figure 1 As shown, the process includes the following steps. In some embodiments, the process may be executed by a processor.

[0052] S1: Obtain the robot state quadruple based on robot motion control data.

[0053] Robot motion control data is the data used to control the movement of a robot. For example, robot motion control data may include environmental images, robot position, joint angles, movement speed, and other data.

[0054] In some embodiments, the processor can acquire digital signals based on robot sensors as robot motion control data.

[0055] A robot state quadruple is a data set that reflects the robot's interaction with its environment. For example, a robot state quadruple can include current state data, action data, next state data, and reward data.

[0056] In some embodiments, the robot can continuously interact with the environment and collect multiple <current state, action, reward, next state> quadruples as robot state quadruples.

[0057] In some embodiments, the processor may implement S1 based on the following steps: the robot observes the environment through sensors and obtains current state data; based on the current state data, the robot uses motion control data to control the robot to perform actions and interact with the environment, thereby obtaining action data; when the robot's actions change the environment, the environment transitions to the next state, thereby obtaining the next state data and the robot's reward data.

[0058] Current state data reflects the robot's current motion state.

[0059] In some embodiments, the processor can acquire digital signals such as images, robot position, joint angles, and motion speed from robot sensors as current state data.

[0060] Motion data is the data used to control the movement of a robot.

[0061] In some embodiments, the processor can control the torque applied between various joints such as the robot's ankle, knee, and wrist, thereby adjusting the robot's direction and speed of movement as motion data for the robot.

[0062] Next state data is data that reflects the robot's motion state in the next stage.

[0063] Reward data is data used to incentivize robots to interact with their environment.

[0064] S2: Construct a reinforcement learning model using a meta-objective function and a two-layer optimization framework; wherein, the reinforcement learning model includes a meta-training layer constructed using a meta-objective function and an actor-critic algorithm, and a meta-optimization layer constructed using reinforcement learning-imitation learning scoring.

[0065] The meta-objective function is the learning objective of the upper meta-optimization layer, generated by the demonstration-based general imitation learning module.

[0066] The two-layer optimization framework is an optimization framework built for training and optimizing reinforcement learning models in sparse reward environments, consisting of a lower meta-training layer and an upper meta-optimization layer.

[0067] In some embodiments, the expression for the two-layer optimization framework is:

[0068]

[0069] Where ω represents the parameters of the meta-loss function, φ * θ represents the optimal actor parameter. * Indicates the best critic parameter. Indicates the original loss. Minimize To obtain the optimal parameters for the meta-loss function, This represents the reinforcement learning loss function output by the critic. Describe the meta-objective function. Indicates minimizing simultaneously and To obtain the parameters φ, L MSBE (θ) represents the mean square Bellman loss function. Minimize L MSBE (θ) to obtain the parameter θ * .

[0070] Reinforcement learning models are neural network models used to analyze the state of a robot and obtain its motion control data. Reinforcement learning can use various neural network models, such as convolutional neural networks and generative adversarial neural networks.

[0071] In some embodiments, the input to the reinforcement learning model can be a robot state quadruple, and the output of the reinforcement learning model can be robot motion control data.

[0072] In some embodiments, the structure of the reinforcement learning model is as follows:

[0073] A reinforcement learning model consists of a meta-training layer and a meta-optimization layer. The output of the meta-optimization layer serves as the input to the meta-training layer, and the output of the meta-training layer serves as the final output of the reinforcement learning model.

[0074] The meta-optimization layer is used to train the meta-objective function using the updated actor and critic parameters, resulting in an updated meta-objective function. The input to the meta-optimization layer can include the updated actor and critic parameters, and the output can include the updated meta-objective function.

[0075] The meta-training layer is used to update the actor and critic parameters. The input to the meta-training layer can include the robot state quadruple and the updated meta-objective function, and the output can include the updated actor and critic parameters.

[0076] In some embodiments, the processor can train the actor in conjunction with the meta-objective function and the critic, and finally the actor analyzes the robot's state, controls the robot's actions, and obtains robot motion control data.

[0077] Robot motion control data is data used to adjust the robot's direction and speed of movement. For example, robot motion control data can include joint angles and applied torques.

[0078] In some embodiments, the processor can adjust the robot's actions (adjusting joint angles and applied torque to control the direction and speed of the robot's movement) based on the robot's motion control data. After the robot moves to a new position in the environment, it obtains state and environmental reward data from the sensors again, and then makes actions again according to the strategy, interacting with the environment in a cyclical manner.

[0079] S3: Input the robot state quadruple into the reinforcement learning model, update the critic parameters in the reinforcement learning model by minimizing the critic parameters and the meta-objective function, and update the actor parameters in the reinforcement learning model by combining the meta-objective function and the reinforcement learning loss function calculated by the critic, to obtain the updated critic parameters and updated actor parameters.

[0080] Actor parameters are strategy parameters used to control the robot's movement.

[0081] The critic parameter is a parameter used to evaluate the quality of an actor's strategy.

[0082] In some embodiments, the processor may implement S3 based on the following steps.

[0083] S310: Store the robot state quadruple in the experience replay buffer, and randomly sample small batches of training data from it as the initial input of the reinforcement learning model.

[0084] The experience replay buffer is a storage unit used to store and reuse the quadruple.

[0085] In some embodiments, memory can be used to construct an experience replay buffer based on the quadruple obtained from the interaction between the processor and the environment.

[0086] Mini-batch training data is a batch of robot state quadruple data used to train reinforcement learning models.

[0087] S320: Using the mini-batch training data and minimizing the mean square Bellman error loss function, update the commentator parameters to obtain the updated commentator parameters.

[0088] In some embodiments, the expression for the updated critic parameter can be:

[0089]

[0090] Where, θ (k) Let θ represent the critic parameter at step k. (k+1) Let α represent the commentator parameters at step k+1, and α represent the model learning rate. This represents the derivative with respect to parameter θ, N represents the batch size of random sampling, s,a,r,s'> is a quadruple randomly sampled from the experience replay buffer, s represents the current state, a represents the action, r represents the reward, s′ represents the next state, D represents the experience replay buffer, and Q represents the next state. θ (s,a) represents the state-action value function calculated by the commentator based on the current state and actions, γ represents the depreciation rate, and Q... θ (s′,φ(s′)) represents the state-action value function calculated by the critic based on the next state and the actor's actions in the next state, φ (k) This represents the actor parameters at step k.

[0091] S330: Using the mini-batch training data, the reinforcement learning loss function calculated by combining the meta-objective function and the updated critic parameters is used to update the actor parameters, thus obtaining the updated actor parameters.

[0092] In some embodiments, the expression for the updated actor parameter can be:

[0093]

[0094] Where, φ (k+1) φ represents the actor parameters at step k+1. (k) This represents the actor parameters at step k. This represents the derivative with respect to the parameter φ, s d Indicates the demonstration state, a d Indicates the demonstration action, Q θ (s d ,φ(s d ) represents the state-action value function calculated by the critic based on the demonstration state and the actions performed by the actor in that state. Let ω represent the meta-objective function. (k) The parameters of the meta-objective function at step k; s d and a d This belongs to mini-batch training data, Q θ (s d ,φ(s d This belongs to the updated commenter parameters.

[0095] S4: Input the quadruple into the upper meta-optimization layer to train and obtain a better meta-objective function. Calculate the meta-loss using the updated critic parameters and the updated actor parameters, update the meta-objective function, and obtain the updated meta-objective function.

[0096] In some embodiments, the reinforcement learning model can be trained using quadruples as training samples. For example, quadruples can be collected by continuously interacting with the environment. A loss function is constructed around the reward in the quadruples, and the parameters of the initial reinforcement learning model are iteratively updated based on the loss function using gradient descent or other methods. The model training is complete when preset conditions are met, resulting in a trained reinforcement learning model. These preset conditions may include loss function convergence, the number of iterations reaching a threshold, etc.

[0097] In some embodiments, training samples may include a <current state, action, reward, next state> quadruple recorded during the robot's interaction with the environment. The state may be digital signals such as images acquired from the robot's sensors, the robot's position, joint angles, and movement speed. The action may be the torque applied between various joints such as the robot's ankle, knee, and wrist, serving as the robot's movement.

[0098] In some embodiments, the processor may implement S4 based on the following steps: using the mini-batch training data, updating the parameters of the actor replica through traditional imitation learning and reinforcement learning to obtain updated actor replica parameters; using the updated actor replica parameters and the updated critic parameters, calculating the meta-loss based on the mini-batch training data; calculating the derivative of the meta-loss with respect to the meta-objective function based on the meta-loss; updating the meta-objective function based on the derivative of the meta-loss with respect to the meta-objective function to obtain the updated meta-objective function.

[0099] The actor parameter copy is an independent copy of the actor parameters before the update.

[0100] In some embodiments, the expression for the actor parameter replica can be:

[0101]

[0102] in, Parameters representing actor clones, The parameters represent the actor clone at step k+1. This represents the parameters of the actor clone at step k. Indicates the parameter Find the derivative, D dem This represents offline demonstration data, (s d ,a d ) is a state-action pair randomly sampled from offline demo data, s d Indicates the demonstration state, a d To demonstrate an action, w rl and w il This represents the hyperparameters that balance reinforcement learning and imitation learning. This represents the state-action value function calculated by the critic based on the demonstration state and the actions performed by the actor's clone in the demonstration state. This represents the objective function in traditional imitation learning.

[0103] Meta-loss is a parameter that reflects the loss of the meta-objective function.

[0104] In some embodiments, the expression for the meta-loss can be:

[0105]

[0106] in, φ represents the elemental loss. (k+1) Let s represent the parameters of the actor clone at step k+1. val surface

[0107] Demonstration state of indicator loss calculation, θ (k+1)This represents the commentator parameter at step k+1. The parameter ω represents the actor clone. (k) This represents the parameters of the meta-objective function at step k.

[0108] In some embodiments, the expression for the derivative of the meta-objective function can be:

[0109]

[0110] in, Let ω represent the parameters of the meta-objective function, and φ represent the actor parameters. The derivative of φ with respect to ω abbreviation, φ (k+1) Let θ represent the actor parameters at step k+1. (k+1) ω represents the critic parameter at step k+1. (k) This represents the parameters of the meta-objective function at step k.

[0111] The updated meta-objective function is used to supervise the training of reinforcement learning models.

[0112] In some embodiments, the expression for the updated meta-objective function can be:

[0113]

[0114] Where, ω (k+1) Let represent the parameters of the meta-objective function at step k+1, and α represent the model learning rate. Denotes the meta-objective function, φ (k) φ represents the actor parameters at step k. (k+1) This represents the actor parameters at step k+1.

[0115] S5: Iteratively execute S3 and S4, alternately training reinforcement learning and the meta-objective function until convergence, determine the final meta-objective function, critic parameters, and actor parameters, and obtain the trained reinforcement learning model, completing the reinforcement learning based on meta-objective learning; wherein, the reinforcement learning model is used to analyze the robot's current state data to obtain updated robot motion control data.

[0116] In some embodiments of this specification, the processor uses quadruples collected during the robot's interaction with the environment to train the reinforcement learning model. The policy generated by the model can analyze environmental state signals and control the robot's motion. The meta-objective function proposed in this invention extracts knowledge from offline demonstration data to improve reinforcement learning, avoiding the constraints of manually crafted objective functions on reinforcement learning policies in traditional imitation learning, and achieving a balance between imitation learning and reinforcement learning under suboptimal demonstration policies.

Claims

1. A reinforcement learning method based on meta-objective learning, characterized in that, include: S1: Based on robot motion control data, obtain the robot state quadruple; where robot motion control data is the data used to control robot motion, including environmental images, robot position, joint angles, and motion speed data; S2: Construct a reinforcement learning model using a meta-objective function and a two-layer optimization framework; wherein, the reinforcement learning model includes a meta-training layer constructed using a meta-objective function and an actor-critic algorithm and a meta-optimization layer constructed using reinforcement learning-imitation learning scoring; S3: Input the robot state quadruple into the reinforcement learning model, update the critic parameters in the reinforcement learning model by minimizing the critic parameters and the meta-objective function, and update the actor parameters in the reinforcement learning model by combining the meta-objective function and the reinforcement learning loss function calculated by the critic, to obtain the updated critic parameters and the updated actor parameters; S4: Input the quadruple into the upper meta-optimization layer to train and obtain a better meta-objective function. Calculate the meta-loss using mini-batch training data and the updated commentator parameters, and update the meta-objective function to obtain the updated meta-objective function. S5: Iteratively execute S3 and S4, alternately training reinforcement learning and the meta-objective function until convergence, determine the final meta-objective function, critic parameters, and actor parameters, and obtain the trained reinforcement learning model, completing the reinforcement learning based on meta-objective learning; wherein, the reinforcement learning model is used to analyze the robot's current state data to obtain updated robot motion control data.

2. The reinforcement learning method based on meta-objective learning according to claim 1, characterized in that, S1 includes: The robot observes its environment through sensors and obtains current state data; Based on the current state data, the robot motion control data is used to control the robot to perform actions and interact with the environment, thereby obtaining motion data. When the robot's actions change the environment, the environment transitions to the next state, and the next state data and the robot's reward data are obtained; wherein, the current state data, the action data, the next state data, and the reward data belong to the robot state quadruple.

3. The reinforcement learning method based on meta-objective learning according to claim 1, characterized in that, The expression for the two-layer optimization framework is: ； ； in, The parameters represent the meta-loss function. Indicates the optimal actor parameters. Indicates the best critic parameter. Indicates the original loss. Minimize To obtain the optimal parameters for the meta-loss function, This represents the reinforcement learning loss function output by the critic. Describe the meta-objective function. Indicates minimizing simultaneously and Get parameters , This represents the mean squared Bellman loss function. Minimize Get parameters .

4. The reinforcement learning method based on meta-objective learning according to claim 1, characterized in that, S3 includes: S310: Store the robot state quadruple in the experience replay buffer, and randomly sample small batches of training data from it as the initial input of the reinforcement learning model; S320: Using the mini-batch training data and minimizing the mean square Bellman error loss function, update the critic parameters to obtain the updated critic parameters; S330: Using the mini-batch training data, the reinforcement learning loss function calculated by combining the meta-objective function and the updated critic parameters is used to update the actor parameters, thus obtaining the updated actor parameters.

5. The reinforcement learning method based on meta-objective learning according to claim 4, characterized in that, The expression for the updated critic parameter is: ； in, Indicates the first The critics' parameters for the step, Indicates the first +1 step of the critic parameter, This represents the model learning rate. Indicates the parameter Find the derivative. Indicates the batch size of the random sampling. These are quadruplets randomly sampled from the experience playback buffer. Indicates the current state. Indicates an action, Indicates a reward. Indicates the next state. This indicates an experience replay buffer. This represents the state-action value function calculated by the commentator based on the current state and actions. Indicates the depreciation rate. This represents the state-action value function calculated by the critic based on the next state and the actor's actions in that next state. Indicates the first The actor parameters for the step; The expression for the updated actor parameters is: ； in, Indicates the first +1 step actor parameters, Indicates the first The parameters of the actors in the step, Indicates the parameter Find the derivative. Indicates the demonstration status. Indicates the demonstration action. This represents the state-action value function calculated by the critic based on the demonstration state and the actions performed by the actor in that state. Describe the meta-objective function. Indicates the first The parameters of the meta-objective function of the step; and This belongs to mini-batch training data. This belongs to the updated commenter parameters.

6. The reinforcement learning method based on meta-objective learning according to claim 1, characterized in that, S4 includes: Using the aforementioned mini-batch training data, the parameters of the actor clone are updated through traditional imitation learning and reinforcement learning, resulting in the updated actor clone parameters: ； in, Parameters representing actor clones, Indicates the first +1 step of actor clone parameters, Indicates the first The parameters of the actor clone in the step, Indicates the parameter Find the derivative. This indicates offline demonstration data. These are state-action pairs randomly sampled from offline demo data. Indicates the demonstration status. Indicates the demonstration action. and This represents the hyperparameters that balance reinforcement learning and imitation learning. This represents the state-action value function calculated by the critic based on the demonstration state and the actions performed by the actor's clone in the demonstration state. This represents the objective function in traditional imitation learning; Using the updated actor replica parameters and the updated critic parameters, and based on the mini-batch training data, the meta-loss is calculated as follows: ； in, Indicates the original loss. This represents the parameters of the actor clone at step k+1. This indicates the demonstration state of the meta-loss calculation. This represents the commentator parameter at step k+1. Parameters representing actor clones, The parameters of the meta-objective function at step k are represented. Based on the aforementioned meta-loss, calculate the derivative of the meta-loss with respect to the meta-objective function: ； in, express, The parameters represent the meta-objective function. Indicates actor parameters, express about derivative abbreviation, This represents the actor parameters at step k+1. This represents the commentator parameter at step k+1. The parameters of the meta-objective function at step k are represented. Based on the derivative of the meta-loss with respect to the meta-objective function, the meta-objective function is updated to obtain the updated meta-objective function: ； ； ； in, The parameters of the meta-objective function at step k+1 are represented. This represents the model learning rate. Describe the meta-objective function. This represents the actor parameters at step k. This represents the actor parameters at step k+1.

Images