Dual robustness enhanced control method suitable for robots to adapt to different tasks

Abstract

The application provides a dual-robust enhanced control method suitable for robots to adapt to different tasks, which is based on meta-reinforcement learning to sample signals in a robot control process on robot control tasks with different goals and potential dynamics for training, while considering that the control signals on each task are relabeled with the reward functions on other tasks to enhance the training data set, and a dual-robust estimation method is used to estimate the state value of the relabeled signals, so that the robot can adaptively solve robot control tasks with different goals or different potential dynamics through a small amount of sampling. The application solves the problems that in the robot control task, the goals and potential dynamics are different in different task scenarios, and the reward signal is sparse, and enhances the adaptability of the robot control method to different environments and task goals, thereby providing a better performance control method for the robot.

Description

Technical Field

[0001] This invention relates to a method in the fields of machine learning and robot control, specifically to a deep meta-reinforcement learning control method, system, medium, and terminal that is applicable to the dynamic adaptation of robots to different tasks with dual robust enhancements. Background Technology

[0002] Deep reinforcement learning has achieved remarkable success in various sequential decision-making tasks, such as Atari games and robot control. Traditional reinforcement learning methods train agent control policies aimed at learning a single task, and therefore cannot quickly adapt even to new tasks with prior experience. In contrast, deep meta-reinforcement learning focuses on "learning how to learn," i.e., learning how to adapt. Meta-reinforcement learning essentially learns underlying common structures from experience gathered from a set of tasks, and then leverages these structures to quickly adapt to similar new tasks with only a few attempts.

[0003] One of the main challenges in deep meta-reinforcement learning is sparse reward settings, a situation prevalent in real-world environments. When rewards become sparse, reinforcement learning agents can only acquire very little task-related information, posing significant difficulties for meta-training and adaptation processes. Furthermore, in reality, robots face different underlying environmental dynamics when performing tasks under varying terrains, weather conditions, and other environmental circumstances. This results in different observation transition probabilities for different tasks, further challenging the generalizability of robot control strategies.

[0004] A search of existing technologies revealed that Alexander Li et al. published an article titled "Generalized hindsight for reinforcement learning" in *Advances in Neural Information Processing Systems*, 33:7754–7767, 2020. This article introduces a post-hoc experience replay method. This method recalculates the reward signal from samples of other target tasks using the target reward function to guide the reinforcement learning agent to achieve different goals in new tasks. However, this method is essentially designed for tasks that differ only in the reward function, assuming that the dynamics of all different tasks are the same.

[0005] The double robust estimator is a method in reinforcement learning to address probability distribution mismatch. Nan Jiang et al. published a paper entitled "Doubly robust off-policy value evaluation for reinforcement learning" in the International Conference on Machine Learning, pages 652–661, PMLR, 2016. This paper introduces a relatively simple double robust estimator to solve the problem of estimating the state value of the target policy using the experience of sampling the behavior policy. However, this method assumes that the task is the same and therefore does not consider different dynamics. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a deep meta-reinforcement learning control method, system, medium, and terminal with dual robust enhancements suitable for robots to dynamically adapt to different tasks. This invention aims to solve the technical problem in actual industrial robot deployments where tasks cannot be completed well due to potential dynamic changes in the environment.

[0007] According to one aspect of the present invention, a dual robust enhancement control method suitable for robots to adapt to different tasks is provided, comprising:

[0008] S1, establish the policy neural network π respectively θ Action Value Neural Network Q θ Inference Neural Network E θ and observation transfer prediction neural network And initialize the network parameters;

[0009] S2, sample multiple different task objectives or potential dynamic parameters, and generate simulation environments for multiple different tasks based on these parameters;

[0010] S3, the robot runs in multiple different simulation environments generated in S2, and the robot follows the strategy neural network π. θ The output control signals execute the task, record the observation signals, control signals and reward signals in the robot simulation control process, and use them as training sample tuples to build a training dataset for each task.

[0011] S4. For each simulation task, training sample tuples of observation signals, control signals, and reward signals are sampled from the training dataset. These training sample tuples are used as inputs to the policy neural network, action value neural network, inference neural network model, and observation transfer prediction neural network, respectively. The loss function of each network is calculated, and the backpropagation method is used to optimize each network model. For each task, the signal trajectory with the large reward signal is sampled from the training dataset. The action value is estimated as the target using a dual robust estimator, and the backpropagation method is used to optimize the action value neural network.

[0012] S5, Deploy the optimized policy neural network into the robot, and repeat S2-S4 to enable the policy neural network to obtain the largest possible reward signal in each task environment.

[0013] Preferably, in S2, the simulation environments for the multiple tasks generated have different task objectives and dynamic parameters, and there is a situation where the reward signal is sparse. That is, the robot will only obtain a reward signal related to the distance to the target position when it is near the target, otherwise it will only obtain a fixed value reward signal.

[0014] Preferably, in S3, the policy neural network π θ The process of outputting control signals includes:

[0015] The inference neural network infers the potential feature information of each task based on the latest batch of data collected from each task.

[0016] The observed signals and the inferred potential feature information of the task are simultaneously input into the policy neural network to obtain the control signal distribution parameters, and the control signal is obtained through sampling.

[0017] Preferably, in S4, the training sample tuples for sampling observation signals, control signals, and reward signals from the training dataset include:

[0018] Data batches are randomly sampled from all training sample tuples stored in the data cache for optimization and training of the policy neural network, the action value neural network, and the observation transition prediction neural network;

[0019] A batch of randomly sampled data from the training sample tuples collected in the most recent simulation in the data cache is used for the optimization and training of the inference neural network model.

[0020] For each task, the signal trajectory with the largest cumulative discount reward signal is searched from the data cache of all tasks. The corresponding training sample tuple is used to further optimize and train the action value neural network using the dual robust estimator.

[0021] Preferably, in S4, the loss function of each network is calculated, and the backpropagation method is used to optimize each network model, including:

[0022] The update of the action value neural network includes two loss functions calculated and updated using the data batch and the signal trajectory. The first loss function is defined as the mean square error of the action value prediction and estimation using the action value neural network and the Bellman formula in the training data batch, respectively, i.e., the Bellman residual. The second loss function is defined as the mean square error of the action value prediction and estimation using the action value neural network and the dual robust estimator.

[0023] The inference neural network E θ The loss function used for the update is the same as the loss function used for the first update of the action value neural network;

[0024] For the update of the policy neural network, its loss function is defined as the KL divergence of the softmax distribution calculated from the distribution of the control signal output by the policy network and the estimated action values ​​of each action.

[0025] For the update of the observation transition prediction neural network, its loss function is defined as the arithmetic mean of the negative log probability values ​​of the data tuples in the data batch;

[0026] When optimizing the action value neural network, the gradient of the first loss function calculated by backpropagation is used only to update the parameters of the action value neural network; when optimizing the inference neural network, the gradient calculated by backpropagation is used only to update the inference neural network E. θ The parameters.

[0027] Preferably, the dual robust estimator, by considering the differences in observation transition probabilities between tasks and the differences in strategies, simultaneously considers the importance ratio of the strategy and the estimated observation transition probability ratio. It solves for the optimal observation transition probability ratio by minimizing the theoretical least square error between the dual robust estimator and the true value, and derives the corresponding interval in which it lies. The estimated observation transition probability ratio is then truncated through interval estimation to improve the accuracy of value estimation.

[0028] Preferably, the calculation process of the dual robust estimator is as follows:

[0029] For all training sample tuples in the signal trajectory, the observation transition prediction network of task j is used. Estimate the probability that, at a certain time step in task j, the observed signal s will change to s' after receiving the observed signal s and using the control signal a; for task i, its observation transition probability follows a Gaussian distribution. The estimation is performed, with the mean being the actual observed signal s' at the next time step and the standard deviation being σ.

[0030] Estimation of the importance ratio for calculating observation shift Calculate each data point The average value is used as For less than Less than The value is used for clipping, that is...

[0031] The importance ratio of the calculation strategy That is, input the same observation signal s t Policy neural network; π θ Under tasks j and i, output control signal a t The ratio;

[0032] The value estimate is calculated based on the importance ratio of observation transfer and the importance ratio of strategy, according to the dual robust estimator formula.

[0033] According to a second aspect of the present invention, a dual robust enhancement control system suitable for robot adaptation to different tasks is provided, comprising:

[0034] Model building module: Establish policy neural networks π respectively θ Action Value Neural Network Q θ Inference Neural Network E θ and observation transfer prediction neural network And initialize the network parameters;

[0035] Environment generation module: Samples multiple different task objectives or potential dynamic parameters, and generates simulation environments for multiple different tasks based on these parameters;

[0036] Dataset construction module: Runs the robot in multiple different simulation environments generated by S2, the robot following the policy neural network π θ The output control signals execute the task, record the observation signals, control signals and reward signals in the robot simulation control process, and use them as training sample tuples to build a training dataset for each task.

[0037] Training Module: For each simulation task, training sample tuples of observation signals, control signals, and reward signals are sampled from the training dataset. These training sample tuples are used as inputs to the policy neural network, action value neural network, inference neural network model, and observation transition prediction neural network, respectively. The loss function of each network is calculated, and the backpropagation method is used to optimize each network model. The signal trajectory with the large reward signal is sampled from the training dataset for each task. The action value is estimated as the target using a dual robust estimator, and the backpropagation method is used to optimize the action value neural network.

[0038] Application module: Deploy the optimized policy neural network into the robot, repeat each module, so that the policy neural network can obtain the largest possible reward signal in each task environment.

[0039] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform the method described therein, or to run the system described therein.

[0040] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described thereon, or to run the system described thereon.

[0041] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0042] The deep meta-reinforcement learning control method and system with dual robust enhancements for dynamic adaptation of robots to different tasks, as described in this invention, solves the problem in robot control tasks where the target and potential dynamics are different in different task scenarios and the reward signal is sparse. It enhances the adaptability of the robot control method to different environments and task objectives, and provides a better-performing control method for robots. Attached Figure Description

[0043] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0044] Figure 1 This is a flowchart of a deep meta-reinforcement learning control method with dual robust enhancements for dynamic adaptation of robots to different tasks, as described in an embodiment of the present invention.

[0045] Figure 2 To model the interaction between the robot training environment and the reinforcement learning agent in the robot in a preferred embodiment of the invention as a Markov decision process;

[0046] Figure 3 This is a schematic diagram of the training and update process of each neural network in a preferred embodiment of the present invention.

[0047] Figure 4 This is a specific application embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0049] Existing meta-reinforcement learning methods for addressing sparse rewards only consider the differences in the task target location, which is reflected in the different reward functions used to calculate the reward signal in reinforcement learning modeling. Therefore, in principle, they cannot adaptively solve different tasks that simultaneously have sparse rewards and potential environmental dynamic mismatches. Existing dual robust estimators only consider the differences in policy networks between two tasks, and therefore, in principle, they cannot be applied to estimating the value of the same signal trajectory in different tasks when there are potential environmental dynamic mismatches and the reward function changes. However, the problem of potential environmental dynamic mismatches is widespread in actual robot deployments in industry, such as changes in the robot's working environment due to different climates and regions, and changes in component wear and physical parameters due to long-term use of the robot.

[0050] To address the technical problem of robots failing to perform tasks effectively in actual industrial deployments due to dynamic changes in the potential environment, this invention employs meta-reinforcement learning to sample signals from the robot control process for training on robot control tasks with different objectives and potential dynamics. Simultaneously, it considers relabeling the control signals from each task using reward functions from other tasks to enhance the training dataset. A dual robust estimation method is used to estimate the state value of the relabeled signals, ultimately enabling the robot to adaptively solve robot control tasks with different objectives or potential dynamics through minimal sampling.

[0051] Based on the above inventive concept, in one embodiment of the present invention, a deep meta-reinforcement learning control method with dual robust enhancements suitable for robot dynamic adaptation to different tasks is provided, such as... Figure 1 As shown, it includes the following steps:

[0052] S1, Establish the policy neural network π θ Action Value Neural Network Q θ Inference Neural Network E θand observation transfer prediction neural network And initialize the network parameters;

[0053] S2, sample multiple different task objectives or potential dynamic parameters, and the robot simulation environment generates multiple simulation environments for different tasks based on different task objectives or potential dynamic parameters;

[0054] S3 involves running the robot in multiple different simulation environments generated in S2. The robot follows a policy neural network π. θ The output control signal executes the task, records the signals (observation signal, control signal and reward signal) in the robot simulation control process, and uses the observation signal, control signal and reward signal as training sample tuples to build its own training dataset for each task;

[0055] S4. For each simulation task, training sample tuples of observation signals, control signals, and reward signals are sampled from the training dataset. These training sample tuples are used as inputs to the policy neural network, action value neural network, inference neural network model, and observation transfer prediction neural network, respectively. The loss function of each network is calculated, and the backpropagation method is used to optimize each network model. For each task, the signal trajectory with the large reward signal is sampled from the training dataset. The action value is estimated as the target using a dual robust estimator, and the action value neural network is optimized using the backpropagation method.

[0056] S5. Deploy the optimized policy neural network into the robot and repeat the operation starting from S2 to enable the policy neural network to obtain the largest possible reward signal in each task environment.

[0057] In control tasks, different task scenarios simultaneously present varying objectives and potential dynamics, and there is also the problem of sparse reward signals. The above embodiments enhance the adaptability of robot control methods to different environments and task objectives, providing robots with better-performing control methods.

[0058] In a preferred embodiment of the present invention, S1 is implemented to establish a policy neural network π. θ Action Value Neural Network Q θ Inference Neural Network E θ and observation transfer prediction neural network The network parameters are then initialized. Policy Neural Network π θ The input is the observation signal, and the output is the control signal; its function is to provide control signals for controlling the robot; the action value neural network Q... θThe input is the observation signal and the control signal, and the output is the scalar function value. Its function is to predict the expected cumulative reward value that the robot can obtain by continuing to control according to the current policy neural network after receiving the input observation signal value and taking the control signal; infer the neural network E θ The input is a sequence of observation signals and control signals, and the output is a one-dimensional vector representing the potential feature information of the current task. Its function is to infer the potential feature information of the current task; the input of the observation transition prediction neural network is the observation signal and the control signal, and the output is the probability of the possible value of the observation signal at the next time step. Its function is to predict the transition of the observation signal at the next time step.

[0059] In a preferred embodiment of the present invention, S2 is implemented. Among the simulation environments generated for multiple different tasks, the task objectives and dynamic parameters are all different, and there is also a situation where the reward signal is sparse, that is, only when the robot is near the solved target will it obtain a reward signal related to the distance from the target position, otherwise it will only obtain a fixed-value reward signal.

[0060] Specifically, multiple different tasks are modeled as follows:

[0061] Based on the reinforcement learning theory, in a certain task i, the operation of the robot in the simulation environment of this task is modeled as a Markov decision process where and respectively represent all possible value spaces of the robot observation signal and all possible value spaces of the control signal, p i represents that in the simulation environment of this task, when the observation signal at a certain time step t is s t and the control signal a is input to the robot t , the observation transition probability that the observation signal at t + 1 is s t+1 . r i is the feedback reward signal obtained by the robot according to the current observation signal and action signal. In the simulation environment with sparse reward signals, when the current observation position and the position d(s t , s g ) < D, that is, when it is close enough, the reward signal is an indication of the distance from the target position r i = -d(s t , s g ) + C, where C is a certain fixed value; when d(s t , s g ) > D, the reward signal is zero. At each time step t during the operation of the task simulation environment, the robot receives the observation signal s of the current environment tAs input to the policy neural network π, the control signal is input to the robot for execution, and then the observation signal s of the environment is obtained at the next time step t+1. t+1 The probability is st +1 ~p i (·|s t ,a t ).

[0062] During the simulation phase, the reward function R for each task is calculated. i From (s,a), we know that the observation transition probability pi is unknown. The proposed policy neural network π aims to obtain the maximum cumulative reward signal in the task simulation environment, i.e., to complete the task objective.

[0063]

[0064] Where γ∈[0,1] is the discount factor, and τ is a complete signal trajectory sampled during the simulation. The multiple tasks can be represented as follows: Where N is the number of all tasks. At a certain time step t during task i, the cumulative expected discount and cumulative reward signal obtained after the robot takes control signal a and continues executing the task according to the policy neural network π after observing a certain observation signal s is defined as the action value function. Define the cumulative expected discount and cumulative reward signal obtained by the robot after observing a certain observed signal s and continuing to execute the task according to the policy neural network π as the state-value function.

[0065] In a preferred embodiment of the present invention, S3 is implemented. Wherein, the policy neural network π... θ The process of outputting control signals is as follows:

[0066] Before a certain task i begins, there is an inference neural network E. θ Input the latest batch of historically collected data. i Inference z is obtained from the latent feature information of task i. i =E θ (c i );

[0067] At each time step t, the observed signal s observed by the robot t The distribution parameters are obtained by inputting into the policy neural network, and the control signal is sampled from this distribution, i.e., a. t ~π θ (·|s t ,z i ).

[0068] Create a corresponding data cache for each task. Used to store observation signals, control signals, and reward signals collected during task simulation as training sample tuples.

[0069] In a preferred embodiment of the present invention, step S4 is implemented. The process of sampling from the training dataset includes three sampling methods as follows:

[0070] From data cache Randomly sampled data from all training sample tuples stored in the database. Used for the optimization and training of the policy neural network, value neural network, and observation transition prediction neural network, where |b i | is data batch b i The number of training sample tuples.

[0071] From data cache The data is randomly sampled from the training sample tuples collected in the most recent simulation. Used for the optimization and training of inference neural network models, where |c i | is data batch c i The number of training sample tuples.

[0072] For each task, search the cumulative discount reward signal from the data cache of all tasks. The largest signal trajectory τ i The corresponding training sample tuples are used to further optimize and train the action value neural network. For example, the signal trajectory τ sampled from task i for task j. i The cumulative discount reward signal has the largest value under the reward function calculation for task j.

[0073] Furthermore, for each task, the cumulative discount reward signal is searched from the data cache of all tasks. The largest signal trajectory τ i The specific execution steps are as follows:

[0074] S10, randomly sample several signal trajectories from the data buffers of all tasks;

[0075] S20, calculate the total cumulative discount reward for the sampled signal trajectory under each task;

[0076] S30 selects the signal trajectory with the highest cumulative discount reward for each task.

[0077] In a preferred embodiment of the present invention, S4 is implemented. The gradient calculations for the policy neural network, value neural network, inference neural network, and observation-transfer-move network models are as follows:

[0078] For updating the action value neural network, there exists a method that utilizes data batch b. i The two updates of the signal trajectory τ, using data batch b i During the update, the loss function is defined as the mean squared error of the prediction and estimation of action values ​​using the action value neural network and the Bellman formula respectively in the training data batch, i.e., the Bellman residual, as follows:

[0079]

[0080] Q θ (s j ,a j ,z j ) utilizes the action value neural network to process state-action pairs (s) j ,a j Action value under task i The predicted value, the reward signal r j And prediction of the observed signal value at the next time step The sum of these values ​​serves as the objective of the action-value neural network, and the network parameters used to calculate the objective value do not participate in backpropagation; this is represented by an overline. i =E θ (c i The inference neural network is based on batch c of data. i The inference of latent feature information for task i serves as the input z to the action value neural network. i This represents the fitting of the action value function for task i. When optimizing the action value neural network, the gradients calculated by backpropagation are only used to update the parameters of the action value neural network.

[0081] Using signal trajectory τ i During the update, the loss function can be defined as the mean squared error of the action value prediction and estimation using the action value neural network and the dual robust estimator in the training dataset, as follows:

[0082]

[0083] in The dual robust estimator is used to evaluate the observed signal s. t+1 The expected cumulative reward obtained afterward is the estimate of the state value.

[0084] The inference neural network E θ The loss function used for parameter updates is the same as that used in the action-value neural network. During optimization, the gradients calculated by backpropagation are used only to update the inference neural network E. θ The parameters.

[0085] For policy network updates, the loss function can be defined as the KL divergence of the softmax distribution calculated from the distribution of the control signal output by the policy network and the estimated action values ​​of each action, as follows:

[0086]

[0087] in It is a softmax distribution calculated based on the action values ​​of each action estimated using the action value neural network.

[0088] For the observation transition prediction neural network Its loss function can be defined as data batch b j The arithmetic mean of the negative log probability values ​​of the data tuples is as follows:

[0089]

[0090] in Predict the current observation state s in task i. j Lower robot input control signal a j Then, the robot observes the signal at the next time step as s' j The probability of.

[0091] In a preferred embodiment of the present invention, the dual robust estimator in S4 is described. Specifically, the calculation formula for the dual robust estimator is as follows:

[0092] in, It is the importance ratio of the strategy, defined as the ratio of the number of observed signals s when inputs are the same. t The strategy neuron outputs control signal a under tasks j and i. t The ratio, i.e. Because the distribution parameters of the policy network output are known. When the same observation signal s is observed t And using the same control signal a t Subsequently, under tasks j and i, the observed signal s is observed at the next time step. t+1 The ratio of observed transition probabilities, i.e. Due to the observation transition probability p i Unknown, utilization Importance ratio of observational shift Make an estimate. j (s,a t ) is the reward signal recalculated using the reward function in task j.

[0093] The dual robust estimator utilizes the signal trajectory from task i. Use the reward signal calculation function R under task j. j The signal trajectory obtained after relabeling Iteratively estimate each time step s t Next state value. Consider. For the dual robust estimator estimate against the true state value, the least squared error between the dual robust estimator estimate and the true state value has the following upper bound:

[0094]

[0095] in Var is the difference between the action value predicted by the action value neural network and the true value. t [r j (s t ,a t )∣a t [Action a is taken at time step t] t Variance of the post-reward signal. Arbitrary state value estimator. For the state value truth value V j The least squares error (st) has the following decomposition:

[0096]

[0097] when When the bias of the dual robust estimator is zero, that is, the first term of the decomposition is zero. This makes the second term of the decomposition, the variance term, zero. The optimal solution for optimizing the variance of the dual robust estimator is as follows:

[0098]

[0099] The optimal solution to this problem for

[0100]

[0101] A preferred implementation indicates that the dual robust estimator estimates the importance ratio of observation transitions. The following optimization problem can be established to minimize the least squared error between the estimated value and the true state value of the dual robust estimator:

[0102]

[0103] The optimization variable is the estimation of the observation transition probability ratio. The optimal value of the optimization problem is:

[0104]

[0105] Because of the optimal value Optimizing the least squares error, according to the decomposition of the least squares error, is equivalent to jointly optimizing the variance and bias, and Therefore, it is in the range Between. The optimal value can be approximated using an interval.

[0106] In a preferred embodiment of the present invention, a specific calculation process for estimating state values ​​using a dual robust estimator is provided:

[0107] (a) For all training sample tuples in the signal trajectory, utilize the observation transition prediction network of task j. Estimate the probability that, at a certain time step in task j, the observed signal s will change to s' after receiving the observed signal s and using the control signal a; for task i, its observation transition probability follows a Gaussian distribution. The estimation is performed, with the mean being the actual observed signal s' at the next time step and the standard deviation being σ.

[0108] (b) Estimation of the importance ratio of observational shift Calculate each data point The average value is used as For less than Less than The value is used for clipping, that is...

[0109] (c) Calculate the importance ratio of the strategy

[0110] (d) Calculate the value estimate based on the dual robust estimator formula.

[0111]

[0112] The value estimation is applied to predict the value of the current state during robot simulation. Specifically, it is the cumulative expected reward signal that can be obtained based on the observed signal at any time and by continuing the task according to the current policy network. Based on this value estimation, the method can update the policy network so that the robot approaches and stays in a high-value state, thereby better completing the task.

[0113] The preferred embodiments of the present invention, by designing a novel dual robust estimator, are used to estimate the state values ​​of signal trajectories with large reward signals after mutual migration between different tasks, thereby helping to accelerate training on various tasks. The novel dual robust estimator employs the importance ratio of the policy and the estimated observation transition probability ratio, while simultaneously considering the differences in observation transition probabilities and policies between tasks. By minimizing the theoretically optimal least-squares error between the dual robust estimator and the true value, the optimal observation transition probability ratio is solved, and its corresponding interval is derived. The estimated observation transition probability ratio is then truncated using interval estimation, thereby obtaining a more accurate value estimate.

[0114] The preferred embodiments of the present invention address the problems in robot control tasks where different objectives and potential dynamics exist simultaneously in different task scenarios, and where reward signals are sparse. This enhances the adaptability of the robot control method to different environments and task objectives, and provides a better-performing control method for robots.

[0115] Based on the same inventive concept, in other embodiments of the present invention, a dual robust enhancement deep meta-reinforcement control system suitable for robot dynamic adaptation to different tasks is provided, comprising:

[0116] Model building module: Building a policy neural network π θ Action Value Neural Network Q θ Inference Neural Network E θ and observation transfer prediction neural network And initialize the network parameters;

[0117] Environment generation module: Samples multiple different task objectives or potential dynamic parameters, and generates simulation environments for multiple different tasks based on these parameters;

[0118] Dataset construction module: Run the robot in multiple different simulation environments generated by S2, and the robot follows the described strategy neural network π. θ The output control signals execute the task, record the observation signals, control signals and reward signals in the robot simulation control process, and use them as training sample tuples to build a training dataset for each task.

[0119] Training Module: For each simulation task, training sample tuples of observation signals, control signals, and reward signals are sampled from the training dataset. These training sample tuples are used as inputs to the policy neural network, action value neural network, inference neural network model, and observation transition prediction neural network, respectively. The loss function of each network is calculated, and the backpropagation method is used to optimize each network model. The signal trajectory with the largest reward signal is sampled from the training dataset for each task. The action value is estimated using a dual robust estimator as the target, and the backpropagation method is used to optimize the action value neural network.

[0120] Application module: Deploy the optimized policy neural network into the robot, repeat each module, so that the policy neural network can obtain the largest possible reward signal in each task environment.

[0121] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the deep meta-reinforcement learning control method with dual robust enhancement applicable to the dynamic adaptation of robots to different tasks in the above embodiments, and will not be repeated here.

[0122] In a specific application embodiment of the present invention, a deep meta-reinforcement learning control method with dual robust enhancements for quadruped robot control and navigation tasks is provided, the process of which is as follows: Figure 4 As shown, it specifically includes:

[0123] First: Build a robot simulator;

[0124] The main tasks involved are task sampling and task simulation. Task sampling involves sampling multiple task objectives and potential dynamic parameters; task simulation involves inputting the task objectives and dynamic parameters into the simulator to generate different training environments.

[0125] Next, for the quadruped robot, a deep meta-reinforcement learning control method with dual robustness enhancement is implemented. The specific process is as follows:

[0126] 1. Network Construction:

[0127] Its like Figure 2 Based on reinforcement learning theory, the interaction between the robot training environment and the reinforcement learning agent in the robot is modeled as a Markov decision process. Robots Figure 2 As shown in Figure X, this is a quadruped robot. The key physical parameter affecting the robot's observation transition probability in the simulation environment is denoted as... Robot observations at time step t t Data obtained from various sensors, including the world coordinate system position information and rotation angles of each joint, is used to input control signals a to the robot. tIt can control the rotation angle of each joint. The objectives of each simulation task are denoted as... The reward signal calculation function at each time step is:

[0128]

[0129] Where dist is a measure of the observed signal s t The corresponding position and the observation s after reaching the target g The spatial distance to the corresponding target point. When the distance between the current observation position and the observation position at the target position is less than D, the reward signal includes information indicating the distance to the target position, i.e., r. i =-dist(s t ,s g )+C; when dist(s t ,s g When )≥D, the reward signal is zero.

[0130] During the simulation, before starting the task, the robot inputs the most recently collected signals from the task into an inference neural network to obtain an inference z of the task's potential feature information. i At each time step t, the robot receives the observed signal s. t , will s t and z i Common input to the policy neural network π θ (·|s t ,z i The distribution parameters of the control signal are obtained, and the control signal a is sampled from this distribution. t Then the observation signal s of the next time step is received. t+1 (s' t ).

[0131] 2. Output storage

[0132] Build a data cache for each task to store the simulated signal trajectory in tuples;

[0133] 3. Network training

[0134] like Figure 3 Build a data cache for each task i Used to store the signal tuples (s) collected during each task simulation. j ,a j ,r j ,s' j Data is sampled from all cached data using the following three methods. Medium-sized samples:

[0135] Sampling method For each task i, from the data cache Randomly sampled data from all training sample tuples stored in the database.

[0136] Sampling method For each task i, from the data cache The data is randomly sampled from the training sample tuples collected in the most recent simulation.

[0137] Sampling method For each task i, search the cumulative discount reward signal from the data cache of all tasks. The largest signal trajectory τ i .

[0138] 4. Construct a dual robust estimator for network training;

[0139] Training is primarily based on minimum mean square error analysis, including the optimal importance ratio of observation transition probabilities with minimum mean square error, and an interval approximation of the optimal importance ratio. Specifically,

[0140] like Figure 3 ,in accordance with Sampled data batch b i The loss functions used to calculate the updates of the value neural network, policy neural network, and observation transition prediction neural network are calculated as follows:

[0141]

[0142]

[0143]

[0144] in accordance with Sampled data batch c i The loss function used to calculate the update of the inference neural network is as follows:

[0145]

[0146] in accordance with The sampled signal trajectory τ i The value estimate will be used to compute the dual robust estimator. The calculation process for the estimated value is as follows:

[0147] (a) For all training sample tuples in the signal trajectory, utilize the observation transition prediction network of task j. Estimate the probability that, at a certain time step in task j, the observed signal s will change to s' after receiving the observed signal s and using the control signal a; for task i, its observation transition probability follows a Gaussian distribution. The estimation is performed, with the mean being the actual observed signal s' at the next time step and the standard deviation being σ.

[0148] (b) Estimation of the importance ratio of observational shift Calculate each data point The average value is used as For less than Less than The value is used for clipping, that is...

[0149] (c) Calculate the importance ratio of the strategy

[0150] (d) The estimated value is calculated according to the dual robust estimator formula, as follows:

[0151]

[0152] estimated value The loss function used to update the value neural network is calculated as follows:

[0153]

[0154] Finally, the optimized policy neural network (control policy) is deployed into the robot in the simulator to enable the policy neural network to obtain the largest possible reward signal in each task environment.

[0155] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the dual robust deep meta reinforcement learning control method described in any of the above embodiments, or to run the dual robust deep meta reinforcement learning control system described in any of the above embodiments.

[0156] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0157] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0158] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0159] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0160] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to execute the dual robust deep meta reinforcement learning control method described in any of the above embodiments, or to run the dual robust deep meta reinforcement learning control system described in any of the above embodiments.

[0161] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0166] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A dual-robust enhanced control method suitable for robots to adapt to different tasks, characterized in that, Comprise: S1, respectively establish a policy neural network , an action value neural network , an inference neural network and an observation transition prediction neural network and initialize network parameters; S2, sample a plurality of different task targets or potential dynamics parameters, and generate a plurality of different task simulation environments based on the same; S3, running the robot in the plurality of different simulation environments generated in S2, the robot following the policy neural network The output control signal performs the task, records the observation signal, the control signal and the reward signal in the robot simulation control process, and serves as a training sample tuple to construct its own training data set for each task; S4, for each simulation task, sample a training sample tuple of observation signals, control signals and reward signals from the training data set, take the training sample tuple as the input of the policy neural network, the action value neural network, the inference neural network model and the observation transition prediction neural network respectively, calculate the loss function of each network, and optimize each network model respectively by using the back propagation method; from the training data set, sample a signal trajectory with a large reward signal for each task, estimate the action value as the target by using a double robust estimator, and optimize the action value neural network by using the back propagation method; S5, deploy the optimized policy neural network to the robot, and repeat S2-S4 to make the policy neural network obtain the largest reward signal in each task environment; The double robust estimator considers the differences between tasks in policy and observation transition probability by the importance ratio of policy and the estimated importance ratio of observation transition; by minimizing the theoretical least square error between the double robust estimator and the true value, the optimal observation transition probability ratio is solved, and the confidence interval of the optimal ratio is derived; The estimated observation transition probability ratio is truncated by using interval estimation to improve the accuracy of action value estimation; The calculation process of the double robust estimator is as follows: For all training sample tuples in the signal trajectory, the task of observation transition prediction network is estimated The probability that the observation signal after adopting the control signal after a certain time step, changes to ; for the task , the observation transition probability is estimated by a Gaussian distribution, that is, the mean is the actual next time step observation signal , and the standard deviation is ; Computing an estimate of the importance ratio of observed transitions , computing the mean value of each data point as , clipping values less than less than , i.e. ; Importance ratio of a computation policy , i.e. input same observation signal , policy neural network At task and task , output control signal ratio Based on the importance ratio of observation transition and the importance ratio of policy, the value estimation is calculated according to the double robust estimator formula.

2. The dual-robust enhanced control method suitable for robots to adapt to different tasks of claim 1, wherein, In S2, the generated simulation environment of a plurality of different tasks has different task targets and dynamics parameters, and there is a sparse reward signal, that is, the robot can only obtain a reward signal related to the distance from the target position when solving the target, otherwise only a fixed value reward signal is obtained. 3.The dual-robust enhanced control method suitable for robots to adapt to different tasks of claim 1, wherein, In S3, the policy neural network The process of outputting the control signal comprises: The inference neural network obtains the inference of the potential feature information of each task according to the latest data batch collected on each task; The observation signal observed by the robot and the inferred potential feature information of the task are input into the policy neural network to obtain the control signal distribution parameter, and the control signal is obtained by sampling. 4.The dual-robust enhanced control method suitable for robots to adapt to different tasks of claim 1, wherein, In S4, the training sample tuple of observation signals, control signals and reward signals sampled from the training data set comprises: Randomly sample a data batch from all training sample tuples stored in the data cache for optimization and training of the policy neural network, the action value neural network and the observation transition prediction neural network; Randomly sample a data batch from the training sample tuples collected in the last simulation in the data cache for optimization and training of the inference neural network model; For each task, search for a signal trajectory with the largest cumulative discounted reward signal from the data cache of all tasks, use the corresponding training sample tuple, and optimize and train the action value neural network by using the double robust estimator.

5. The dual-robust enhanced control method suitable for robots to adapt to different tasks of claim 4, wherein, In S4, calculating the loss function of each network and optimizing each network model by using the back propagation method comprises: For the update of the action-value neural network, including two loss function calculations and updates using data batches and signal trajectories, the first loss function is defined as the mean square error of the action value prediction and estimation in the training data batch using the action-value neural network and using the Bellman formula, i.e. the Bellman residual; the second loss function is defined as the mean square error of the action value prediction and estimation using the action-value neural network and using the double robust estimator; the inference neural network the loss function employed for the update of the action-value neural network is identical to the first loss function definition used for the update of the inference neural network; For the update of the policy neural network, the loss function is defined as the KL divergence between the distribution of the control signal output by the policy neural network and the softmax distribution calculated from the estimated action value of each action; For the update of the observation transition prediction neural network, the loss function is defined as the arithmetic mean of the negative logarithmic probability values of the data tuples in the data batch; In optimizing the action-value neural network, the computed gradients of the first loss function are only used to update parameters of the action-value neural network; in optimizing the inference neural network, the computed gradients are only used to update parameters of the inference neural network .

6. A dual robustly enhanced control system suitable for a robot to adapt to different tasks, employing the dual robustly enhanced control method suitable for a robot to adapt to different tasks according to any one of claims 1-5, characterized in that, Comprising: Model construction module: establish a policy neural network , an action-value neural network , an inference neural network , and an observation transition prediction neural network and initialize network parameters; An environment generation module: sampling multiple different task targets or potential dynamics parameters and generating multiple different task simulation environments based thereon; A dataset construction module: run the robot in multiple different simulation environments generated in S2, and the robot follows the policy neural network The output control signal performs the task, records the observation signal, the control signal and the reward signal in the process of robot simulation control, and serves as a training sample tuple, and constructs its own training dataset for each task; A training module: for each simulation task, sampling a training sample tuple of observation signals, control signals and reward signals from the training data set, taking the training sample tuple as the input of the policy neural network, action-value neural network, inference neural network model and observation transition prediction neural network respectively, calculating the loss function of each network, and optimizing each network model using the backpropagation method; from the training data set, sample signal trajectories with large reward signals for each task, estimate the action value using the double robust estimator as the target, and optimize the action-value neural network using the backpropagation method; An application module: deploy the optimized policy neural network to the robot, repeat each module, and make the policy neural network obtain as large a reward signal as possible in each task environment.

7. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program, which can be used to execute the method of any one of claims 1-5, or run the system of claim 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor, which can be used to execute the method of any one of claims 1-5, or run the system of claim 6. The program is executed by the processor, which can be used to execute the method of any one of claims 1-5, or run the system of claim 6.

Images