An adaptive control method for robots in dynamic environments based on offline reinforcement learning

Abstract

This invention relates to a method for adaptive robot control in dynamic environments based on offline reinforcement learning, belonging to the field of intelligent robot control technology. During the robot's behavioral strategy learning phase, an offline reinforcement learning robot control method is proposed that utilizes a historical behavior reuse strategy to improve behavior learning. This method enables the robot to utilize expert datasets containing significant noise for behavioral learning, improving learning speed and sample utilization, and reducing the interaction cost between the robot and the environment during intelligent robot control. During the robot's strategy evaluation phase, a stable strategy evaluation method is proposed that effectively assesses the robot's behavioral strategy.

Description

Technical Field

[0001] The present invention relates to a dynamic environment robot adaptive control method based on offline reinforcement learning, belonging to the technical field of robot intelligent control. Background Art

[0002] A robot refers to a comprehensive system that integrates multiple functions such as environmental perception and recognition, planning and decision-making, behavior and control. It can receive control signals from the designer, operate according to pre-programmed instructions, and act according to rules and strategies formulated by artificial intelligence technology.

[0003] Currently, intelligent robot behavior and decision-making control are key areas of development. With the widespread adoption of mobile robots, their operational scenarios are becoming increasingly diverse and complex. Enabling robots to operate stably in complex scenarios, while also improving their efficiency and intelligence in production operations, has become a key research focus in this field.

[0004] Mobile robots are required to possess stable and robust operational capabilities. Since their operating environments often exhibit a degree of uncertainty, they must possess robustness and adaptability from perception to control. Furthermore, due to environmental uncertainty, the robot and its environment form a dynamic game system to some extent. Consequently, various artificial intelligence methods, such as deep reinforcement learning, have been continuously proposed and gradually applied to practical engineering fields. Deep reinforcement learning, among others, is a general learning framework that combines the perception capabilities of deep learning with the learning capabilities of reinforcement learning. It enables direct control from raw sensory input to decision-making action output through end-to-end learning. When using deep reinforcement learning to solve control problems, as long as the state transitions in the environment conform to a Markov process, reinforcement learning can be used to adaptively learn a relatively robust policy. Furthermore, the learning process does not require a completely deterministic environment.

[0005] However, existing model-free deep reinforcement learning methods require online interaction with the environment to learn policies. The data generated during this learning process is very costly, making this learning approach a significant challenge for practical robots. Therefore, leveraging expert datasets or previously collected historical experience datasets for policy learning has become a major challenge. Furthermore, expert datasets are difficult to obtain for complex robots. Therefore, leveraging noisy expert datasets for policy learning has become a key technical challenge in the field of intelligent robot control. Summary of the Invention

[0006] The purpose of this invention is to solve the technical problems of low data utilization and poor learning stability faced in the process of robot intelligent control, and creatively propose a dynamic environment robot adaptive control method based on offline reinforcement learning.

[0007] The innovation of this invention lies in the following: During the robot's behavioral strategy learning phase, an offline reinforcement learning robot control method is proposed that uses historical behavior reuse strategies to improve behavior. This enables the robot to learn behaviors using expert datasets containing significant noise, improving learning speed and sample utilization, while reducing the cost of interaction between the robot and the environment. Furthermore, to effectively evaluate the robot's behavioral strategy, a stable strategy evaluation method is proposed during the robot's strategy evaluation phase.

[0008] The present invention is implemented by adopting the following technical solutions.

[0009] A method for adaptive control of a robot in a dynamic environment based on offline reinforcement learning, comprising the following steps:

[0010] Define the infinite time Markov process as Among them, S represents a finite set of continuous state spaces, A represents a finite set of continuous action spaces, Indicates that in a given state s t ∈S, take action a t ∈A and then transfer to state s t+1 ∈S probability;

[0011] Define π to represent the strategy of the agent, and define π(a t |s t ) indicates that the agent is in a given state s t Next take action a t probability;

[0012] Define φ as the parameter of the policy model, then π φ Represents the Gaussian strategy model of the neural network of the agent, and the mean and standard deviation of the strategy are output by the neural network; define r t Indicates that the agent is in state s t Take action a t The reward signal returned to the agent by the environment, γ represents the discount factor of the infinite time Markov process, γ∈[0,1]; the action value function is: The state value function is:

[0013] Step 1: Construct an expert dataset for robot behavior learning.

[0014] The construction content includes the robot's state space, action space and reward function.

[0015] The state vector s is designed based on the information the robot can perceive, and the action space a is designed based on the robot's actuators. The state space s and action space a are independently defined by the technicians based on the robot's characteristics and engineering practices.

[0016] At the same time, the reward function r(s) in the robot behavior learning process is designed t ,a t ), s t 、a t Represent the state and action of the robot at time t respectively. According to the expert strategy, the robot interacts with the environment and collects tuples (s t ,a t ,r t ,s t+1 ) and store it in the data set, r t Indicates that the agent is in state s t Take action a t The reward signal returned to the agent by the environment,

[0017] Since expert strategies are not easy to obtain, the final dataset is integrated with the same amount of interaction data as the expert strategy model, and the data is obtained by interacting with the environment in a uniformly distributed manner.

[0018] Step 2: The robot learns the behavior strategy model from the noisy expert dataset constructed in step 1 and performs intelligent control.

[0019] Specifically, a batch of state-action pairs (s t ,a t ), and use the evaluation function under this state-action pair to improve the robot's behavior strategy.

[0020] Among them, the policy gradient loss function of the policy model parameters is As shown in the following formula:

[0021]

[0022] in, Expresses expectation, E b Indicates the prepared data set; BC() function indicates the Box-Cox data distribution transformation of the input variable, BC(logπ φ (a t |s t )) represents the Box-Cox data distribution transformation of the logarithmic probability of the action collected from the experience replay pool together with the state under the current policy model; π represents the policy of the agent, π(a t |s t) indicates that the agent is in state s t Next take action a t The probability of; φ is the parameter of the strategy model, π φ Represents the Gaussian strategy model of the agent's neural network, and the mean and standard deviation of the strategy are output by the neural network; clip n The (·) function indicates that the input variable is clipped. If the input value is less than zero, it is set to zero, and the clipped value is divided by the standard deviation of the batch of input data.

[0023] In the strategy improvement using the state-action pairs in the dataset, the collected tanh(·) compressed actions are converted into the corresponding actions a sampled from the Gaussian strategy by the following formula: G :

[0024]

[0025] Among them, a tanh represents an action that has been compressed by tanh(·).

[0026] The logarithmic probability density function of the compression action is calculated by the following formula:

[0027]

[0028] Where α represents the logarithmic probability scaling factor of the action in the dataset in the current policy model, for example, α = 10 -6 ; D represents the dimension of the motion vector; μ(a G |s) is the probability density function corresponding to the Gaussian strategy, is a G The i-th element in , s represents the state.

[0029] Step 3: Evaluate the robot's behavioral policy model.

[0030] Unified evaluation network model structure, including state input layer, action input layer, action expansion layer, hidden layer and value function output layer;

[0031] In the predicted state value function V π (s t ), define an action a t A vector a with all zero elements of the same shape e When the input of the action input layer is detected to be zero, the output of the action expansion layer is set to zero, and the state value function is obtained by the unified evaluation network: V π (s t )=Q(s t ,a e );

[0032] The variable θ is the model parameter of the unified evaluation network. When this unified evaluation network model is used to simultaneously approximate the state value function and the action value function, the policy gradient loss function of the model is shown as follows:

[0033]

[0034]

[0035] in,

[0036]

[0037] Where ξ represents the scaling factor of the action-value function under the current policy; Represented by a policy parameter The state value function predicted by the target evaluation network, Represents the state value function V π The gradient loss function, represents the gradient loss function of the action value function Q; Q θ (s t ,a t ), They represent the action-value function and state-value function represented by the unified evaluation network represented by the parameter θ;

[0038] Step 4: Conduct robot training.

[0039] The robot collects data from the dataset and repeats steps 2 and 3 every other step. Then, the robot collects data from the dataset again, thus completing the autonomous learning of the robot's behavior.

[0040] Step 5: Real-time control of the robot.

[0041] After the robot behavior autonomous learning is completed, the robot will send the current perceived state s t Input to the trained policy model, the policy model gives the action a corresponding to the current state t , to achieve control of the robot.

[0042] Beneficial effects

[0043] Compared with the prior art, the method of the present invention has the following advantages:

[0044] 1. In offline reinforcement learning control of robots, this paper proposes an offline reinforcement learning robot control method that utilizes a historical behavior reuse strategy. This method enables the robot to learn behaviors using expert datasets containing significant noise, thereby improving learning speed and sample utilization, and reducing the interaction cost between the robot and the environment during intelligent control.

[0045] 2. In the stage of evaluating the robot's behavioral strategy model, the present invention proposes a stable strategy evaluation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a diagram of the BipedalWalkerHardcore-v3 robot training environment;

[0047] Figure 2 It is the unified evaluation network used in the embodiment. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0049] Example

[0050] like Figure 1 As shown in FIG, the method of the present invention is described by taking the "BipedalWalkerHardcore" robot behavior learning in the Gym environment as an example.

[0051] A method for adaptive control of a robot in a dynamic environment based on offline reinforcement learning, comprising the following steps:

[0052] Step 1: Construct the robot's state space, action space, and reward function.

[0053] Specifically, the "BipedalWalkerHardcore" robot learns a series of skills such as running, crossing pits, overcoming obstacles, and going down stairs in complex road conditions through interaction with the environment.

[0054] The robot perceives its environment through its sensors, and the policy model determines the robot's actions based on the state variables constructed from this perception information. The "BipedalWalkerHardcore" robot's perception information includes component angular velocity, horizontal velocity, vertical velocity, joint position, leg contact with the ground, and measurements from 10 LiDAR rangefinders. This perception information is concatenated into a 24-dimensional state vector s. The robot's 4-dimensional action space a is constructed using the torques of its two hip joints and the knee joint. Because the action space in this environment is a 4-dimensional continuous action space, the "BipedalWalkerHardcore" robot's goal is to reach the destination quickly while minimizing the torques on all four joints. The closer the robot gets to the destination, the greater the reward function returned by the environment. If the robot falls, the environment returns a reward of -100. The environment is designed based on this rule and directly returns the corresponding reward function r(s,a) to the robot.

[0055] Step 2: The robot uses the noisy expert dataset for intelligent learning control.

[0056] Randomly sample a batch of state-action pairs (s t ,a t ), using the evaluation function under this state-action pair to improve the robot's behavior strategy;

[0057] Among them, the policy gradient loss function of the policy model parameters is As shown in formula 1:

[0058]

[0059] Among them, E b represents the prepared data set; BC(·) function represents the Box-Cox data distribution transformation of the input variable, BC(logπ φ (a t |s t )) represents the Box-Cox data distribution transformation of the logarithmic probability of the action collected from the experience replay pool together with the state under the current policy model; clip n The (·) function indicates that the input variable is clipped. If the input value is less than zero, it is set to zero, and the clipped value is divided by the standard deviation of the input data of this batch.

[0060] In the strategy improvement using the state-action pairs in the dataset, the collected tanh(·) compressed actions are converted into the corresponding actions a sampled from the Gaussian strategy through Equation 2. G :

[0061]

[0062] Among them, a tanh represents the action compressed by tanh(·); and the logarithmic probability density function of the compressed action is calculated by Equation 3:

[0063]

[0064] Where, given α=10 -6 ; D represents the dimension of the motion vector; μ(a G |s) is the probability density function corresponding to the Gaussian strategy, is a G The i-th element in ;

[0065] Step 3: Evaluation of robot behavior control strategy.

[0066] Define the variable θ as the model parameter of the unified evaluation network; when this unified evaluation network model is used to simultaneously approximate the state value function and the action value function, the policy gradient loss function of the model is shown in Equation 4:

[0067]

[0068]

[0069] in,

[0070]

[0071] Represented by a policy parameter The state value function predicted by the target evaluation network, Represents the state value function V π The gradient loss function is used, and the clipped double-Q technique is used to limit the overestimation of the action value function. represents the gradient loss function of the action value function Q; Q θ (s t ,a t ), They represent the action-value function and state-value function represented by the unified evaluation network represented by the parameter θ;

[0072] Step 4: Robot training.

[0073] The robot collects data from the dataset, repeats steps 2 and 3 every other step, and then collects data from the dataset again, thereby completing the autonomous learning of the robot's behavior.

[0074] During the training process, first set the relevant hyperparameters. Figure 2 The figure shows the unified evaluation network. The number of neural network units in the unified evaluation network extension layer is set to 84.

[0075] Step 5: Perform real-time control of the robot.

[0076] After the robot behavior autonomous learning is completed, the robot will send the current perceived state s t Input to the trained policy model, the policy model gives the action a corresponding to the current state t , to achieve control of the robot.

Claims

1. A method for adaptive control of a robot in a dynamic environment based on offline reinforcement learning, characterized in that: The following steps are involved: Define the infinite time Markov process as Among them, S represents a finite set of continuous state spaces, A represents a finite set of continuous action spaces, In a given state s t ∈S, take action a t ∈A and then transfer to state s t+1 ∈S probability; Define π to represent the strategy of the agent, and define π(a t |s t ) indicates that the agent is in a given state s t Next take action a t probability; Define φ as the parameter of the policy model, then π φ Represents the Gaussian strategy model of the neural network of the agent, and the mean and standard deviation of the strategy are output by the neural network; define r t Indicates that the agent is in state s t Take action a t The reward signal returned to the agent by the environment, γ represents the discount factor of the infinite time Markov process, γ∈[0,1]; the action value function is: The state value function is: Step 1: Construct an expert dataset for robot behavior learning; The construction content includes the robot's state space, action space and reward function; Design the state vector s based on the information the robot can perceive; design the action space a based on the robot's actuators. The state space s and action space a are independently defined by technicians based on the robot's characteristics and engineering practices. At the same time, the reward function r(s) in the robot behavior learning process is designed t ,a t ), s t 、a t Represent the state and action of the robot at time t respectively; according to the expert strategy, the robot interacts with the environment and collects tuples (s t ,a t ,r t ,s t+1 ) and store it in the data set, r t Indicates that the agent is in state s t Take action a t The reward signal returned to the agent by the environment, The final dataset is integrated with the same amount of interaction data as the expert strategy model, and the data is obtained by interacting with the environment in a uniformly distributed manner. Step 2: The robot learns the behavior strategy model from the noisy expert dataset constructed in step 1 and performs intelligent control; Randomly sample a batch of state-action pairs (s t ,a t ), using the evaluation function under this state-action pair to improve the robot's behavior strategy; Among them, the policy gradient loss function of the policy model parameters is As shown in the following formula: in, Expresses expectation, E b Indicates the prepared data set; BC() function indicates the Box-Cox data distribution transformation of the input variable, BC(logπ φ (a t |s t )) represents the Box-Cox data distribution transformation of the logarithmic probability of the action collected from the experience replay pool together with the state under the current policy model; π represents the policy of the agent, π(a t |s t ) indicates that the agent is in state s t Next take action a t The probability of; φ is the parameter of the strategy model, π φ Represents the Gaussian strategy model of the agent's neural network, and the mean and standard deviation of the strategy are output by the neural network; clip n The (·) function means clipping the input variable. If the input value is less than zero, it will be set to zero, and the clipped value will be divided by the standard deviation of the batch of input data. In the strategy improvement using the state-action pairs in the data set, the collected tanh(·) compressed actions are converted into the corresponding actions a sampled from the Gaussian strategy by the following formula: G : Among them, a tanh represents an action that has been compressed by tanh(·); The logarithmic probability density function of the compression action is calculated by the following formula: Where α represents the logarithmic probability scaling factor of the action in the data set in the current policy model, α=10 -6 ; D represents the dimension of the motion vector; μ(a G |s) is the probability density function corresponding to the Gaussian strategy, is a G The i-th element in , s represents the state; Step 3: Evaluate the robot’s behavioral strategy model; Unified evaluation network model structure, including state input layer, action input layer, action expansion layer, hidden layer and value function output layer; In the predicted state value function V π (s t ), define an action a t A vector a with all zero elements of the same shape e ; When it is detected that the input of the action input layer is zero, the output of the action expansion layer is set to zero, and the state value function is obtained by the unified evaluation network: V π (s t )=Q(s t ,a e ); The variable θ is the model parameter of the unified evaluation network. When this unified evaluation network model is used to simultaneously approximate the state value function and the action value function, the policy gradient loss function of the model is shown as follows: in, Where ξ represents the scaling factor of the action-value function under the current policy; Represented by a policy parameter The state value function predicted by the target evaluation network, Represents the state value function V π The gradient loss function, represents the gradient loss function of the action value function Q; Q θ (s t ,a t ), They represent the action-value function and state-value function represented by the unified evaluation network represented by the parameter θ; Step 4: Conduct robot training; The robot collects data from the dataset and repeats steps 2 and 3 every other step. Then, the robot collects data from the dataset again to complete the autonomous learning of the robot's behavior. Step 5: Real-time control of the robot; After the robot behavior autonomous learning is completed, the robot will send the current perceived state s t Input to the trained policy model, the policy model gives the action a corresponding to the current state t , to achieve control of the robot.

Images