Robot throwing motion control method and device based on deep reinforcement learning

Abstract

The invention relates to a robot throwing motion planning method based on deep reinforcement learning. The method comprises the following steps: constructing a data set of robot throwing tasks through a virtual simulation environment; the data set is used for training a neural network to predict a drop point of the thrown object; on the basis of a deep reinforcement learning algorithm, performing trial and error learning continuously through a robot and a deep neural network of a predicted drop point to obtain physical parameters of a throwing behavior; the physical parameters of the throwing behavior output by the network are directly deployed in the physical environment, and then the robot is controlled to execute the actual throwing task. According to the method, the throwing behavior of the virtual simulation environment is fitted through the deep neural network, and the problem that training time is too long when the virtual simulation environment is adopted in reinforcement learning is avoided. The virtual simulation environment and the physical environment are restored according to the ratio of 1: 1, the safety problem caused by direct training on a physical robot can be avoided, a trained model can be directly migrated to a physical platform, and the robot can autonomously execute a throwing task.

Description

Technical Field

[0001] The present invention relates to a robot throwing motion control method and device based on deep reinforcement learning, belonging to the technical field of robot trajectory planning. Background Art

[0002] With the advancement of science and technology, robotics technology has been fully developed and has been widely used in factory automation, medical health, intelligent services and other fields. As a type of robot, the robotic arm can fully replace humans to complete tedious and complex production and manufacturing tasks in a structured environment with its high precision, high efficiency and high intelligence, and has great development potential.

[0003] Learning the throwing skill of the robot can give the robot a larger workspace and faster picking speed. There are many solutions to the current robot throwing problem. Many of the traditional solutions are based on the traditional modeling method, which solves the specific problem through modeling, solves the throwing trajectory, and then calculates the control method of each joint controller based on the trajectory tracking method. However, this method has the problems of poor adaptability and weak generalization.

[0004] In recent years, with the development of artificial intelligence, deep reinforcement learning has emerged and applied to the field of robot control, such as learning skills such as grasping, obstacle avoidance, shaft hole assembly and throwing. Through trial and error learning between the agent and the environment, the reward value is used as feedback information from the environment to guide the robot to learn an optimal strategy. However, reinforcement learning algorithms have problems such as slow convergence speed, low learning efficiency, and the interaction between the robot and the environment requires a large amount of sample data. Summary of the invention

[0005] The technical problem to be solved by the present invention is: traditional modeling methods and deep reinforcement learning methods have their own advantages and disadvantages. How to give full play to the advantages of the two methods in the robot throwing motion control method and seek a method with strong adaptability, good generalization, high algorithm efficiency and short training time is still a key problem to be solved in the robot throwing motion control method.

[0006] In order to solve the above technical problems, one aspect of the present invention is to provide a robot throwing motion control method based on deep reinforcement learning, which is characterized by comprising the following steps:

[0007] S1: Construct a dataset for robot throwing tasks through a virtual simulation environment;

[0008] S2: using the data set to train a neural network to predict the landing point of an object thrown by the robot, wherein the input of the neural network is the throwing position and throwing speed of the thrown object, and the output is the landing point;

[0009] S3: Based on the deep reinforcement learning algorithm, the robot and the neural network are continuously learning by trial and error in the environment to obtain the physical parameters of the throwing behavior;

[0010] S4: The physical parameters of the throwing behavior output by the neural network are directly deployed into the physical environment, thereby controlling the robot to perform the actual throwing task.

[0011] Preferably, in step S1, the data set is collected in a virtual simulation environment, and a data sample in the data set includes a throwing position, a throwing speed and a landing point of a thrown object, which is expressed as:

[0012]

[0013] The throwing position is The throwing speed is [v x , v y , v z ], the landing point is

[0014] Preferably, in step S2, when training the neural network, the minimum square error is used as the loss function, and the Adam optimizer is used to optimize the network parameters, wherein the training set used includes 2000 throwing data collected from the virtual simulation environment, and the test set includes 1000 throwing data collected from the virtual simulation environment.

[0015] Preferably, the step S3 specifically includes the following steps:

[0016] S31: Determine the parameters of the throwing task, including state s, action a and reward function r;

[0017] S32: Initialization phase, constructing four neural networks, including the online strategy network μ(s t θ μ ), online value network Q(s, a; θ Q ) and the target policy network and the target value network, where θ μ is the network parameter of the online policy network, θ Q are the network parameters of the online value network; initialize the online policy network and the online value network, copy the network parameters of the online policy network and the online value network to the corresponding target policy network and the target value network, and initialize the experience replay pool R;

[0018] S33: Experience data collection phase, initialize the random process for action exploration, initialize the environment to obtain the initial state value, and then initialize the online policy network μ(s t θ μ ) Select an action according to the following formula:

[0019]

[0020] Among them, the behavior strategy is based on the current online strategy network μ(s t θ μ ) and random noise The random process generated and the action value a is obtained by sampling from the random process t , the action a generated by the environment execution t , and returns the reward r t and the new state s t+1 , the agent converts the state of the environment into a quadruple (s t , a t , r t ,s t+1 ) is stored in the experience replay pool R as a training dataset for the online policy network;

[0021] S34: During the network training phase, the agent randomly selects N conversion experience samples from the experience replay pool R as the online policy network μ(s t θ μ ), online value network Q(s, a; θ Q ) of small batch training sample data, and calculate the online value network Q(s, a; θ Q ) is the gradient L(θ Q ), as shown below:

[0022]

[0023] Where N is the number of historical experience data randomly extracted from the experience replay pool R when updating parameters by stochastic gradient descent, and the online value network Q(s, a; θ Q ) uses the mean square error as the loss function, y i is the target Q value, calculated by Bellman equation;

[0024] Update the network parameters θ of the online value network using stochastic gradient descent Q and the network parameters θ of the online policy network μ , the formula for the policy gradient is:

[0025]

[0026] in, is the derivative of the value network output value for action a, is the derivative of the policy network output action with respect to the policy network θμ;

[0027] Finally, the target strategy network and target value network are updated using the soft update method, as shown in the following formula:

[0028]

[0029] Among them, θ μ ′ is the network parameter of the updated online policy network, θ Q ′ is the network parameter of the updated online value network, and τ is the weight.

[0030] Preferably, in step S34, y i Calculated by the following formula:

[0031] y i =r i +ρQ′(s i+1 ,μ′(s i+1 θ μ′ );θ Q ′)

[0032] Among them, ρ is the discount factor, which is a constant between [0, 1], Q′() is the updated online value network, and μ′() is the updated online strategy network.

[0033] Preferably, in step S31, the state s includes the joint angle, the joint angular velocity, the position of the end effector, the velocity of the end effector, the position of the target point, and the position error, and a state is expressed as:

[0034]

[0035] Among them, [θ1, θ2, θ3, θ4, θ5, θ6] are the six joint angle values, [w1, w2, w3, w4, w5, w6] are the six joint angle velocities, [e x , e y ] is the error between the target position and the landing point; the action includes the joint angle and joint angular velocity during throwing;

[0036] An action a is represented by: a t =[θ1, θ2, θ3, θ4, w1, w2, w3, w4];

[0037] The reward function r depends on the position error and is expressed as

[0038] Preferably, in step S4, the robot is controlled to perform the actual throwing task. First, a camera is used on the physical platform to identify the position of the target point, and the state information of the physical platform is input into the trained DDPG algorithm model to obtain the physical parameters of the robot's motion process. The control parameters are fed back to the robot motion planner to enable the robot to complete the throwing task.

[0039] Another aspect of the present invention is to provide a robot throwing motion control device based on deep reinforcement learning, characterized in that it is used to implement the above method, including a UR5e six-degree-of-freedom robot, a control cabinet, a base, a Robotiq2F 140 gripper, a target basket, a PC and a throwing object.

[0040] The present invention proposes a robot throwing motion control method based on deep reinforcement learning. The throwing task of a six-degree-of-freedom robot is taken as the research object. Through the DDPG deep reinforcement learning algorithm model, the robot and the deep learning fitting environment are quickly interactively learned. The physical parameters obtained by the trained algorithm model are input into the motion controller, so that the robot completes the throwing task, and the six-degree-of-freedom robot UR5e can be realized to throw objects.

[0041] Compared with the prior art, the advantages of the present invention are: by fitting the throwing behavior of the simulation environment through a neural network, the problem of long training time in the simulation environment in reinforcement learning is avoided, and the training efficiency of the algorithm is improved. Compared with the traditional modeling method, the method of deep reinforcement learning used in the present invention has strong adaptability and good generalization; the simulation environment and the physical environment are restored 1:1, which can avoid the safety problems of direct training on the physical robot and prevent damage to the physical robot, and can also directly migrate the trained model to the physical platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0043] Figure 2 The virtual simulation environment constructed by Coppeliasim in the present invention is illustrated;

[0044] Figure 3 is a flow chart of the DDPG algorithm of the present invention;

[0045] Figure 4 The neural network model structure used to fit the environment in the present invention is illustrated;

[0046] Figure 5 It is a diagram of the principle of reinforcement learning training adopted by the present invention;

[0047] Figure 6 The neural network model of the DDPG algorithm framework of the present invention is illustrated. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0049] like Figure 1 As shown, one aspect of an embodiment of the present invention provides a robot throwing motion control method based on deep reinforcement learning, and the method flow refers to Figure 3 , including the following steps:

[0050] Step 1: Use CoppeliaSim to build a dataset for the robot throwing task.

[0051] In step 1, the data set of the throwing task is collected in the simulation software CoppeliaSim, and the scene built in the simulation environment is as follows Figure 2 As shown in the figure, 1 is the UR5e six-degree-of-freedom robot, 2 is the Robotiq 2F gripper, 3 is the object to be thrown, 4 is the base and control cabinet, and 5 is the target area. The data collected from Coppeliasim includes the throwing position, throwing speed, and landing point of the thrown object. A data set can be expressed as:

[0052]

[0053] The throwing position is The throwing speed is [v x , v y , v z ], the landing point is

[0054] Step 2: Use the dataset to train a neural network to predict the landing point of a thrown object.

[0055] In step 2, the data set is used to train a neural network to predict the landing point of a thrown object, including preprocessing of the data set, designing the network structure, optimizing parameters, and evaluating network performance with a test set. First, the data collected from Coppeliasim is preprocessed to remove abnormal data. The structure of the neural network is as follows: Figure 4 As shown in the figure, the input is the throwing position and throwing speed of the thrown object, and the output is the landing point. The minimum square error is used as the loss function, and the Adam optimizer is used to optimize the network parameters. The training set includes 2000 throwing data collected from Coppeliasim, and the test set includes 1000 throwing data collected from Coppeliasim. Finally, the performance of the neural network model after parameter adjustment is verified on the test set.

[0056] Step 3: Based on the DDPG deep reinforcement learning algorithm, the robot and the neural network are continuously trained through trial and error in the environment to obtain the physical parameters of the throwing behavior.

[0057] The process of continuous interaction and optimization between the robot and the deep neural network predicting the landing point in step 3 specifically includes the following steps:

[0058] Step 301: The throwing task can be solved by reinforcement learning. The components of reinforcement learning are as follows: Figure 5 As shown, it includes the agent and the environment, which interact through states, actions, and rewards. First, we need to determine the parameters of the throwing task, including state s, action a, and reward function r, where:

[0059] The state s specifically includes the joint angle, joint angular velocity, end effector position, end effector velocity, target point position, and position error. A state is expressed as:

[0060]

[0061] Among them, [θ1, θ2, θ3, θ4, θ5, θ6] are the angle values ​​of the six joint angles, [w1, w2, w3, w4, w5, w6] are the angular velocities of the six joint angles, [e x , e y ] is the error between the target position and the landing point position;

[0062] Action a includes the joint angle and joint angular velocity during throwing. An action can be expressed as:

[0063] a t =[θ1, θ2, θ3, θ4, w1, w2, w3, w4]

[0064] Among them, the network only outputs the joint information of the base, shoulder, elbow and wrist 1;

[0065] The reward function r depends on the position error and can be expressed as:

[0066]

[0067] Step 302: In the initialization phase, two deep neural networks, the target network and the online network, are used for both the policy network and the value network to construct four neural networks, such as Figure 6 As shown, including the online strategy network μ(s t θ μ ), online value network Q(s, a; θ Q) as well as the target policy network and the target value network to improve the stability of the algorithm and solve the overestimation or underestimation problem caused by bootstrapping in the temporal difference method. Initialize the online policy network and the online value network, copy the parameters of the online network to the corresponding target network parameters, and initialize the experience replay pool R.

[0068] Step 303: In the experience data collection phase, the random process is initialized to perform action exploration, the environment is initialized to obtain the initial state value, and the online network selects an action according to the following formula:

[0069]

[0070] The behavior strategy is a random process generated according to the current online strategy network μ and random OU noise, and the action value a is sampled from the random process. t , the action a generated by the environment execution t , and returns the reward r t and the new state s t+1 , the agent converts the state of the environment into a quadruple (s t , a t , r t , S t+1 ) is stored in the experience replay pool R as the training data set for the online network model. μ represents the parameters of the online policy network, It indicates that the Ornstein-Uhlenbeck process generates time-dependent exploration.

[0071] Step 304: In the network training phase, the agent randomly selects N conversion experience samples from the experience playback pool as small batch training sample data for the online policy network and the online value network, calculates the gradient of the online value network, and feeds the small batch randomly sampled samples into the neural network for training, removes the temporal correlation between samples, and reduces the deviation generated by the value function estimation after function approximation, thereby solving the correlation and non-static distribution problems between data and making the algorithm easier to converge. As described in the following formula:

[0072]

[0073] Among them, N is the number of historical experience data randomly extracted from the experience replay pool when updating parameters by stochastic gradient descent. The online value network Q(s i , a i θ Q ) uses the mean square error, θ Q represents the parameters of the online value network, y i is the target Q value, calculated by the Bellman equation:

[0074] yi =r i +ρQ′(s i+1 ,μ′(s i+1 θ μ′ );θ Q ′)

[0075] ρ represents the discount factor, which is a constant between [0, 1]. Use stochastic gradient descent to update the parameters θ of the online value network Q and the parameters θ of the online policy network μ , the formula for the policy gradient is:

[0076]

[0077] It is expressed as the derivative of the value network output value to action a, It is represented as the policy network output action to the policy network θ μ The derivative of .

[0078] Finally, the target policy network and target value network are updated using the soft update method.

[0079]

[0080] τ represents the update rate, which is a constant between [0, 1].

[0081] Repeatedly execute a single experience trajectory.

[0082] Step 4: Deploy the physical parameters of the network output throwing behavior directly into the physical environment, thereby controlling the robot to perform the actual throwing task.

[0083] In step 4, the robot is controlled to perform the actual throwing task. First, the position of the target point is identified by using a camera on the physical platform. The state information of the physical platform is input into the trained DDPG algorithm model to obtain the physical parameters of the robot's motion process. The control parameters are fed back to the robot motion planner to enable the robot to complete the throwing task.

[0084] Another aspect of an embodiment of the present invention is to provide a robot throwing motion control device based on deep reinforcement learning, comprising the following components: a UR5e six-degree-of-freedom robot, a control cabinet, a base, a Robotiq 2F 140 gripper, a target basket, a PC and a throwing object, wherein the device is used to implement the above-mentioned method.

Claims

1. A robot throwing motion control method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Construct a dataset for robot throwing tasks through a virtual simulation environment; S2: using the data set to train a neural network to predict the landing point of an object thrown by the robot, wherein the input of the neural network is the throwing position and throwing speed of the thrown object, and the output is the landing point; S3: Based on the deep reinforcement learning algorithm, the robot and the neural network are continuously learning by trial and error in the environment to obtain the physical parameters of the throwing behavior; S4: The physical parameters of the throwing behavior output by the neural network are directly deployed into the physical environment, thereby controlling the robot to perform the actual throwing task.

2. A robot throwing motion control method based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S1, the data set is collected in a virtual simulation environment, and a data sample in the data set includes a throwing position, a throwing speed, and a landing point of a thrown object, which is expressed as: The throwing position is The throwing speed is [v x ,v y ,v z ], the landing point is 3. A robot throwing motion control method based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S2, when training the neural network, the minimum square error is used as the loss function, and the Adam optimizer is used to optimize the network parameters, wherein the training set used includes 2000 throwing data collected from the virtual simulation environment, and the test set includes 1000 throwing data collected from the virtual simulation environment.

4. A robot throwing motion control method based on deep reinforcement learning as claimed in claim 1, characterized in that: The step S3 specifically comprises the following steps: S31: Determine the parameters of the throwing task, including state s, action a and reward function r; S32: Initialization phase, constructing four neural networks, including the online strategy network μ(s t θ μ ), online value network Q(s,a;θ Q ) and the target policy network and the target value network, where θ μ is the network parameter of the online policy network, θ Q are the network parameters of the online value network; initialize the online policy network and the online value network, copy the network parameters of the online policy network and the online value network to the corresponding target policy network and the target value network, and initialize the experience replay pool R; S33: Experience data collection phase, initialize the random process for action exploration, initialize the environment to obtain the initial state value, and then initialize the online policy network μ(s t θ μ ) Select an action according to the following formula: Among them, the behavior strategy is based on the current online strategy network μ(s t θ μ ) and random noise The random process generated and the action value a is obtained by sampling from the random process t , the action a generated by the environment execution t , and returns the reward r t and the new state s t+1 , the agent converts the state of the environment into a quadruple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool R as a training dataset for the online policy network; S34: During the network training phase, the agent randomly selects N conversion experience samples from the experience replay pool R as the online policy network μ(s t θ μ ), online value network Q(s,a;θ Q ) of small batch training sample data, and calculate the online value network Q(s,a;θ Q ) is the gradient L(θ Q ), as shown below: Where N is the number of historical experience data randomly extracted from the experience replay pool R when updating parameters by stochastic gradient descent, and the online value network Q(s,a;θ Q ) uses the mean square error as the loss function, y i is the target Q value, calculated by Bellman equation; Update the network parameters θ of the online value network using stochastic gradient descent Q and the network parameters θ of the online policy network μ , the formula for the policy gradient is: in, is the derivative of the value network output value for action a, The output action of the policy network is the policy network θ μ The derivative of Finally, the target strategy network and target value network are updated using the soft update method, as shown in the following formula: Among them, θ μ′ is the network parameter of the updated online policy network, θ Q′ is the network parameter of the updated online value network, and τ is the weight.

5. A robot throwing motion control method based on deep reinforcement learning as claimed in claim 4, characterized in that: In step S34, y i Calculated by the following formula: y i =r i +ρQ′(s i+1 ,μ′(s i+1 ;θ μ′ );i Q′ ) Among them, ρ is the discount factor, which is a constant between [0,1], Q′() is the updated online value network, and μ′() is the updated online policy network.

6. A robot throwing motion control method based on deep reinforcement learning as claimed in claim 4, characterized in that: In step S31, the state s includes the joint angle, the joint angular velocity, the position of the end effector, the velocity of the end effector, the position of the target point, and the position error. Then one state is represented as: Among them, [θ1,θ2,θ3,θ4,θ5,θ6] are the six joint angle values, [w1,w2,w3,w4,w5,w6] are the six joint angle velocities, [e x ,e y ] is the error between the target position and the landing point; the action includes the joint angle and joint angular velocity during throwing; An action a is represented by: a t =[θ1,θ2,θ3,θ4,w1,w2,w3,w4]; The reward function r depends on the position error and is expressed as 7. A robot throwing motion control method based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S4, the robot is controlled to perform the actual throwing task. First, the position of the target point is identified by using a camera on the physical platform, and the state information of the physical platform is input into the trained DDPG algorithm model to obtain the physical parameters of the robot's motion process. The control parameters are fed back to the robot motion planner to enable the robot to complete the throwing task.

8. A robot throwing motion control device based on deep reinforcement learning, characterized in that: Used to implement the method of any one of claims 1-7, comprising a UR5e six-degree-of-freedom robot, a control cabinet, a base, a Robotiq 2F 140 gripper, a target basket, a PC and a throwing object.

Images