Dual-arm robot intelligent obstacle avoidance and path planning method

Abstract

The application provides a dual-arm robot intelligent obstacle avoidance and path planning method, which detects the number of times that obstacles cross the safety boundary of each module of the mechanical arm and the number of times that each module collides with each other when performing actions and interacting with the environment, adjusts the weight of the adaptive reward and punishment function, obtains action information, joint angle of the mechanical arm after interaction, spatial position information of each module of the mechanical arm, obstacles and targets and rewards, and generates soft Q values to guide the Actor and Critic network update through three experience pool sampling according to the adaptive reward and punishment change trend, detects whether the end of the actuator of the mechanical arm reaches the target area or reaches the maximum step at the same time, judges whether the current state continues to interact with the environment, detects whether the maximum training number is reached, judges whether the mechanical arm, obstacles and targets are initialized, and deploys the improved SAC algorithm optimized to the dual-arm robot for motion control, thereby solving the problems of lack of real-time and complex modeling in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, and particularly relates to a dual-arm robot intelligent obstacle avoidance and path planning method. BACKGROUND

[0002] With the wide application of deep reinforcement learning (DRL) in dual-arm robot control and intelligent decision-making, a series of advanced algorithms have emerged, such as deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and proximal policy optimization (PPO). These algorithms can not only learn complex task strategies autonomously, but also optimize trajectory planning and control strategies in dynamic environments, providing more efficient solutions for human-robot collaboration. However, in actual human-robot collaboration precise positioning and dynamic obstacle avoidance tasks, deep reinforcement learning still faces the following challenges:

[0003] Firstly, in the reinforcement learning framework, robots need to optimize strategies through reward signals. However, in human-robot collaboration environments, rewards are usually composed of multiple factors, such as accuracy requirements for reaching target points, safety requirements for obstacle avoidance, and time constraints for task execution. These rewards may conflict with each other. Therefore, how to construct a reasonable reward function in a dynamic interactive environment so that the robot can achieve a reasonable balance between precise positioning and dynamic obstacle avoidance without causing local optimum or training difficulty to converge is still a difficult point.

[0004] Secondly, although deep reinforcement learning methods show strong adaptive ability in complex human-robot collaboration tasks, they usually require a large amount of training data to learn effective strategies. In dynamic environments, the movement of obstacles, the uncertainty of human operation, and other factors increase the complexity of strategy learning, making it difficult for the model to quickly adapt or causing strategy oscillation due to reward signal conflicts, affecting task execution efficiency. Therefore, how to improve the generalization ability of deep reinforcement learning so that it can stably converge in different collaboration tasks and dynamic environments is a problem to be solved. SUMMARY

[0005] The technical scheme of the present application aims to: in a complex human-machine interaction dynamic environment, make the dual-arm robot avoid dynamic obstacles while completing precise positioning control operation, complete dual-arm robot control in a complex dynamic interaction scene, and ensure the safety and accuracy of the human-machine interaction process. At the same time, the reasonable reward mechanism is constructed in the dynamic interaction environment of deep reinforcement learning, so that the dual-arm robot reaches the optimal balance between precise positioning and dynamic obstacle avoidance, and its generalization ability and convergence stability, solving the problem of quickly learning stable strategies in a high dynamic environment.

[0006] The technical scheme of the present application provides a dual-arm robot intelligent obstacle avoidance and path planning method, the dual-arm robot comprising left and right mechanical arms, and the activity area of the left and right mechanical arms being defined as a collision space area, the method comprising the following steps:

[0007] initializing the initial joint angle of the left and right mechanical arms, setting a randomly direction moving obstacle in the collision space area, and setting a random target point in the reachable range of the end of the left and right mechanical arms;

[0008] inputting the current time sequence state information into the Actor and Critic network based on the state representation of the motion trend to perform action and environment interaction, and detecting the number of times each module of the left and right mechanical arms is crossed over the safety boundary, the number of times the obstacle collides, and the number of times each module of the left and right mechanical arms collides with each other, the current time sequence state information comprising the joint angle of the left and right mechanical arms, the position of the target point, the position of the obstacle, the shortest distance between the obstacle and each module of the left and right mechanical arms, and the end effector pose of the left and right mechanical arms;

[0009] According to the number of times each module is crossed over the safety boundary, the number of times the obstacle collides, and the number of times each module collides with each other, the collision reward function weight of each module of the left and right mechanical arms and the obstacle, and the end effector position reward function weight of the left and right mechanical arms are adaptively adjusted;

[0010] obtain the action information of the left and right mechanical arms, the joint angle and the spatial position information of the target point, the spatial position information of each module and the target point, the spatial position information of the obstacle and the target point, the collision reward function of each module of the left and right mechanical arms and the obstacle, the end effector position reward function of the left and right mechanical arms, the mutual collision reward function between each module, and the target pose maintenance reward function;

[0011] According to the adaptive reward and punishment change trend, the soft Q value of the time sequence state information and the action is obtained by three experience pool sampling, guiding the Actor and Critic network to update;

[0012] Detect whether the left and right manipulator end effector reaches the target area at the same time with the specified pose or reaches the maximum step to determine whether to continue to interact with the environment based on the current state, detect whether the maximum training round is reached to determine whether to initialize the left and right manipulator, obstacle and target point, and perform optimization.

[0013] The improved SAC algorithm after optimization is deployed to the dual-arm robot for motion control.

[0014] Preferably, the Actor and Critic network comprises a double bidirectional LSTM layer and a three-layer fully connected layer.

[0015] Preferably, the current time state information is used to describe the information of the current time state and the previous two continuous time states.

[0016] Preferably, the action information comprises joint angle increments corresponding to joint angles of the left and right manipulators.

[0017] Preferably, the adaptive reward and punishment function weight adjustment formula is as follows:

[0018]

[0019] In the formula, is the left and right manipulator end effector position reward function weight, is the left and right manipulator module and obstacle collision reward function weight, t l , t r is the left and right manipulator end effector position reward function weight, is the left and right manipulator and obstacle distance, d0 is the set safety boundary.

[0020] The technical scheme of the present application proposes a dual-arm robot intelligent obstacle avoidance and path planning method, which detects the current state of the action and the environment interaction, the number of times the obstacle crosses the safety boundary of each module of the manipulator, and the number of times each module collides with each other, and adjusts the adaptive reward and punishment function weight, obtains the action information, the joint angle of the manipulator after interaction, the spatial position information of the modules of the manipulator, the obstacle and the target point, and the reward, and samples the three experience pools according to the adaptive reward and punishment change trend to generate soft Q value to guide the Actor and Critic network update, detects whether the end effector of the manipulator reaches the target area at the same time with the specified pose or reaches the maximum step to determine whether the current state continues to interact with the environment, detects whether the maximum training round is reached to determine whether to initialize the manipulator, obstacle and target point, and deploys the improved SAC algorithm after optimization to the dual-arm robot for motion control, solving the problems of lack of real-time and complex modeling in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of a dual-arm robot intelligent obstacle avoidance and path planning method based on deep reinforcement learning is provided for an embodiment of the present application.

[0022] Figure 2 An algorithm framework diagram of a dual-arm robot intelligent obstacle avoidance and path planning based on deep reinforcement learning is provided for an embodiment of the present application.

[0023] Figure 3 A scene diagram of a dual-arm robot intelligent obstacle avoidance and path planning based on deep reinforcement learning is provided for an embodiment of the present application.

[0024] Figure 4 A device diagram of a Baxter dual-arm robot mechanical arm model is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0025] The present application will be further described with reference to the following specific embodiments. It should be understood that these embodiments are intended to illustrate the present application and are not intended to limit the scope of the present application. Furthermore, it should be understood that one of ordinary skill in the art can make various modifications or changes to the present application after reading the content taught in the present application, and these equivalent forms also fall within the scope of the appended claims of the present application.

[0026] As shown in Figure 1 , the present embodiment provides a dual-arm robot intelligent obstacle avoidance and path planning method, which adopts a completely centralized training framework as shown in Figure 2 , that is, the input state of the value network and the policy network is the global state, and the output is the global value and the global policy, and the dual-arm robot intelligent obstacle avoidance and path planning method comprises the following steps:

[0027] Initialize the left and right two seven-axis mechanical arms of the dual-arm robot and the obstacles and target points. Among them, the mechanical arm includes the initial joint angle, the obstacles are set in the possible collision space region of the mechanical arm and are randomly moved in the direction, and the target point is randomly generated in the part of the space region which can be reached by the end of the mechanical arm.

[0028] Input the current time sequence state information into the Actor and Critic network based on the motion trend state representation, and the Actor and Critic network based on the motion trend state representation respectively comprises a double bidirectional LSTM layer and a three-layer fully connected layer.

[0029] The end effector of the robot left and right arm position reward function weight is denoted as μ l, the robot left and right arm module and obstacle collision reward function weight is denoted as μ r, and the robot left and right arm module and itself collision reward function weight is denoted as μ s.

[0030] According to the number of times of crossing the safety boundary of each module of the left and right two mechanical arms and the number of times of mutual collision of each module of the left and right two mechanical arms, the adaptive reward and punishment function weight is adjusted, as follows:

[0031]

[0032] In the formula, μ l is the end effector of the robot left and right arm position reward function weight, The robot left and right arm module and obstacle collision reward function weight is denoted as μ r, and the robot left and right arm module and itself collision reward function weight is denoted as μ s. l r The end effector of the robot left and right arm position reward function weight is denoted as μ l, the robot left and right arm module and obstacle collision reward function weight is denoted as μ r, and the robot left and right arm module and itself collision reward function weight is denoted as μ s. The robot left and right arm module and obstacle collision reward function weight is denoted as μ r, and the robot left and right arm module and itself collision reward function weight is denoted as μ s.

[0033] The action information and the joint angle of the left and right two seven-axis mechanical arms of the dual-arm robot, the spatial position information of the obstacle and the target point, and the reward after interaction are obtained. The action information is the joint angle increment of the 14 joints of the two seven-axis mechanical arms as follows:

[0034]

[0035] The reward includes the end effector position reward function of the left and right two mechanical arms of the robot, the collision reward function of each module of the left and right two mechanical arms and the obstacle, the collision reward function of each module of the left and right two mechanical arms and itself, and the target pose maintenance reward function.

[0036] Specifically, the end effector position reward function of the robot left and right arm is related to the Euclidean distance of the left and right two mechanical arms from the target point. The reward increases as the target point is approached, and the reward decreases as the target point is approached. The expression is as follows:

[0037]

[0038] In the formula, μ l is the end effector of the robot left and right arm position reward function weight, l r ​​​respectively, are the rate of change of the left and right manipulator end effector position reward function weight, respectively, are the distance error of the left and right manipulator end effector position and target point position.

[0039] The robot left and right two manipulator module and obstacle collision reward function is not only related to the number of times the safety distance of each module is crossed, but also related to the crossing safety distance. When the safety distance is crossed, the punishment is given, and the more the safety distance is crossed, the higher the punishment is. Its expression is as follows:

[0040]

[0041]

[0042] The robot left and right two manipulator module and obstacle collision reward function is not only related to the number of times the safety distance of each module is crossed, but also related to the crossing safety distance. When the safety distance is crossed, the punishment is given, and the more the safety distance is crossed, the higher the punishment is. Its expression is as follows:

[0043]

[0044] In the above formula, C and C respectively, are the collision penalty coefficients of the left and right arm modules of the dual-arm robot, C li and C ri respectively, are the collision conditions of the left and right arm modules of the dual-arm robot, 1 when collision, 0 when no collision.

[0045] The robot target pose maintenance reward function is that the left and right two manipulator end effector can obtain a reward when maintaining in the target area at a specified pose. Its expression is as follows:

[0046]

[0047] In the above formula, a l , a r respectively, are the position error punishment coefficients of the left and right manipulator of the dual-arm robot, which are positive numbers, b l , b r respectively, are the attitude error punishment coefficients of the left and right manipulator of the dual-arm robot, which are positive numbers, respectively, are the attitude error, the value range is (0-1), and r is the target area radius.

[0048] Three experience pools are sampled according to the adaptive reward and punishment change trend. Specifically, the three experience pools include:

[0049] Experience replay pool, which stores experience data of the agent interacting with the environment wherein, represents the time series state information of the agent at time t, a tdenotes the action taken by the agent at time t, r t denotes the reward obtained after taking the action at time t, denotes the state information reached at time t+1 after the agent takes the action, and done denotes a flag indicating whether the task has been completed.

[0050] The expert pose experience pool is an expert experience pool in which the experience data is artificially taught to consider only pose control and ignore obstacle avoidance.

[0051] The interaction experience pool is an experience pool in which data is continuously generated during interaction; the obstacle avoidance and positioning experience pool is an experience pool in which data in which the end of the dual-arm actuator is maintained at a target region with a certain pose and no collision with obstacles occurs throughout the round is stored in the interaction experience pool.

[0052] The training process first uses expert data sampling and continuously transitions to sampling from the interaction experience pool, separates high-quality data in the interaction experience pool and stores it in the obstacle avoidance and positioning experience pool, and after storing to a specified number, the sampling method is based on combined sampling of the interaction experience pool and the obstacle avoidance and positioning experience pool and then transitions to sampling based on the interaction experience pool. The sampling process expression is:

[0053]

[0054] In the above formula, denotes the number of data sampled from the interaction experience pool, the expert pose experience pool and the obstacle avoidance and positioning experience pool. N is the batch data size sampled from the experience pool at each time step t, T s is the decay step number, used to control the speed of sampling change, is the size of the time step corresponding to the expert experience pool data of the obstacle avoidance and positioning expert experience pool which is not 0, are all non-negative integers.

[0055] The soft Q value is generated based on knowledge distillation to guide the Actor and Critic network update.

[0056] The soft Q value is based on the proportional weighted sum of the target Q value and the teacher target Q value, and the proportion coefficient is determined by the TD error size of a single sampling batch data, and the expression is:

[0057]

[0058] In the above formula, is the TD error standardized to (0-1), and ω is the proportion factor to adjust importance, is the sampled state information.

[0059] When the main Q network, the target Q network, the teacher target Q network and the policy network are updated, the training and update of the policy network and the value network are first started by sampling the expert pose experience pool, and data is collected and stored in the interaction experience pool and sampled to the interaction experience pool, and high-quality data is separated in the interaction experience pool for network training and update, and the specific update process comprises:

[0060] First, the current state sequence The action a is obtained by combining the policy network with the LSTM, and the next state sequence And the reward r are stored in the interaction experience pool, and the sampling data And the action a predicted by the policy network ′ Two main Q networks, two target Q networks and two teacher target Q networks are used to avoid overestimation, and the minimum values Q, Q target , Q T-target , Q are taken respectively, Q is used to update the policy network, and Q target , Q T-target are generated based on the average TD error of the sampling batch data. soft Q is used to update the main Q network.

[0061] Whether the left and right manipulator end effectors simultaneously reach the target area or reach the maximum step at the specified pose is detected, and whether to continue to interact with the environment based on the current state is judged;

[0062] Whether the maximum training number is reached and the training is completed is detected, and whether the left and right seven-axis manipulators of the dual-arm robot and the obstacles and target points need to be initialized again is judged;

[0063] The improved SAC algorithm is deployed to the dual-arm robot for motion control.

[0064] The simulation environment of the dual-arm robot intelligent obstacle avoidance and path planning is built in CoppeliaSim, and Python and CoppeliaSim are combined for simulation, and the simulation scene is as shown in Figure 3 The Baxter dual-arm robot is selected, and the mechanical arm parameters thereof are used as the simulation mechanical arm parameters, as shown in Figure 4 The simulation experiment considers that the left and right manipulator end effectors of the dual-arm robot simultaneously reach the target area and maintain for a period of time, and the modules of the left and right manipulator end effectors of the dual-arm robot avoid collision with dynamic obstacles, and the pose error between the left and right manipulator end effectors and the target point and the collision times of the left and right manipulators are used as the main indexes for evaluating the control strategy.

[0065] The improved SAC algorithm is used for positioning control of the dual-arm robot intelligent obstacle avoidance, and the current time sequence state information The Actor and Critic network based on the motion trend state representation is inputted, so that the motion decision of the dual-arm robot is based on the current motion trend, and the update and optimization of the whole Actor and Critic network are based on the time sequence state information at the same time This method can alleviate the partially observable Markov decision process (POMDP) problem to be more consistent with the dynamic real-time obstacle avoidance scene.

[0066] The technical scheme of the present application sets up an adaptive reward and punishment function in the design of the reward, and in the dynamic interactive environment of deep reinforcement learning, the modules of the dual-arm robot can adaptively adjust the position reward weight of the end effectors of the left and right arms, the collision reward weight of the modules of the left and right arms of the robot and the obstacles, so as to effectively achieve the optimal balance between precise positioning and dynamic obstacle avoidance, and the generalization ability and convergence stability of the dual-arm robot.

[0067] The technical scheme of the present application designs a three-experience pool sampling method in combination with the characteristics of task transition under the change of reward and punishment weight in experience pool sampling.

[0068] In the aspect of updating the Actor network and the Critic network, the technical scheme of the present application combines knowledge distillation to generate soft Q values by reasonably distributing target Q values and teacher target Q values according to the TD error of the sampling batch data, so as to effectively alleviate the influence of the non-steady state of the environment under the adaptive change of the reward and punishment function on the stability of the algorithm.

[0069] By applying the improved SAC algorithm, the dual-arm robot can avoid dynamic obstacles while completing precise positioning control operation in a complex human-robot interactive dynamic environment, complete the control of the dual-arm robot in a complex dynamic interactive scene, and ensure the safety and accuracy of the human-robot interactive process.

[0070] The embodiment of the application provides a dual-arm robot intelligent obstacle avoidance and path planning method, which comprises the following steps: initializing a left seven-axis mechanical arm and a right seven-axis mechanical arm of a dual-arm robot, an obstacle and a target point; inputting current time sequence state information into an Actor and a Critic network based on motion trend state representation; performing action and environment interaction, and detecting the number of times that the obstacle crosses the safety boundary of each module of the left and right mechanical arms and the number of times that each module of the left and right mechanical arms collides with the self; adjusting the weight of an adaptive reward and punishment function according to the number of times that the obstacle crosses the safety boundary; obtaining action information, joint angles of the left and right seven-axis mechanical arms of the dual-arm robot after interaction, spatial position information of each module of the mechanical arms, the obstacle and the target point, and a reward; performing three experience pool sampling according to the adaptive reward and punishment change trend; generating a soft Q value to guide the Actor and the Critic network update based on knowledge distillation; detecting whether the end of the left and right mechanical arm actuators simultaneously reaches the target area or reaches the maximum step with a specified pose, and judging whether to continue to interact with the environment based on the current state; detecting whether the maximum training number of training is reached, and judging whether the left and right seven-axis mechanical arms of the dual-arm robot and the obstacle and the target point need to be initialized again; and deploying the optimized improved SAC algorithm to the dual-arm robot for motion control.

[0071] The embodiment of the application provides a dual-arm robot intelligent obstacle avoidance and path planning method, which solves the problems of lack of real-time and modeling complexity in the process of mechanical arm dynamic obstacle avoidance based on a traditional intelligent algorithm, and based on the improved SAC algorithm, the dual-arm robot can also adaptively realize positioning control under obstacle avoidance in a complex human-computer interaction dynamic environment, and overcomes the problems of complex reward mechanism, difficulty in manually setting reward amplitude and slow convergence or even difficulty in convergence of an intelligent agent in a complex non-steady-state environment, so that dual-arm mechanical arm control in a complex dynamic interaction scene is completed, and the safety and accuracy of the human-computer interaction process are ensured.

Claims

1. A method for intelligent obstacle avoidance and path planning of a dual-arm robot, characterized in that, The double-arm robot comprises left and right mechanical arms, and a collision space region is defined by the movable region of the left and right mechanical arms, and the method comprises the following steps: initializing initial joint angles of the left and right mechanical arms, setting an obstacle moving in a random direction in the collision space region, and setting a random target point in the reachable range of the end of the left and right mechanical arms; inputting current time sequence state information into an Actor and Critic network based on state representation of motion trends to perform action and environment interaction, and detecting the number of times each module of the left and right mechanical arms is crossed over a safety boundary, the number of times the obstacle collides, and the number of times each module of the left and right mechanical arms collides, wherein the current time sequence state information comprises joint angles of the left and right mechanical arms, positions of the target points, positions of the obstacles, shortest distances between the obstacles and each module of the left and right mechanical arms, and end effector pose information of the left and right mechanical arms; adapting the collision reward function weight of each module of the left and right mechanical arms and the obstacle, and the end effector position reward function weight of the left and right mechanical arms according to the number of times each module is crossed over the safety boundary, the number of times the obstacle collides, and the number of times each module collides; the adaptive reward and punishment function weight adjustment formula is as follows: ， ， , In the formula, , is the left and right robot arm end effector position reward function weight, , is the left and right robot arm each module and obstacle collision reward function weight, , is the initialization of the left and right robot arm end effector position reward function weight, , is the left and right robot arm and obstacle distance, is the set safety boundary; obtaining action information of the left and right mechanical arms, joint angles and target point space position information, module and target point space position information, obstacle and target point space position information, collision reward functions of each module of the left and right mechanical arms and the obstacle, end effector position reward functions of the left and right mechanical arms, mutual collision reward functions between each module, and target pose maintenance reward functions; sampling time sequence state information and action according to the adaptive reward and punishment change trend through three experience pools, generating soft Q values based on knowledge distillation to guide the Actor and Critic network to update; the three experience pools comprise an experience replay pool, an expert pose experience pool, and an interaction experience pool; detecting whether the end of the left and right mechanical arm actuators reaches the target region or reaches the maximum step at the same time with the specified pose to determine whether to continue to interact with the environment based on the current state, and detecting whether the maximum training number is reached to determine whether to initialize the left and right mechanical arms, the obstacle and the target point for optimization; deploying the improved SAC algorithm after optimization to the double-arm robot for motion control.

2. The dual-arm robot intelligent obstacle avoidance and path planning method of claim 1, wherein, The Actor and Critic network comprises a double-layer bidirectional LSTM layer and a three-layer fully connected layer.

3. The dual-arm robot intelligent obstacle avoidance and path planning method of claim 1, wherein, The current time sequence state information is used to describe the information of the current time state and the previous two continuous time states.

4. The dual-arm robot intelligent obstacle avoidance and path planning method of claim 3, wherein, The action information comprises joint angle increments corresponding to the joint angles of the left and right mechanical arms.

Images