Robotic arm autonomous dynamic obstacle avoidance method and system based on deep reinforcement learning
By using deep reinforcement learning to generate an experience replay pool and train a deep learning network, the problem of low obstacle avoidance efficiency of robotic arms in complex environments is solved, and the ability to autonomously adapt to obstacle avoidance in dynamic environments is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAINAN POWER SUPPLY CO OF STATE GRID ANHUI ELECTRIC POWER CORPORATIO
- Filing Date
- 2022-11-29
- Publication Date
- 2026-06-02
AI Technical Summary
Existing robotic arm trajectory planning methods are inefficient in complex work scenarios, have poor environmental adaptability, low level of intelligence, and are unable to learn and adapt to unstructured environments autonomously.
An autonomous dynamic obstacle avoidance method for robotic arms based on deep reinforcement learning is adopted. By initializing environmental parameters and an experience replay pool, selecting actions and updating reward function values, generating an experience replay pool, and training a deep learning network, the generality of the obstacle avoidance algorithm is improved.
It enables the robotic arm to autonomously avoid obstacles in dynamic environments, improving the adaptability and versatility of obstacle avoidance algorithms.
Smart Images
Figure CN116100539B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of automation and robotics, specifically to a method and system for autonomous dynamic obstacle avoidance of a robotic arm based on deep reinforcement learning. Background Technology
[0002] Trajectory planning is a fundamental problem in the motion control of robotic arms, and the results of trajectory planning can measure whether the robot has performed its task well. Currently, most trajectory planning methods for robotic arms are based on specific constraints such as velocity and acceleration in joints or Cartesian space, and calculate the trajectory through polynomial interpolation, or based on sensor information, such as using artificial potential field methods and sampling algorithms for trajectory planning.
[0003] However, the methods listed above have the following problems: 1. They are limited in the types of work scenarios they can handle. When faced with complex work scenarios, the planning efficiency is low, and they are prone to getting stuck in local minima or failing to reach optimal convergence; 2. They have poor environmental adaptability and can only perform trajectory planning for a single scenario; 3. They have a low level of intelligence, lack self-learning ability, and are limited to application in unstructured environments. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for autonomous dynamic obstacle avoidance of a robotic arm based on deep reinforcement learning, which can improve the versatility of obstacle avoidance algorithms.
[0005] To achieve the above objectives, embodiments of the present invention provide an autonomous dynamic obstacle avoidance method for a robotic arm based on deep reinforcement learning, comprising:
[0006] Initialize the current environment parameters and experience replay pool;
[0007] Select the action to be performed;
[0008] The selected action is sent to the server to update the robotic arm's status;
[0009] Update the reward function value based on the updated robotic arm status;
[0010] Store the state before the action is executed, the selected action, the state after the action is executed, and the reward value into the experience replay pool;
[0011] Determine whether the amount of data in the experience replay pool has reached the minimum number of samples;
[0012] If the amount of data in the experience replay pool reaches the minimum number of samples, the deep learning network is trained using the experience replay pool, and the obstacle avoidance accuracy of the deep learning network is determined.
[0013] If it is determined that the amount of data in the experience replay pool has not reached the minimum number of samples, return to the step of selecting the action to be performed.
[0014] Determine whether the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is greater than a preset value;
[0015] If the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is greater than a preset value, return to the step of selecting the action to be performed.
[0016] If the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is less than a preset value, the deep learning network is output.
[0017] Optionally, the action to be performed includes:
[0018] The selected action is represented by formulas (1) and (2).
[0019] (1)
[0020] in, It is a dynamic scaling factor. For the current training round, The maximum number of training rounds;
[0021] (2)
[0022] in, The action selected for the current state. For the random action to be explored in the current state, This is the vector representation of the guiding term.
[0023] Optionally, sending the selected action to the server to update the robotic arm's state includes:
[0024] The selected action is sent to the CoppeliaSim server via the CoppeliaSim and Python co-simulation framework.
[0025] The CoppeliaSim server decodes and executes the selected action to bring the robotic arm to the next state.
[0026] Optionally, the reward function value is updated based on the updated robotic arm state, including:
[0027] Update the reward function value according to formulas (3) to (7).
[0028] (3)
[0029] in, To attract field reward functions, The distance between the end effector of the robotic arm and the target object. The target object's safe distance threshold.
[0030] Optionally, the reward function value is updated based on the updated robotic arm state, including:
[0031] (4)
[0032] in, For the repulsive field reward function, The safe distance threshold for obstacles. This is the distance between the end effector of the robotic arm and the dynamic obstacle.
[0033] Optionally, the reward function value is updated based on the updated robotic arm state, including:
[0034] (5)
[0035] in, For energy reward function, Let be the second normal form quantity of the six joints of the robotic arm.
[0036] Optionally, the reward function value is updated based on the updated robotic arm state, including:
[0037] (6)
[0038] in, This is the time-based reward function.
[0039] Optionally, the reward function value is updated based on the updated robotic arm state, including:
[0040] (7)
[0041] in, The reward function value, , , , represents the weights of each reward function, and represents the hyperparameters.
[0042] On the other hand, the present invention also provides an autonomous dynamic obstacle avoidance system for robotic arms based on deep reinforcement learning, the autonomous dynamic obstacle avoidance system for robotic arms including a processor, the processor being used to execute the autonomous dynamic obstacle avoidance method for robotic arms as described above.
[0043] In another aspect, the present invention also provides a computer-readable storage medium storing instructions for being read by a machine to cause the machine to perform the autonomous dynamic obstacle avoidance method for a robotic arm as described above.
[0044] Through the above technical solution, the autonomous dynamic obstacle avoidance method and system for robotic arms based on deep reinforcement learning provided by this invention generates an experience replay pool by combining environmental parameters, traversing each action, and calculating the corresponding reward function value. Finally, by training the deep learning network using the experience replay pool, the goal of adapting to dynamic environments is achieved, improving the versatility of the obstacle avoidance algorithm.
[0045] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description
[0046] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:
[0047] Figure 1 This is a flowchart of an autonomous dynamic obstacle avoidance method for a robotic arm based on deep reinforcement learning according to an embodiment of the present invention.
[0048] Figure 2 This is a 3D rendering of a six-degree-of-freedom robotic arm and an obstacle scene according to one embodiment of the present invention. Detailed Implementation
[0049] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.
[0050] like Figure 1 The diagram shows a flowchart of an autonomous dynamic obstacle avoidance method for a robotic arm based on deep reinforcement learning, according to an embodiment of the present invention. Figure 1 In this context, the robotic arm's autonomous dynamic obstacle avoidance method may include:
[0051] In step S10, the current environmental parameters and experience replay pool are initialized;
[0052] In step S11, select the action that needs to be performed;
[0053] In step S12, the selected action is sent to the server to update the state of the robotic arm;
[0054] In step S13, the reward function value is updated based on the updated robotic arm state;
[0055] In step S14, the state before the action is executed, the selected action, the state after the action is executed, and the reward value are stored in the experience replay pool;
[0056] In step S15, it is determined whether the amount of data in the experience playback pool has reached the minimum number of samples;
[0057] In step S16, if it is determined that the amount of data in the experience replay pool has reached the minimum number of samples, the deep learning network is trained using the experience replay pool, and the obstacle avoidance accuracy of the deep learning network is determined.
[0058] If it is determined that the amount of data in the experience replay pool has not reached the minimum number of samples, return to the step of selecting the action to be performed.
[0059] In step S17, it is determined whether the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is greater than a preset value.
[0060] If the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is greater than a preset value, return to the step of selecting the action to be performed.
[0061] In step S18, if the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is less than a preset value, the deep learning network is output.
[0062] In such Figure 1 In the illustrated autonomous dynamic obstacle avoidance method for the robotic arm, step S10 can be used to initialize the current environmental parameters and the experience replay pool. The current environmental parameters can be expressed as:
[0063] .in, , , , , and It can represent the angle values of a six-degree-of-freedom robotic arm. Indicates the current position of the robotic arm. This indicates the current position of the six-degree-of-freedom robotic arm. (Location at time) This represents the distance to dynamic obstacles. The experience replay pool can be a training set and dataset used to train deep learning networks. A 3D rendering of the six-DOF robotic arm and obstacle scene can be, for example,... Figure 2 As shown.
[0064] Step S11 is used to select the action to be performed. Specifically, considering that the trajectory planning of a six-DOF robotic arm is a high-dimensional planning problem, the problem becomes even more complex when the goal of real-time avoidance of dynamic obstacles needs to be achieved. Therefore, a dynamic action selection strategy is needed to solve the inefficient action exploration problem. Specifically, this can be achieved by first obtaining the angle values of each joint of the robotic arm and the spatial position of the target, and then using the Jacobian matrix to calculate the numerical solution of the inverse kinematics of the robotic arm. As a guiding term, a suitable motion detection direction for the robotic arm is provided. Then, the guiding term is combined with randomly sampled motion terms containing noise from a deep reinforcement learning algorithm, and a dynamic scaling factor is used to... Integrate the selected actions. More specifically, in one example of the invention, step S11 may represent the selected actions according to formulas (1) and (2).
[0065] (1)
[0066] in, It is a dynamic scaling factor. For the current training round, The maximum number of training rounds;
[0067] (2)
[0068] in, The action selected for the current state. For the random action to be explored in the current state, This is the vector representation of the guiding term.
[0069] Step S12 can send the selected action to the server to update the state of the robotic arm. Specifically, step S12 can first send the selected action to the CoppeliaSim server through the CoppeliaSim and Python co-simulation framework, and then the CoppeliaSim server decodes and executes the selected action to bring the robotic arm to the next state.
[0070] Step S13 can be used to update the reward function value based on the updated state of the robotic arm. Specifically, step S13 can be updated according to formulas (3) to (7).
[0071] (3)
[0072] in, To attract field reward functions, The distance between the end effector of the robotic arm and the target object. The target object's safe distance threshold.
[0073] (4)
[0074] in, For the repulsive field reward function, The safe distance threshold for obstacles. This is the distance between the end effector of the robotic arm and the dynamic obstacle.
[0075] (5)
[0076] in, For energy reward function, Let be the second normal form quantity of the six joints of the robotic arm.
[0077] (6)
[0078] in, This is the time-based reward function.
[0079] Optionally, the reward function value is updated based on the updated robotic arm state, including:
[0080] (7)
[0081] in, For the reward function value, , , , represents the weights of each reward function, and represents the hyperparameters.
[0082] On the other hand, the present invention also provides an autonomous dynamic obstacle avoidance system for robotic arms based on deep reinforcement learning, the autonomous dynamic obstacle avoidance system for robotic arms including a processor, the processor being used to execute the autonomous dynamic obstacle avoidance method for robotic arms as described above.
[0083] In another aspect, the present invention also provides a computer-readable storage medium storing instructions for being read by a machine to cause the machine to perform the autonomous dynamic obstacle avoidance method for a robotic arm as described above.
[0084] Through the above technical solution, the autonomous dynamic obstacle avoidance method and system for robotic arms based on deep reinforcement learning provided by this invention generates an experience replay pool by combining environmental parameters, traversing each action, and calculating the corresponding reward function value. Finally, by training the deep learning network using the experience replay pool, the goal of adapting to dynamic environments is achieved, improving the versatility of the obstacle avoidance algorithm.
[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0086] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0089] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0090] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0091] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0092] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0093] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for autonomous dynamic obstacle avoidance of a robot arm based on deep reinforcement learning, characterized in that, The robotic arm's autonomous dynamic obstacle avoidance method includes: Initialize the current environment parameters and experience replay pool; Select the action to be performed; The selected action is sent to the server to update the robotic arm's status; Update the reward function value based on the updated robotic arm status; Store the state before the action is executed, the selected action, the state after the action is executed, and the reward value into the experience replay pool; Determine whether the amount of data in the experience replay pool has reached the minimum number of samples; If the amount of data in the experience replay pool reaches the minimum number of samples, the deep learning network is trained using the experience replay pool, and the obstacle avoidance accuracy of the deep learning network is determined. If it is determined that the amount of data in the experience replay pool has not reached the minimum number of samples, return to the step of selecting the action to be performed. Determine whether the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is greater than a preset value; If the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is greater than a preset value, return to the step of selecting the action to be performed. If the difference between the obstacle avoidance accuracy in the current iteration and the obstacle avoidance accuracy in the previous iteration is less than a preset value, the deep learning network is output.
2. The method of claim 1, wherein, The action to be performed includes: The selected action is represented by formulas (1) and (2). ，（1） in, It is a dynamic scaling factor. For the current training round, The maximum number of training rounds; ，（2） in, The action selected for the current state. For the random action to be explored in the current state, This is the vector representation of the guiding term.
3. The autonomous dynamic obstacle avoidance method for a robotic arm according to claim 1, characterized in that, Sending the selected action to the server to update the robotic arm's status includes: The selected action is sent to the server via the co-simulation framework; The server decodes and executes the selected action to bring the robotic arm to the next state.
4. The autonomous dynamic obstacle avoidance method for a robotic arm according to claim 1, characterized in that, Based on the updated robotic arm status, the updated reward function values include: Update the reward function value according to formulas (3) to (7). ，（3） in, To attract field reward functions, The distance between the end effector of the robotic arm and the target object. The target object's safe distance threshold.
5. The autonomous dynamic obstacle avoidance method for a robotic arm according to claim 4, characterized in that, Based on the updated robotic arm status, the updated reward function values include: ，（4） in, For the repulsive field reward function, The safe distance threshold for obstacles. This is the distance between the end effector of the robotic arm and the dynamic obstacle.
6. The autonomous dynamic obstacle avoidance method for a robotic arm according to claim 5, characterized in that, Based on the updated robotic arm status, the updated reward function values include: ，（5） in, For energy reward function, Let be the second normal form quantity of the six joints of the robotic arm.
7. The autonomous dynamic obstacle avoidance method for a robotic arm according to claim 6, characterized in that, Based on the updated robotic arm status, the updated reward function values include: ，（6） in, This is the time-based reward function.
8. The autonomous dynamic obstacle avoidance method for a robotic arm according to claim 7, characterized in that, Based on the updated robotic arm status, the updated reward function values include: ，（7） in, The reward function value, , , , represents the weights of each reward function, and represents the hyperparameters.
9. An autonomous dynamic obstacle avoidance system for a robotic arm based on deep reinforcement learning, characterized in that, The robotic arm autonomous dynamic obstacle avoidance system includes a processor, which is used to execute the robotic arm autonomous dynamic obstacle avoidance method as described in any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that are read by a machine to cause the machine to perform the autonomous dynamic obstacle avoidance method for a robotic arm as described in any one of claims 1 to 8.