A robotic arm autonomous obstacle avoidance method, device and medium based on deep reinforcement learning

Abstract

The present invention relates to a method, device and medium for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning. The method addresses the problem of obstacle avoidance of the robotic arm in the process of moving from an initial position to a target position through a reward function that is comprehensively designed in deep reinforcement learning. The method comprehensively considers multiple aspects such as obstacle avoidance at the end of the robotic arm, obstacle avoidance of the robotic arm body, avoiding self-collision, accurately reaching the target position and smoothness of the motion trajectory. The method has strong adaptability when the environmental map is uncertain or the obstacles change dynamically. The working area of ​​the robotic arm is divided into a safe area, a warning area and a dangerous area, so that the robotic arm can adopt a more stringent obstacle avoidance strategy in the dangerous area to avoid collisions with obstacles. Compared with the existing technology, the present invention has the advantages of high success rate, wide scope of application and strong reliability.

Description

Technical Field

[0001] The present invention relates to robotic arm obstacle avoidance technology, and in particular to a robotic arm autonomous obstacle avoidance method, equipment, and medium based on deep reinforcement learning. Background Art

[0002] With the rapid development of robotic arm technology, its application scenarios have expanded from traditional structured environments to dynamic and unpredictable unstructured environments. In unstructured environments, robotic arms face many challenges, such as the random motion of obstacles and targets. These factors place higher demands on motion planning algorithms. Motion planning for robotic arms in unstructured environments must meet two key requirements. First, the algorithm must adapt to the uncertainty of the environment map and the dynamic changes of obstacles. Second, the algorithm must be highly real-time to quickly adapt to immediate environmental changes. These requirements together form the core goal of the design of dynamic obstacle avoidance algorithms for robotic arms: to enable efficient and real-time operation of robotic arms in complex environments. Implementing dynamic obstacle avoidance for robotic arms in unstructured environments not only facilitates task completion but also plays a crucial role in ensuring safe human-robot interaction.

[0003] Traditional motion planning algorithms include sampling-based rapid randomized exploration trees (RRTs), probabilistic roadmap methods (PRMs), artificial potential fields (APFs), and numerical optimization methods such as model predictive control (MPC). These methods have certain limitations in real-time adaptive obstacle avoidance. In recent years, deep reinforcement learning (DRL) has utilized neural networks to handle tasks in high-dimensional, complex environments. It eliminates the need for precise environmental modeling and autonomously masters behavioral strategies through trial-and-error learning. Its application has rapidly expanded in robotics, such as drone flight, autonomous vehicle navigation, and robotic arm manipulation. Therefore, how to improve the autonomous obstacle avoidance capabilities of robotic arms in complex environments based on deep reinforcement learning is a technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method, equipment and medium for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning. By focusing on the obstacle avoidance problem of the end effector of the robotic arm, comprehensively considering multiple key aspects such as obstacle avoidance at the end of the robotic arm, obstacle avoidance of the robotic arm body, avoiding self-collision, accurately reaching the target position and smoothness of the motion trajectory, the method can achieve autonomous obstacle avoidance of the robotic arm that adapts to uncertain environments and is real-time.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to one aspect of the present invention, a method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning is provided, and the specific steps include:

[0007] S1. Collecting the status data of the robot arm and obstacles: Collecting the joint angles and joint velocities of the robot arm through the real-time data exchange interface; detecting the coding marks through the camera and collecting obstacle information; setting the target position of the robot arm;

[0008] S2. Process the collected data to obtain status information: Based on the current joint angles and joint velocities of the manipulator, the position and posture of the manipulator's end effector are calculated through the robot's forward kinematics. The image processing library is used to perform coded marker recognition to obtain the coordinates of obstacles on the image.

[0009] S3. Action planning and execution: The obtained state information of the robotic arm, obstacle and target position is input into the trained neural network model for action planning. The planned action obtained is the joint speed, which is sent to the robotic arm through the robotic arm speed control interface to actually execute the planned action.

[0010] In S3, after the robot arm executes the planned action, it is determined whether the current end position of the robot arm reaches the set target position. If so, the task is successful; otherwise, it enters the next control cycle and repeats S1, S2 and S3.

[0011] In S2, the coordinates in the camera coordinate system are obtained through the image processing library; the camera coordinate system is converted into 3D coordinates in the robot base coordinate system through hand-eye calibration technology; and further converted into the world coordinate system to obtain the 3D position of the obstacle in the world coordinate system.

[0012] The reward function of the neural network model in S3 includes the obstacle avoidance reward function r at the end of the robot tool t o , body obstacle avoidance reward function r t b , reaching the target reward function r t g , self-collision reward function r t f , motion smoothness reward function r t m , round end reward function r t c , the total reward function R t The expression is:

[0013] R t =r t o +r t b +r t g +r t f +r t m +rt c .

[0014] To avoid obstacles at the end of the robot arm tool, the workspace is divided into a safe area, a warning area, and a danger area according to the location of obstacles and the robot arm tool;

[0015] When the robot tool is in the dangerous area, the actual movement direction of the robot tool Encouraged to move closer to the desired direction of motion of the end-of-arm tool Dangerous area reward function r t d The expression is:

[0016]

[0017] Among them, the desired movement direction of the robot arm tool end The direction of the resultant force of the attraction of the target position on the robot arm tool and the repulsion of the obstacle on the robot arm tool;

[0018] When the end of the robot tool is in the safe area, the reward function r t s The expression is:

[0019]

[0020] Among them, P t g is the target position; P t e is the position of the robot tool; P t o is the obstacle position; d(P t g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position;

[0021] When the end of the robot tool is in the warning area, the reward function r t e According to the reward function design of the safe area and the dangerous area, the closer the end of the robot tool is to the dangerous area, the greater the weight assigned to the reward function of the dangerous area. t e The expression is:

[0022]

[0023] Where d1 is the boundary radius between the safety area and the warning area; d2 is the boundary radius between the warning area and the danger area;

[0024] The obstacle avoidance reward function r of the robot tool end t o The expression is:

[0025]

[0026] To prevent collisions with the robot, 11 key points were selected on the robot, including six joints, the center of the tool end, and four key points on the long connecting rods. A spherical envelope was used to cover the key points, and a first safety distance threshold was set. When the distance from the key point to the obstacle was less than the first safety distance threshold, a reward was given to maintain a larger distance; otherwise, no reward was given.

[0027] The body obstacle avoidance reward function r t b The expression is:

[0028]

[0029] Wherein, d3 is the first safety distance threshold; P t o is the obstacle position; P i,t c is the position of the i-th key point; N is the number of key points of the robotic arm.

[0030] Set the reward function so that the end of the tool faces vertically downward during the movement of the robot arm, decelerates to a stop when approaching the target position, and reaches the target position in the shortest time. The reward function r is used to reach the target position. t g , the expression is:

[0031] r t g =r t z +r t v +r t p ,

[0032] Among them, set the reward function r t z , so that during the movement of the robot arm, the positive direction of the z-axis of the tool end coordinate system is encouraged to be close to the negative direction of the z-axis of the world coordinate system. The origin of the tool end coordinate system is located at the center point of the tool gripper finger, and the positive direction of the z-axis is the direction of the tool. t z The expression is:

[0033]

[0034] is the positive direction vector of the z-axis of the robot tool end coordinate system; is the unit vector in the negative direction of the z-axis of the world coordinate system;

[0035] Set the reward function r t v , so that when the robot arm approaches the target position, the joint speed slows down until it stops. The expression is:

[0036]

[0037] d4 is the second safety distance threshold; d(P t g , P t e ) is the distance from the target position to the robot tool position; u t is the joint velocity;

[0038] Set the reward function r t p , so that the robot arm reaches the target position in the shortest time, the expression is:

[0039]

[0040] t real is the time step actually used to reach the target, t total is the maximum time step allowed for an episode.

[0041] The reward function also includes a self-collision reward function r t f , motion smoothness reward function r t m and the end-of-epoch reward function r t c , where the self-collision reward function r t f The expression is:

[0042]

[0043] Among them, d(P i,t c , P j,t c ) is the distance between two points selected from the key points; N is the number of key points;

[0044] The motion smoothness reward function r t m , so that the difference in joint velocity of continuous moment planning is reduced, the expression is:

[0045]

[0046] Among them, u t is the current joint velocity at time t; u t-1 is the joint velocity at the previous moment;

[0047] The end-of-epoch reward function r t c In order to give positive rewards for successfully completing the task at the end, penalties are imposed on other termination conditions, including collision, exceeding the workspace range, self-collision, and exceeding the maximum round time. The expression is:

[0048]

[0049] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method when executing the program.

[0050] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) Improve the success rate of the robot arm in completing tasks: By setting a multi-objective reward function, we focus on the obstacle avoidance problem of the end tool of the robot arm, and comprehensively consider multiple aspects such as obstacle avoidance at the end of the robot arm, obstacle avoidance of the robot arm itself, avoiding self-collision, accurately reaching the target position, and the smoothness of the motion trajectory, so that the robot arm can complete the task more safely and stably during the obstacle avoidance process, thereby improving the success rate.

[0053] (2) Strong adaptability and wide range of applications: This robotic arm autonomous obstacle avoidance method is not only suitable for scenarios with simple obstacle movement patterns, but also can cope with environments with more complex and random obstacle movement patterns. When the environmental map is uncertain or the obstacles change dynamically, it has strong adaptability and can be applied to various occasions that require the robotic arm to perform real-time dynamic obstacle avoidance.

[0054] (3) Improve the safety and reliability of the robotic arm: By dividing the safe area, warning area and danger area, the robotic arm can adopt a more stringent obstacle avoidance strategy in the danger area to avoid collision with obstacles; in addition, the application of key point selection and spherical envelope ideas further enhances the obstacle avoidance capability of the robotic arm itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a framework diagram of the robotic arm's autonomous obstacle avoidance method based on deep reinforcement learning;

[0056] Figure 2 This is a schematic diagram of the area division of the robotic arm working scene;

[0057] Figure 3 Schematic diagram of the reward function designed for obstacle avoidance in the dangerous area of ​​the robot arm tool end;

[0058] Figure 4 is the position diagram of the key points of the robotic arm;

[0059] Figure 5 Schematic diagram of the simulation environment of the robotic arm;

[0060] Figure 6 is the average reward curve for training;

[0061] Figure 7 is the average success rate curve of training;

[0062] Figure 8 It is a curve diagram of the difference between the end of the tool and the target position when the obstacle moves along the x-axis;

[0063] Figure 9 This is the velocity curve of the robot arm joint when the obstacle moves along the x-axis;

[0064] Figure 10 This is the motion trajectory of the robot arm tool end and the obstacle when the obstacle moves along the x-axis;

[0065] Figure 11 It is a curve diagram of the difference between the end of the tool and the target position when the obstacle moves along the y-axis;

[0066] Figure 12 This is the velocity curve of the robot arm joint when the obstacle moves along the y-axis;

[0067] Figure 13 This is the motion trajectory of the robot arm tool end and the obstacle when the obstacle moves along the y-axis. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0069] like Figure 1 The figure shows the framework of the autonomous obstacle avoidance method for a robotic arm based on deep reinforcement learning. The specific steps include:

[0070] S1. Collecting the status data of the robot arm and obstacles: Collecting the joint angles and joint velocities of the robot arm through the real-time data exchange interface; detecting the coding marks through the camera and collecting obstacle information; setting the target position of the robot arm;

[0071] S2. Process the collected data to obtain status information: Based on the current joint angles and joint velocities of the manipulator, the position and posture of the manipulator's end effector are calculated through the robot's forward kinematics. The image processing library is used to perform coded marker recognition to obtain the coordinates of obstacles on the image.

[0072] S3, motion planning and execution: The obtained state information of the robotic arm, obstacles and target positions is input into the trained neural network model for motion planning. The planned action is obtained as the joint speed, which is sent to the robotic arm through the robotic arm speed control interface to actually execute the planned action.

[0073] In S3, after the robotic arm executes the planned action, it determines whether the current robotic arm end position has reached the set target position. If so, the mission is successful; otherwise, it enters the next control cycle and repeats S1, S2, and S3. In S2, the coordinates in the camera coordinate system are obtained through the image processing library; the camera coordinate system is converted to 3D coordinates in the robotic arm base coordinate system through hand-eye calibration technology; and then further converted to the world coordinate system to obtain the 3D position of the obstacle in the world coordinate system.

[0074] The reward functions of the neural network model in S3 include the obstacle avoidance reward function r at the end of the robot tool t o , body obstacle avoidance reward function r t b , reaching the target reward function r t g , self-collision reward function r t f , motion smoothness reward function r t m , round end reward function r t c , the total reward function R t The expression is:

[0075] R t =r t o +r t b +r t g +r t f +r t m +r t c .

[0076] like Figure 2 As shown, the robot arm tool end avoids obstacles, and the workspace is divided into a safe area, a warning area, and a dangerous area according to the position of the obstacle and the robot arm tool.

[0077] When the robot tool is in a dangerous area, the distance between the robot tool and the obstacle is relatively close, and the obstacle avoidance requirements are strict. Figure 3 As shown, the attraction of the target position to the robot tool is set to The repulsive force of the obstacle on the robot tool is Combined Force The direction of attraction is the desired movement direction of the end of the robot tool. and repulsive force The expression is:

[0078]

[0079] Among them, P t g is the target position; P t e is the position of the robot tool; P t o is the obstacle position; d(P t g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position. The actual movement direction of the robot tool Encouraged to move closer to the desired direction of motion of the end-of-arm tool Dangerous area reward function r t d The expression is:

[0080]

[0081] When the end of the robot tool is in the safe area, the reward function r t s The expression is:

[0082]

[0083] Among them, P t g is the target position; P t e is the position of the robot tool; P t o is the obstacle position; d(Pt g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position.

[0084] When the end of the robot tool is in the warning area, the reward function r t e According to the reward function design of the safe area and the dangerous area, the closer the end of the robot tool is to the dangerous area, the greater the weight assigned to the reward function of the dangerous area. t e The expression is:

[0085]

[0086] Where d1 is the boundary radius between the safety area and the warning area; d2 is the boundary radius between the warning area and the danger area.

[0087] Therefore, the obstacle avoidance reward function r t o The expression is:

[0088]

[0089] To avoid collision between the robot arm and the body, Figure 4 As shown in the figure, 11 key points are selected on the robot arm, including 6 joint points, the center point of the tool end, and 4 key points on the long connecting rod. The key points are covered by the sphere envelope and the first safety distance threshold is set. When the distance from the key point to the obstacle position is less than the first safety distance threshold, a reward is given to maintain a larger distance, otherwise no reward is given. The body obstacle avoidance reward function r t b The expression is:

[0090]

[0091] Where d3 is the first safety distance threshold; P t o is the obstacle position; P i,t c is the position of the i-th key point; N is the number of key points of the robotic arm.

[0092] Set the reward function so that the end of the tool faces vertically downward during the movement of the robot arm, decelerates to a stop when approaching the target position, and reaches the target position in the shortest time. The reward function r is used to reach the target position. tg , the expression is:

[0093] r t g =r t s +r t v +r t p ,

[0094] Among them, set the reward function r t z , so that during the movement of the robot arm, the positive direction of the z-axis of the tool end coordinate system is encouraged to be close to the negative direction of the z-axis of the world coordinate system. The origin of the tool end coordinate system is located at the center point of the tool gripper finger, and the positive direction of the z-axis is the direction of the tool. t z The expression is:

[0095]

[0096] is the positive direction vector of the z-axis of the robot tool end coordinate system; is the unit vector in the negative direction of the z-axis of the world coordinate system;

[0097] Set the reward function r t v , so that when the robot arm approaches the target position, the joint speed slows down until it stops. The expression is:

[0098]

[0099] d4 is the second safety distance threshold; d(P t g , P t e ) is the distance from the target position to the robot tool position; u t is the joint velocity.

[0100] Set the reward function r t p , so that the robot arm reaches the target position in the shortest time, the expression is:

[0101]

[0102] t real is the time step actually used to reach the target, t total is the maximum time step allowed for an episode.

[0103] Self-collision reward function r t f The expression is:

[0104]

[0105] Among them, d(P i,t c , P j,t c ) is the distance between two points selected from the key points; N is the number of key points;

[0106] Motion smoothness reward function r t m , so that the difference in joint velocity of continuous moment planning is reduced, the expression is:

[0107]

[0108] Among them, u t is the current joint velocity at time t; u t-1 is the joint velocity at the previous moment.

[0109] End-of-epoch reward function r t c In order to give positive rewards for successfully completing the task at the end, penalties are imposed on other termination conditions, including collision, exceeding the workspace range, self-collision, and exceeding the maximum round time. The expression is:

[0110]

[0111] like Figure 5 As shown in the figure, a simulation environment for the robot arm is built. In the simulation training, the environment is reset at the beginning of the round, and the starting point of the tool end is in area A. I Randomly generated, the initial position of the obstacle is in area A O Randomly generated, the simulator's collision detection mechanism ensures that no collision occurs, and the target point is in the target area A at the end of the tool G Random generation. The speed and direction of the obstacle are randomly set on the x-axis and y-axis, and are randomly selected at the beginning of each round. The speed of the obstacle has two modes: fixed speed and variable speed mode. In variable speed mode, the speed changes randomly at each sampling interval. When the obstacle reaches the boundary, it moves in the opposite direction until the end of the round. This example trains for a total of 10 million time steps and 111,341 rounds. The average reward of the training is as follows: Figure 6 As shown, the average success rate is Figure 7 shown.

[0112] In this embodiment, the obstacle is set as a cube with a main diagonal of 0.3 meters, which is enclosed by a sphere with a radius of 0.15 meters. The robot arm obtains real-time information through a real-time data exchange interface, and the external application and the robot arm controller are synchronized. In the experimental test scenario of this embodiment, the obstacle moves along the x-axis at a speed of 0.18m / s, or along the y-axis at a speed of 0.18m / s. The initial positions of the robot arm tool starting point, target point and obstacle are set to different fixed values ​​according to different situations. The test results are as follows Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 、 Figure 11 、 Figure 12 and Figure 13 As shown in the figure, no matter whether the obstacle moves along the x-axis or the y-axis, the robot arm can effectively avoid dynamic obstacles and successfully guide the robot arm to the predetermined target position. Even under the influence of dynamic obstacles, the error between the tool end and the target position can eventually converge to zero stably, which fully proves that the robot arm can accurately reach the target position. In addition, Figure 8 and Figure 12 As shown in the figure, the change curve of each joint speed maintains good smoothness during the entire movement process, and the speed drops to zero when the robot arm reaches the target position. That is, the robot arm can not only successfully reach the target position, but also stop at the target point accurately without exceeding the predetermined target range.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0115] Multiple components in the device are connected to the I / O interface, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as magnetic disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method of the present invention by any other suitable means (e.g., by means of firmware).

[0116] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A robotic arm autonomous obstacle avoidance method based on deep reinforcement learning, characterized in that: The specific steps include: S1. Collecting the status data of the robot arm and obstacles: Collecting the joint angles and joint velocities of the robot arm through the real-time data exchange interface; detecting the coding marks through the camera and collecting obstacle information; setting the target position of the robot arm; S2. Process the collected data to obtain status information: Based on the current joint angles and joint velocities of the manipulator, the position and posture of the manipulator's end effector are calculated through the robot's forward kinematics. The image processing library is used to perform coded marker recognition to obtain the coordinates of obstacles on the image. S3. Action planning and execution: The obtained state information of the robotic arm, obstacle and target position is input into the trained neural network model for action planning. The planned action obtained is the joint speed, which is sent to the robotic arm through the robotic arm speed control interface to actually execute the planned action.

2. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: In S3, after the robot arm executes the planned action, it is determined whether the current end position of the robot arm reaches the set target position. If so, the task is successful; otherwise, it enters the next control cycle and repeats S1, S2 and S3.

3. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: In S2, the coordinates in the camera coordinate system are obtained through the image processing library; the camera coordinate system is converted into 3D coordinates in the robot base coordinate system through hand-eye calibration technology; and further converted into the world coordinate system to obtain the 3D position of the obstacle in the world coordinate system.

4. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: The reward function of the neural network model in S3 includes the obstacle avoidance reward function r at the end of the robot tool t o , body obstacle avoidance reward function r t b , reaching the target reward function r t g , self-collision reward function r t f , motion smoothness reward function r t m , round end reward function r t c , the total reward function R t The expression is: R t ＝r t o +r t b +r t g +r t f +r t m +r t c 。 5. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: To avoid obstacles at the end of the robot arm tool, the workspace is divided into a safe area, a warning area, and a danger area according to the location of obstacles and the robot arm tool; When the robot tool is in the dangerous area, the actual movement direction of the robot tool Encouraged to move closer to the desired direction of motion of the end-of-arm tool Dangerous area reward function r t d The expression is: Among them, the desired movement direction of the robot arm tool end The direction of the resultant force of the attraction of the target position on the robot arm tool and the repulsion of the obstacle on the robot arm tool; When the end of the robot tool is in the safe area, the reward function r t s The expression is: Among them, P t g is the target position; P t e is the position of the robot tool; P t o is the obstacle position; d(P t g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position; When the end of the robot tool is in the warning area, the reward function r t e According to the reward function design of the safe area and the dangerous area, the closer the end of the robot tool is to the dangerous area, the greater the weight assigned to the reward function of the dangerous area. t e The expression is: Where d1 is the boundary radius between the safety area and the warning area; d2 is the boundary radius between the warning area and the danger area; The obstacle avoidance reward function r of the robot tool end t o The expression is:

6. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: To prevent collisions with the robot, 11 key points were selected on the robot, including six joints, the center of the tool end, and four key points on the long connecting rods. A spherical envelope was used to cover the key points, and a first safety distance threshold was set. When the distance from the key point to the obstacle was less than the first safety distance threshold, a reward was given to maintain a larger distance; otherwise, no reward was given. The body obstacle avoidance reward function r t b The expression is: Wherein, d3 is the first safety distance threshold; P t o is the obstacle position; P i,t c is the position of the i-th key point; N is the number of key points of the robotic arm.

7. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: Set the reward function so that the end of the tool faces vertically downward during the movement of the robot arm, decelerates to a stop when approaching the target position, and reaches the target position in the shortest time. The reward function r is used to reach the target position. t g , the expression is: r t g ＝r t z +r t v +r t p ， Among them, set the reward function r t z , so that during the movement of the robot arm, the positive direction of the z-axis of the tool end coordinate system is encouraged to be close to the negative direction of the z-axis of the world coordinate system. The origin of the tool end coordinate system is located at the center point of the tool gripper finger, and the positive direction of the z-axis is the direction of the tool. t z The expression is: is the positive direction vector of the z-axis of the robot tool end coordinate system; is the unit vector in the negative direction of the z-axis of the world coordinate system; Set the reward function r t v , so that when the robot arm approaches the target position, the joint speed slows down until it stops. The expression is: d4 is the second safety distance threshold; d(P t g , P t e ) is the distance from the target position to the robot tool position; u t is the joint velocity; Set the reward function r t p , so that the robot arm reaches the target position in the shortest time, the expression is: t real is the time step actually used to reach the target, t total is the maximum time step allowed for an episode.

8. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: The reward function also includes a self-collision reward function r t f , motion smoothness reward function r t m and the end-of-epoch reward function r t c , where the self-collision reward function r t f The expression is: Among them, d(P i,t c , P j,t c ) is the distance between two points selected from the key points; N is the number of key points; The motion smoothness reward function r t m , so that the difference in joint velocity of continuous moment planning is reduced, the expression is: Among them, u t is the current joint velocity at time t; u t-1 is the joint velocity at the previous moment; The end-of-epoch reward function r t c In order to give positive rewards for successfully completing the task at the end, penalties are imposed on other termination conditions, including collision, exceeding the workspace range, self-collision, and exceeding the maximum round time. The expression is:

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images