Mechanical arm autonomous obstacle avoidance method and device based on deep reinforcement learning and medium

Abstract

The invention relates to a mechanical arm autonomous obstacle avoidance method and device based on deep reinforcement learning and a medium, and solves the obstacle avoidance problem of a mechanical arm in the process from an initial position to a target position through a reward function comprehensively designed in the deep reinforcement learning. Multiple aspects of obstacle avoidance of the tail end of the mechanical arm, obstacle avoidance of the mechanical arm body, self-collision avoidance, accurate reaching of a target position, smoothness of a movement track and the like are comprehensively considered. When an environment map is uncertain or obstacles dynamically change, the method has high adaptability. The working area of the mechanical arm is divided into the safety area, the warning area and the dangerous area, so that the mechanical arm can adopt a stricter obstacle avoidance strategy when the mechanical arm is in the dangerous area, and collision between the mechanical arm and the obstacle is avoided. Compared with the prior art, the method has the advantages of high success rate, wide application range, high reliability and the like.

Description

Technical Field

[0001] The present invention relates to a robot arm obstacle avoidance technology, and in particular to a robot arm autonomous obstacle avoidance method, equipment and medium based on deep reinforcement learning. Background Art

[0002] With the rapid development of robotic arm technology, its application scenarios have expanded from traditional structured environments to dynamic and unpredictable unstructured environments. In unstructured environments, robotic arms face many challenges, such as the random movement of obstacles and targets, which place higher requirements on motion planning algorithms. In unstructured environments, the motion planning of robotic arms needs to meet two key requirements. First, the algorithm must adapt to the uncertainty of the environment map and the dynamic changes of obstacles. Second, the algorithm must have a high degree of real-time performance so that it can quickly adapt to the immediate changes in the environment. These requirements together constitute the core goal of the design of the dynamic obstacle avoidance algorithm for robotic arms, that is, to achieve efficient and real-time operation of robotic arms in complex environments. Realizing dynamic obstacle avoidance of robotic arms in unstructured environments makes the completion of tasks smoother, and is also of great significance to ensuring the safety of human-computer interaction.

[0003] Traditional motion planning algorithms include sampling-based rapid random exploration trees (RRT), probabilistic roadmap methods (PRM), artificial potential field methods (APF), and numerical optimization methods such as model predictive control (MPC). These methods have certain limitations in real-time adaptive obstacle avoidance. In recent years, deep reinforcement learning (DRL) uses neural networks to handle high-dimensional complex environment tasks without precise environmental modeling. It learns to master behavioral strategies autonomously through trial and error, and has rapidly expanded its application in the field of robotics such as drone flight, autonomous vehicle navigation, and robotic arm control. Therefore, how to improve the autonomous obstacle avoidance capability of robotic arms in complex environments based on deep reinforcement learning is a technical problem that needs to be solved. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method, device and medium for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning. By focusing on the obstacle avoidance problem of the end effector of the robotic arm, it comprehensively considers multiple key aspects such as obstacle avoidance at the end of the robotic arm, obstacle avoidance of the robotic arm body, avoiding self-collision, accurately reaching the target position and smoothness of the motion trajectory, so as to achieve autonomous obstacle avoidance of the robotic arm that can adapt to uncertain environments and in real time.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to one aspect of the present invention, a method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning is provided, and the specific steps include:

[0007] S1. Collect the status data of the robot arm and obstacles: collect the joint angles and joint speeds of the robot arm through the real-time data exchange interface; detect the coding mark through the camera and collect the obstacle information; set the target position of the robot arm;

[0008] S2. Process the collected data to obtain status information: Based on the current joint angle and joint speed of the robot arm, the position and posture of the end effector of the robot arm are obtained through the forward kinematics calculation of the robot; the image processing library is used to perform coded mark recognition to obtain the coordinates of the obstacles on the image;

[0009] S3, action planning and execution: The obtained state information of the robotic arm, obstacle and target position is input into the trained neural network model for action planning. The obtained planned action is the joint speed, which is sent to the robotic arm through the robotic arm speed control interface to actually execute the planned action.

[0010] In S3, after the robot arm executes the planned action, it is determined whether the current end position of the robot arm reaches the set target position. If so, the task is successful; otherwise, the next control cycle is entered, and S1, S2 and S3 are repeated.

[0011] In S2, the coordinates in the camera coordinate system are obtained through the image processing library; the camera coordinate system is converted into 3D coordinates in the robot base coordinate system through the hand-eye calibration technology; and further converted into the world coordinate system to obtain the 3D position of the obstacle in the world coordinate system.

[0012] The reward function of the neural network model in S3 includes the obstacle avoidance reward function r at the end of the robot tool t o , body obstacle avoidance reward function r t b , reaching the target reward function r t g , self-collision reward function r t f , motion smoothness reward function r t m , round termination reward function r t c , the total reward function R t The expression is:

[0013] R t =r t o +r t b +r t g +r t f +r t m +rt c .

[0014] To avoid obstacles at the end of the robot tool, the workspace is divided into a safe area, a warning area, and a danger area according to the position of the obstacle and the robot tool;

[0015] When the robot tool is in the dangerous area, the actual movement direction of the robot tool Encouraged to move closer to the desired direction of motion of the end-of-arm tool Dangerous area reward function r t d The expression is:

[0016]

[0017] Among them, the desired movement direction of the robot arm tool end is the direction of the combined force of the attraction of the set target position on the robot arm tool and the repulsion of the obstacle on the robot arm tool;

[0018] When the end of the robot tool is in the safe area, the reward function r t s The expression is:

[0019]

[0020] Among them, P t g is the target position; P t e is the tool position of the robot arm; P t o is the obstacle position; d(P t g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position;

[0021] When the end of the robot tool is in the warning area, the reward function r t e According to the reward function design of the safe area and the dangerous area, the closer the end of the robot tool is to the dangerous area, the greater the weight assigned to the reward function of the dangerous area. t e The expression is:

[0022]

[0023] Among them, d1 is the boundary radius between the safety area and the warning area; d2 is the boundary radius between the warning area and the danger area;

[0024] The robot tool end obstacle avoidance reward function r t o The expression is:

[0025]

[0026] In order to avoid collisions with the robot body, 11 key points are selected on the robot, including 6 joint points, the center point of the tool end and 4 key points on the long connecting rods; the key points are covered by a spherical envelope, and the first safety distance threshold is set; when the distance from the key point position to the obstacle position is less than the first safety distance threshold, a reward is given to maintain a larger distance, otherwise there is no reward;

[0027] The body obstacle avoidance reward function r t b The expression is:

[0028]

[0029] Where d3 is the first safety distance threshold; P t o is the obstacle position; P i,t c is the position of the i-th key point; N is the number of key points of the robot arm.

[0030] Set the reward function so that the end of the tool faces vertically downward during the movement of the robot arm, decelerates to stop when approaching the target position, and reaches the target position in the shortest time. The reward function r t g , the expression is:

[0031] r t g =r t z +r t v +r t p ,

[0032] Among them, set the reward function r t z , so that during the movement of the robot arm, the positive direction of the z-axis of the tool end coordinate system is encouraged to be close to the negative direction of the z-axis of the world coordinate system. The origin of the tool end coordinate system is located at the center point of the tool gripper finger, and the positive direction of the z-axis is the direction of the tool. t z The expression is:

[0033]

[0034] is the positive direction vector of the z-axis of the robot tool end coordinate system; is the unit vector in the negative direction of the z-axis of the world coordinate system;

[0035] Set the reward function r t v , so that when the robot arm approaches the target position, the joint speed slows down until it stops. The expression is:

[0036]

[0037] d4 is the second safety distance threshold; d(P t g , P t e ) is the distance from the target position to the robot tool position; u t is the joint velocity;

[0038] Set the reward function r t p , so that the robot arm reaches the target position in the shortest time, the expression is:

[0039]

[0040] t real is the time step actually used to reach the target, t total is the maximum time step allowed for an episode.

[0041] The reward function also includes a self-collision reward function r t f , motion smoothness reward function r t m and the round-end reward function r t c , where the self-collision reward function r t f The expression is:

[0042]

[0043] Among them, d(P i,t c , P j,t c ) is the distance between two points selected from the key points; N is the number of key points;

[0044] The motion smoothness reward function r t m , so that the difference in joint speeds planned at consecutive moments is reduced, and the expression is:

[0045]

[0046] Among them, u t is the current joint velocity at time t; u t-1 is the joint velocity at the previous moment;

[0047] The round end reward function r t c In order to give positive rewards for successfully completing the task at the end, other termination situations, including collision, exceeding the workspace range, self-collision, and exceeding the maximum round time, are penalized. The expression is:

[0048]

[0049] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0050] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described above when executed by a processor.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) Improve the success rate of the robot arm in completing tasks: By setting a multi-objective reward function, we focus on the obstacle avoidance problem of the end tool of the robot arm, and comprehensively consider multiple aspects such as obstacle avoidance at the end of the robot arm, obstacle avoidance of the robot body, avoiding self-collision, accurately reaching the target position, and the smoothness of the motion trajectory. This allows the robot arm to complete tasks more safely and stably during the obstacle avoidance process, thereby improving the success rate.

[0053] (2) Strong adaptability and wide range of applications: This robotic arm autonomous obstacle avoidance method is not only suitable for scenarios with simple obstacle movement patterns, but can also cope with environments with more complex and random obstacle movement patterns. When the environmental map is uncertain or the obstacles change dynamically, it has strong adaptability and can be applied to various occasions that require the robotic arm to perform real-time dynamic obstacle avoidance.

[0054] (3) Improving the safety and reliability of the robotic arm: By dividing the robot into safe areas, warning areas, and dangerous areas, the robot arm can adopt more stringent obstacle avoidance strategies in dangerous areas to avoid collisions with obstacles. In addition, the application of key point selection and spherical enveloping ideas further enhances the obstacle avoidance capability of the robot arm itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a framework diagram of the autonomous obstacle avoidance method for a robotic arm based on deep reinforcement learning;

[0056] Figure 2 This is a schematic diagram of the area division of the robot arm working scene;

[0057] Figure 3 Schematic diagram of the reward function designed for obstacle avoidance in the dangerous area at the end of the robot tool;

[0058] Figure 4 is the position diagram of the key points of the robot arm;

[0059] Figure 5 This is a schematic diagram of the simulation environment of the robotic arm;

[0060] Figure 6 This is the average reward curve for training;

[0061] Figure 7 It is the average success rate curve of training;

[0062] Figure 8 It is a curve diagram of the difference between the end of the robot tool and the target position when the obstacle moves along the x-axis;

[0063] Fig. 9 It is the velocity curve of the robot joint when the obstacle moves along the x-axis;

[0064] Fig.10 It is the motion trajectory diagram of the robot tool end and the obstacle when the obstacle moves along the x-axis;

[0065] Fig.11 It is a curve diagram of the difference between the end of the robot tool and the target position when the obstacle moves along the y-axis;

[0066] Fig.12 It is the velocity curve of the robot joint when the obstacle moves along the y-axis;

[0067] Fig.13 This is the motion trajectory diagram of the robot tool end and the obstacle when the obstacle moves along the y-axis. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0069] like Figure 1 As shown in the figure, it is a framework diagram of the autonomous obstacle avoidance method of the robotic arm based on deep reinforcement learning, and its specific steps include:

[0070] S1. Collect the status data of the robot arm and obstacles: collect the joint angles and joint speeds of the robot arm through the real-time data exchange interface; detect the coding mark through the camera and collect the obstacle information; set the target position of the robot arm;

[0071] S2. Process the collected data to obtain status information: Based on the current joint angle and joint speed of the robot arm, the position and posture of the end effector of the robot arm are obtained through the forward kinematics calculation of the robot; the image processing library is used to perform coded mark recognition to obtain the coordinates of the obstacles on the image;

[0072] S3, action planning and execution: The obtained state information of the robotic arm, obstacles and target positions is input into the trained neural network model for action planning. The obtained planned action is the joint speed, which is sent to the robotic arm through the robotic arm speed control interface to actually execute the planned action.

[0073] In S3, after the robot arm executes the planned action, it determines whether the current robot arm end position reaches the set target position. If so, the task is successful; otherwise, it enters the next control cycle and repeats S1, S2 and S3. In S2, the coordinates in the camera coordinate system are obtained through the image processing library; the camera coordinate system is converted into 3D coordinates in the robot arm base coordinate system through hand-eye calibration technology; and further converted to the world coordinate system to obtain the 3D position of the obstacle in the world coordinate system.

[0074] The reward functions of the neural network model in S3 include the obstacle avoidance reward function r at the end of the robot tool t o , body obstacle avoidance reward function r t b , reaching the target reward function r t g , self-collision reward function r t f , motion smoothness reward function r t m , round termination reward function r t c , the total reward function R t The expression is:

[0075] R t =r t o +r t b +r t g +r t f +r t m +r t c .

[0076] like Figure 2 As shown, the robot arm tool end obstacle avoidance is performed, and the workspace is divided into a safe area, a warning area, and a dangerous area according to the position of the obstacle and the robot arm tool.

[0077] When the robot tool is in a dangerous area, the distance between the robot tool and the obstacle is relatively close, and the obstacle avoidance requirement is strict. Figure 3 As shown, the attraction of the target position to the robot tool is set to The repulsive force of the obstacle on the robot tool is Join forces The direction of attraction is the desired movement direction of the robot tool end. and repulsive The expression is:

[0078]

[0079] Among them, P t g is the target position; P t e is the tool position of the robot arm; P t o is the obstacle position; d(P t g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position. The actual movement direction of the robot tool Encouraged to move closer to the desired direction of motion of the end-of-arm tool Dangerous area reward function r t d The expression is:

[0080]

[0081] When the end of the robot tool is in the safe area, the reward function r t s The expression is:

[0082]

[0083] Among them, P t g is the target position; P t e is the tool position of the robot arm; P t o is the obstacle position; d(Pt g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position.

[0084] When the end of the robot tool is in the warning area, the reward function r t e According to the reward function design of the safe area and the dangerous area, the closer the end of the robot tool is to the dangerous area, the greater the weight assigned to the reward function of the dangerous area. t e The expression is:

[0085]

[0086] Wherein, d1 is the boundary radius between the safety area and the warning area; d2 is the boundary radius between the warning area and the danger area.

[0087] Therefore, the obstacle avoidance reward function r t o The expression is:

[0088]

[0089] To avoid collision between the robot arm and the body, Figure 4 As shown in the figure, 11 key points are selected on the robot arm, including 6 joint points, the center point of the tool end and 4 key points on the long connecting rods. The key points are covered by the sphere envelope and the first safety distance threshold is set; when the distance from the key point position to the obstacle position is less than the first safety distance threshold, a reward is given to maintain a larger distance, otherwise there is no reward. Body obstacle avoidance reward function r t b The expression is:

[0090]

[0091] Where d3 is the first safety distance threshold; P t o is the obstacle position; P i,t c is the position of the i-th key point; N is the number of key points of the robot arm.

[0092] Set the reward function so that the end of the tool faces vertically downward during the movement of the robot arm, decelerates to stop when approaching the target position, and reaches the target position in the shortest time. The reward function r tg , the expression is:

[0093] r t g =r t s +r t v +r t p ,

[0094] Among them, set the reward function r t z , so that during the movement of the robot arm, the positive direction of the z-axis of the tool end coordinate system is encouraged to be close to the negative direction of the z-axis of the world coordinate system. The origin of the tool end coordinate system is located at the center point of the tool gripper finger, and the positive direction of the z-axis is the direction of the tool. t z The expression is:

[0095]

[0096] is the positive direction vector of the z-axis of the robot tool end coordinate system; is the unit vector in the negative direction of the z-axis of the world coordinate system;

[0097] Set the reward function r t v , so that when the robot arm approaches the target position, the joint speed slows down until it stops. The expression is:

[0098]

[0099] d4 is the second safety distance threshold; d(P t g , P t e ) is the distance from the target position to the robot tool position; u t is the joint velocity.

[0100] Set the reward function r t p , so that the robot arm reaches the target position in the shortest time, the expression is:

[0101]

[0102] t real is the time step actually used to reach the target, t total is the maximum time step allowed for an episode.

[0103] Self-collision reward function r t f The expression is:

[0104]

[0105] Among them, d(P i,t c , P j,t c ) is the distance between two points selected from the key points; N is the number of key points;

[0106] Motion smoothness reward function r t m , so that the difference in joint speeds planned at consecutive moments is reduced, and the expression is:

[0107]

[0108] Among them, u t is the current joint velocity at time t; u t-1 is the joint velocity at the previous moment.

[0109] Episode end reward function r t c In order to give positive rewards for successfully completing the task at the end, other termination situations, including collision, exceeding the workspace range, self-collision, and exceeding the maximum round time, are penalized. The expression is:

[0110]

[0111] like Figure 5 As shown in the figure, a simulation environment for the robot arm is built. In the simulation training, the environment is reset at the beginning of the round, and the starting point of the tool end is in area A. I Randomly generated, the initial position of the obstacle is in area A O Randomly generated, the collision detection mechanism of the simulator ensures that no collision occurs, and the target point is in the target area A at the end of the tool G Randomly generated. The speed and direction of the obstacle are randomly set on the x-axis and y-axis, and are randomly selected at the beginning of each round. The speed of the obstacle has two modes: fixed speed and variable speed mode. In variable speed mode, the speed changes randomly at each sampling interval. When the obstacle reaches the boundary, it moves in the opposite direction until the end of the round. This example trains for a total of 10 million time steps and 111341 rounds. The average reward of the training is as follows Figure 6 As shown, the average success rate is Figure 7 shown.

[0112] In this embodiment, the obstacle is set as a cube with a main diagonal of 0.3 meters, surrounded by a sphere with a radius of 0.15 meters. The robot arm obtains real-time information through a real-time data exchange interface, and the external application and the robot arm controller are synchronized. In the experimental test scenario of this embodiment, the obstacle moves along the x-axis at a speed of 0.18m / s, or along the y-axis at a speed of 0.18m / s. The initial positions of the robot arm tool starting point, target point and obstacle are set to different fixed values ​​according to different situations. The test results are as follows Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 and Fig.13 As shown in the figure, no matter whether the obstacle moves along the x-axis or the y-axis, the robot arm can effectively avoid dynamic obstacles and successfully guide the robot arm to the predetermined target position. Even under the influence of dynamic obstacles, the error between the tool end and the target position can eventually converge to zero stably, which fully proves that the robot arm can accurately reach the target position. In addition, Figure 8 and Fig.12 As shown, the change curve of each joint speed maintains good smoothness during the entire movement process, and the speed drops to zero when the robot arm reaches the target position, that is, the robot arm can not only successfully reach the target position, but also stop at the target point accurately without exceeding the predetermined target range.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0114] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0115] A plurality of components in the device are connected to the I / O interface, including: an input unit, such as a keyboard, a mouse, etc.; an output unit, such as various types of displays, speakers, etc.; a storage unit, such as a disk, an optical disk, etc.; and a communication unit, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit allows the device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to perform the method of the present invention in any other appropriate manner (e.g., by means of firmware).

[0116] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A robotic arm autonomous obstacle avoidance method based on deep reinforcement learning, characterized in that: The specific steps include: S1. Collect the status data of the robot arm and obstacles: collect the joint angle and joint speed of the robot arm through the real-time data exchange interface; detect the coding mark through the camera and collect the obstacle information; set the target position of the robot arm; S2. Process the collected data to obtain status information: Based on the current joint angle and joint speed of the robot arm, the position and posture of the end effector of the robot arm are obtained through the forward kinematics calculation of the robot; the image processing library is used to perform coded mark recognition to obtain the coordinates of the obstacles on the image; S3, action planning and execution: The obtained state information of the robotic arm, obstacle and target position is input into the trained neural network model for action planning. The obtained planned action is the joint speed, which is sent to the robotic arm through the robotic arm speed control interface to actually execute the planned action.

2. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: In S3, after the robot arm executes the planned action, it is determined whether the current end position of the robot arm reaches the set target position. If so, the task is successful; otherwise, the next control cycle is entered, and S1, S2 and S3 are repeated.

3. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: In S2, the coordinates in the camera coordinate system are obtained through the image processing library; the camera coordinate system is converted into 3D coordinates in the robot base coordinate system through the hand-eye calibration technology; and further converted into the world coordinate system to obtain the 3D position of the obstacle in the world coordinate system.

4. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: The reward function of the neural network model in S3 includes the obstacle avoidance reward function r at the end of the robot tool t o , body obstacle avoidance reward function r t b , reaching the target reward function r t g , self-collision reward function r t f , motion smoothness reward function r t m , round termination reward function r t c , the total reward function R t The expression is: R t ＝r t o +r t b +r t g +r t f +r t m +r t c 。 5. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: To avoid obstacles at the end of the robot tool, the workspace is divided into a safe area, a warning area, and a danger area according to the position of the obstacle and the robot tool; When the robot tool is in the dangerous area, the actual movement direction of the robot tool Encouraged to move closer to the desired direction of motion of the end-of-arm tool Dangerous area reward function r t d The expression is: Among them, the desired movement direction of the robot arm tool end is the direction of the combined force of the attraction of the set target position on the robot arm tool and the repulsion of the obstacle on the robot arm tool; When the end of the robot tool is in the safe area, the reward function r t s The expression is: Among them, P t g is the target position; P t e is the tool position of the robot arm; P t o is the obstacle position; d(P t g , P t e ) is the distance from the target position to the robot tool position; d(P t e , P t o ) is the distance from the robot tool position to the obstacle position; When the end of the robot tool is in the warning area, the reward function r t e According to the reward function design of the safe area and the dangerous area, the closer the end of the robot tool is to the dangerous area, the greater the weight assigned to the reward function of the dangerous area. t e The expression is: Among them, d1 is the boundary radius between the safety area and the warning area; d2 is the boundary radius between the warning area and the danger area; The obstacle avoidance reward function r of the robot tool end t o The expression is:

6. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: In order to avoid collisions with the robot body, 11 key points are selected on the robot, including 6 joint points, the center point of the tool end and 4 key points on the long connecting rods; the key points are covered by a spherical envelope, and the first safety distance threshold is set; when the distance from the key point position to the obstacle position is less than the first safety distance threshold, a reward is given to maintain a larger distance, otherwise there is no reward; The body obstacle avoidance reward function r t b The expression is: Where d3 is the first safety distance threshold; P t o is the obstacle position; P i,t c is the position of the i-th key point; N is the number of key points of the robot arm.

7. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: Set the reward function so that the end of the tool faces vertically downward during the movement of the robot arm, decelerates to a stop when approaching the target position, and reaches the target position in the shortest time. The reward function r t g , the expression is: r t g ＝r t z +r t v +r t p ， Among them, set the reward function r t z , so that during the movement of the robot arm, the positive direction of the z-axis of the tool end coordinate system is encouraged to be close to the negative direction of the z-axis of the world coordinate system. The origin of the tool end coordinate system is located at the center point of the tool gripper finger, and the positive direction of the z-axis is the direction of the tool. t z The expression is: is the positive direction vector of the z-axis of the robot tool end coordinate system; is the unit vector in the negative direction of the z-axis of the world coordinate system; Set the reward function r t v , so that when the robot arm approaches the target position, the joint speed slows down until it stops. The expression is: d4 is the second safety distance threshold; d(P t g , P t e ) is the distance from the target position to the robot tool position; u t is the joint velocity; Set the reward function r t p , so that the robot arm reaches the target position in the shortest time, the expression is: t real is the time step actually used to reach the target, t total is the maximum time step allowed for an episode.

8. The method for autonomous obstacle avoidance of a robotic arm based on deep reinforcement learning according to claim 4, characterized in that: The reward function also includes a self-collision reward function r t f , motion smoothness reward function r t m and the round-end reward function r t c , where the self-collision reward function r t f The expression is: Among them, d(P i,t c , P j,t c ) is the distance between two points selected from the key points; N is the number of key points; The motion smoothness reward function r t m , so that the difference in joint speeds planned at consecutive moments is reduced, and the expression is: Among them, u t is the current joint velocity at time t; u t-1 is the joint velocity at the previous moment; The round end reward function r t c In order to give positive rewards for successfully completing the task at the end, other termination situations, including collision, exceeding the workspace range, self-collision, and exceeding the maximum round time, are penalized. The expression is:

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images