A method and system for planning the trajectory of a robotic arm

Abstract

A robot arm trajectory planning method and system, relating to the field of robot control technology, includes the following steps: S1, receiving environmental data collected from the observation space based on the robot arm and a virtual laser sensor, wherein the virtual laser sensor is set at a preset position on the robot arm and is relatively fixed to the robot arm; S2, processing the environmental data based on a pre-trained model and outputting continuous values ​​for controlling the motion trajectory; based on virtual laser scanning as observation data, under given target and observation data conditions, a DRL agent trained by a deep reinforcement learning-based robot arm motion planning algorithm can provide a collision-free path toward the target; avoiding the need for extensive computation and hyperparameter setting in complex environments, and the need for resetting when migrating to a new environment.

Description

Technical Field

[0001] This application relates to the field of robot control technology, specifically to a method and system for planning the trajectory of a robot arm. Background Technology

[0002] Industrial robots are widely used in manufacturing processes. Although robots can automatically perform specific tasks, they still require manual programming and setup beforehand. One common problem is how industrial robots can avoid collisions with obstacles or other robot arms. Commonly used sampling-based path planning methods can handle simple environments, but in complex environments, they require a lot of calculations and tedious engineering debugging to set various hyperparameters, which is time-consuming and costly. When migrating to a new environment, offline programming and other methods are also needed for reconfiguration. Summary of the Invention

[0003] (a) Purpose of application

[0004] In view of this, the purpose of this application is to provide a robot arm trajectory planning method and system to solve the technical problems of how to avoid the need for a lot of calculations and hyperparameter settings in complex environments, and the need to reset when migrating to a new environment.

[0005] (II) Technical Solution

[0006] This application discloses a method for planning the trajectory of a robotic arm, including the following steps:

[0007] S1. Receive environmental data collected from the observation space based on the robot arm and the virtual laser sensor, wherein the virtual laser sensor is set at a preset position on the robot arm and is fixed relative to the robot arm;

[0008] S2. The pre-trained model processes the environmental data and outputs continuous values ​​to control the motion trajectory.

[0009] In one possible implementation, the preset location includes: the top of the end effector of the robotic arm, the outer surface, and the top of the wrist joint.

[0010] In one possible implementation, the observation space of the robotic arm includes: the current joint angle vector of the robotic arm, a first scalar value, a second scalar value, and a third scalar value, wherein the first scalar value is the difference between the current position and the target position in Cartesian coordinates, the second scalar value is the current end effector pose described in Euler angles, and the third scalar value is the distance from the current position to the target position; the observation space of the virtual laser sensor includes: the number and angle of rays of the virtual laser sensor.

[0011] In one possible implementation, the observation space of the robotic arm is a 13-dimensional vector, wherein the current joint angle vector is a 6-dimensional vector, the first scalar value is a 3-dimensional vector, the second scalar value is a 3-dimensional vector, and the third scalar value is a 1-dimensional vector.

[0012] In one possible implementation, the pre-trained model includes an actor-critic network, which outputs continuous values ​​for controlling the motion trajectory.

[0013] As a second aspect of this application, a robotic arm trajectory planning system is also disclosed, comprising:

[0014] An environmental data receiving module is used to receive environmental data collected from the observation space based on the robot arm and a virtual laser sensor. The virtual laser sensor is set at a preset position on the robot arm and is fixed relative to the robot arm.

[0015] An environmental data processing module is used to process the environmental data based on a pre-trained model and output continuous values ​​for controlling the motion trajectory.

[0016] In one possible implementation, the preset location includes: the top of the end effector of the robotic arm, the outer surface, and the top of the wrist joint.

[0017] In one possible implementation, the observation space of the robotic arm includes: the current joint angle vector of the robotic arm, a first scalar value, a second scalar value, and a third scalar value, wherein the first scalar value is the difference between the current position and the target position in Cartesian coordinates, the second scalar value is the current end effector pose described in Euler angles, and the third scalar value is the distance from the current position to the target position; the observation space of the virtual laser sensor includes: the number and angle of rays of the virtual laser sensor.

[0018] In one possible implementation, the observation space of the robotic arm is a 13-dimensional vector, wherein the current joint angle vector is a 6-dimensional vector, the first scalar value is a 3-dimensional vector, the second scalar value is a 3-dimensional vector, and the third scalar value is a 1-dimensional vector.

[0019] In one possible implementation, the pre-trained model includes an actor-critic network, which outputs continuous values ​​for controlling the motion trajectory.

[0020] (III) Beneficial Effects

[0021] Based on virtual laser scanning as observation data, given the target and observation data, the DRL agent trained by the deep reinforcement learning-based robotic arm motion planning algorithm can provide a collision-free path toward the target; avoiding the need for a lot of computation and hyperparameter setting in complex environments, and the need to reset when migrating to a new environment.

[0022] Other advantages, objectives, and features of this application will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from practice of this application. The objectives and other advantages of this application can be realized and obtained through the following description. Attached Figure Description

[0023] The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain and illustrate this application, and should not be construed as limiting the scope of protection of this application.

[0024] Figure 1 This is the system flowchart of this application;

[0025] Figure 2 This is a flowchart of the model training process in this application;

[0026] Figure 3 This is a system structure diagram of this application.

[0027] The module consists of: 1. Environmental data receiving module; 2. Environmental data processing module. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0031] In the above description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] like Figure 1 As shown, this embodiment provides a robot arm trajectory planning method, including the following steps:

[0033] S1. Receive environmental data collected from the observation space based on the robot arm and the virtual laser sensor, wherein the virtual laser sensor is set at a preset position on the robot arm and is fixed relative to the robot arm.

[0034] In some embodiments, the preset location includes: the top of the end effector of the robotic arm, the outer surface, and the top of the wrist joint.

[0035] In some embodiments, the observation space of the robot arm includes: the current joint angle vector of the robot arm, a first scalar value, a second scalar value, and a third scalar value. The first scalar value is the difference between the current position and the target position in Cartesian coordinates. The second scalar value is the pose of the current end effector described in Euler angles. The third scalar value is the distance from the current position to the target position. The observation space of the robot arm is a 13-dimensional vector, wherein the current joint angle vector is a 6-dimensional vector, the first scalar value is a 3-dimensional vector, the second scalar value is a 3-dimensional vector, and the third scalar value is a 1-dimensional vector. It can be seen that the environmental data collected based on the observation space of the robot arm is the target position and the current pose of the robot arm. The observation space of the virtual laser sensor includes: the number and angle of the virtual laser sensor. The observation space of the virtual laser sensor is a vector composed of the hit scores of 129 rays. The exact number and angle of the rays are configured according to a specific end effector or robot model. Taking a specific robot model as an example, the virtual laser sensors are arranged at the top, outer surface, and wrist joint of the end effector of the robot arm in the following manner. First, the virtual laser sensor for the wrist joint consists of 24 rays, each placed vertically at an angular distance of [missing information]. The polar angle is Then, the end effector's surrounding outer surface is equipped with a total of 80 rays, with 10 rays horizontally spaced apart. Finally, 25 rays were placed at the top of the end effector, with a polar angle of [missing information]. Horizontal interval is In this embodiment, the received environmental data is in the form of a series of environmental data collected from the observation space of the robotic arm and the virtual laser sensor.

[0036] S2. The pre-trained model processes the environmental data and outputs continuous values ​​to control the motion trajectory.

[0037] Based on virtual laser scanning as observation data, given the target and observation data, the DRL agent trained by the deep reinforcement learning-based robotic arm motion planning algorithm can provide a collision-free path toward the target; avoiding the need for a lot of computation and hyperparameter setting in complex environments, and the need to reset when migrating to a new environment.

[0038] In some embodiments, the pre-trained model includes an actor-critic network, which outputs continuous values ​​to control the motion trajectory. The pre-trained model implements an industrial robot task in the 3D simulator Bullet3. Virtual laser sensors are placed on the robot arm at the preset positions described above to collect environmental data. A dominant actor-critic network with a reward mechanism is used based on deep reinforcement learning to train a DRL agent to output continuous values ​​to predict changes in the robot's current posture. In the prior art, deep reinforcement learning (DRL) is an end-to-end method with the potential to learn complex behaviors in unknown environments. DRL-based methods have been used to plan collision-free robot arm trajectories and perform long-distance movements of ground robots. However, existing DRL methods rely on static sensors to perceive the environment and take corresponding actions. For industrial tasks such as welding, the robot may move outside the range of static sensors or into areas that static sensors cannot detect in narrow workspaces. Furthermore, training efficiency is further improved by simultaneously simulating multiple agent environments for parallel training and setting dynamic augmentation targets.

[0039] The pre-training model process is as follows: Figure 2 As shown, a deep reinforcement learning DRL agent is trained using a pre-trained model to perform collision-free motion planning for an industrial robot using virtual laser scanning observations. Both training and testing are performed in the Bullet3 simulator. Training involves calculating each step based on the robot's motion using inverse robot kinematics and damped least squares, repeating this process until the robot reaches the target. Testing involves sending the final action into a virtual testing environment for testing.

[0040] The pre-trained model includes an actor-critic network, where the actor provides the policy and the critic provides the value function. During training, the pre-trained model is trained on continuous action states within the workspace using training environment data collected from the observation space based on the robotic arm and virtual laser sensors, to improve the flexibility and smoothness of the actions. The workspace A is defined as follows:

[0041] ;

[0042] The specific steps for training a DRL agent using the PPO algorithm optimized by the proximal strategy are as follows: reset the environment and obtain the initial state. Further through operational strategies Choose an action Further actions to be performed in the environment Further rewards and the next state Further collection and conversion Trajectory set for trajectory memory In the middle; further calculation of rewards Further based on the current value function Estimated Advantage Further maximize the PPO cutoff objective and update the value function.

[0043] PPO uses an Advantage Actor-Critic structure, where actors provide the policy and critics provide the value function. In each iteration, the value function is calculated to assist in policy updates.

[0044] In this embodiment, the actor and critic share a feature extractor of a specific size to extract features from the input, saving computation time. Parallel training is achieved by simultaneously simulating multiple DRL agents and a virtual training environment. To further accelerate training, the data acquired by the virtual laser sensor and the setting of augmented targets are simplified. The augmented target refers to an adaptive strategy that adjusts the reachability of a target. The size of the augmented target is modified based on the training results; if the DRL agent consistently fails to reach the target, the target size increases; otherwise, the target size decreases. The formula is:

[0045] ;

[0046] ;

[0047] ;

[0048] in, express If smaller than the default size, then This is the initial size; This means that if the number of times the target is reached is less than the success threshold, and less than The maximum range will then be the previous one. The value is updated by adding the increment value; This means that if the number of times the target is reached is not less than the success threshold and is greater than... The minimum range, then the previous one Value incremented by subtraction; If the enhancement objective is met once, the current value will be assigned the default size. In other cases The value remains unchanged; It is the size of the enhanced target of set e. It is an episode whose size remains unchanged. and These represent the target size increment and decrement values, respectively. Indicates the success threshold. This represents the number of times the enhancement goal was successfully achieved in the past k actions. This represents the success or failure of the DRL agent in the current operation. Pt is the current position of the arm, and Ptarget is the target position. If it is smaller than the size of the augmentation set, it is considered a success.

[0049] The DRL agent is trained in a randomized virtual training environment. After each iteration, static and dynamic obstacles are randomly generated. The number of obstacles increases or decreases depending on the success rate of the DRL agent.

[0050] The network's reward system It is designed as follows, which will affect the DRL agent behavior.

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, It is the reward for success, when the distance to the target location is... Less than the distance threshold At that time, they considered it a success. For other cases, It's a collision penalty. It's a distance penalty. This is a penalty that exceeds the scope of the workspace. It is a penalty for running out of steps in a single process, preventing the robot from abandoning space exploration by avoiding collisions. It is a penalty for uneven motion. This indicates the number of times the robot moves back and forth in the last 10 steps. For workspace Working coordinates in the axial direction The working coordinates are located in the y-axis direction of the workspace. For workspace Working coordinates in the axial direction This represents the minimum value of the working coordinate in the x-axis direction of the workspace. This represents the maximum value of the working coordinate in the x-axis direction of the workspace. This represents the minimum value of the working coordinate in the y-axis direction of the workspace. This represents the maximum value of the working coordinate in the y-axis direction of the workspace. This represents the minimum value of the working coordinate in the z-axis direction of the workspace. This represents the maximum value of the working coordinate in the z-axis direction of the workspace. The number of steps in one process, It is the maximum number of steps allowed.

[0059] When the actor network calculates the optimal action based on received sensor data and control signals using certain algorithms and models, it then transmits this action to the machine kinematics. The machine kinematics, based on this action and the robot's kinematic model, calculates the motion parameters of the machine actuators, thereby enabling the robot's autonomous movement and interaction with the environment. Robot control algorithms rely on kinematics to calculate accurate commands. A common method for modeling robots composed of physical links and joints is the Denavit-Hartenberg (DH) representation, which uses a homogeneous transformation matrix (robot configuration q) to describe the relationship between the coordinate systems of two adjacent links. For a given robot, these matrices are chained together to produce a transformation from the robot's base links to its end effector, which is the control target in most applications. For a six-DOF robot, the transformation from the base to the end effector is shown in the following equation.

[0060] ;

[0061] ;

[0062] The formula can be understood as the robot's forward kinematics; for a given configuration q, the transformation... The orientation and position of the end effector within the robot's chassis are given. However, most tasks in a robot are defined in the workspace, not the configuration space. This necessitates inverse kinematics so that the trajectory defined in the workspace can be mapped to the configuration space and subsequently used as control input. Generally, in inverse kinematics, the desired workspace displacement is... With the initially unknown configuration space displacement The relevant formula is shown below:

[0063] ;

[0064] in, For the Jacobian determinant of the robot;

[0065] Since an exact solution cannot be obtained, an approximate method is used to solve the equation. The damped least squares method is used to solve this problem, as shown in the following equation:

[0066] ;

[0067] in, It is a damping factor. Let I be the initially unknown configuration space displacement, and let I be the identity matrix. This is the Jacobian determinant of the robot; this damping factor can be reduced when the robot enters a configuration near the singularity. , It is a dot product. It can adjust the damping factor for each singular vector in the singular value decomposition of the Jacobian matrix according to the difficulty of reaching the target position.

[0068] As a second aspect of this application, a robotic arm trajectory planning system is also disclosed, comprising:

[0069] The environmental data receiving module 1 is used to receive environmental data collected from the observation space based on the robot arm and the virtual laser sensor. The virtual laser sensor is set at a preset position on the robot arm and is fixed relative to the robot arm.

[0070] The environmental data processing module 2 is used to process the environmental data based on the pre-trained model and output continuous values ​​for controlling the motion trajectory.

[0071] In one possible implementation, the preset location includes: the top of the end effector of the robotic arm, the outer surface, and the top of the wrist joint.

[0072] In one possible implementation, the observation space of the robotic arm includes: the current joint angle vector of the robotic arm, a first scalar value, a second scalar value, and a third scalar value, wherein the first scalar value is the difference between the current position and the target position in Cartesian coordinates, the second scalar value is the current end effector pose described in Euler angles, and the third scalar value is the distance from the current position to the target position; the observation space of the virtual laser sensor includes: the number and angle of rays of the virtual laser sensor.

[0073] In one possible implementation, the observation space of the robotic arm is a 13-dimensional vector, wherein the current joint angle vector is a 6-dimensional vector, the first scalar value is a 3-dimensional vector, the second scalar value is a 3-dimensional vector, and the third scalar value is a 1-dimensional vector.

[0074] In one possible implementation, the pre-trained model includes an actor-critic network, which outputs continuous values ​​for controlling the motion trajectory.

[0075] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments.

[0076] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0077] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application. All such modifications and variations fall within the scope defined by the appended claims. The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this invention, and these modifications or substitutions should all be covered within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for planning the trajectory of a robotic arm, characterized in that, Includes the following steps: S1. Receive environmental data collected from the observation space based on the robot arm and the virtual laser sensor, wherein the virtual laser sensor is set at a preset position on the robot arm and is fixed relative to the robot arm; S2. Based on the pre-trained model, the environmental data is processed and continuous values ​​are output to control the motion trajectory. The preset positions include: the top of the end effector of the robot arm, its outer surface, and the top of the wrist joint; The observation space of the robotic arm includes: the current joint angle vector of the robotic arm, a first scalar value, a second scalar value, and a third scalar value. The first scalar value is the difference between the current position and the target position in Cartesian coordinates. The second scalar value is the current end effector pose described in Euler angles. The third scalar value is the distance from the current position to the target position. The observation space of the virtual laser sensor includes: the number and angle of the rays of the virtual laser sensor. The observation space of the robot arm is a 13-dimensional vector, wherein the current joint angle vector is a 6-dimensional vector, the first scalar value is a 3-dimensional vector, the second scalar value is a 3-dimensional vector, and the third scalar value is a 1-dimensional vector; The pre-trained model includes an actor-critic network, which outputs continuous values ​​to control the motion trajectory.

2. A robot arm trajectory planning system, characterized in that, include: An environmental data receiving module is used to receive environmental data collected from the observation space based on the robot arm and a virtual laser sensor. The virtual laser sensor is set at a preset position on the robot arm and is fixed relative to the robot arm. An environmental data processing module is used to process the environmental data based on a pre-trained model and output continuous values ​​for controlling the motion trajectory. The preset positions include: the top of the end effector of the robot arm, its outer surface, and the top of the wrist joint; The observation space of the robotic arm includes: the current joint angle vector of the robotic arm, a first scalar value, a second scalar value, and a third scalar value. The first scalar value is the difference between the current position and the target position in Cartesian coordinates. The second scalar value is the current end effector pose described in Euler angles. The third scalar value is the distance from the current position to the target position. The observation space of the virtual laser sensor includes: the number and angle of the rays of the virtual laser sensor. The observation space of the robot arm is a 13-dimensional vector, wherein the current joint angle vector is a 6-dimensional vector, the first scalar value is a 3-dimensional vector, the second scalar value is a 3-dimensional vector, and the third scalar value is a 1-dimensional vector; The pre-trained model includes an actor-critic network, which outputs continuous values ​​to control the motion trajectory.

Images