Reinforcement learning method for mechanical arm path planning based on action smoothing mechanism and application

Abstract

The application discloses a kind of mechanical arm path planning reinforcement learning method and application based on action smoothing mechanism, constructs the state space and action space of mechanical arm;Based on flexible actor-critic architecture, the strategy network is constructed to train mechanical arm path planning, with the state space, action space, immediate reward and next time state space of mechanical arm at each time as sample, experience pool is formed, and the sample of experience pool is sampled in the training process by introducing priority experience playback mechanism;Action smoothing mechanism based on obstacle distance self-adaption is introduced in the strategy network update stage, the action change penalty related to the distance between mechanical arm and obstacle is added in strategy loss function, the trajectory smoothness dynamic adjustment of mechanical arm path planning process is realized by constraining the action difference value of adjacent time of mechanical arm.The application can effectively inhibit the sudden action shock when mechanical arm is close to obstacle, and significantly improve the motion stability and system reliability in mechanical arm obstacle avoidance task.

Description

Technical Field

[0001] This invention relates to a reinforcement learning method and application for robotic arm path planning based on motion smoothing mechanism, belonging to the field of robotic arm path planning technology. Background Technology

[0002] With the development of intelligent manufacturing and human-machine collaboration technologies, robotic arms are widely used in complex operation scenarios such as assembly, grinding, spraying, and grasping. The performance of the path planning algorithm for robotic arms directly affects their motion efficiency, trajectory smoothness, and execution safety, and is a key link in achieving high-precision and high-stability control. Currently, path planning schemes based on traditional control methods such as interpolation planning, optimal control, and model predictive control can generate relatively stable trajectories in known environments. However, when faced with dynamic obstacles, complex spatial constraints, or uncertain tasks, their generalization ability and adaptive performance are insufficient, making it difficult to meet the requirements of real-time performance and robustness.

[0003] In recent years, reinforcement learning (RL) technology has been widely introduced into the path planning problem of robotic arms. Chinese patent application CN119328765A discloses a deep reinforcement learning-based method for optimizing robotic arm trajectories. Through interactive learning with the environment, the reinforcement learning algorithm can achieve adaptive path generation without relying on an accurate model. Typical methods include Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC). However, existing reinforcement learning methods still have the following problems in robotic arm control applications: First, robotic arm control is a high-dimensional continuous motion space problem. Reinforcement learning networks have complex structures and high state space dimensions. Small fluctuations in the network output are easily amplified in the control signal, leading to significant jitter in joint angles or angular velocities, affecting trajectory smoothness and control stability. Secondly, frequent changes in movement cause the motor to be in a state of high-frequency start-stop and repeated acceleration and deceleration for a long time, increasing the wear and fatigue load on mechanical components. This not only reduces the service life of the system but may also cause problems such as motor overheating and abnormal temperature rise, affecting the overall safety and stability of operation. This is especially true in obstacle avoidance tasks, where the movement of the robotic arm must be smooth. If the changes in movement are too drastic, the trajectory may deviate from the expected path, increasing the risk of collision. At the same time, excessive speed fluctuations can cause delayed response or overcompensation in the control loop, affecting obstacle avoidance accuracy and system stability.

[0004] To address the aforementioned issues, there is an urgent need for a path planning method that can balance safety, smoothness, and stability within a reinforcement learning framework. Summary of the Invention

[0005] The technical problem solved by this invention is to address the issues of trajectory smoothness and control stability in the continuous motion space of robotic arms by existing reinforcement algorithms, and to provide a reinforcement learning method and application for robotic arm path planning based on motion smoothing mechanism.

[0006] This invention is achieved using the following technical solution:

[0007] This invention first discloses a reinforcement learning method for robotic arm path planning based on a motion smoothing mechanism, comprising the following steps:

[0008] Construct the state space and motion space of the robotic arm;

[0009] The strategy network is constructed based on the flexible actor-critic architecture to train the path planning of the robotic arm. The state space, action space, immediate reward and the state space of the next moment of the robotic arm at each moment are used as samples to form an experience pool. During the training process, a priority experience replay mechanism is introduced to sample the samples in the experience pool and predict the action space of the robotic arm at the next moment.

[0010] The update phase of the policy network introduces an action smoothing mechanism based on obstacle distance adaptation. The policy loss function adds a penalty for action changes related to the distance between the robotic arm and the obstacle. The mechanism uses the action difference between adjacent moments of the robotic arm as a constraint to achieve dynamic adjustment of the trajectory smoothness of the robotic arm path planning process.

[0011] In a reinforcement learning method for robotic arm path planning based on motion smoothing mechanism of the present invention, the state space of the robotic arm further includes the angle of each joint of the robotic arm at each time, the angular velocity of each joint, the distance between the position of the robotic arm end effector and the position of the moving target, and the distance between the robotic arm end effector and the center of the obstacle; the motion space of the robotic arm includes the expected motion commands of each joint of the robotic arm output by the policy network.

[0012] In a reinforcement learning method for robotic arm path planning based on motion smoothing mechanism of the present invention, the immediate reward is further obtained through the reward function of the policy network, and the reward function includes the reward for reaching the target at the end of the robotic arm and the reward for safe distance from obstacles.

[0013] The robotic arm's end-effector reaches the target reward. for:

[0014] ,

[0015] in, The target position for the movement of the robotic arm's end effector. The current position of the robotic arm's end effector; reward for reaching the target at the end effector. Defined as the negative of the Euclidean distance between the current position of the robotic arm's end effector and the position of the moving target;

[0016] The obstacle distance safety reward Defined as:

[0017] ,

[0018] in, This is the current distance between the end effector of the robotic arm and the center of the obstacle. The minimum safe distance is set. This is the penalty coefficient.

[0019] In a reinforcement learning method for robotic arm path planning based on a motion smoothing mechanism according to the present invention, the reward function further incorporates collision penalty and time penalty, expressed as:

[0020] ,

[0021] For the overall reward function, As a penalty for collision, As a time penalty.

[0022] In a reinforcement learning method for path planning of a robotic arm based on a motion smoothing mechanism according to the present invention, further, during the training of the policy network, the robotic arm optimizes its policy through continuous interaction with the environment. The environment is modeled using a Markov decision process and consists of the state space, action space, immediate reward, and state space of the robotic arm at each time step. First, the robotic arm obtains the current state space through interaction with the environment, outputs the corresponding action space according to the policy network, and executes it in the environment. The environment provides immediate reward feedback and updates the state space of the robotic arm, and stores the updated interaction data as samples in the experience pool. The policy parameters of the policy network are updated by sampling in the experience pool.

[0023] In a reinforcement learning method for robotic arm path planning based on a motion smoothing mechanism according to the present invention, further, in the priority experience replay mechanism, the importance of each sample is measured by the TD error of each sample, the sampling priority of each sample is evaluated according to its importance, and the sampling probability of the sample is calculated according to the sample priority during sampling:

[0024] ,

[0025] in, The sampling probability of the sample. For the priority of sample i, Represents the priority of all samples. Represents the total number of samples

[0026] In a reinforcement learning method for robotic arm path planning based on motion smoothing mechanism of the present invention, the sampling probability of the sample is further corrected and updated by means of importance sampling weight.

[0027] In a reinforcement learning method for robotic arm path planning based on a motion smoothing mechanism according to the present invention, the policy loss function is further defined as:

[0028] ,

[0029] in, To introduce the policy loss term after the action smoothing mechanism is introduced, For the original strategy loss term of the flexible actor-critic architecture, For motion smoothing constraints, = ,in, and Let these represent the robot arm motion vectors at the current time t and the previous time t-1, respectively. It is a norm 2. This indicates the distance between the end effector or key joint of the robotic arm and the center of the obstacle. For distance adaptive smoothing coefficient, , This is a proportional adjustment parameter. .

[0030] In a reinforcement learning method for robotic arm path planning based on a motion smoothing mechanism according to the present invention, the gradient of the change in robotic arm motion in the motion smoothing constraint term is further used for policy parameter updating, and the corresponding gradient is:

[0031] ,

[0032] in, For the gradient of policy parameter changes, This refers to the change in motion of the robotic arm. The policy parameters are updated following the gradient descent rule.

[0033] , in Let be the policy parameters of the policy network at the current time t. These are the policy parameters that the policy network will update at the next time step t+1. This is the learning rate.

[0034] The present invention also discloses a robotic arm control method, wherein the robotic arm uses a policy network trained by the above-described robotic arm path planning reinforcement learning method of the present invention for path planning control.

[0035] This invention introduces a motion smoothing mechanism that adaptively correlates the distance between the robotic arm and obstacles during the strategy update phase of the flexible actor-critic architecture. By dynamically adjusting the amplitude of the robotic arm's motion changes, it achieves smooth constraints on the joint movements of the robotic arm during the obstacle avoidance phase, thereby effectively improving motion stability during path planning, reducing joint motor impact and wear of the robotic arm, and enhancing the safety and reliability of the system.

[0036] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0037] (1) To address the high-dimensional continuous nature of the robotic arm path planning task, a state-space model integrating its own motion state and environmental perception information is constructed. The state space includes key parameters such as the joint angles, joint angular velocities, end effector poses, relative positions of obstacles, and environmental features of the robotic arm, to comprehensively describe the current motion and environmental state of the system. The motion space is defined as the control quantities of each joint of the robotic arm, including the angular velocity or torque commands of each joint, which are output by the policy network and used to directly drive the robotic arm to achieve path planning and obstacle avoidance control.

[0038] (2) A comprehensive reward function is constructed for the policy network, which includes multiple indicators such as trajectory tracking error, motion smoothness constraint, and safe distance constraint. By dynamically adjusting the weight coefficients, the algorithm can adaptively optimize motion performance during training and generate smooth, energy-efficient, and safe motion sequences.

[0039] (3) To improve the sample utilization efficiency and training stability of reinforcement learning, a Prioritized Experience Replay (PER) mechanism is adopted for data sampling. Different sampling probabilities are assigned to experience samples according to the magnitude of the TD error, so that key experience samples in the experience pool receive higher training frequency. At the same time, importance sampling weights are introduced to correct the bias and maintain the unbiasedness of the estimation.

[0040] (4) In the policy network update phase, an obstacle distance-based motion smoothing constraint mechanism is introduced. By designing distance-adaptive motion smoothing constraint terms, penalties are imposed on the motion difference between adjacent time steps. When the robotic arm approaches an obstacle, the smoothing constraint is automatically strengthened, thereby limiting abrupt motion changes, realizing dynamic adjustment of the robotic arm's motion output, suppressing joint oscillations, improving the continuity and execution stability of the robotic arm's trajectory, thereby enhancing the robotic arm's motion performance and control safety in complex environments, and maintaining continuous and stable motion trajectories.

[0041] In summary, the present invention, through the above-mentioned solution, can effectively suppress sudden motion oscillations when the robotic arm approaches an obstacle, ensuring the continuity and stability of the trajectory, and maintaining efficient response and energy-saving movement when moving away from the obstacle. Thus, it achieves a dynamic balance between safety, smoothness and flexibility, significantly improving the motion stability and system reliability of the robotic arm in obstacle avoidance tasks.

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the process for training the robotic arm path planning using the policy network in this invention.

[0044] Figure 2 This is the robotic arm simulation platform constructed in the embodiment.

[0045] Figure 3 The figure shows the curves of the change in the angular velocity of the robotic arm joints during the obstacle avoidance task in the robotic arm simulation platform in the embodiment. The shaded area represents the mechanical obstacle avoidance stage. Detailed Implementation

[0046] Example

[0047] The reinforcement learning method for robotic arm path planning of the present invention is applied to the training of a robotic arm path planning model. The model uses a flexible actor-commentator architecture to construct a policy network for robotic arm path planning training. The state space, action space, immediate reward, and next-time state space of the robotic arm at each moment are selected as samples from the constructed robotic arm state space and action space data model. The samples corresponding to all moments in the robotic arm's movement path form an experience pool. During the training process of the policy network for robotic arm path planning, a priority experience replay mechanism is introduced to sample the samples in the experience pool. In the update phase of the policy network, an action smoothing mechanism based on obstacle distance adaptation is introduced. The action change penalty related to the distance between the robotic arm and the obstacle is added to the policy loss function. The trajectory smoothness of the robotic arm path planning process is dynamically adjusted by constraining the action difference between adjacent moments of the robotic arm.

[0048] In the construction of the state space and motion space data model of the robotic arm, the state space of the robotic arm includes the angles of each joint of the robotic arm, the angular velocity of each joint, the distance between the current position of the robotic arm end effector and the position of the moving target, and the distance between the robotic arm end effector and the center of the obstacle. The obstacle position information is obtained by a depth RGB camera installed on the robotic arm. The motion space of the robotic arm includes the expected motion commands of each joint of the robotic arm output by the policy network.

[0049] To fully describe the motion state and task objective of a robotic arm, taking a six-degree-of-freedom robotic arm as an example, the state space of the robotic arm... , This represents the state space of the robotic arm at time t. These represent the angles of each joint in a six-degree-of-freedom robotic arm. , Represents the angular velocities of each joint in a six-degree-of-freedom robotic arm. . The target position for the robotic arm's movement is set. This is the current position of the robotic arm. This represents the distance between the end effector of the robotic arm and the center of the obstacle at time t.

[0050] To ensure consistency in numerical scale across different dimensions of state variables and to guarantee the training stability and convergence efficiency of the algorithm, the state space variables are normalized as follows: ,in, This refers to the raw state data acquired at the current moment. The minimum value in the history of this state. This represents the maximum value in the history of this state.

[0051] The robotic arm's motion space , ,in This refers to the control commands output by the robotic arm, i.e., the desired action instructions issued by the policy network to each joint of the robotic arm. The update relationship is... ,in For time intervals, This represents the upper limit value of the hardware parameters (such as angular velocity) corresponding to each joint in the robotic arm control commands. The robotic arm state space and motion space constructed above ensure that the state and motion spaces can fully reflect the motion characteristics of the robotic arm, while providing stable and efficient input and output for subsequent reinforcement learning algorithms.

[0052] The immediate rewards included in the sample are obtained through a reward function of the policy network. This reward function includes a reward for reaching the target at the robotic arm's end effector, a reward for safe obstacle distance, a collision penalty, and a time penalty. This reward function ensures that the robotic arm can reach the designated location during its movement and successfully complete the task while avoiding obstacles.

[0053] Specifically, the target arrival reward at the end of the robotic arm is used to drive the end effector of the robotic arm to continuously move closer to the target position. It is defined as the negative number of the Euclidean distance between the current position of the end of the robotic arm and the position of the moving target, as follows:

[0054] .

[0055] in, Rewards are given for reaching the target at the end effector of the robotic arm. The target position for the movement of the robotic arm's end effector. This is the current position of the robotic arm's end effector. When the Euclidean distance between the current position of the robotic arm's end effector and the position of the moving target decreases, this reward value increases accordingly, thereby encouraging the robotic arm to quickly approach the target.

[0056] The obstacle distance safety bonus is a safety constraint introduced to ensure that the robotic arm can maintain a safe distance during obstacle avoidance tasks and prevent collisions between the joints or end effector and obstacles. It is defined as follows:

[0057] .

[0058] in, This is the current distance between the end effector of the robotic arm and the center of the obstacle. The minimum safe distance is set. The penalty coefficient is... The reward is applied to obstacle distance safety. When the robotic arm's end effector approaches an obstacle, this reward generates a strong penalty signal, forcing the policy network to adjust its actions to move away from the obstacle; when the robotic arm's end effector is within a safe range, no additional constraint is generated.

[0059] Although safety rewards effectively prevent actions that approach obstacles, collisions may still occur during actual training. To reinforce safety constraints, the reward function also introduces a collision penalty term, defined as:

[0060] .

[0061] in, The collision penalty coefficient is applied immediately when any part of the robotic arm comes into contact with an obstacle. This factor can significantly suppress dangerous collision actions during the robotic arm's path planning process and accelerate the strategy's learning of action patterns that meet physical constraints and safety requirements.

[0062] To prevent the robotic arm from pausing for extended periods or making redundant movements during a task, a time penalty term is introduced into the reward function. , ,in A fixed penalty constant is applied for each time step. This incentive policy network efficiently completes the robot arm's path planning task within a finite time, avoiding excessively long trajectories or ineffective movements, thereby improving overall task execution efficiency.

[0063] The overall reward function of the final policy network is expressed as:

[0064] .

[0065] For the overall reward function, Rewards are given for reaching the target at the end effector of the robotic arm. As a safety reward for obstacle distance, As a penalty for collision, As a time penalty.

[0066] During the training of the policy network, the robotic arm optimizes its policy through continuous interaction with the environment, which is modeled using a Markov Decision Process (MDP) and consists of the robotic arm's state space at each time step. Action space Instant rewards and the state space at the next moment The structure involves the robotic arm first obtaining its current state space through interaction with the environment, then outputting the corresponding action space according to the policy network and executing it in the environment. The environment provides immediate feedback, rewards the robotic arm, and updates its state space, and then transmits the updated interaction data ( , , , The samples are stored in the experience pool, and the policy parameters of the policy network are updated by sampling in the experience pool, thereby gradually improving the policy performance.

[0067] In the policy network of this invention, samples from the experience pool are sampled through a priority experience replay mechanism. Priority experience replay is used to break the correlation between samples and improve data utilization. The sampling probability of each experience is dynamically adjusted according to its contribution to policy improvement, thereby accelerating training convergence and improving the utilization value of samples. This solves the problem that traditional random sampling methods cannot distinguish the importance of different experiences, resulting in low policy update efficiency.

[0068] Assume the experience pool stores the first The sample is ,in For the state space of the robotic arm, For the robotic arm's motion space, For instant rewards, For the state space of the robotic arm in the next moment, the priority experience replay mechanism measures the importance of a sample by the TD error of each sample, as defined below:

[0069] .

[0070] in, For TD error, As a discount factor, To evaluate network parameters for the target, For the next action, For the current evaluation of network parameters, This is the action value function. A larger TD error indicates a greater potential for improvement to the current policy from that sample. Sample sampling is prioritized based on the importance of each sample. , ,in, To prevent smoothing constants with zero priority, Control the intensity of priority sampling. To control the hyperparameters of priority sampling intensity, when When the value is 0, it degenerates into random sampling.

[0071] During sampling, the sampling probability of a sample is calculated based on its priority.

[0072] .

[0073] in, The sampling probability of the sample. For the priority of sample i, Represents the priority of all samples. This represents the total number of samples, making it more likely that samples with high TD errors will be selected.

[0074] To eliminate sampling bias, the priority experience replay mechanism of this invention corrects the sampling probability of samples by updating the gradient through importance sampling weights, defined as follows: , in, For the capacity of the experience pool, For sampling weights, Used to control the degree of correction for sampling bias. When It is not corrected at the time, but when When the value is gradually increased to 1, sufficient correction of high-priority samples can be achieved.

[0075] This invention aims to suppress motion oscillations in a robotic arm during obstacle avoidance and improve trajectory continuity and execution stability. It introduces an obstacle distance-adaptive motion smoothing mechanism during the policy network update stage. This mechanism, based on the original flexible actor-critic architecture's reinforcement learning policy loss function, adds a penalty for motion changes related to obstacle distance. By constraining the difference in motion between adjacent time steps, it achieves dynamic adjustment of trajectory smoothness. The improved policy loss function of the policy network is defined as follows:

[0076] .

[0077] in, To introduce the policy loss term after the action smoothing mechanism is introduced, For the original strategy loss term of the flexible actor-critic architecture, ,in, The temperature coefficient is used to control the exploration weights. Represents a policy network. Representative evaluation network action value function This indicates that the average value is taken from the state samples stored in the experience pool. For motion smoothing constraints, = ,in, and Let these represent the robot arm motion vectors at the current time t and the previous time t-1, respectively. It is a norm 2. This indicates the distance between the end effector or key joint of the robotic arm and the center of the obstacle. This refers to the distance adaptive smoothing coefficient, used to achieve distance adaptive adjustment. Using an inverse proportional function form, , This is a proportional adjustment parameter. This design ensures that when the robotic arm approaches an obstacle, When the distance is smaller, the adaptive smoothing coefficient increases significantly, thus strengthening the penalty for abrupt movement changes and suppressing drastic joint changes; while when moving away from obstacles, The larger the distance, the more the adaptive smoothing coefficient automatically decreases, allowing the strategy to maintain high operational flexibility and response speed.

[0078] The gradient of the change in the robot arm's motion in the motion smoothing constraint term is used for policy parameter updates, thereby automatically strengthening the smoothing constraint when approaching an obstacle. The corresponding gradient is:

[0079] .

[0080] in, For the gradient of policy parameter changes, This refers to the change in motion of the robotic arm. .

[0081] Obtaining the gradient of policy parameter changes Subsequently, the policy parameters are updated following the rules of gradient descent. , in Let be the policy parameters of the policy network at the current time t. These are the policy parameters that the policy network will update at the next time step t+1. This is the learning rate.

[0082] Therefore, the final policy optimization objective of the policy network of this invention can be expressed as:

[0083] .

[0084] Through the above design, when planning the movement path of the robotic arm, the present invention can effectively suppress sudden movement oscillations when the robotic arm is close to an obstacle, ensuring the continuity and stability of the trajectory; and maintain efficient response and energy-saving movement when it is far away from the obstacle, thereby achieving a dynamic balance between safety, smoothness and flexibility, and significantly improving the movement stability and system reliability of the robotic arm in obstacle avoidance tasks.

[0085] See Figure 1 The diagram illustrates the process of training a robotic arm path planning system using the reinforcement learning method of this invention. First, the robotic arm pose, obstacle positions, and target position are initialized. Then, the current robotic arm state information is input into a policy network based on a flexible actor-critic architecture to execute actions and interact with the environment, detecting whether the robotic arm collides with obstacles. Next, the next moment's robotic arm joint angle state, end-effector spatial position information, and reward are generated. A priority replay experience mechanism is used to sample from the experience pool, and the sampled information is input into the policy network for training. Based on the distance between the end-effector and the obstacle, a motion smoothing mechanism is used to update the policy network. The updated policy network adjusts the input... To obtain a more accurate next-moment motion state of the robotic arm, the updated next-moment robotic arm state information is repeatedly input into a policy network based on a flexible actor-critic architecture for repeated training. During this repeated training process, it is detected in real time whether the robotic arm end effector has reached the target area or the maximum number of steps, i.e., the number of times it interacts with the environment in one training round. If the target area or the maximum number of steps has not been reached, the above training is repeated. If the target area or the maximum number of steps has been reached, it is detected whether the maximum number of detection rounds of the policy network has been reached. After the maximum number of training rounds of the policy network has been reached, the trained policy network is deployed to the robotic arm for path planning and control, ultimately realizing the movement control of the robotic arm.

[0086] The reinforcement learning algorithm described above is applied to the obstacle avoidance task of a robotic arm, and a robotic arm simulation platform is constructed as follows: Figure 2 As shown, by Figure 3 The curves showing the change in angular velocity of the robotic arm joints are shown. The shaded area represents the obstacle avoidance phase. Within the obstacle avoidance area, the average control speed of each joint of the robotic arm is effectively smoothed, and motion oscillation is significantly reduced, indicating that the proposed method significantly improves the trajectory continuity during the obstacle avoidance phase.

[0087] The above embodiments describe the basic principles and main features of the present invention, as well as its advantages. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the specific working principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the present invention as claimed, which is defined by the appended claims and their equivalents.

Claims

1. A reinforcement learning method for robotic arm path planning based on motion smoothing mechanism, characterized by: The process includes the following: Construct the state space and motion space of the robotic arm; The strategy network is constructed based on the flexible actor-critic architecture to train the path planning of the robotic arm. The state space, action space, immediate reward and the state space of the next moment of the robotic arm at each moment are used as samples to form an experience pool. During the training process, a priority experience replay mechanism is introduced to sample the samples in the experience pool and predict the action space of the robotic arm at the next moment. The immediate reward is obtained through a reward function of a policy network. This reward function includes a reward for reaching the target at the robotic arm's end effector, a reward for safe obstacle distance, a collision penalty, and a time penalty, expressed as follows: ， For the overall reward function, As a penalty for collision, As a time penalty; The robotic arm's end-effector reaches the target reward. for: ， in, The target position for the movement of the robotic arm's end effector. The current position of the robotic arm's end effector; reward for reaching the target at the end effector. Defined as the negative of the Euclidean distance between the current position of the robotic arm's end effector and the position of the moving target; The obstacle distance safety reward Defined as: ， in, This is the current distance between the end effector of the robotic arm and the center of the obstacle. The minimum safe distance is set. This is the penalty coefficient; The update phase of the policy network introduces an action smoothing mechanism based on obstacle distance adaptation. The policy loss function adds a penalty for action changes related to the distance between the robotic arm and the obstacle. The mechanism uses the action difference between adjacent moments of the robotic arm as a constraint to achieve dynamic adjustment of the trajectory smoothness of the robotic arm path planning process. The policy loss function is defined as follows: ， in, To introduce the policy loss term after the action smoothing mechanism is introduced, For the original strategy loss term of the flexible actor-critic architecture, For motion smoothing constraints, = ,in, and Let these represent the robot arm motion vectors at the current time t and the previous time t-1, respectively. It is a norm 2. This indicates the distance between the end effector or key joint of the robotic arm and the center of the obstacle. For distance adaptive smoothing coefficient, , This is a proportional adjustment parameter. ; The gradient of the change in the robot arm's motion in the motion smoothing constraint term is used for policy parameter updates. The corresponding gradient is: ， in, For the gradient of policy parameter changes, This refers to the change in motion of the robotic arm. The policy parameters are updated following the gradient descent rule. , in Let be the policy parameters of the policy network at the current time t. These are the policy parameters that the policy network will update at the next time step t+1. This is the learning rate.

2. The reinforcement learning method for robotic arm path planning based on motion smoothing mechanism according to claim 1, characterized in that: The state space of the robotic arm includes the angles of each joint of the robotic arm at each moment, the angular velocities of each joint, the distance between the position of the robotic arm end effector and the position of the moving target, and the distance between the robotic arm end effector and the center of the obstacle; the motion space of the robotic arm includes the expected motion commands of each joint of the robotic arm output by the policy network.

3. The reinforcement learning method for robotic arm path planning based on motion smoothing mechanism according to claim 1, characterized in that: During the training of the policy network, the robotic arm optimizes its policy through continuous interaction with the environment. The environment is modeled using a Markov decision process and consists of the robotic arm's state space, action space, immediate reward, and next-time state space at each moment. First, the robotic arm obtains its current state space through interaction with the environment. Then, it outputs the corresponding action space according to the policy network and executes it in the environment. The environment provides immediate rewards and updates the robotic arm's state space. The updated interaction data is stored as samples in the experience pool. The policy network's policy parameters are updated by sampling from the experience pool.

4. The reinforcement learning method for robotic arm path planning based on motion smoothing mechanism according to claim 1, characterized in that: In the aforementioned priority experience replay mechanism, the importance of each sample is measured by its TD error, and the sampling priority of samples is determined according to the importance of each sample. During sampling, the sampling probability of a sample is calculated based on its priority. ， in, The sampling probability of the sample. For the priority of sample i, Represents the priority of all samples. Represents the total number of samples.

5. The reinforcement learning method for robotic arm path planning based on motion smoothing mechanism according to claim 4, characterized in that: The sampling probability of a sample is corrected and the gradient is updated by using importance sampling weights.

Images