Mechanical arm trajectory optimization method based on deep reinforcement learning

Abstract

The invention relates to a mechanical arm track optimization method based on deep reinforcement learning. The mechanical arm track optimization method comprises the following steps that firstly, kinematics modeling is conducted on a mechanical arm; 2, multi-objective optimization problem modeling is carried out, trajectory planning of the mechanical arm is regarded as a multi-objective optimization problem, and the main objective of optimization comprises three aspects of trajectory precision, trajectory smoothness and energy consumption; and 3, optimizing a PPO reinforcement learning algorithm by using a near-end strategy, and designing a trajectory planning algorithm.

Description

Technical Field

[0001] The invention belongs to the technical field of robot trajectory planning, and in particular relates to a robot arm trajectory optimization method. Background Art

[0002] A robotic arm is a device used to perform various automated tasks. It usually has multiple degrees of freedom and can achieve accurate path planning and motion control in complex environments. Traditional robotic arm motion planning methods mainly rely on teaching methods, that is, the robotic arm is guided step by step through manual operation and its motion trajectory is recorded. However, although the teaching method can achieve basic motion control of the robotic arm, it is difficult to achieve adaptive and efficient planning in multi-task scenarios and dynamic environments. Especially when facing complex tasks, the teaching method relies on manual operation, the operation process is cumbersome and time-consuming, lacks flexibility, and cannot meet the needs of modern industry for automation and diversified operations. The trajectory planning of the robotic arm not only needs to consider obstacles and target points, but also needs to solve the path optimization problem of multiple degrees of freedom in high-dimensional space. Compared with the path planning of mobile robots in two-dimensional or three-dimensional space, the trajectory planning of the robotic arm is usually carried out in a high-dimensional configuration space. This high-dimensional planning problem increases the computational complexity and is prone to problems such as uneven motion and vibration. Therefore, how to improve the efficiency of trajectory planning and the smoothness of the trajectory while considering energy consumption, the multi-objective trajectory planning problem of the robotic arm is gradually replacing the single optimal problem and becoming a challenging hot issue in the field of robotic arm control.

[0003] Researchers at home and abroad have conducted a lot of research on the multi-objective trajectory planning problem of robotic arms. Scholars from the Royal Institute of Technology in Sweden studied a trajectory competition multi-objective particle swarm optimization algorithm to solve the trajectory planning problem of collaborative robots from three aspects: time, energy and pulse optimization. (Journal: Procedia Manufacturing; Author: S.Liu, Y.Wang, XVWang, L.Wang; Publication Year: 2018; Article Title: Energy-efficient trajectory planning for an industrial robot using a multi-objective optimisation approach; Pages: 517-525); Scholars from the Beijing University of Aeronautics and Astronautics proposed an innovative time-energy consumption optimization trajectory planning dynamic identification method, and verified the effectiveness of the method using simulation and real experiments. (Journal: Industrial Robot: An International Journal; Author: R. Hou, J. Niu, Y. Guo, T. Ren, B. Han, X. Yu, Q. Ma, J. Wang, R. Qi; Publication Year: 2022; Article Title: Multi-objective optimal trajectory planning of customized industrial robot based on reliable dynamic identification for improving control accuracy); Scholars from the Indian National Institute of Technology studied a multi-objective ant lion optimization algorithm to obtain the optimal trajectory by minimizing the time-torque ratio. (Journal: Journal of the Institution of Engineers (India): Series C; Author: A. Rout, GB Mahanta, D. Bbvl, BB Biswal; Publication Year: 2020; Article Title: Kinematic and dynamic optimal trajectory planning of industrial robot using improved multi-objective ant lion optimizer; Pages: 559-569). Scholars from South China University of Technology used the NSGA-II algorithm to propose a time-pulse integrated optimal trajectory planning method for industrial robots.(Journal: Mechanism and Machine Theory; Author: J. Huang, P. Hu, K. Wu, M. Zeng; Publication Year: 2018; Article Title: Optimal time-jerk trajectory planning for industrial robots; Pages: 530-544).

[0004] The existing technology has the following disadvantages in multi-objective optimization:

[0005] 1) Low trajectory accuracy: Traditional algorithms often focus on a certain goal, such as the shortest path or minimum energy consumption, and it is difficult to take into account multiple performance indicators.

[0006] 2) High computational complexity: In a dynamic environment, the existing algorithms take a long time to compute, making it difficult to achieve real-time planning.

[0007] 3) Poor trajectory smoothness: In the planning process of some algorithms, trajectory smoothness is difficult to ensure, resulting in large vibrations during the movement of robot joints. Summary of the invention

[0008] The purpose of the present invention is to address the limitations of traditional trajectory planning methods in complex dynamic environments. It is difficult to meet the accuracy requirements while taking into account energy consumption and motion smoothness. A robot arm trajectory optimization method based on deep reinforcement learning is provided to achieve comprehensive optimization of accuracy, smoothness and energy consumption, while improving the real-time performance of planning. The present invention combines deep reinforcement learning (PPO algorithm) and a multi-objective optimization model to comprehensively consider trajectory accuracy, energy consumption and smoothness, so that the robot can dynamically adjust the trajectory according to environmental changes in real-time tasks, ensuring that the robot can not only accurately reach the target position when performing tasks, but also reduce energy consumption and maintain smooth movement, reducing vibration and impact caused by joint movement. The technical solution is as follows:

[0009] A robot arm trajectory optimization method based on deep reinforcement learning includes the following steps:

[0010] Step 1: Kinematic modeling of the robotic arm;

[0011] Step 2: Modeling of multi-objective optimization problem,The trajectory planning of the robot is regarded as a multi-objective optimization problem. The main optimization objectives include three aspects: trajectory accuracy, trajectory smoothness and energy consumption;

[0012] Step 3: Use the proximal strategy to optimize the PPO reinforcement learning algorithm and design a trajectory planning algorithm.

[0013] Furthermore, the method of step one is as follows: each link of the robot arm is characterized by four kinematic parameters, two of which describe the link itself, and the other two represent the connection relationship between the links. The kinematic model of the robot arm is described using the Denavit-Hartenberg method. For the robot arm, a transformation formula for the link from the base to the end effector is established using a homogeneous transformation matrix. The kinematic equation is used to convert the angles of each joint of the robot arm into the position and posture of the end effector. The goal of the optimization process is to achieve the movement of the end effector along a predetermined path by controlling the angle of each joint; the joint angle is calculated through the position and posture of the end effector, and the joint is precisely controlled in motion. An inverse kinematic equation is established to link the velocity of the end effector to the angular velocity of the joint, thereby achieving conversion between joint space and Cartesian space.

[0014] Furthermore, in step 2, the overall optimization objective function is set:

[0015]

[0016] Among them, θ = [θ1, θ2, θ3, θ4, θ5, θ6,…, θ n ] is the joint angle of the robot arm, f k (θ) is the kth objective function, λ k Indicates the weight coefficient corresponding to the function, h kj (θ) represents the jth constraint condition of the kth joint of the robot arm, and the total number of robot arm joints is n;

[0017] The trajectory accuracy refers to the distance error between the actual trajectory of the end effector of the robot and the expected trajectory. The trajectory accuracy optimization objective function is defined as:

[0018] f a (θ)=||p d -p|| 2 (5)

[0020] Among them, p d =[x d ,y d , z d ], p = [x, y, z] represents the three-dimensional coordinate position of the end effector of the desired trajectory and the actual trajectory in the Euclidean space, and the optimization goal is to minimize the error between the desired position and the actual position;

[0021] The trajectory smoothness optimization objective function is defined as follows:

[0022] in, is the velocity of the joint, is the acceleration of the joint, λv and λ a is the corresponding weight coefficient;

[0023] The energy consumption optimization considers the square of the angle change and the sum of the joint torque in discrete unit time, and the energy consumption cost function is expressed as:

[0024]

[0025] Among them, Δθ k is the angle change of the kth joint, τ k is the torque of the kth joint.

[0026] Furthermore, step three includes the design of observation space and action space. The state variables included in the observation space include joint angles, joint velocities, position and posture information of the end effector; the state variables include the current state q of each joint of the robot arm start 、Current location p c , target state q end and the target position p d , and the Euclidean distance d between the current end pose and the target point e ; When the motion state of the robot exceeds its own constraints, the specific joint state beyond the observation space will be adjusted to its maximum limit value; The setting of the action space and the speed of each joint Correspondingly, the state variables in the observation space are updated with each provided action.

[0027] Furthermore, step three includes reward function design, taking into account the time factor and the inherent complexity of the robot planning task, and proposes a reward function:

[0028]

[0029] Where t represents the number of iterations, and the component of the reward function They are related to accuracy, smoothness and energy consumption respectively, and are obtained from the optimization function in the previous article:

[0030]

[0031]

[0032]

[0033]

[0034] Among them, w a and σ a is the weight coefficient of the accuracy reward, w s , v , acc is the weight coefficient of the smooth reward, we Represents the weight coefficient of energy consumption reward.

[0035] Furthermore, additional rewards are provided when the predetermined minimum distance between the current and desired end points is reached.

[0036]

[0037] Furthermore, step three includes the policy update principle: setting and optimizing a truncated objective function, which limits the amplitude of the policy update and prevents the policy from changing drastically during the update process.

[0038] Furthermore, step three includes the reinforcement learning training process:

[0039] a. Sampling and Interaction

[0040] In each training cycle, according to the current strategy π of the robot θ Interact with the simulation environment and collect multiple state transition samples. The state transition sample format is {s t ,a t ,r t ,s t+1}, as the input for subsequent strategy updates; where s t is the current state, a t It is an action. t is the reward after executing the action, s t+1 is the next state, and the state transition data is used as a training sample in the reinforcement learning process;

[0041] b. Actor-Critic Model, including

[0042] Actor Network: According to the current state s t , generate a probability distribution and select the best action a from it t , thereby determining the next operation of the robot arm; in this process, the PPO algorithm uses policy gradient optimization to enable the actor network to continuously improve the strategy of selecting actions;

[0043] Critic network: According to the current state s t and reward r t To estimate the value of the state V(s t ), and calculate the advantage function of the action; the critic network is used to give an evaluation of the current strategy;

[0044] c. Policy Update

[0045] PPO measures the current action a by calculating the advantage function t The relative merits of other actions, and feed this information back to the actor network to guide its updated strategy;

[0046] d. Value Update

[0047] The critic network calculates the current state s t The value of V(s t ) and updates its parameters based on the actual accumulated rewards.

[0048] Furthermore, step three includes a decay mechanism, which dynamically adjusts the training cycle step size to improve

[0049] High computational efficiency:

[0050] T new =max(T old α,T min )

[0051] Where: T new is the updated training cycle step size, T old is the current training cycle step size, α is a decay coefficient, less than 1, which controls the speed of step size reduction, T min is the minimum step size.

[0052] Compared with the existing trajectory planning methods based on RRT or PSO, the advantages of the present invention include:

[0053] Strong adaptability: This method can achieve real-time adjustment to complex dynamic environments without relying on precise mathematical models.

[0054] Multi-objective optimization capability: Through deep reinforcement learning, multiple performance indicators such as accuracy, smoothness, and energy consumption can be optimized simultaneously;

[0055] Efficiency: After the attenuation mechanism is introduced, the algorithm converges faster and can effectively reduce the training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of DH parameter representation of the robot arm

[0057] Figure 2 Schematic diagram of the trajectory planning algorithm method framework based on the PPO algorithm DETAILED DESCRIPTION

[0058] The multi-objective robot arm trajectory optimization method based on deep reinforcement learning proposed in the present invention aims to achieve comprehensive optimization of trajectory accuracy, smoothness and energy consumption through online learning and real-time adjustment strategies. This method first establishes a kinematic model of a six-degree-of-freedom robot arm, and combines the forward kinematics and inverse kinematics models to describe it through the relationship between the joint angle and the end effector position. Then, the proximal policy optimization (PPO) algorithm in deep reinforcement learning is used for multi-objective optimization, and the accuracy, smoothness and energy consumption of the trajectory are used as optimization objective functions, and the robot's learning is guided by designing a suitable reward function. A decreasing mechanism is introduced during the training process, so that the algorithm can quickly approach the optimal solution, while ensuring that the generated trajectory is smooth, low in energy consumption and high in accuracy.

[0059] The specific steps are as follows

[0060] Step 1: Kinematic modeling of the robotic arm

[0061] In order to accurately control the position of the robot arm, a kinematic model of the robot arm must be established. Each link of the robot arm can be characterized by four kinematic parameters, two of which describe the link itself, and the other two represent the connection relationship between the links. The kinematic model of the robot arm is usually described using the Denavit–Hartenberg (DH) method. The DH parameters are as follows: Figure 1 shown.

[0062] For a six-DOF robot, the transformation formula of the connecting rod from the base to the end effector can be expressed by the homogeneous transformation matrix q = [q1, q2, ..., q n ]express:

[0063]

[0064] in Defined as:

[0065]

[0066] is the transformation matrix from joint i-1 to joint i, q is the joint angle vector, and T represents the transformation matrix. This kinematic equation is used to convert the angles of each joint of the robot arm into the position and posture of the end effector. This model is crucial for subsequent trajectory optimization because the goal of the optimization process is to control the angle of each joint and ultimately achieve the movement of the end effector along a predetermined path.

[0067] Since the end effector of the robot arm has a specific task configuration, it is usually necessary to infer a specific configuration space based on the position space. This process is called inverse kinematics. The joint angle can be inferred from the position and posture of the end effector, thereby accurately controlling the motion of the joint. The inverse kinematics equation is:

[0068]

[0069] Among them, J(q) is the Jacobian matrix, which relates the velocity of the end effector to the angular velocity of the joint. Through this modeling, it is possible to accurately convert between joint space and Cartesian space, providing a mathematical basis for subsequent trajectory optimization.

[0070] Step 2: Modeling of multi-objective optimization problems

[0071] In this paper, the trajectory planning of the robot is regarded as a multi-objective optimization problem. The main optimization objectives include three aspects: trajectory accuracy, smoothness and energy consumption. We integrate these objectives into an overall optimization objective function:

[0072]

[0073] Among them, θ = [θ1, θ2, θ3, θ4, θ5, θ6,…, θ n ] is the joint angle of the robot arm, f i (θ) is the i-th objective function, λ i Indicates the weight coefficient corresponding to the function, h ij (θ) represents the jth constraint condition of the i-th joint of the robot arm, and the total number of robot arm joints is n.

[0074] ①Trajectory accuracy optimization

[0075] Trajectory accuracy refers to the distance error between the actual trajectory of the end effector of the robot and the expected trajectory. In order to ensure that the end effector can accurately reach the target position, the trajectory accuracy optimization objective function is defined:

[0076] f a (θ)=||p d -p|| 2 (5)

[0078] Among them, p d =[x d ,y d , z d ], p = [x, y, z] represents the three-dimensional coordinate position of the end effector of the desired trajectory and the actual trajectory in the Euclidean space respectively. The optimization goal is to minimize the error between the desired position and the actual position, so as to improve the accuracy of the trajectory.

[0079] ②Trajectory smoothness optimization

[0080] In order to achieve smooth trajectory and reduce the vibration impact caused by the movement of each joint of the robot, the smoothness optimization objective function is defined as follows:

[0081]

[0082] in, is the velocity of the kth joint, is the acceleration of the kth joint, λ v and λ a is the corresponding weight coefficient. By optimizing the smoothness function, the vibration and sharp changes of the robot joints during movement can be reduced to ensure the continuity and smoothness of the trajectory.

[0083] ③Energy consumption optimization

[0084] The energy consumption cost function mainly deals with the energy consumption related to the joint motion of the robot arm. Considering the square of the angle change and the sum of the joint torque in discrete unit time, the energy consumption cost function is expressed as:

[0085]

[0086] Among them, Δθ k is the angle change of the kth joint, τ k is the torque of the kth joint. By optimizing the energy consumption target, the energy consumption of the joint in performing the task can be reduced, the working time of the equipment can be extended, and the execution efficiency of the task can be improved.

[0087] Step 3: Design of trajectory planning algorithm based on reinforcement learning

[0088] ①Observation space and action space

[0089] Using the PPO reinforcement learning algorithm, the states of the observation space include information such as joint angles, joint velocities, and the position and posture of the end effector. Specifically, the state variables include the current states q of the six joints of the robot arm. start 、Current location p c , target state q end and the target position p d , and the Euclidean distance d between the current end pose and the target point e Furthermore, when the motion state of the robot exceeds its own constraints, the specific joint states beyond the observation space will be adjusted to their maximum limit values.

[0090] Setting the action space and the speed of each joint Correspondingly, the state variables in the observation space are updated with each provided action.

[0091] Taking into account the time factor and the inherent complexity of the robot planning task, the following reward function is proposed:

[0092]

[0093] Where t represents the number of iterations, and the component of the reward function They are related to accuracy, smoothness and energy consumption respectively, and are obtained from the optimization function in the previous article:

[0094]

[0095]

[0096]

[0097]

[0098] Among them, w a and σ a is the weight coefficient of the accuracy reward, w s , v , acc is the weight coefficient of the smooth reward, w e Represents the weight coefficient of energy consumption reward.

[0099] Provide additional rewards when the current training performance reaches a specified threshold, such as when a predetermined minimum distance between the current and desired endpoints is reached

[0100]

[0101] ③Strategy update principle

[0102] In the present invention, the trajectory planning problem is modeled as a reinforcement learning problem, and the proximal policy optimization (PPO) algorithm is used to update the policy. PPO is an algorithm based on policy gradient. Through interaction with the environment, the robot arm gradually optimizes the trajectory, while limiting the update amplitude of the policy to ensure that the policy does not fluctuate too much during the update process, thereby improving the stability of learning.

[0103] The key to policy updating is to optimize a truncated objective function, which limits the magnitude of policy updates and prevents drastic changes in the policy during the update process. The policy update optimization objective function is as follows:

[0104]

[0105] in:

[0106] · It is the probability ratio of the new and old strategies, indicating the degree of change of the current strategy relative to the previous round of strategy.

[0107] ·π θ (a t |s t) is the current strategy selection action a t The probability of is the probability of choosing the same action under the previous strategy.

[0108] · It is the advantage function, which measures the quality of the current action relative to the benchmark strategy. The advantage function is used to guide the robot arm to choose a better behavior in the strategy update.

[0109] ∈ is a hyperparameter that controls the policy update amplitude, which is used to prevent the policy from updating too quickly or over-adjusting.

[0110] ④Reinforcement learning training process

[0111] a. Sampling and Interaction

[0112] In each training cycle, the robot arm follows the current policy π θ Interact with the simulation environment and collect multiple state transition samples. The sample format is {s t ,a t ,r t ,s t+1}, these data will be saved in an experience pool as input for subsequent strategy updates. t is the current state, a t It is an action. t is the reward after executing the action, s t+1 is the next state, and these state transition data will be used as training samples in the reinforcement learning process.

[0113] b. Actor-Critic Model

[0114] Actor Neural Network: This neural network is based on the current state s t , generate a probability distribution and select the best action a from it t , thereby determining the next operation of the robot arm. In this process, the PPO algorithm uses policy gradient optimization to enable the actor network to continuously improve the strategy of selecting actions.

[0115] Critic Neural Network: The critic network is based on the current state s t and reward r t To estimate the value of the state V(s t ), and calculate the advantage of the action That is, how good the action is relative to other possible actions. The main task of the critic network is to give an evaluation of the current strategy.

[0116] c. Policy Update

[0117] PPO measures the current action a by calculating the advantage function t The pros and cons of one action relative to other actions are measured and this information is fed back to the actor network to guide its updated strategy.

[0118] d. Value Update

[0119] The critic network calculates the current state s t The value of V(s t ), and update its parameters according to the actual accumulated rewards. Loss function L V (θ) is used to minimize the difference between the estimated value and the actual return.

[0120] ⑤Establish attenuation mechanism

[0121] By gradually shortening the step size of each training episode, the robot can focus more on optimizing the details of the trajectory during later training instead of conducting long and ineffective exploration.

[0122] The decay mechanism improves computational efficiency by dynamically adjusting the training step size. In the early stages, the robot needs to perform a lot of exploration to learn the basic strategy for trajectory planning, so a longer episode step size is set. However, as training progresses, the robot gradually learns how to plan trajectories, and the exploration step size can be reduced.

[0123] The dynamic adjustment is performed through the following formula:

[0124] T new =max(T old α,T min )

[0125] Where: T new is the updated episode step length, T old is the current episode step size, α is a decay coefficient, usually less than 1, which controls the speed of step size reduction, T min It is the minimum step size set to ensure that the step size is not too short, so that the robot can perform necessary fine-tuning optimization in the later stages of training. This mechanism helps the robot reduce unnecessary exploration in the later stages of training and focus on optimizing the details of the trajectory.

Claims

1. A robot arm trajectory optimization method based on deep reinforcement learning, comprising the following steps: Step 1: Kinematic modeling of the robotic arm; Step 2: Modeling of multi-objective optimization problem,The trajectory planning of the robot is regarded as a multi-objective optimization problem. The main optimization objectives include three aspects: trajectory accuracy, trajectory smoothness and energy consumption; Step 3: Use the proximal strategy to optimize the PPO reinforcement learning algorithm and design a trajectory planning algorithm.

2. The robot arm trajectory optimization method according to claim 1, characterized in that: The method of step one is as follows: each link of the robot arm is characterized by four kinematic parameters, two of which describe the link itself, and the other two represent the connection relationship between the links. The kinematic model of the robot arm is described using the Denavit–Hartenberg method. For the robot arm, the transformation formula of the link from the base to the end effector is established using the homogeneous transformation matrix. The kinematic equation is used to convert the angles of each joint of the robot arm into the position and posture of the end effector. The goal of the optimization process is to achieve the movement of the end effector along a predetermined path by controlling the angle of each joint; the joint angle is calculated through the position and posture of the end effector, and the joint is precisely controlled in motion. The inverse kinematic equation is established to link the velocity of the end effector with the angular velocity of the joint to achieve conversion between joint space and Cartesian space.

3. The robot arm trajectory optimization method according to claim 1, characterized in that: In step 2, set the overall optimization objective function: Among them, θ = [θ1, θ2, θ3, θ4, θ5, θ6,…,θ n ] is the joint angle of the robot arm, f k (θ) is the kth objective function, λ k Indicates the weight coefficient corresponding to the function, h kj (θ) represents the jth constraint condition of the kth joint of the robot arm, and the total number of robot arm joints is n; The trajectory accuracy refers to the distance error between the actual trajectory of the end effector of the robot and the expected trajectory. The trajectory accuracy optimization objective function is defined as: f a (θ)=||p d -p|| 2 (5) Among them, p d =[x d ,y d ,z d ], p = [x, y, z] represents the three-dimensional coordinate position of the end effector of the desired trajectory and the actual trajectory in the Euclidean space, and the optimization goal is to minimize the error between the desired position and the actual position; The trajectory smoothness optimization objective function is defined as follows: in, is the velocity of the joint, is the acceleration of the joint, λ v and λ a is the corresponding weight coefficient; The energy consumption optimization considers the square of the angle change and the sum of the joint torque in discrete unit time, and the energy consumption cost function is expressed as: Among them, Δθ k is the angle change of the kth joint, τ k is the torque of the kth joint.

4. The robot arm trajectory optimization method according to claim 3, characterized in that: Step 3 includes the design of observation space and action space. The state variables included in the observation space include joint angles, joint velocities, and the position and posture information of the end effector. The state variables include the current state q of each joint of the robot arm. start 、Current location p c , target state q end and the target position p d , and the Euclidean distance d between the current end pose and the target point e ;When the motion state of the robot arm exceeds its own constraints, the specific joint state beyond the observation space will be adjusted to its maximum limit value; Setting the action space and the speed of each joint Correspondingly, the state variables in the observation space are updated with each provided action.

5. The robot arm trajectory optimization method according to claim 3, characterized in that: Step three involves reward function design, Taking into account both the time factor and the inherent complexity of the robot planning task, a reward function is proposed: Where t represents the number of iterations, and the component of the reward function They are related to accuracy, smoothness and energy consumption respectively, and are obtained from the optimization function in the previous article: Among them, w a and σ a is the weight coefficient of the accuracy reward, w s , v , acc is the weight coefficient of the smooth reward, w e Represents the weight coefficient of energy consumption reward.

6. The robot arm trajectory optimization method according to claim 5, characterized in that: Provides additional rewards when a predetermined minimum distance is reached between the current and desired end point distances 7. The robot arm trajectory optimization method according to claim 1, characterized in that: Step 3 contains the principle of policy updating: setting and optimizing a truncated objective function that limits the magnitude of policy updates and prevents drastic changes in the policy during the update process.

8. The robot arm trajectory optimization method according to claim 1, characterized in that: Step 3 includes the reinforcement learning training process: a. Sampling and Interaction In each training cycle, according to the current strategy π of the robot θ Interact with the simulation environment and collect multiple state transition samples. The state transition sample format is {s t ,a t ,r t ,s t+1 }, as the input for subsequent strategy updates; where s t is the current state, a t It is an action. t is the reward after executing the action, s t+1 is the next state, and the state transition data is used as a training sample in the reinforcement learning process; b. Actor-Critic Model, including Actor Network: According to the current state s t , generate a probability distribution and select the best action a from it t , thereby determining the next operation of the robot arm; in this process, the PPO algorithm uses policy gradient optimization to enable the actor network to continuously improve the strategy of selecting actions; Critic network: According to the current state s t and reward r t To estimate the value of the state V(s t ), and calculate the advantage function of the action; the critic network is used to give an evaluation of the current strategy; c. Policy Update PPO measures the current action a by calculating the advantage function t The relative merits of other actions, and feed this information back to the actor network to guide its updated strategy; d. Value Update The critic network calculates the current state s t The value of V(s t ) and updates its parameters based on the actual accumulated rewards.

9. The robot arm trajectory optimization method according to claim 1, characterized in that: Step 3 includes a decay mechanism that dynamically adjusts the training cycle step size to improve computational efficiency through the following formula: T new =max(T old ·α,T min ) Where: T new is the updated training cycle step size, T old is the current training cycle step size, α is a decay coefficient, less than 1, which controls the speed of step size reduction, T min is the minimum step size.

Images