A method for trajectory optimization of a robot arm based on deep reinforcement learning

Abstract

The application relates to a kind of trajectory optimization methods of mechanical arm based on deep reinforcement learning, comprising the following steps: step one: mechanical arm kinematics modeling;Step two: multi-objective optimization problem modeling, the trajectory planning of mechanical arm is regarded as multi-objective optimization problem, and the main optimization target includes three aspects: trajectory accuracy, trajectory smoothness and energy consumption;Step three: using proximal policy optimization PPO reinforcement learning algorithm, trajectory planning algorithm is designed.

Description

Technical Field

[0001] This invention belongs to the field of robot trajectory planning technology, and specifically relates to a method for optimizing the trajectory of a robotic arm. Background Technology

[0002] A robotic arm is a device used to perform various automated tasks. It typically possesses multiple degrees of freedom and can achieve precise path planning and motion control in complex environments. Traditional robotic arm motion planning methods primarily rely on teaching, where the robotic arm is gradually guided by human intervention, and its trajectory is recorded. However, while teaching can achieve basic motion control, it struggles to achieve efficient adaptive planning in multi-task scenarios and dynamic environments. Especially when facing complex tasks, teaching relies on manual operation, which is cumbersome, time-consuming, and lacks flexibility, failing to meet the demands of modern industry for automation and diverse operations. Robotic arm trajectory planning not only needs to consider obstacles and target points but also requires solving multi-degree-of-freedom path optimization problems in high-dimensional space. Compared to path planning for mobile robots in two- or three-dimensional space, robotic arm trajectory planning is usually performed in a high-dimensional configuration space. This high-dimensional planning problem increases computational complexity and is prone to problems such as motion unevenness and vibration. Therefore, improving the efficiency and smoothness of trajectory planning while also considering energy consumption is crucial. The multi-objective trajectory planning problem for robotic arms is gradually replacing the single optimal problem and becoming a challenging and hot topic in the field of robotic arm control.

[0003] Numerous studies have been conducted by researchers both domestically and internationally on the multi-objective trajectory planning problem for robotic arms. Scholars at the Royal Institute of Technology in Sweden developed a trajectory competition multi-objective particle swarm optimization algorithm to address the trajectory planning problem for collaborative robots from three aspects: time, energy, and impulse optimization. (Journal: Procedia Manufacturing; Authors: S. Liu, Y. Wang, XV Wang, L. Wang; Publication Date: 2018; Title: Energy-efficient trajectory planning for an industrial robot using a multi-objective optimization approach; Pages: 517-525); Researchers at Beijing University of Aeronautics and Astronautics proposed an innovative time-energy consumption optimized trajectory planning dynamic recognition method and verified its effectiveness using simulations and real experiments. (Journal: Industrial Robot: An International Journal; Authors: R. Hou, J. Niu, Y. Guo, T. Ren, B. Han, X. Yu, Q. Ma, J. Wang, R. Qi; Publication Date: 2022; Title: Multi-objective optimal trajectory planning of customized industrial robot based on reliable dynamic identification for improving control accuracy); Researchers at the Indian National Institute of Technology have studied a multi-objective ant lion optimization algorithm to obtain the optimal trajectory by minimizing the time-torque ratio. (Journal: Journal of the Institution of Engineers (India): Series C; Authors: A. Rout, GB Mahanta, D. Bbvl, BBB Biswal; Publication Date: 2020; Title: Kinematic and dynamic optimal trajectory planning of industrial robot using improved multi-objective ant lion optimizer; Pages: 559-569). Scholars from South China University of Technology have proposed a time-pulse integrated optimal trajectory planning method for industrial robots using the NSGA-II algorithm.(Journal: Mechanism and Machine Theory; Authors: J. Huang, P. Hu, K. Wu, M. Zeng; Publication Date: 2018; Article Title: Optimal time-jerktrajectoryplanning for industrial robots; Pages: 530-544).

[0004] Existing technologies have the following drawbacks in multi-objective optimization:

[0005] 1) Low trajectory accuracy: Traditional algorithms often focus on a single objective, such as the shortest path or minimum energy consumption, making it difficult to balance multiple performance metrics.

[0006] 2) High computational complexity: In dynamic environments, existing algorithms take a long time to compute, making it difficult to achieve real-time planning.

[0007] 3) Poor trajectory smoothness: Some algorithms have difficulty ensuring trajectory smoothness during the planning process, resulting in large vibrations during robot joint movement. Summary of the Invention

[0008] The purpose of this invention is to address the limitations of traditional trajectory planning methods in complex dynamic environments, which struggle to simultaneously meet accuracy requirements while considering energy consumption and motion smoothness. This invention provides a deep reinforcement learning-based robotic arm trajectory optimization method to achieve comprehensive optimization of accuracy, smoothness, and energy consumption, while improving the real-time performance of the planning. By combining deep reinforcement learning (PPO algorithm) and a multi-objective optimization model, this invention comprehensively considers trajectory accuracy, energy consumption, and smoothness, enabling the robot to dynamically adjust its trajectory according to environmental changes during real-time tasks. This ensures that the robot not only accurately reaches the target position but also reduces energy consumption and maintains stable motion, minimizing vibration and impact caused by joint movements. The technical solution is as follows:

[0009] A method for optimizing the trajectory of a robotic arm based on deep reinforcement learning includes the following steps:

[0010] Step 1: Kinematic modeling of the robotic arm;

[0011] Step 2: Modeling a multi-objective optimization problem. The trajectory planning of the robotic arm is regarded as a multi-objective optimization problem. The main optimization objectives include three aspects: trajectory accuracy, trajectory smoothness, and energy consumption.

[0012] Step 3: Optimize the PPO reinforcement learning algorithm using a proximal strategy and design a trajectory planning algorithm.

[0013] Further, the method in step one is as follows: Each link of the robotic arm is characterized by four kinematic parameters, two of which describe the link itself and the other two represent the connection relationship between the links. The kinematic model of the robotic arm is described using the Denavit-Hartenberg method. For the robotic arm, a transformation formula from the base to the end effector is established using a homogeneous transformation matrix. The kinematic equations are used to convert the angles of each joint of the robotic arm into the position and orientation of the end effector. The goal of the optimization process is to achieve the movement of the end effector along a predetermined path by controlling the angle of each joint. The joint angles are calculated by the position and orientation of the end effector, and the joints are precisely controlled. An inverse kinematic equation is established to link the velocity of the end effector with the angular velocity of the joint, realizing the conversion between joint space and Cartesian space.

[0014] Furthermore, in step two, the overall optimization objective function is defined as follows:

[0015]

[0016] Among them, θ = [θ1, θ2, θ3, θ4, θ5, θ6,…, θ n [f] represents the joint angle of the robotic arm. k (θ) is the k-th objective function, λ k h represents the weight coefficient corresponding to this function. kj (θ) represents the j-th constraint of the k-th joint of the robotic arm, and the total number of joints of the robotic arm is denoted as n;

[0017] The trajectory accuracy refers to the distance error between the actual trajectory and the desired trajectory of the robotic arm's end effector. The objective function for trajectory accuracy optimization is defined as follows:

[0018] f a (θ)=||p d -p|| 2 (5)

[0019] Where, p d =[x d y d , z d ], p = [x, y, z] represent the three-dimensional coordinate positions of the end effectors of the desired trajectory and the actual trajectory in Euclidean space, respectively. The optimization objective is to minimize the error between the desired position and the actual position;

[0020] The objective function for optimizing trajectory smoothness is defined as follows:

[0021] in, It's the speed of the joint. It is the acceleration of the joint, λv and λ a These are the corresponding weighting coefficients;

[0022] The energy consumption optimization considers the sum of the square of the angle change and the joint torque per discrete unit time. The energy consumption cost function is expressed as:

[0023]

[0024] Where, Δθ k τ is the change in angle of the k-th joint. k It is the torque of the k-th joint.

[0025] Furthermore, step three includes the design of the observation space and motion space. The state variables included in the observation space include joint angles, joint velocities, and the position and orientation information of the end effector; the state variables include the current state q of each joint of the robotic arm. start Current position p c Target state q end and target position p d And the Euclidean distance d between the current pose and the target point. e When the robotic arm's motion exceeds its own constraints, specific joint states that are outside the observation space will be adjusted to their maximum limits; the motion space setting and the speed of each joint... Correspondingly, the state variables in the observation space are updated with each provided action.

[0026] Furthermore, step three involves designing a reward function, taking into account both time factors and the inherent complexity of the robotic arm's planning task, and proposing the following reward function:

[0027]

[0028] Where t represents the iteration number and the components of the reward function. These are related to accuracy, smoothness, and energy consumption, respectively, and are obtained from the optimization function mentioned earlier:

[0029]

[0030]

[0031]

[0032] Among them, w a and σ a w is the weighting coefficient for accuracy rewards. s , λ v , λ acc To smooth the weighting coefficients of the rewards, w e This represents the weighting coefficient for energy consumption rewards.

[0033] Furthermore, an additional reward is provided when a predetermined minimum distance is reached between the current and the desired destination distance.

[0034]

[0035] Furthermore, step three includes the policy update principle: setting and optimizing the truncation objective function, which limits the magnitude of policy updates and prevents drastic changes in the policy during the update process.

[0036] Furthermore, step three includes a reinforcement learning training process:

[0037] a. Sampling and Interaction

[0038] In each training cycle, based on the robotic arm's current strategy π θ Interact with the simulation environment to collect multiple state transition samples. The state transition sample format is {s t ,a t ,r t ,s t+1}, as input for subsequent policy updates; where s t This is the current state, a t It's an action, r t It is the reward after performing the action, s t+1 The next state is the state transition data, which serves as the training sample in the reinforcement learning process.

[0039] b. Actor-Critic Model, including

[0040] Actor Network: Based on current status t Generate a probability distribution and select the optimal action a from it. t This determines the next action of the robotic arm; in this process, the PPO algorithm optimizes the policy gradient, enabling the actor network to continuously improve its action selection strategy.

[0041] Critics Network: Based on the current state s t and reward r t To estimate the value V(s) of this state t The algorithm calculates the advantage function of the action; the critic network is used to give an evaluation of the current strategy.

[0042] c. Strategy Update

[0043] PPO measures the current action a by calculating the advantage function. t The advantages and disadvantages of different actions are compared, and this information is fed back to the actor network to guide them in updating their strategies;

[0044] d. Value Renewal

[0045] The critic network calculates the current state s t Value V(s) t ), and update its parameters based on the actual cumulative rewards.

[0046] Furthermore, step three includes a decay mechanism, which dynamically adjusts the training cycle step size using the following formula to improve performance.

[0047] High computational efficiency:

[0048] T new =max(T) old ·α,T min )

[0049] Wherein: T new It is the updated training cycle step size, T old This is the current training cycle step size, α is a decay coefficient, less than 1, controlling the rate at which the step size decreases, and T. min It is the set minimum step size.

[0050] Compared with existing trajectory planning methods based on RRT or PSO, the advantages of this invention include:

[0051] • High adaptability: This method can achieve real-time adjustment to complex dynamic environments without relying on precise mathematical models.

[0052] • Multi-objective optimization capability: Through deep reinforcement learning, multiple performance metrics, such as accuracy, smoothness, and energy consumption, can be optimized simultaneously.

[0053] • Efficiency: With the introduction of the decay mechanism, the algorithm converges faster and can effectively reduce training time. Attached Figure Description

[0054] Figure 1 Schematic diagram of DH parameters of robotic arm

[0055] Figure 2 Schematic diagram of the trajectory planning algorithm framework based on PPO algorithm Detailed Implementation

[0056] This invention proposes a multi-objective robotic arm trajectory optimization method based on deep reinforcement learning, aiming to achieve comprehensive optimization of trajectory accuracy, smoothness, and energy consumption through online learning and real-time policy adjustment. The method first establishes a kinematic model of a six-DOF robotic arm, combining forward and inverse kinematic models, and describes it through the relationship between joint angles and end effector positions. Then, it utilizes the proximal policy optimization (PPO) algorithm in deep reinforcement learning for multi-objective optimization, using trajectory accuracy, smoothness, and energy consumption as the optimization objective functions, and guiding the robot's learning by designing a suitable reward function. A decrementing mechanism is introduced during training, enabling the algorithm to quickly approach the optimal solution while ensuring that the generated trajectory is smooth, low-energy, and highly accurate.

[0057] The specific steps are as follows:

[0058] Step 1: Kinematic Modeling of the Robotic Arm

[0059] To precisely control the position of a robotic arm, a kinematic model of the arm must be established. Each link of the robotic arm can be characterized by four kinematic parameters: two describing the link itself and the other two representing the connection between links. The kinematic model of the robotic arm is typically described using the Denavit-Hartenberg (DH) method. DH parameters are as follows: Figure 1 As shown.

[0060] For a six-degree-of-freedom robotic arm, the transformation formula from the base to the end effector can be expressed by the homogeneous transformation matrix q = [q1, q2, ..., q]. n ]express:

[0061]

[0062] in Defined as:

[0063]

[0064] is the transformation matrix from joint i-1 to joint i, q is the joint angle vector, and T represents the transformation matrix. This kinematic equation is used to convert the angles of each joint of the robotic arm into the position and orientation of the end effector. This model is crucial for subsequent trajectory optimization because the goal of the optimization process is to ultimately achieve the movement of the end effector along a predetermined path by controlling the angle of each joint.

[0065] Because robotic arm end effectors have specific task configurations, it is usually necessary to infer a specific configuration space from the position space. This process is called inverse kinematics. Joint angles can be calculated from the position and orientation of the end effector, thereby enabling precise motion control of the joints. The inverse kinematics equations are:

[0066]

[0067] Here, J(q) is the Jacobian matrix, which relates the velocity of the end effector to the angular velocity of the joint. This modeling allows for precise transformation between joint space and Cartesian space, providing a mathematical foundation for subsequent trajectory optimization.

[0068] Step 2: Modeling the Multi-Objective Optimization Problem

[0069] In this invention, the trajectory planning of the robotic arm is considered a multi-objective optimization problem. The main optimization objectives include three aspects: trajectory accuracy, smoothness, and energy consumption. We integrate these objectives into a single overall optimization objective function:

[0070]

[0071] Among them, θ = [θ1, θ2, θ3, θ4, θ5, θ6,…, θ n [f] represents the joint angle of the robotic arm. i (θ) is the i-th objective function, λ i h represents the weight coefficient corresponding to this function. ij (θ) represents the j-th constraint of the i-th joint of the robotic arm, and the total number of joints of the robotic arm is denoted as n.

[0072] ①Trajectory accuracy optimization

[0073] Trajectory accuracy refers to the distance error between the actual trajectory and the desired trajectory of the robotic arm's end effector. To ensure that the end effector can accurately reach the target position, a trajectory accuracy optimization objective function is defined:

[0074] f a (θ)=||p d -p|| 2 (5)

[0075] Where, p d =[x d y d , z d Let p = [x, y, z] represent the three-dimensional coordinates of the end effectors of the desired trajectory and the actual trajectory in Euclidean space, respectively. The optimization goal is to minimize the error between the desired position and the actual position, thereby improving the accuracy of the trajectory.

[0076] ② Trajectory smoothness optimization

[0077] To achieve smooth trajectory and reduce vibration and impact caused by the movement of the robotic arm's joints, the smoothness optimization objective function is defined as follows:

[0078]

[0079] in, It is the velocity of the k-th joint. λ is the acceleration of the k-th joint. v and λ a These are the corresponding weighting coefficients. By optimizing the smoothness function, the vibration and abrupt changes of the robotic arm joints during movement can be reduced, ensuring the continuity and smoothness of the trajectory.

[0080] ③ Energy consumption optimization

[0081] The energy consumption cost function primarily deals with energy consumption related to the joint motion of the robotic arm. Considering the square of the angle change and the sum of joint torques per discrete unit time, the energy consumption cost function is expressed as:

[0082]

[0083] Where, Δθ k τ is the change in angle of the k-th joint. k This is the torque of the k-th joint. By optimizing energy consumption targets, the energy consumption of joints during task execution can be reduced, extending the equipment's operating time and improving task execution efficiency.

[0084] Step 3: Design of a Reinforcement Learning-Based Trajectory Planning Algorithm

[0085] ①Observation space and action space

[0086] Using the PPO reinforcement learning algorithm, the states incorporated into the observation space include information such as joint angles, joint velocities, and the position and orientation of the end effector. Specifically, the state variables include the current states q of the six joints of the robotic arm. start Current position p c Target state q end and target position p d And the Euclidean distance d between the current pose and the target point. e Furthermore, when the robotic arm's motion exceeds its own constraints, specific joint states that are outside the observation space will be adjusted to their maximum limit values.

[0087] The setting of the motion space and the speed of each joint Correspondingly, the state variables in the observation space are updated with each provided action.

[0088] Taking into account both the time factor and the inherent complexity of the robotic arm's planning task, the following reward function is proposed:

[0089]

[0090] Where t represents the iteration number and the components of the reward function. These are related to accuracy, smoothness, and energy consumption, respectively, and are obtained from the optimization function mentioned earlier:

[0091]

[0092]

[0093]

[0094] Among them, w a and σ a w is the weighting coefficient for accuracy rewards. s , λ v , λ acc To smooth the weighting coefficients of the rewards, w e This represents the weighting coefficient for energy consumption rewards.

[0095] Additional rewards are provided when the current training performance reaches a specified threshold, such as when a predetermined minimum distance is reached between the current and desired endpoint distances.

[0096]

[0097] ③ Strategy Update Principle

[0098] In this invention, the trajectory planning problem is modeled as a reinforcement learning problem, using the Proximal Policy Optimization (PPO) algorithm to update the policy. PPO is a policy gradient-based algorithm that, through interaction with the environment, allows the robotic arm to progressively optimize the trajectory while limiting the policy update magnitude to ensure that the policy does not fluctuate excessively during the update process, thereby improving the stability of the learning process.

[0099] The key to policy updates is optimizing a truncated objective function that limits the magnitude of policy updates, preventing drastic changes in the policy during the update process. The policy update optimization objective function is as follows:

[0100]

[0101] in:

[0102] · It is the probability ratio of the new and old strategies, representing the degree of change in the current strategy relative to the previous strategy.

[0103] ·π θ (a t |s t ) is the current strategy selection action a t The probability, It represents the probability of choosing the same action under the previous strategy.

[0104] · It is the advantage function, which measures the quality of the current action relative to the baseline policy. The advantage function is used to guide the robotic arm to choose a better behavior when updating the policy.

[0105] •∈ is a hyperparameter that controls the magnitude of policy updates, used to prevent policy updates from being too fast or over-adjusted.

[0106] ④ Strengthen the learning and training process

[0107] a. Sampling and Interaction

[0108] In each training cycle, the robotic arm follows the current strategy π θ Interact with the simulation environment to collect multiple state transition samples, with the sample format being {s t ,a t ,r t ,s t+1 These data will be saved to an experience pool as input for subsequent policy updates. Wherein, s t This is the current state, a t It's an action, r t It is the reward after performing the action, s t+1 This is the next state, and these state transition data will serve as training samples in the reinforcement learning process.

[0109] b. Actor-Critic Model

[0110] Actor Neural Network: This neural network adjusts its operation based on the current state s. t Generate a probability distribution and select the optimal action a from it. t This determines the next action of the robotic arm. During this process, the PPO algorithm optimizes the policy gradient, enabling the actor network to continuously improve its action selection strategy.

[0111] Critic Neural Network: The critic network is based on the current state s t and reward r t To estimate the value V(s) of this state t ), and calculate the advantages of the action. This refers to the merits of the action relative to other possible actions. The main task of the critic network is to provide an evaluation of the current strategy.

[0112] c. Strategy Update

[0113] PPO measures the current action a by calculating the advantage function.t The strengths and weaknesses of different actions are compared, and this information is fed back to the actor network to guide them in updating their strategies.

[0114] d. Value Renewal

[0115] The critic network calculates the current state s t Value V(s) t The loss function L is updated based on the actual cumulative reward. V (θ) is used to minimize the difference between the estimated value and the actual return.

[0116] ⑤ Establish an attenuation mechanism

[0117] By gradually shortening the step size of each training episode, the robotic arm can focus more on optimizing the details of the trajectory during later training sessions, rather than engaging in lengthy and ineffective exploration.

[0118] The decay mechanism improves computational efficiency by dynamically adjusting the training step size. Initially, the robotic arm needs to explore extensively to learn basic trajectory planning strategies, so a relatively long episode step size is set. However, as training progresses, the robotic arm gradually learns how to plan trajectories, at which point the exploration step size can be reduced.

[0119] Specifically, dynamic adjustments are made using the following formula:

[0120] T new =max(T) old ·α,T min )

[0121] Wherein: T new It is the updated episode step size, T old This is the current episode step size, α is a decay coefficient, usually less than 1, which controls the rate of step size reduction, T min This is a set minimum step size to ensure that the step size is not too short, allowing the robotic arm to make necessary fine-tuning and optimizations later in the training process. This mechanism helps the robotic arm reduce unnecessary exploration in the later stages of training and focus on optimizing the details of the trajectory.

Claims

1. A method for trajectory optimization of a robotic arm based on deep reinforcement learning, comprising the following steps: Step one: Kinematic modeling of the robotic arm; Step two: Multi-objective optimization problem modeling, the trajectory planning of the robotic arm is regarded as a multi-objective optimization problem, the main optimization objectives include three aspects: trajectory accuracy, trajectory smoothness and energy consumption, including: Setting the total optimization objective function: (4) wherein, is a joint angle of the robot arm, is a first objective function, represents a first corresponding to the function, represents a first joint of the robot arm, a total number of joints of the robot arm is denoted as ; The trajectory accuracy refers to the distance error between the actual trajectory of the end effector of the robotic arm and the desired trajectory, and the trajectory accuracy optimization objective function is defined as: (5) wherein, respectively represent the three-dimensional coordinate positions of the end effector in Euclidean space for the desired trajectory and the actual trajectory, respectively, and the goal of the optimization is to minimize the error between the desired and actual positions; The trajectory smoothness optimization objective function is defined as: (6) wherein, is the velocity of the joint, is the acceleration of the joint, and are corresponding weight coefficients; The energy consumption optimization considers the sum of the square of the angle change and the sum of the joint torque in discrete unit time, and the energy consumption cost function is represented as: (7) wherein, is the angle change amount of the joints, is the torque of the joints; Step three: Using the Proximal Policy Optimization (PPO) reinforcement learning algorithm to design the trajectory planning algorithm: (1) The observation space and action space design are included. The state variables included in the observation space include joint angles, joint velocities, end effector position and attitude information; the state variables include the current state of each joint of the robot arm , the current position , the target state , the target position , and the Euclidean distance between the current end pose and the target point ; when the motion state of the robot arm exceeds its own constraints, the specific joint state exceeding the observation space will be adjusted to its maximum limit value; the action space is set according to the speed of each joint, and the state variables in the observation space are updated with each provided action; (2) The reward function design considers both the time factor and the inherent complexity of the robotic arm planning task, and proposes a reward function: (8) where t represents the iteration number, the components of the reward function , , are related to accuracy, smoothness and energy consumption, respectively, and are obtained from the optimization function: (9) (10) (11) wherein, and is a weight coefficient of the precision reward, is a weight coefficient of the smoothing reward, represents a weight coefficient of the energy consumption reward; Provide an additional reward when a predetermined minimum distance is achieved between the current and desired end point distance : (12)。 2. The method of claim 1, wherein, The method of step one is as follows: Each link of the robotic arm is represented by four kinematic parameters, two of which describe the link itself and the other two represent the connection relationship between the links. The kinematic model of the robotic arm is described using the Denavit-Hartenberg method. For the robotic arm, the transformation formula from the base to the end effector of the link is established using the homogeneous transformation matrix. The kinematic equation is used to convert the angles of each joint of the robotic arm into the position and attitude of the end effector. The goal of the optimization process is to achieve the movement of the end effector along the predetermined path by controlling the angles of each joint. The joint angles are calculated based on the position and attitude of the end effector, and precise motion control of the joints is achieved. The inverse kinematic equation is established to link the velocity of the end effector with the angular velocity of the joints, and the conversion between the joint space and the Cartesian space is realized.

3. The method of claim 1, wherein, Step three includes the principle of policy update: set and optimize the clipping objective function, which limits the amplitude of policy update and prevents drastic changes in policy during the update process.

4. The method of claim 1, wherein, Step three includes the reinforcement learning training process: a. Sampling and interaction In each training cycle, according to the current strategy of the robot arm Interact with the simulation environment, collect a plurality of state transition samples, and the state transition sample format is , as the input of subsequent policy update; wherein, is the current state, is the action, is the reward after executing the action, is the next state, and the state transition data is used as the training sample in the reinforcement learning process; b. Actor-critic model, including Actor network: generate a probability distribution according to the current state , and select the optimal action from it , so as to determine the next operation of the mechanical arm; in this process, the PPO algorithm is optimized through the policy gradient, so that the actor network continuously improves the strategy of selecting actions; Critics Network: Based on the current status and rewards To estimate the value of the state And calculate the advantage function of the action; the critic network is used to give an evaluation of the current strategy; c. Policy update PPO measures the current action by computing an advantage function relative to other actions, and feeds this information back to the actor network to guide its policy updates; d. Value update The critic network calculates the current state value And update its parameters based on the actual cumulative rewards.

5. The method of claim 1, wherein, Step three contains a decay mechanism to improve computational efficiency by dynamically adjusting the training cycle step size with the following equation: wherein: is the updated training cycle step size, is the current training cycle step size, is a decay factor, less than 1, that controls the speed of step size reduction, is a set minimum step size.

Images