A reinforcement learning-based multi-objective trajectory planning method for industrial robots

Abstract

This invention discloses a reinforcement learning-based multi-objective trajectory planning method for industrial robots, belonging to the field of industrial robot trajectory planning technology. Based on constraints and optimization objectives, a reward function is determined for each optimization objective in the trajectory planning process. Parallel action networks and a judge network are constructed. The Pareto front set is obtained by non-dominated ranking of the action network strategies based on the dominance function output by the judge network. From the Pareto front set, several state-action pairs are randomly selected. The maximum entropy search algorithm is used to train the parameters of each action network for each objective function, and the gradient algorithm is used to train the parameters of the judge network for each objective, thus obtaining industrial robot trajectories satisfying multiple constraints and objectives. This invention improves the stability and convergence of the obtained industrial robot trajectories by continuously updating the parameters through the maximum entropy search algorithm to train the parameters of each action network for each objective function.

Description

Technical Field

[0001] This invention belongs to the field of industrial robot trajectory planning technology, specifically relating to a reinforcement learning multi-objective trajectory planning method for industrial robots. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, deep neural networks, represented by reinforcement learning, have gradually demonstrated their advantages in the field of robot trajectory planning, providing new ideas and methods for efficient and autonomous trajectory planning of robotic arms. Compared with traditional motion trajectory planning methods, thanks to the powerful learning ability of neural networks for trajectory planning strategies and the adjustable exploration mechanism during interaction with the environment, they can achieve rapid online target planning in continuous motion space, thus possessing better stability and convergence. However, at the same time, existing motion planning based on reinforcement learning is mostly single-objective planning or plans the robot's path configuration points without providing its end-effector trajectory, making it difficult to meet the multi-objective optimization needs of trajectory planning in the field of industrial robots.

[0003] The Chinese patent disclosure (application number: 202211184744.4) discloses a multi-objective trajectory optimization method for industrial robots based on an improved particle swarm optimization algorithm. This method optimizes multi-objective trajectories by using the robot's time, energy, average pulsation, and the average rate of change of joint torques as objective functions. It finds the optimal solution based on Pareto dominance and utilizes Logistic chaotic mapping for population initialization and inertia weight allocation. The resulting optimal trajectory solution set exhibits minimal pulsation variation, short processing time, and high efficiency. However, this method does not consider the stability of the obtained target trajectory. Therefore, improving the stability of the obtained target trajectory is the technical problem this invention aims to solve. Summary of the Invention

[0004] The purpose of this invention is to provide a reinforcement learning multi-objective trajectory planning method for industrial robots, so as to solve the problems and achieve the effects mentioned in the background art.

[0005] The objective of this invention is achieved as follows: a reinforcement learning multi-objective trajectory planning method for industrial robots, characterized in that the channel estimation scheme includes the following steps:

[0006] Step S1: Determine the reward function for the trajectory planning process for each optimization objective based on the constraints and optimization objectives;

[0007] Step S2: For each optimization objective, construct a parallel action network and a judgment network. Based on the advantage function output by the judgment network, perform non-dominated ranking of the policies of the action network to obtain the Pareto front set.

[0008] Step S3: Determine the objective functions of the action network and the judge network based on the maximum entropy proximal policy optimization algorithm. From the obtained Pareto front set, randomly select several sets of state-action pairs. Use the maximum entropy search algorithm to train the parameters of each action network for each objective function. Use the gradient method to train the parameters of each judge network for each objective, thereby obtaining the industrial robot trajectory that satisfies multiple objectives under various constraints.

[0009] Preferably, in step S1, the optimization objectives are to minimize the time, kinetic energy, and impact force of the industrial robot trajectory, satisfying position, velocity, and acceleration constraints, and the reward function is set as follows:

[0010]

[0011]

[0012]

[0013] Where, r time r energy r impact Let represent the reward functions for the three optimization objectives of minimizing time, minimizing kinetic energy, and minimizing impact, respectively. Let represent the three optimization objectives: minimum time, minimum kinetic energy, and minimum impact, respectively. Let d represent the distance between the end effector position and the target position of the industrial robot. Constants α>0 and β>0 are used to adjust the convergence speed of the algorithm to prevent it from getting trapped in local optima. Let i represent the i-th joint, and t... i This represents the running time of the i-th joint. Let M(q) represent the generalized joint angular position, generalized joint angular velocity, and generalized joint angular acceleration of a robot with n degrees of freedom, respectively, where M(q)∈R. n×n Let N represent the robot's inertia matrix, and let N represent the robot's running time.

[0014] Preferably, the action networks all adopt a two-layer network with the same structure, where the first layer is used to represent the first probability distribution of new actions, and the second layer is used to represent the second probability distribution of old actions;

[0015] The evaluation network adopts a two-layer network with the same structure. The first layer network is used to represent the display network, and the second layer network is used to represent the target network.

[0016] Preferably, in step S2, the Pareto front set is obtained by non-dominated ranking of the action network's policies based on the dominance function output by the evaluation network. The specific steps are as follows:

[0017] Step S2-1: Set the state s of the action network corresponding to each target at time k. i(k) represents the normalized joint angular position and joint angular velocity at time k; the output a of the motion network at time k is set. i (k) represents the angular acceleration of each joint, and the action used satisfies a normal distribution with a mean of 0 and a variance of 1;

[0018] Step S2-2: For each target, the action network generates an action a during each training iteration. i (k) plus the quasi-random variable ξ described by the Logistic mapping i Repeat this process multiple times, and each repetition yields action a. i_j (k)=a i (k)+ξ j (k), and calculate each a i_j (k) corresponds to the state value function V i_j (k), select the largest V i_j (k) corresponds to action a i_j (k) serves as the action output a for this training. i (k);

[0019] Step S2-3: The real network and the target network represent the current state value function V of the i-th target, respectively. i (s(k)), the prediction V of the next state value function of the i-th objective. i (s(k+1)), to obtain the evaluation network output advantage function for the i-th target;

[0020] Step S2-4: Perform Pareto non-dominated ranking on the advantage function of each objective. The action process executed by the real network in each action network is called policy π. i The corresponding advantage function is R. i R i The larger the value, the more likely the execution strategy π is to be implemented. i The better the optimization objective value obtained, the better the Pareto front set is obtained, and the Pareto front set is saved.

[0021] Preferably, the objective function in step S3 is:

[0022]

[0023] in, clip(·) is a commonly used clipping function. It is a suitable constant, ρ>0; R i For strategy π i The corresponding advantage function.

[0024] Preferably, the dominance function in steps S2-3 is:

[0025] Ri (k)=V i (s(k))+r i (k)-δ i V i (s(k+1));

[0026] Among them, V i (s) represents the value function of state s, r i (k) represents the reward function of the evaluation network for this action at time k, δ i Denotes the discount factor, and 0 < δ i <1.

[0027] Preferably, in step S3, the maximum entropy search algorithm is used to train the network parameters for each action of each objective function. Specifically, the operation is as follows:

[0028]

[0029] Where j represents the value function at the current moment, and the V function is sorted from largest to smallest in the previous μ iterations; λ i This represents the weight of the network for the i-th action of each objective.

[0030] Preferably, in step S3, the optimization objective of the real network for each evaluation network of each optimization objective is:

[0031]

[0032] Where, θ i This represents the parameters of the i-th evaluation network for each objective function, 0 < δ. i <1;V i (s) represents the value function of state s.

[0033] Compared with the prior art, the present invention has the following improvements and advantages: 1. By training the network parameters of each action of each objective function through the maximum entropy search algorithm, the parameters are continuously updated to obtain the industrial robot trajectory of multiple objectives, thereby improving the stability and convergence of obtaining the industrial robot trajectory.

[0034] 2. The optimization objective is determined by the objective function, and the Pareto non-dominated sorting is performed on each optimization objective using the dominance function to obtain the Pareto front set. By obtaining the dominance relationship with high accuracy, the dominance level and crowding distance are selected to obtain a set of solutions with diversity and balance, thereby improving the stability and convergence of the industrial robot trajectory. Attached Figure Description

[0035] Figure 1 This is the overall flowchart of the present invention.

[0036] Figure 2This is a flowchart of the present invention.

[0037] Figure 3 This is a rendering of the invention. Detailed Implementation

[0038] The invention will be further summarized below with reference to the accompanying drawings.

[0039] See Figure 1 , 2 A reinforcement learning-based multi-objective trajectory planning method for industrial robots, comprising the following steps:

[0040] Step S1: Determine the reward function for the trajectory planning process for each optimization objective based on the constraints and optimization objectives;

[0041] Based on the determined geometry and kinematic model of the robotic arm, as well as the constraints on position, velocity, and acceleration, the geometry and kinematic model of the robotic arm are determined according to the type of robotic arm. When determining the constraints on position, velocity, and acceleration, the physical characteristics and safety requirements of the robotic arm are considered.

[0042] Define the objective function and constraint functions. For the three optimization objectives of minimizing time, minimizing energy consumption, and minimizing impact force, the corresponding objective functions need to be defined. The constraint functions for position, velocity, and acceleration need to be defined to ensure that the generated trajectory meets the constraint conditions.

[0043] Therefore, based on the optimization objectives of minimizing the time, kinetic energy, and impact force of the industrial robot trajectory, and satisfying the constraints of position, velocity, and acceleration, the reward function is set as follows:

[0044]

[0045]

[0046]

[0047] Where, r time r energy r impact Let represent the reward functions for the three optimization objectives of minimizing time, minimizing kinetic energy, and minimizing impact, respectively. Let represent the three optimization objectives: minimum time, minimum kinetic energy, and minimum impact, respectively. Let d represent the distance between the end effector position and the target position of the industrial robot. Constants α>0 and β>0 are used to adjust the convergence speed of the algorithm to prevent it from getting trapped in local optima. Let i represent the i-th joint, and t... i Let q represent the running time of the i-th joint, where q ∈ R. n , Let M(q) represent the generalized joint angular position, generalized joint angular velocity, and generalized joint angular acceleration of a robot with n degrees of freedom, respectively, where M(q)∈R. n×n Let N represent the robot's inertia matrix, and let N represent the robot's running time.

[0048] A smooth trajectory is generated using a suitable sampling method and motion interpolation algorithm; a multi-objective optimization algorithm is used to iteratively optimize the initial population to obtain the Pareto front set. In each iteration, the algorithm evaluates the population using the defined objective function and constraint functions, and updates the individuals in the population according to the optimization algorithm's strategy.

[0049] Step S2: For each optimization objective, construct a parallel action network and a judgment network. Based on the advantage function output by the judgment network, perform non-dominated ranking of the policies of the action network to obtain the Pareto front set.

[0050] In step S2, the Pareto front set is obtained by non-dominated ranking of the policies of the action network based on the dominance function output by the evaluation network. The specific steps are as follows:

[0051] Step S2-1: Set the state s of the action network corresponding to each target at time k. i (k) represents the normalized joint angular position and joint angular velocity at time k; the output a of the motion network at time k is set. i (k) represents the angular acceleration of each joint, and the action used satisfies a normal distribution with a mean of 0 and a variance of 1;

[0052] Step S2-2: All action networks use a two-layer network with the same structure. The first layer represents the first probability distribution of new actions, and the second layer represents the second probability distribution of old actions. For each target, the action network, to enrich the exploratory behavior of the actions, incorporates actions 'a' obtained during each training iteration. i (k) plus the quasi-random variable ξ described by the Logistic mapping i Repeat this process multiple times, and each repetition yields action a. i_j (k)=a i (k)+ξ j (k), and calculate each a i_j (k) corresponds to the state value function V i_j (k), select the largest V i_j (k) corresponds to action a i_j (k) serves as the action output a for this training. i (k);

[0053] Step S2-3: The evaluation network corresponding to each target adopts a two-layer network with the same structure. The first layer network is used to represent the display network, and the second layer network is used to represent the target network. The display network and the target network represent the current state value function V of the i-th target, respectively. i (s(k)), the prediction V of the next state value function of the i-th objective. i (s(k+1)), to obtain the evaluation network output advantage function for the i-th target;

[0054] R i (k)=V i (s(k))+rx(k)-δ i V i (s(k+1))

[0055] Among them, V i (s) represents the value function of state s, r i (k) represents the reward function of the evaluation network for this action at time k, δ i Denotes the discount factor, and 0 < δ i <1.

[0056] Step S2-4: Perform Pareto non-dominated ranking on the advantage function of each objective. The action process executed by the real network in each action network is called policy π. i The corresponding advantage function is R. i R i The larger the value, the more likely the execution strategy π is to be implemented. i The better the optimization objective value obtained, the better the Pareto front set is obtained, and the Pareto front set is saved.

[0057] Based on the constraints of the problem, a set of initial solutions (robot trajectories) is generated using a stochastic method. These solutions are ensured to satisfy the position, velocity, and acceleration constraints. Smooth trajectories can be generated using appropriate sampling methods and motion interpolation algorithms.

[0058] Performing a multi-objective optimization algorithm: In MATLAB, a multi-objective optimization algorithm is used to iteratively optimize the initial population to obtain the Pareto front set. In each iteration, the algorithm evaluates the population using the defined objective function and constraint functions, and updates the individuals in the population according to the optimization algorithm's strategy.

[0059] Non-dominated sorting and crowding distance calculation: Based on the calculation results of the dominance function, the Pareto front set is non-dominatedly sorted. Non-dominated sorting divides the solutions into different levels, making the superiority-inferiority relationship of the solutions clearer.

[0060] Selecting Pareto Front Solutions: Based on requirements, select an appropriate number of Pareto front solutions as feasible solutions. The selection is based on the solution's non-dominated level and crowding distance to obtain a set of solutions with diversity and equilibrium.

[0061] Step S3: Determine the objective functions of the action network and the judge network based on the maximum entropy proximal policy optimization algorithm. From the obtained Pareto front set, randomly select several sets of state-action pairs. Use the maximum entropy search algorithm to train the parameters of each action network for each objective function. Use the gradient method to train the parameters of each judge network for each objective, thereby obtaining the industrial robot trajectory that satisfies multiple objectives under various constraints.

[0062] The objective function of the real network for each target action is:

[0063]

[0064] in, clip(·) is a commonly used clipping function. It is a suitable constant, ρ > 0; R i For strategy π i The corresponding advantage function.

[0065] The maximum entropy search algorithm is used to train the network parameters for each action of each objective function. The specific steps are as follows:

[0066]

[0067] Where j represents the value function at the current moment, and V represents the sorting of the V function from largest to smallest during the previous μ iterations; λ i This represents the weight of the network for the i-th action of each objective.

[0068] The optimization objective of the real network for each evaluation network with each optimization objective is:

[0069]

[0070] Where, θ i This represents the parameters of the i-th evaluation network for each objective function, where 0 < δ. i <1; V i (s) represents the value function of state s.

[0071] The gradient method is used to train the parameters of each evaluation network for each objective. The specific operation is as follows:

[0072] First, it is necessary to define the network structure and objective function of the action network and the judge network;

[0073] Two-layer Action Network: The goal of the action network is to generate the trajectory movements of a robotic arm; a deep neural network is used as the structure of the action network; the two-layer action network is a network structure used in algorithms for deep reinforcement learning. In traditional Q-learning algorithms, a Q-network is used to estimate the value function (Q-value function) for each action, and then the optimal action is selected based on these value functions. However, Q-learning suffers from an overestimation problem, where the value function tends to overestimate the value of some actions. To address this issue, the two-layer action network introduces two independent Q-networks: one for selecting actions and the other for evaluating the value of the selected actions. During training, by periodically updating these two networks, the overestimation problem can be reduced, thereby improving the learning effect. The input is the current state of the robotic arm, and the output is the next action of the robotic arm. The objective function is selected using a maximum entropy proximal optimization strategy.

[0074] A two-layer evaluation network: The goal of the evaluation network is to assess the quality of the generated trajectory. A deep neural network is used as the structure of the evaluation network. The two-layer evaluation network is a policy improvement technique in reinforcement learning algorithms. In traditional Q-learning algorithms, the next action is determined by selecting the maximum Q-value in the current state. However, due to the inaccuracy of the estimated Q-value, this may lead to policy selection bias. The two-layer evaluation network separates action selection and value evaluation in the current state. One network selects the best action, and the other network evaluates the value of the selected action. This reduces the impact of estimation errors on policy improvement and improves the stability of learning. The input is the trajectory of the robotic arm, and the output is the trajectory quality score, such as time, energy consumption, and impact force. The objective function is chosen to be the maximum entropy proximal optimization policy consistent with the action network.

[0075] Secondly, set the parameters and hyperparameters. During training, it is necessary to set some parameters and hyperparameters, such as learning rate, batch size, and number of training iterations.

[0076] The selection of these parameters will affect the experimental results and can be adjusted according to the specific circumstances.

[0077] Policy gradient method training: Using the policy gradient method algorithm provided in MATLAB, the parameters are trained based on the objective function of a two-layer action network and a two-layer evaluator network. During training, the parameters of the action network are updated by interacting with the evaluator network to maximize the trajectory quality in the Pareto front set.

[0078] Set the number of iterations and perform iterative training: Based on experimental requirements, set the number of training iterations as the termination condition. In each iteration, generate a trajectory based on the selected Pareto front solution and update the parameters using the policy gradient method. As training progresses, the performance of the action network and the judge network will gradually improve.

[0079] Using a trained action network, the robot's trajectory is generated based on the initial and target states. The initial state can be input into the action network, and the generated actions can be used to infer the robot's next state, then the complete trajectory can be generated iteratively.

[0080] Check if the generated trajectory meets the constraints for position, velocity, and acceleration. If the trajectory does not meet the constraints, you can try adjusting the optimization parameters, modifying the network structure, or increasing the number of training iterations to further optimize the results.

[0081] Working Principle: The system optimizes for three objectives in industrial robot trajectory planning: minimizing time, energy consumption, and impact force, while satisfying constraints on position, velocity, and acceleration. It calculates the dominance functions for each strategy, ranking these functions non-dominated to obtain the Pareto front set. Actions are randomly selected from this Pareto front set, and a two-layer action network and a two-layer evaluator network are constructed using a maximum entropy proximal optimization strategy. The parameters of the action and evaluator networks are trained using the policy gradient method, with the number of iterations as the termination condition. Finally, an industrial robot trajectory satisfying multiple constraints and optimizing multiple objectives is obtained. Extending the action-evaluator reinforcement learning structure using the maximum entropy proximal optimization strategy to the field of multi-objective optimization industrial robot trajectory planning yields better stability and convergence, providing diverse solutions for multi-objective optimization industrial robot trajectory planning.

[0082] Simulation experiment:

[0083] Using the simulation software CoppeliaSim Edu, users can simulate and interact with various robots and robotic systems in a virtual environment. It supports various robot types, including mobile robots, industrial robotic arms, drones, etc., and provides a rich set of sensor and actuator models.

[0084] This invention conducts a trajectory joint trajectory tracking simulation control experiment using MATLAB and CoppeliaSim Edu, realizing the control of the running trajectory of the UR5 robotic arm in Verp via mouse in MATLAB.

[0085] In reinforcement learning, the average reward during training reflects the convergence of the algorithm: the average reward can be used to determine whether the training process is gradually converging to a higher level. If the average reward gradually increases and stabilizes at a high value during training, it can be said that the agent's policy is continuously improving and converging to a better state. Figure 3It can be seen that when the number of training iterations reaches 3200, the algorithm converges, outputs the algorithm with parameters, and conducts simulation experiments. The traditional method uses PD control to control the position of a specified point in space and records the completion time of each task.

[0086] Table 1 is a comparison table between the traditional method and the method of the present invention.

[0087] Traditional methods Improved Algorithm Average time 7.2s 4.5s Fastest time 6.7s 3.6s

[0088] The comparison shows that the algorithm used in this invention improves efficiency by an average of 37.5%, proving the superiority of the improved algorithm; while realizing the multi-target trajectory task of the robotic arm, it improves the efficiency of the task and reduces energy consumption.

[0089] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A reinforcement learning-based multi-objective trajectory planning method for industrial robots, characterized in that: The method includes the following steps: Step S1: Based on the constraints and optimization objectives, determine the reward function for the trajectory planning process for each optimization objective. The specific steps are as follows: Satisfying position, velocity, and acceleration constraints, the reward function is defined as follows: ； ； ； in, , , Let represent the reward functions for the three optimization objectives of minimizing time, minimizing kinetic energy, and minimizing impact, respectively. , , These represent the three optimization objectives: minimum time, minimum kinetic energy, and minimum impact, respectively. This represents the distance between the end effector position and the target position of the industrial robot, a constant. ,constant Adjust the convergence speed of the algorithm to prevent it from getting trapped in local optima. Indicates the first One joint, Indicates the first The time it takes for each joint to operate. , , They represent the degrees of freedom as follows: The robot's generalized joint angular position, generalized joint angular velocity, and generalized joint angular acceleration. The inertia matrix of the robot. Indicates the robot's running time; Step S2: For each optimization objective, construct a parallel action network and a judge network. Based on the advantage function output by the judge network, perform a non-dominated ranking of the policies in the action network to obtain the Pareto front set. The specific steps are as follows: Step S2-1: Setting The state of the action network for each target at any given time for The joint angle position and joint angular velocity are normalized at all times; settings are applied. Output of the Moment Action Network The angular accelerations of each joint are given, and the motions used follow a normal distribution with a mean of 0 and a variance of 1. Step S2-2: The action network corresponding to each target, in each training iteration, generates the actions... Adding quasi-random variables described by the Logistic mapping Repeat this process multiple times, and each repetition yields an action. and calculate each Corresponding state value function Select the largest The corresponding action As the output of this training ; Step S2-3: The real network and the target network represent the first... Current state value function of each target , No. Prediction of the next state value function for each objective , obtained the The evaluation network for each objective outputs an advantage function; Step S2-4: Perform Pareto non-dominated ranking on the advantage function of each objective, and define the action process executed by the real network in each action network as the policy. ,Strategy The corresponding advantage function is , The larger the value, the more likely it is to be an execution strategy. The better the obtained optimization objective value, the better the Pareto front set is, and the Pareto front set is saved; Step S3: Determine the objective functions of the action network and the judge network based on the maximum entropy proximal policy optimization algorithm. From the obtained Pareto front set, randomly select several sets of state-action pairs. Use the maximum entropy search algorithm to train the parameters of each action network for each objective function. Use the gradient method to train the parameters of each judge network for each objective, thereby obtaining the industrial robot trajectory that satisfies multiple objectives under various constraints.

2. The reinforcement learning multi-objective trajectory planning method for industrial robots according to claim 1, characterized in that: The objective function in step S3 is: ； in, , It is a commonly used clipping function. It is a suitable constant. ; For strategy The corresponding advantage function.

3. The reinforcement learning multi-objective trajectory planning method for industrial robots according to claim 1, characterized in that: The dominance function in step S2-3 is: ； in, Representing state The value function, express It is time to evaluate the network's reward function. Represents the discount factor, and .

4. The reinforcement learning multi-objective trajectory planning method for industrial robots according to claim 1, characterized in that: In step S3, the maximum entropy search algorithm is used to train the network parameters for each action of each objective function. The specific operation is as follows: ； in, The function representing the value at the current moment is in front. During the next iteration Sort the functions from largest to smallest. This represents the weight of the network for the i-th action of each objective.

5. The reinforcement learning multi-objective trajectory planning method for industrial robots according to claim 1, characterized in that: The real-world network optimization objective for each evaluation network in step S3 is: ； in, This represents the parameters of the i-th evaluation network for each objective function. ; Representing state The value function.

Images