Robot motion control method and system

Abstract

The invention discloses a robot motion control method and system, and the method comprises the steps: obtaining the environment information of a robot before motion, and planning the optimal track of the robot through a track planning algorithm according to the environment information; constructing a track simulation reward function according to the optimal track, and adding the track simulation reward function to a self-adaptive dynamic reward function to obtain a target self-adaptive dynamic reward function; controlling the robot to simulate the optimal track based on the target adaptive dynamic reward function; and training and learning the motion track of the robot according to a PPO deep reinforcement learning algorithm of an improved cutting mechanism, and finally obtaining an optimal control strategy of the robot. The self-adaptive dynamic reward mechanism can dynamically adjust the reward weight according to the environment condition, the reference track reward provides directional guidance, and the combination can flexibly cope with the real-time change of the environment while the track accuracy is kept.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a robot motion control method and system. Background Art

[0002] Robot dynamic target grasping technology is widely used in industrial production lines and is a research hotspot in the field of robotics. In complex industrial environments, reinforcement learning overcomes the shortcomings of traditional robot control methods that have poor adaptability to changes in environmental states, and has become an important direction for robot intelligence research. At present, robot motion control based on reinforcement learning faces the problems of high robot action dimensions and complex environmental states, difficulty in value assessment in the face of a large number of states and actions, and sparse rewards, that is, the robot can only be rewarded when it reaches the specified position, resulting in long training time and poor results. Summary of the invention

[0003] The present invention provides a robot motion control method and system, which are used to solve the technical problem that the robot can only be rewarded when it reaches a specified position, resulting in long training time and poor effect.

[0004] In a first aspect, the present invention provides a robot motion control method, comprising: Acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; Constructing a trajectory imitation reward function according to the optimal trajectory, and adding the trajectory imitation reward function to an adaptive dynamic reward function to obtain a target adaptive dynamic reward function; Controlling the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The motion trajectory of the robot is trained and learned according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally the optimal control strategy of the robot is obtained, wherein the expression of the improved clipping mechanism is: , In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

[0005] In a second aspect, the present invention provides a robot motion control system, comprising: An acquisition module is configured to acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; A construction module is configured to construct a trajectory imitation reward function according to the optimal trajectory, and add the trajectory imitation reward function to the adaptive dynamic reward function to obtain a target adaptive dynamic reward function; an imitation module, configured to control the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The output module is configured to train and learn the motion trajectory of the robot according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally obtain the optimal control strategy of the robot, wherein the expression of the improved clipping mechanism is: , In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the robot motion control method of any embodiment of the present invention.

[0007] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor is caused to execute the steps of the robot motion control method of any embodiment of the present invention.

[0008] The robot motion control method and system of the present application have the following beneficial effects: 1. It can improve the efficiency of robot reinforcement learning. In reinforcement learning, the path generated by the trajectory planning algorithm is used as a reference path. The algorithm can obtain clearer direction guidance, thereby reducing the time wasted by random exploration. At the same time, the adaptive dynamic mechanism can make timely adjustments when the path deviates or the environment changes, making the learning process more efficient and converging faster; 2. It can enhance the robot's adaptability to dynamic environments. The adaptive dynamic reward mechanism can dynamically adjust the reward weight according to the environmental conditions, while the reference trajectory reward provides directional guidance. This combination can flexibly respond to real-time changes in the environment while maintaining trajectory accuracy; 3. It can improve the robustness of the robot. When the environment changes dynamically or new obstacles appear, the adaptive dynamic reward function combined with the reference trajectory can better cope with emergencies and ensure the rationality and consistency of the training trajectory. Even in a complex environment, the robot can stably complete the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flowchart of a robot motion control method provided by one embodiment of the present invention; Figure 2 A structural block diagram of a robot motion control system provided by one embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] See also Figure 1 , which shows a flow chart of a robot motion control method of the present application.

[0013] like Figure 1 As shown, the robot motion control method specifically includes the following steps: Step S101, obtaining environmental information before the robot moves, and planning the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information and dynamic target information.

[0014] In this step, the initial state includes the angles of each joint of the robot, the angular velocity, the position of the robot and the speed of the robot; the obstacle information includes the position and geometric dimensions of the obstacle; the dynamic target information includes the geometric dimensions of the dynamic target, the position of the dynamic target and the motion range of the dynamic target.

[0015] Specifically, use The trajectory planning algorithm plans the trajectory of the robot from the initial state to the target state in space. Since the trajectory obtained by the preliminary planning may not be smooth enough or inefficient in the robot movement, the use The spline smoothing algorithm is used to smooth the trajectory; on the basis of satisfying all operation constraints, the numerical optimization algorithm is used to optimize the trajectory to reduce the motion time, energy consumption or improve the safety and stability of the trajectory.

[0016] Step S102: construct a trajectory imitation reward function according to the optimal trajectory, and add the trajectory imitation reward function to the adaptive dynamic reward function to obtain a target adaptive dynamic reward function.

[0017] In this step, the trajectory is built to mimic the reward function It mainly includes position error part, speed and acceleration matching part and time synchronization part.

[0018] Among them, the position error part contains the local refined position error reward function and trajectory coverage reward function , specifically: The expression of the local refined position error reward function is: , In the formula, is the number of key points, is the time weight of the key point of the i-th reference trajectory, is the key point of the actual trajectory, is the key point of the reference trajectory; The expression of the trajectory coverage reward function is: , In the formula, is the total number of time steps, is the threshold value; The speed and acceleration matching part includes the speed error reward function , acceleration error reward function Matching reward function with velocity direction , the specific contents are: By calculating the error between the actual trajectory speed and the reference trajectory speed, the robot is encouraged to move at a speed consistent with the reference trajectory. The speed error reward function is expressed as: , In the formula, is the actual trajectory speed, is the reference trajectory speed; The error between the actual trajectory acceleration and the reference trajectory acceleration is calculated to encourage the robot to move at an acceleration consistent with the reference trajectory. The expression of the acceleration error reward function is: , In the formula, is the actual trajectory acceleration, is the reference trajectory acceleration; The cross product of velocity is used to measure the direction error. The larger the value, the greater the direction deviation. Normalization can avoid the influence of velocity magnitude on the result and ensure that the robot's movement direction is consistent with the reference trajectory. The expression of the velocity direction matching reward function is: , The time synchronization part includes the trajectory time arc length reward function , the specific contents are: The expression of trajectory time arc length reward function is: , , In the formula, is the trajectory time arc length, is the actual trajectory time arc length, is the reference trajectory time arc length.

[0019] The trajectory time arc length reflects the overall time distribution and speed comprehensive characteristics of the trajectory, and encourages the consistency of the actual trajectory with the reference trajectory in terms of the overall time attributes.

[0020] The trajectory imitation reward function is a weighted combination of the above three parts, specifically: , In the formula, is the trajectory imitation reward function, , , , , , are the importance of different reward functions, is the local refined position error reward function, is the trajectory coverage reward function, is the speed error reward function, is the acceleration error reward function, is the velocity direction matching reward function, is the trajectory time arc length reward function.

[0021] Step S103: controlling the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function.

[0022] In this step, the adaptive dynamic reward function includes a dynamic reward function and a speed constraint reward function, wherein the dynamic reward function includes a target distance reward function , trajectory imitation reward function , obstacle avoidance reward function And the completion state reward function ; Among them, the expression of the adaptive dynamic reward function is: , In the formula, is an adaptive dynamic reward function, is a factor based on the dynamic change of the distance between the robot and the obstacle; The dynamic reward function The expression is: , In the formula, is the total number of time steps, is the discount factor; The expression of the target distance reward function is: , , , In the formula, is the dynamic weight, and These are adjustment parameters used to control the speed and turning point of dynamic weight changes. represents the position of the robot, Indicates the location of the target point; The expression of the obstacle avoidance reward function is: , In the formula, To control the coefficient of obstacle avoidance penalty, is the distance influence coefficient, is the distance between the robot and the obstacle; The expression of the completion state reward function is: , In the formula, , are weight factors that control the degree of reward. The ideal time for the robot to complete the task, is the total number of time steps; Speed ​​Constraint Reward Function The expression is: , In the formula, , are weight factors, is the number of robot joints, is the velocity of the jth joint, is the acceleration of the jth joint.

[0023] Step S104, training and learning the motion trajectory of the robot according to the PPO deep reinforcement learning algorithm with improved clipping mechanism, and finally obtaining the optimal control strategy of the robot.

[0024] In this step, the expression of the PPO deep reinforcement learning algorithm is: , , In the formula, is the objective function, To find the expected symbol, is the traditional clipping function, is the advantage function, is the hyperparameter of the cutting function; The expression of the improved clipping mechanism is: , In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

[0025] In this embodiment, when the policy update ratio Beyond the crop center, that is, much larger or much smaller than 1, use the exponential decay function to quickly limit the update. Close to the cropping center, that is, close to 1, using a smoothing function Flexibly trim updates when policy updates are smaller than , using the original updated value , without any clipping. In order to adapt to different task stages, the clipping width and attenuation slope of the multi-stage clipping function can be dynamically adjusted over time to achieve more detailed control. For the case where the update amplitude is too large, rapid attenuation is performed to avoid instability in the optimization process. For the case where the update amplitude is close to the clipping boundary, smooth clipping is performed to prevent excessive restrictions on strategy optimization. Different functions are used to transition between different regions to avoid gradient discontinuity, thereby improving the smoothness of the optimization process.

[0026] In summary, the method of the present application uses the path generated by the trajectory planning algorithm as a reference path, and the algorithm can obtain clearer directional guidance, thereby reducing the time wasted by random exploration. At the same time, the adaptive dynamic mechanism can make timely adjustments when the path deviates or the environment changes, making the learning process more efficient and converging faster; the adaptive dynamic reward mechanism can dynamically adjust the reward weight according to the environmental conditions, and the reference trajectory reward provides directional guidance. This combination can flexibly respond to real-time changes in the environment while maintaining trajectory accuracy; when the environment changes dynamically or new obstacles appear, the adaptive dynamic reward function combined with the reference trajectory can better respond to emergencies and ensure the rationality and consistency of the training trajectory. Even in a complex environment, the robot can stably complete the task.

[0027] See also Figure 2 , which shows a structural block diagram of a robot motion control system of the present application.

[0028] like Figure 2 As shown, the robot motion control system 200 includes an acquisition module 210 , a construction module 220 , an imitation module 230 and an output module 240 .

[0029] The acquisition module 210 is configured to acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; A construction module 220 is configured to construct a trajectory imitation reward function according to the optimal trajectory, and add the trajectory imitation reward function to the adaptive dynamic reward function to obtain a target adaptive dynamic reward function; An imitation module 230 is configured to control the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The output module 240 is configured to train and learn the motion trajectory of the robot according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally obtain the optimal control strategy of the robot, wherein the expression of the improved clipping mechanism is: , In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

[0030] It should be understood that Figure 2 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 2 The modules in it will not be described in detail here.

[0031] In some other embodiments, the embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor is caused to execute the robot motion control method in any of the above method embodiments; As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows: Acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; Constructing a trajectory imitation reward function according to the optimal trajectory, and adding the trajectory imitation reward function to an adaptive dynamic reward function to obtain a target adaptive dynamic reward function; Controlling the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The motion trajectory of the robot is trained and learned according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally the optimal control strategy of the robot is obtained, wherein the expression of the improved clipping mechanism is: , In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

[0032] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the robot motion control system, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the robot motion control system via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 The example of the bus connection is taken. The memory 320 is the computer-readable storage medium mentioned above. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the robot motion control method of the above method embodiment is realized. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the robot motion control system. The output device 340 may include display devices such as display screens.

[0034] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0035] As an implementation mode, the electronic device is applied to a robot motion control system and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: Acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; Constructing a trajectory imitation reward function according to the optimal trajectory, and adding the trajectory imitation reward function to an adaptive dynamic reward function to obtain a target adaptive dynamic reward function; Controlling the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The motion trajectory of the robot is trained and learned according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally the optimal control strategy of the robot is obtained, wherein the expression of the improved clipping mechanism is: , In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

[0036] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot motion control method, characterized in that: include: Acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; Constructing a trajectory imitation reward function according to the optimal trajectory, and adding the trajectory imitation reward function to an adaptive dynamic reward function to obtain a target adaptive dynamic reward function; Controlling the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The motion trajectory of the robot is trained and learned according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally the optimal control strategy of the robot is obtained, wherein the expression of the improved clipping mechanism is: ， In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

2. A robot motion control method according to claim 1, characterized in that: in, The initial state includes the angles, angular velocities, position and speed of each joint of the robot; the obstacle information includes the position and geometric dimensions of the obstacle; and the dynamic target information includes the geometric dimensions of the dynamic target, the position and the motion range of the dynamic target.

3. A robot motion control method according to claim 1, characterized in that: The step of planning the optimal trajectory of the robot using a trajectory planning algorithm according to the environmental information includes: According to the trajectory planning algorithm, a trajectory from the initial state to the target state is planned in space; The trajectory is smoothed according to the smoothing algorithm, and on the basis of satisfying all operation constraints, the trajectory is optimized using the numerical optimization algorithm to obtain the optimal trajectory of the robot.

4. A robot motion control method according to claim 1, characterized in that: The expression of the trajectory imitation reward function is: ， In the formula, is the trajectory imitation reward function, , , , , , are the importance of different reward functions, is the local refined position error reward function, is the trajectory coverage reward function, is the speed error reward function, is the acceleration error reward function, is the velocity direction matching reward function, is the trajectory time arc length reward function; Among them, the expression of the local refined position error reward function is: ， In the formula, is the number of key points, is the time weight of the key point of the i-th reference trajectory, is the key point of the actual trajectory, is the key point of the reference trajectory; The expression of the trajectory coverage reward function is: ， In the formula, is the total number of time steps, is the threshold value; The expression of the speed error reward function is: ， In the formula, is the actual trajectory speed, is the reference trajectory speed; The expression of the acceleration error reward function is: ， In the formula, is the actual trajectory acceleration, is the reference trajectory acceleration; The expression of the velocity direction matching reward function is: ， The expression of trajectory time arc length reward function is: ， ， In the formula, is the trajectory time arc length, is the actual trajectory time arc length, is the reference trajectory time arc length.

5. A robot motion control method according to claim 1, characterized in that: The adaptive dynamic reward function includes a dynamic reward function and a speed constraint reward function, and the dynamic reward function includes a target distance reward function , trajectory imitation reward function , obstacle avoidance reward function And the completion state reward function ; Among them, the expression of the adaptive dynamic reward function is: ， In the formula, is an adaptive dynamic reward function, is a factor based on the dynamic change of the distance between the robot and the obstacle; The dynamic reward function The expression is: ， In the formula, is the total number of time steps, is the discount factor; The expression of the target distance reward function is: ， ， ， In the formula, is the dynamic weight, and These are adjustment parameters used to control the speed and turning point of dynamic weight changes. represents the position of the robot, Indicates the location of the target point; The expression of the obstacle avoidance reward function is: ， In the formula, To control the coefficient of obstacle avoidance penalty, is the distance influence coefficient, is the distance between the robot and the obstacle; The expression of the completion state reward function is: ， In the formula, , are weight factors that control the degree of reward. The ideal time for the robot to complete the task, is the total number of time steps; Speed ​​Constraint Reward Function The expression is: ， In the formula, , are weight factors, is the number of robot joints, is the velocity of the jth joint, is the acceleration of the jth joint.

6. A robot motion control system, characterized in that: include: An acquisition module is configured to acquire environmental information before the robot moves, and plan the optimal trajectory of the robot using a trajectory planning algorithm based on the environmental information, wherein the environmental information includes the initial state of the robot, obstacle information, and dynamic target information; A construction module is configured to construct a trajectory imitation reward function according to the optimal trajectory, and add the trajectory imitation reward function to the adaptive dynamic reward function to obtain a target adaptive dynamic reward function; an imitation module, configured to control the robot to imitate the optimal trajectory based on the target adaptive dynamic reward function; The output module is configured to train and learn the motion trajectory of the robot according to the PPO deep reinforcement learning algorithm of the improved clipping mechanism, and finally obtain the optimal control strategy of the robot, wherein the expression of the improved clipping mechanism is: ， In the formula, To improve the clipping mechanism, , , , are the attenuation slopes of different clipping intervals, , , are the cropping widths of different cropping intervals, , , is the strategy update ratio.

7. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images