Robot speed planning method, system and device based on deep reinforcement learning

Abstract

The invention relates to a robot speed planning method, system and device based on deep reinforcement learning, and the method comprises the steps: constructing a deep reinforcement learning environment, and defining a state space and an action space of deep reinforcement learning; providing a comprehensive reward function; in a deep reinforcement learning environment, performing speed planning training and optimization processing on the robot based on a comprehensive reward function to obtain an optimal speed planning strategy; a speed decision-making model is generated according to the optimal speed planning strategy, the speed decision-making model is deployed to an intelligent decision-making module of a robot control system, and the robot control system obtains the current real state space and action space of the robot body and determines the optimal speed planning strategy of the robot body; and according to the optimal speed planning strategy, generating an instruction sequence of each joint speed to drive the robot body to move. According to the robot speed planning method, intelligent weighing and optimization can be carried out among a plurality of performance indexes.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method, system and device for robot speed planning based on deep reinforcement learning. Background Technology

[0002] With the continuous development of industrial automation and robotics, robots are being used more and more widely in many fields such as production, logistics, and services. Robot speed planning, as a key component of robot control systems, directly affects the efficiency, accuracy, and stability of robot task execution. Traditional robot speed planning methods are usually based on analytical models or empirical formulas. These methods often have many limitations when facing complex task scenarios, dynamically changing environments, and uncertainties in the robot's own parameters, making it difficult to meet the demands of modern robot systems for efficient, accurate, and adaptive speed planning.

[0003] On the one hand, traditional methods rely heavily on the robot's dynamic model, requiring precise model parameters for velocity planning. However, in practical applications, robot model parameters may deviate due to factors such as load variations and mechanical wear, leading to a decrease in the accuracy and reliability of velocity planning. On the other hand, traditional velocity planning methods struggle to effectively balance and optimize multiple performance indicators, failing to meet the diverse needs of complex task scenarios. Furthermore, when facing dynamically changing environments and tasks, traditional methods often require tedious manual parameter adjustments and repeated offline programming, lacking adaptive capabilities and reducing the robot's intelligence level and deployment flexibility. Summary of the Invention

[0004] This invention provides a method, system, and apparatus for robot speed planning based on deep reinforcement learning, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of this invention is a speed planning method for robots based on deep reinforcement learning, which includes:

[0006] Construct a deep reinforcement learning environment, establish the kinematics and approximate dynamics model of the robot in the deep reinforcement learning environment, and define the state space and action space of the deep reinforcement learning.

[0007] Provides a comprehensive reward function for velocity planning of agents;

[0008] In the deep reinforcement learning environment, the agent is trained and optimized for speed planning based on the comprehensive reward function to obtain the optimal speed planning strategy.

[0009] A speed decision model is generated based on the optimal speed planning strategy. The weight parameters of the speed decision model are solidified and saved. The speed decision model is then deployed to the intelligent decision module of the robot control system. The robot control system obtains the current real state space and motion space of the robot body. Based on the current real state space and motion space, the intelligent decision module determines the optimal speed planning strategy for the robot body.

[0010] The robot body generates a sequence of instructions for the speed of each joint based on the optimal speed planning strategy of the robot body to drive the robot body to move.

[0011] According to some embodiments of the present invention, the construction of a deep reinforcement learning environment, the establishment of a kinematic and approximate dynamic model of the robot in the deep reinforcement learning environment, and the definition of the state space and action space of the deep reinforcement learning include:

[0012] The open-source MuJoCo physics engine was selected as the simulation platform, and a deep reinforcement learning environment was built by combining it with the OpenAI Gym framework.

[0013] A kinematic and approximate dynamic model of the robot is established in the deep reinforcement learning environment. The kinematic and approximate dynamic model of the robot includes the robot's DH parameters, mass, inertia, and joint limit information.

[0014] The state space for deep reinforcement learning is defined based on the current angle and angular velocity of each joint of the robot, the error between the target path point and the current end effector position, the error between the target path point and the current end effector posture, and several key point information or curvature information of the future path segment.

[0015] The action space for deep reinforcement learning is defined based on the target angular acceleration or target angular velocity of each joint of the robot in the next control cycle.

[0016] According to some embodiments of the present invention, the comprehensive reward function provided for velocity planning of the agent includes:

[0017] Set the agent's task completion time reward, end effector trajectory tracking accuracy reward, motion stability reward, task completion time weight, end effector trajectory tracking accuracy weight, and motion stability weight;

[0018] A comprehensive reward function is obtained by multiplying the task completion time reward by its weight, the end effector trajectory tracking accuracy reward by its weight, and the motion stability reward by its weight. The expression for the comprehensive reward function is as follows:

[0019] Rt =w time r time,t +w path r path,t +w smooth r smooth,t

[0020] Among them, R t For a comprehensive reward function, w time As the weight of task completion time, r time,t As a reward for completing the task in time, w path r is the weight for the trajectory tracking accuracy of the end effector. path,t As a reward for the end effector trajectory tracking accuracy, w smooth r represents the weight for motion stability. smooth,t Rewards for smoother movement.

[0021] According to some embodiments of the present invention, the step of training and optimizing the agent's velocity planning based on the comprehensive reward function in the deep reinforcement learning environment to obtain the optimal velocity planning strategy includes:

[0022] The actor-critic algorithm was chosen as the deep reinforcement learning algorithm for processing continuous action space to construct the corresponding neural network structure;

[0023] In the deep reinforcement learning environment, the agent selects actions based on the current state and interacts with the environment to generate new states and immediate rewards.

[0024] The effects of the agent's chosen action, the new state, and the immediate reward are evaluated using the comprehensive reward function.

[0025] The parameters of the neural network are adjusted according to the update rules of the actor-critic algorithm until a speed planning strategy that maximizes the cumulative expected reward while satisfying various constraints is learned, or until the speed planning strategy performance converges or reaches the preset number of training rounds and performance indicators, thus obtaining the optimal speed planning strategy.

[0026] According to some embodiments of the present invention, the actor-critic algorithm includes an actor network and a critic network, wherein in the deep reinforcement learning environment, the agent selects an action based on the current state, interacts with the environment to generate a new state and an immediate reward, including:

[0027] The actor network outputs an action based on the agent's current state. The action includes a first action or a second action, wherein the first action is generated by a random policy and the second action is generated by a deterministic policy.

[0028] The actor network evaluates the value of the current state or the value of the current state-action pair;

[0029] In the deep reinforcement learning environment, the agent begins to perform a task. At each time step, the agent observes the current state, obtains the target action through the actor network, performs the target action and obtains a new state, and returns an immediate reward.

[0030] Based on the current state, the target action, the immediate reward, and the new state, an experience tuple is generated and stored in the experience replay buffer or used directly for online updates.

[0031] According to some embodiments of the present invention, adjusting the parameters of the neural network according to the update rules of the actor-critic algorithm until a speed planning strategy that maximizes the cumulative expected reward while satisfying various constraints is learned, or until the speed planning strategy performance converges or reaches a preset number of training rounds and performance indicators, to obtain the optimal speed planning strategy, includes:

[0032] The loss function of the actor network weights is processed by gradient descent to obtain a minimized loss function, which is then used to update the parameters of the actor network and the critic network.

[0033] Through interactive trial-and-error learning, the agent's speed planning strategy is continuously trained and optimized in order to maximize the cumulative expected reward in a complete task round, i.e., the objective function. The expression of the objective function is:

[0034]

[0035] Where J(θ) represents the objective function, θ represents the actor network weight parameters, τ represents a complete trajectory (s0, a0, r0, s1, ...), γ∈[0,1] is the discount factor, T represents the maximum step size of the round, and r t This represents the immediate reward the agent receives from the environment after performing an action at time step t.

[0036] The speed planning strategy of the agent is continuously optimized and trained, and the comprehensive reward function is used to guide the agent to learn the correct behavior pattern from historical experience data until the performance of the speed planning strategy converges or reaches the preset number of training rounds and performance indicators.

[0037] According to some embodiments of the present invention, the step of generating a speed decision model based on the optimal speed planning strategy, and solidifying and saving the weight parameters of the speed decision model, includes:

[0038] A speed decision model is generated based on the optimal speed planning strategy, and the speed decision model is composed of a neural network based on an actor-critic deep reinforcement learning framework;

[0039] The actor network generates the action with the greatest expected reward in the future based on the current state of the agent. The current state includes a vector of the robot's joint angles, angular velocities, and task information. The action with the greatest expected reward in the future includes the target angular velocity or target angular acceleration of each joint of the robot.

[0040] The critic network evaluates the long-term value of performing the action that maximizes the expected future reward in the current state and guides the updating of the actor network.

[0041] The weight parameters of the updated actor network are solidified and saved.

[0042] According to some embodiments of the present invention, the robot control system includes a state perception module, which acquires the current real state space and motion space of the robot body, and the intelligent decision-making module determines the optimal speed planning strategy for the robot body based on the current real state space and motion space, including:

[0043] The robot control system obtains the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body in real time through the state perception module. Based on the actual joint angles, joint angular velocities, end effector poses, and current task information, it generates the current real state space and transmits the current real state space as input to the intelligent decision module.

[0044] After receiving the current real state space, the intelligent decision-making model performs a fast forward reasoning calculation and outputs a normalized action vector.

[0045] The action vectors are denormalized to obtain the action space;

[0046] Based on the current real state space and the action space, the optimal speed planning strategy for the robot body is determined.

[0047] The technical solution of this invention also relates to a speed planning system for robots based on deep reinforcement learning, which includes:

[0048] The robot itself;

[0049] The robot control system includes a speed planning module, a state perception module, and an intelligent decision-making module. The intelligent decision-making module has a built-in intelligent decision-making model trained by deep reinforcement learning. The intelligent decision-making module is used to output the optimal speed planning strategy output by the intelligent decision-making model. The speed planning module is used to generate optimal joint speed parameters based on the optimal speed planning strategy output by the intelligent decision-making module. The state perception module includes state data for collecting the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body, as well as for receiving and transmitting the command execution status of each joint speed.

[0050] The present invention also relates to a computer device, including a memory and a processor, wherein the processor performs the above-described method when executing a computer program stored in the memory.

[0051] The present invention also relates to a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0052] The robot velocity planning method, system, and apparatus based on deep reinforcement learning provided in this invention have at least one of the following advantages or beneficial effects:

[0053] A deep reinforcement learning environment is constructed, within which a kinematic and approximate dynamic model of the robot is established. The state and action spaces for deep reinforcement learning are defined, thus completing the setup of the environment required for deep reinforcement learning training. This includes the precise loading of the robot model, the setting of task paths, the definition of physical constraints, and the concretization of the state and action spaces, laying a solid foundation for the subsequent learning of the agent. A comprehensive reward function is provided for the agent's velocity planning, guiding the deep reinforcement learning agent to learn effective velocity planning strategies in diverse tasks and environments during velocity planning training and optimization. The optimal behavior pattern is extracted from historical interaction experience, and the comprehensive reward function incentivizes the agent to reach the target quickly and accurately. Within the deep reinforcement learning environment, the agent's velocity planning is trained and optimized based on the comprehensive reward function. The network structure and hyperparameters of the model are adjusted according to the feedback from the comprehensive reward function to optimize the training effect and obtain the optimal velocity planning strategy.

[0054] Subsequently, a speed decision model is generated based on the optimal speed planning strategy. The weight parameters of the speed decision model are then fixed and saved, and the speed decision model is integrated into the intelligent decision module of the robot control system. In actual operation scenarios, the robot control system acquires the current state space and motion space of the robot body. Based on the current real state space and motion space, the intelligent decision module determines the optimal speed planning strategy for the robot body. The target speed or target angular velocity of each joint is calculated according to the optimal speed planning strategy and converted into a sequence of command sequences for each joint speed. This sequence of command sequences drives the robot body's movement, ensuring that the speed decision module can respond quickly to meet the real-time requirements of the robot body's motion.

[0055] Furthermore, additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the overall process of a robot speed planning method based on deep reinforcement learning, as provided in an embodiment of the present invention.

[0057] Figure 2 This is a detailed flowchart of step S100 in the robot speed planning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0058] Figure 3 This is a detailed flowchart of step S300 in the robot speed planning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0059] Figure 4 This is a detailed flowchart of step S320 in the robot speed planning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0060] Figure 5 This is a detailed flowchart of step S340 in the robot speed planning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0061] Figure 6 This is a flowchart of an embodiment of a robot speed planning method based on deep reinforcement learning provided by an embodiment of the present invention;

[0062] Figure 7 This is a first detailed flowchart of step S400 in the robot speed planning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0063] Figure 8 This is a second detailed flowchart of step S400 in the robot velocity planning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0064] Figure 9 This is a schematic diagram of the execution method of the robot velocity planning system based on deep reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0065] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0066] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a," "described," and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention. The term "and / or" as used herein includes any combination of one or more of the associated listed items.

[0067] It should be understood that although the terms first, second, third, etc., may be used to describe various elements in this invention, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of the invention, and similarly, a second element may also be referred to as a first element. Any and all instances or exemplary language (“e.g.,” “such as,” etc.) provided herein are intended only to better illustrate embodiments of the invention and, unless otherwise required, do not impose a limitation on the scope of the invention.

[0068] With the continuous development of industrial automation and robotics, robots are being used more and more widely in many fields such as production, logistics, and services. Robot speed planning, as a key component of robot control systems, directly affects the efficiency, accuracy, and stability of robot task execution. Traditional robot speed planning methods are usually based on analytical models or empirical formulas. These methods often have many limitations when facing complex task scenarios, dynamically changing environments, and uncertainties in the robot's own parameters, making it difficult to meet the demands of modern robot systems for efficient, accurate, and adaptive speed planning.

[0069] On the one hand, traditional methods rely heavily on the robot's dynamic model, requiring precise model parameters for velocity planning. However, in practical applications, robot model parameters may deviate due to factors such as load variations and mechanical wear, leading to a decrease in the accuracy and reliability of velocity planning. On the other hand, traditional velocity planning methods struggle to effectively balance and optimize multiple performance indicators, failing to meet the diverse needs of complex task scenarios. Furthermore, when facing dynamically changing environments and tasks, traditional methods often require tedious manual parameter adjustments and repeated offline programming, lacking adaptive capabilities and reducing the robot's intelligence level and deployment flexibility.

[0070] Based on this, embodiments of the present invention provide a robot speed planning method, system, and device based on deep reinforcement learning, which can effectively cope with complex task scenarios, dynamic environmental changes, and the robot's own uncertainties, and can intelligently balance and optimize multiple performance indicators for robot speed planning.

[0071] Please refer to the following. Figures 1 to 7 The velocity planning method, system, and apparatus for robots based on deep reinforcement learning provided in the embodiments of the present invention will be further described.

[0072] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the overall process of a robot speed planning method based on deep reinforcement learning, as provided in this embodiment of the invention. The robot speed planning method based on deep reinforcement learning includes, but is not limited to, steps S100 to S500. Specifically,

[0073] S100: Construct a deep reinforcement learning environment, establish the kinematics and approximate dynamics model of the robot in the deep reinforcement learning environment, and define the state space and action space of the deep reinforcement learning.

[0074] S200: Provides a comprehensive reward function for velocity planning of agents;

[0075] S300: In a deep reinforcement learning environment, the agent is trained and optimized for speed planning based on a comprehensive reward function to obtain the optimal speed planning strategy.

[0076] S400: Generates a speed decision model based on the optimal speed planning strategy, solidifies and saves the weight parameters of the speed decision model, and deploys the speed decision model to the intelligent decision module of the robot control system. The robot control system obtains the current real state space and motion space of the robot body. Based on the current real state space and motion space, the intelligent decision module determines the optimal speed planning strategy for the robot body.

[0077] S500: Generates a sequence of instructions for the speed of each joint based on the optimal speed planning strategy of the robot body to drive the movement of the robot body.

[0078] In some embodiments of the present invention, the robot speed planning method based on deep reinforcement learning includes: constructing a deep reinforcement learning environment, establishing a kinematic and approximate dynamic model of the robot in the deep reinforcement learning environment, and defining the state space and action space of the deep reinforcement learning, thereby completing the construction of the environment required for deep reinforcement learning training, including the accurate loading of the robot model, the setting of the task path, the definition of physical constraints, and the concretization of the state and action spaces, laying a solid foundation for the subsequent learning of the intelligent agent.

[0079] It provides a comprehensive reward function for speed planning of the agent. The comprehensive reward function is the core of reinforcement learning and determines the agent's behavioral goal. Through the comprehensive reward function, the deep reinforcement learning agent learns a strategy that can effectively plan speed in diverse tasks and environments during subsequent speed planning training and optimization. It can also extract the optimal behavior pattern from historical interaction experience. The comprehensive reward function can motivate the agent to reach the goal quickly and accurately, ensure the smoothness and efficiency of the movement process, and strictly abide by physical constraints.

[0080] In a deep reinforcement learning environment, a speed planning training and optimization process is performed on an agent based on a comprehensive reward function, for example, using a deep reinforcement learning algorithm to train the agent. The agent learns how to choose actions to maximize cumulative rewards through interaction with the environment. During training, the agent continuously tries different action strategies and adjusts them based on feedback from the comprehensive reward function. Then, experience replay is used to improve learning efficiency, adjusting the model's network structure and hyperparameters (such as learning rate and discount factor) to optimize training results. Through these training and optimization methods, the optimal speed planning strategy is obtained.

[0081] After training, a speed decision model is generated based on the optimal speed planning strategy. The weight parameters of the speed decision model are then fixed and saved, and the speed decision model is integrated into the intelligent decision module of the robot control system. In actual operation scenarios, the robot control system acquires the robot's current state space and motion space, including the actual angles and angular velocities of each joint, as well as current task information. This information is input into the intelligent decision module, which determines the optimal speed planning strategy for the robot based on the current real state space and motion space.

[0082] The target velocity or target angular velocity of each joint is calculated based on the optimal velocity planning strategy. The target velocity or target angular velocity is converted into a sequence of instructions for each joint velocity. The robot body motion is driven by the sequence of instructions for each joint velocity, ensuring that the velocity decision module can respond quickly to meet the real-time requirements of the robot body motion.

[0083] In summary, this invention utilizes deep reinforcement learning to directly learn velocity planning strategies from interactions with the environment, significantly reducing reliance on robot dynamics models. For real-world conditions such as uncertain model parameters, dynamically changing loads, or minor mechanical wear on the robot, the velocity planning achieved through deep reinforcement learning exhibits stronger robustness and adaptability. Through a comprehensive reward function, intelligent trade-offs and optimizations can be made among multiple, even conflicting, performance metrics (such as motion time, trajectory accuracy, motion stability, energy consumption, and strict adherence to physical constraints), resulting in a significant improvement in the robot's overall operational performance. By endowing the robot with the ability to autonomously learn and optimize velocity planning strategies, it can continuously adapt to complex tasks and dynamically changing working environments through interaction with simulation or real-world environments. This reduces reliance on tedious manual parameter adjustments and repetitive offline programming, thereby significantly improving the robot's intelligence level and deployment flexibility. By deploying the speed decision model into the intelligent decision module of the robot control system, the robot control system obtains the current real state space and motion space of the robot body. Through the learned optimal speed planning strategy, the robot body can complete the specified task in a smoother and more efficient manner, effectively shortening the operation cycle time. In complex task environments, it can efficiently, accurately, smoothly and adaptively plan the speed of each joint, while reducing unnecessary impacts and vibrations, effectively improving the production cycle and the quality of processed products, and also reducing mechanical wear and extending the service life of the robot body.

[0084] It is understood that, in some embodiments of the present invention, kinematic models are used to describe the relationship between robot joint positions and velocities, and are typically represented by forward kinematics (calculating end-effector position from joint angles) and inverse kinematics (calculating joint angles from end-effector position). Approximate dynamic models take into account the robot's mass and inertia, but can be simplified to linear or nonlinear models for predicting the impact of actions on the robot's state.

[0085] Reference Figure 2 As shown, Figure 2 This is a detailed flowchart of step S100 in the robot velocity planning method based on deep reinforcement learning provided in this embodiment of the invention. Step S100 includes, but is not limited to, steps S110 to S140. Specifically,

[0086] S110: The open-source MuJoCo physics engine is used as the simulation platform, and a deep reinforcement learning environment is built in conjunction with the OpenAI Gym framework;

[0087] S120: Establish the kinematic and approximate dynamic model of the robot in a deep reinforcement learning environment. The kinematic and approximate dynamic model of the robot includes the robot's DH parameters, mass, inertia, and joint limit information.

[0088] S130: Define the state space for deep reinforcement learning based on the current angle and angular velocity of each joint of the robot, the error between the target path point and the current end effector position, the error between the target path point and the current end effector posture, and several key point information or curvature information of the future path segment.

[0089] S140: Define the action space for deep reinforcement learning based on the target angular acceleration or target angular velocity of each joint of the robot in the next control cycle.

[0090] In some embodiments of the present invention, in step S100, a deep reinforcement learning environment is constructed, and a kinematic and approximate dynamic model of the robot is established in the deep reinforcement learning environment. The state space and action space of the deep reinforcement learning are defined. This step mainly completes the construction of the environment required for deep reinforcement learning training, including the accurate loading of the robot model, the setting of the task path, the definition of physical constraints, and the concretization of the state and action spaces, laying a solid foundation for the subsequent learning of the robot body.

[0091] First, a platform supporting high-fidelity physical simulation is selected. In this embodiment of the invention, the open-source MuJoCo physics engine is chosen as the simulation platform, and a reinforcement learning environment is built using the OpenAI Gym framework. The robot's kinematics and approximate dynamics model is loaded into this physical simulation platform. This model should include information such as the robot's DH parameters, mass, inertia, and joint constraints. The quality of the empirical data obtained through the simulation environment directly affects the learning effect.

[0092] Based on the current angle and angular velocity of each joint of the robot, the error between the target path point and the current end effector position, the error between the target path point and the current end effector posture, and several key point or curvature information of the future path segment, the state space of deep reinforcement learning is defined. It can be understood that the end effector position error is the error between the target path point and the current end effector position, the end effector posture error is the error between the target path point and the current end effector posture, and the key point or curvature information of the future path segment is used to provide look-ahead information for the path.

[0093] The action space of deep reinforcement learning is defined by the target angular acceleration or target angular velocity of each joint of the robot in the next control cycle. It can be understood that the target angular acceleration is the target angular acceleration of each joint in the next control cycle, and the target angular velocity is the target angular velocity of each joint in the next control cycle.

[0094] By defining the state space and action space of deep reinforcement learning, we can define the physical constraints and concretize the state and action spaces, laying a solid foundation for the subsequent learning of the robot itself.

[0095] In one embodiment of the invention, one or more typical task paths are defined for the robot to perform. For example, a typical task path might be: the robot's end effector starts from point P. start =(x s ,y s ,z s Move along a straight line segment to the midpoint P. mid =(x m ,y m ,z m Then move along another straight line segment to the target point P. end =(x e ,y e ,z e ).

[0096] Among them, P start This represents the starting position of the robot's end effector in a three-dimensional Cartesian coordinate system, with coordinates (x, y, y). s ,y s ,z s ) represent the specific positions of the starting point on the X, Y, and Z axes, respectively; P mid This represents an intermediate point or waypoint on the path, with coordinates (x, y, y). m ,y m ,z m The location of the intermediate point is defined by P. end This indicates the final target location that the robot's end effector needs to reach, and its coordinates (x, y, y). e ,y e ,z e This clarifies the final location of the target point.

[0097] In some tasks, paths may also contain circular arcs or more complex curves, such as B-spline curves. These paths can be represented by a series of discrete target points in Cartesian or joint space, or defined by parameterized curves. Path information is fed into the agent as part of its state.

[0098] Next, the state space and action space of the deep reinforcement learning agent are specifically defined. The state can include: the current angle and angular velocity of each joint of the robot, the error between the target path point and the current end effector position, the error between the target path point and the current end effector posture, and several key point or curvature information of the future path segment. Actions can be defined as the target angular acceleration or target angular velocity of each joint of the robot in the next control cycle. If the target angular velocity is chosen as the action, the formulas for the state space and action space can be written as follows:

[0099]

[0100] Among them, for the state-space formula: s t Let q be the state vector at time t, containing all the information needed for the agent's decision-making. t Let be the vector composed of the current angles of each joint of the robot at time t. Let e ​​be a vector consisting of the current angular velocities of each joint of the robot at time t. p,t Let e ​​be the position error vector between the current position of the robot's end effector and the target path point at time t. o,t Let p be the attitude error vector between the current posture of the robot's end effector and the target path point at time t. info,t This represents information related to future path segments at time t, such as the coordinates of several key points or the curvature of the path, used for path look-ahead. S is the state space, containing all possible state vectors s. t The set that constitutes the composition. The state vector represents a D S A real vector space of dimension 1.

[0101] A is the action space, which is the set of all allowed actions. The action output by the agent is the target angular velocity vector of all joints of the robot in the next control cycle. The target angular velocity vector is an n-dimensional real vector, where n is the number of joints (degrees of freedom) of the robot. This refers to the j-th component of the target angular velocity vector, i.e., the target angular velocity of the j-th joint. Let be the minimum allowable angular velocity of the j-th joint, which is its physical constraint lower limit. Let be the maximum allowed angular velocity of the j-th joint, which is its physical constraint upper limit. This indicates that the angular velocity limit applies to all joints from the 1st to the nth joint.

[0102] For a typical six-degree-of-freedom serial industrial robot, its motion vector a t At time t, it is usually defined as a 6-dimensional real vector, and the action vector a is... tThe expression is:

[0103] a t =(a 1,t ,a 2,t ,a 3,t ,a 4,t ,a 5,t ,a 6,t ) T ,

[0104] Among them, a t Indicates in a t At time t, the actor's complete motion vector output by the network, with the superscript T indicating transpose. j,t (where j = 1, ..., n represents the action vector a) t The j-th component in the expression corresponds to the motion command value of the j-th joint of the robot at time t, and each component is a scalar. For example, a 1,t This refers to the movement of the robot's first joint, a 2,t This refers to the movement of the second joint, and so on, up to the sixth joint.

[0105] If the motion is defined as the target angular velocity, then the j-th component a j,t The target angular velocity corresponding to the j-th joint Action components output by neural networks of kinematic and approximate dynamic models It is usually normalized to the range [-1, 1]. Therefore, it is necessary to determine the maximum physical angular velocity (absolute value) of the j-th joint. Perform inverse normalization; the expression for inverse normalization is:

[0106]

[0107] in, Let t be the target angular velocity sent to the j-th joint of the robot. This is the component corresponding to the j-th joint in the original output value of the policy network. The maximum physical angular velocity (absolute value) of the j-th joint.

[0108] If the motion is defined as target angular acceleration Similarly, based on the maximum angular acceleration Normalization and denormalization are performed. Finally, the neural network parameters and other hyperparameters related to the learning algorithm, such as the learning rate, discount factor γ, and exploration rate ε, are initialized for the deep reinforcement learning agent.

[0109] In some embodiments of the present invention, step S200 in the robot velocity planning method based on deep reinforcement learning includes, but is not limited to, steps S210 to S220. Specifically,

[0110] S210: Set the agent's task completion time reward, end effector trajectory tracking accuracy reward, motion stability reward, task completion time weight, end effector trajectory tracking accuracy weight, and motion stability weight;

[0111] S220: Based on the product of the task completion time reward and the task completion time weight, the product of the end effector trajectory tracking accuracy reward and the end effector trajectory tracking accuracy weight, and the product of the motion stability reward and the motion stability weight, a comprehensive reward function is obtained. The expression of the comprehensive reward function is:

[0112] R t =w time r time,t +w path r path,t +w smooth r smooth,t

[0113] Among them, R t For a comprehensive reward function, w time As the weight of task completion time, r time,t As a reward for completing the task in time, w path r is the weight for the trajectory tracking accuracy of the end effector. path,t As a reward for the end effector trajectory tracking accuracy, w smooth r represents the weight for motion stability. smooth,t Rewards for smoother movement.

[0114] In this embodiment of the invention, in order to guide the deep reinforcement learning agent to learn a strategy that can effectively plan speed in diverse tasks and environments during the training process in step S300, and to extract the optimal behavior pattern from historical interaction experience, a comprehensive reward function R that can fully reflect the quality of speed planning is provided. t This function not only motivates the agent to reach the target quickly and accurately, but also ensures the smoothness and efficiency of the motion process and strictly adheres to physical constraints.

[0115] The comprehensive reward function provided for velocity planning of the agent specifically includes:

[0116] Set the agent's task completion time reward, end effector trajectory tracking accuracy reward, motion stability reward, task completion time weight, end effector trajectory tracking accuracy weight, and motion stability weight, and define these parameters:

[0117] r time,t The task completion time reward incentivizes the agent to move towards the target point;

[0118] rpath,t End-effector trajectory tracking accuracy bonus: This bonus is awarded to the target point for reducing the error between itself and the trajectory during movement, thereby improving the accuracy of the trajectory.

[0119] r smooth,t Motion smoothness rewards encourage agents to reduce trajectory fluctuations in the end effector, resulting in smoother motion.

[0120] Rewards based on task completion time. time,t Weight w of task completion time time The product of the end effector trajectory tracking accuracy reward r path,t Weight w of end effector trajectory tracking accuracy path The product of motion stability and reward r smooth,t With motion stability weight w smooth The product of these factors yields the comprehensive reward function.

[0121] The adjustment process for the parameters of the comprehensive reward function can be carried out as follows: First, a relatively large weight can be set for the most important objective, such as the trajectory tracking accuracy of the end effector, such as w. path =1.0, while the weights of other optimization terms are initially set to smaller values, such as w time =0.1,w smooth =0.05). After training the agent for a period of time, by observing the agent's response to different weights, for example, if the end effector has high trajectory tracking accuracy but slow movement speed, the weights can be adjusted so that w path Decrease, w time Increase w to improve the agent's speed during movement. If the agent exhibits excessive jitter or uneven acceleration during movement, w can be increased. smooth The weights are adjusted to ensure smoother iteration. During this process, the weights can be gradually adjusted based on different optimization requirements and the analysis of cumulative or individual rewards until satisfactory overall performance is achieved. This comprehensive reward function will serve as the core evaluation criterion for the agent during the learning process in step S300, guiding the optimization direction of its velocity planning strategy.

[0122] Reference Figure 3 As shown, Figure 3 This is a detailed flowchart of step S300 in the robot velocity planning method based on deep reinforcement learning provided in this embodiment of the invention. Step S300 includes, but is not limited to, steps S310 to S340. Specifically,

[0123] S310: Select the actor-critic algorithm as the deep reinforcement learning algorithm for processing continuous action space to construct the corresponding neural network structure;

[0124] S320: In a deep reinforcement learning environment, the agent selects actions based on the current state and interacts with the environment to generate new states and immediate rewards.

[0125] S330: Evaluates the effects of the agent's chosen actions, new states, and immediate rewards through a comprehensive reward function;

[0126] S340: Adjust the parameters of the neural network according to the update rules of the actor-critic algorithm until a speed planning strategy that maximizes the cumulative expected reward under the premise of satisfying various constraints is learned, or until the speed planning strategy performance converges or reaches the preset number of training rounds and performance indicators, thus obtaining the optimal speed planning strategy.

[0127] In some embodiments of the present invention, in a deep reinforcement learning environment, the agent is trained and optimized for velocity planning based on a comprehensive reward function to obtain an optimal velocity planning strategy. This includes: selecting a deep reinforcement learning algorithm suitable for handling continuous action spaces. Since the actor-critic algorithm is suitable for robot motion in continuous spaces, this embodiment of the present invention selects the actor-critic algorithm as the training algorithm and constructs a corresponding neural network structure. The actor-critic algorithm includes an actor network and a critic network. It can be understood that the actor-critic algorithm is a reinforcement learning algorithm that combines a policy gradient method (actor) and a value function estimation (critic). It is implemented through two neural networks. The actor network is responsible for selecting actions based on the agent's current state and outputting the probability distribution of the actions or directly outputting the action values. The critic network is responsible for evaluating the quality of the current velocity planning strategy and outputting a state value function or a state-action value function.

[0128] Initialize the parameters of the actor network and the critic network, and define the optimizer (e.g., Adam) and loss function (e.g., mean squared error loss). At each time step t, the agent adjusts its current state s... t Use actor network to select action a t Execute action a t It interacts with the environment to obtain a new state s t+1 and instant rewards t Using a critic network, the action 'a' chosen by the agent is evaluated based on a comprehensive reward function. t The execution effect, the new state s t+1 and instant rewards t An evaluation will be conducted to assess the effectiveness of the current speed planning strategy.

[0129] Training continues, and in each training round, starting from the initial state, action a is performed. tData is collected, and the actor and critic networks are updated using this data. This process is repeated until the policy converges or a preset number of training epochs are reached. Throughout the learning process, the reward function r... t It constantly guides the agent to learn correct behavior patterns from historical experience data, enabling it to learn to maximize operating speed while meeting the requirements of stable operation, high precision, and compliance with physical constraints, thereby generating high-performance speed plans under various operating conditions.

[0130] By training the agent in a deep reinforcement learning environment using the actor-critic algorithm, the optimal speed planning strategy is obtained.

[0131] Reference Figure 4 As shown, Figure 4 This is a detailed flowchart of step S320 in the robot velocity planning method based on deep reinforcement learning provided in this embodiment of the invention. Step S320 includes, but is not limited to, steps S321 to S324. Specifically,

[0132] S321: The actor network outputs an action based on the agent's current state. The action includes a first action or a second action. The first action is generated by a random policy, and the second action is generated by a deterministic policy.

[0133] S322: The actor network evaluates the value of the current state or the value of the current state-action pair;

[0134] S323: Start the task in a deep reinforcement learning environment. At each time step, the agent observes the current state, obtains the target action through the actor network, executes the target action and obtains a new state, and returns an immediate reward.

[0135] S324: Generate experience tuples based on the current state, target action, immediate reward, and new state, and store the experience tuples in the experience replay buffer or use them directly for online updates.

[0136] Reference Figure 4 and Figure 6 As shown, in some embodiments of the present invention, in a deep reinforcement learning environment, the agent selects an action based on the current state, interacts with the environment to generate a new state and immediate reward, including:

[0137] The actor network is based on the agent's current state s t Output action a t Action a t This includes either a first action or a second action. The first action is generated by a random policy and is used to explore the environment, increasing policy diversity. The first action is expressed as: a t ~π θ (a t |s tThe second action is generated by a deterministic policy and is used to execute the current optimal policy. The second action is expressed as a. t =μ θ (s t The critic network is responsible for evaluating the agent's current state. t value or state-action pair (s t ,a t The value of )

[0138] Afterwards, the agent begins to perform tasks in the deep reinforcement learning environment. At each time step t, the agent observes the current state s. t The target action a is obtained through the actor network. t Execute target action a t Afterwards, the environment transitions to a new state s. t+1 and return an instant reward r t Among them, the instant reward r t Calculated using a comprehensive reward function. Based on the current state s. t Target action a t Instant rewards t and new state s t+1 Generate empirical tuples, the expression for which is (s t ,a t ,r t ,s t+1 ), will the empirical tuple (s t ,a t ,r t ,s t+1 The experience tuples are stored in the experience replay buffer (for off-policy algorithms) or used directly for online updates (for online policy algorithms), and a batch of experience tuples are randomly sampled from the buffer for updates.

[0139] Through the above steps, the agent is trained to perform velocity planning in a deep reinforcement learning environment using the actor-critic algorithm, and the optimal velocity planning strategy is finally obtained.

[0140] Reference Figure 5 As shown, Figure 5 This is a detailed flowchart of step S340 in the robot velocity planning method based on deep reinforcement learning provided in this embodiment of the invention. Step S340 includes, but is not limited to, steps S341 to S343. Specifically,

[0141] S341: The loss function of the actor network weights is processed by gradient descent to obtain the minimized loss function, and the parameters of the actor network and the critic network are updated using the minimized loss function;

[0142] S342: The speed planning strategy of the agent is continuously trained and optimized through interactive trial and error learning in order to maximize the cumulative expected reward in a complete task round, i.e., the objective function.

[0143] S343: Continuously optimize and train the agent's speed planning strategy, using a comprehensive reward function to guide the agent to learn the correct behavior patterns from historical experience data until the speed planning strategy performance converges or reaches the preset number of training rounds and performance indicators.

[0144] Reference Figure 5 and Figure 6 As shown, in some embodiments of the present invention, the parameters of the neural network are adjusted according to the update rules of the actor-critic algorithm until a speed planning strategy that maximizes the cumulative expected reward while satisfying various constraints is learned, or until the speed planning strategy performance converges or reaches a preset number of training rounds and performance indicators, thus obtaining the optimal speed planning strategy. This includes: the agent processes the loss function of the actor network weights using gradient descent based on collected empirical data to obtain a minimized loss function, and updates the parameters of the actor network and the critic network by minimizing the weight loss function L(θ). This process typically involves gradient descent, for example... Where α is the learning rate, θ is the set of weight parameters of the neural network to be updated, and L(θ) is the loss function with parameter θ as the variable, used to quantify the error between the prediction result and the true target. This represents the gradient of the loss function L(θ) with respect to the parameter θ. This formula defines how to apply the gradient of the loss function L(θ). The weight parameters θ of the actor network are updated with a learning rate α as the step size, thereby continuously optimizing the agent's speed planning strategy toward maximizing the cumulative reward.

[0145] Through extensive interactive trial-and-error learning, the agent's speed planning strategy is continuously improved in order to maximize the cumulative expected reward in a complete task round, i.e., the objective function. The expression for the objective function is:

[0146]

[0147] Where J(θ) represents the objective function, which is the objective function under policy π. θ The expected cumulative reward that can be obtained in a complete mission round. The parameter θ represents the weight parameters of the actor network. π θ The policy is defined by parameter θ. It is a function that outputs a probability distribution of action a or a definite action based on the current state s. τ represents a complete trajectory (s0, a0, r0, s1, ...). γ∈[0,1] is a discount factor used to balance the importance of short-term and long-term rewards. When γ is close to 1, the agent values ​​future long-term rewards more.t This is the immediate reward the agent receives from the environment after performing an action at time step t. By maximizing this objective function, the agent can learn an optimal velocity planning strategy π. θ This strategy can maximize long-term cumulative rewards while satisfying various constraints.

[0148] The agent's velocity planning strategy is continuously optimized and trained until its performance converges or reaches a preset number of training epochs / performance metrics. Throughout the learning process, the reward function r... t It constantly guides the agent to learn correct behavior patterns from historical experience data, enabling it to improve its operating speed while ensuring smooth operation, high precision, and compliance with physical constraints, thereby generating high-performance speed plans under various operating conditions.

[0149] Reference Figure 7 As shown, Figure 7 This is a first detailed flowchart of step S400 in the robot velocity planning method based on deep reinforcement learning provided in this embodiment of the invention. Step S400 includes, but is not limited to, steps S410 to S440. Specifically,

[0150] S410: Generate a speed decision model based on the optimal speed planning strategy. The speed decision model consists of a neural network based on an actor-critic deep reinforcement learning framework.

[0151] S420: The actor network generates the action with the greatest expected reward in the future based on the current state of the agent. The current state includes vectors of the robot's joint angles, angular velocities, and task information. The action with the greatest expected reward in the future includes the target angular velocity or target angular acceleration of each joint of the robot.

[0152] S430: Evaluate the long-term value of performing the action with the greatest expected future reward in the current state through a critic network, and guide the updating of the actor network;

[0153] S440: Solidify and save the weight parameters of the updated actor network.

[0154] In some embodiments of the present invention, a speed decision model is generated based on an optimal speed planning strategy, and the weight parameters of the speed decision model are solidified and saved. This includes: generating a speed decision model that generates high-quality speed planning for a given robot and task type based on the optimal speed planning strategy. The speed decision model is composed of a neural network based on an actor-critic deep reinforcement learning framework. The actor network determines the speed decision model based on the current state s. t Generate the action with the highest expected future reward. The action with the highest expected reward in the future The mathematical expression of can be defined as Current state s t Vectors containing the robot's joint angles, angular velocities, and task information; the action with the highest expected reward in the future. This refers to the target angular velocity or target angular acceleration of each joint of the robot.

[0155] The neural network structure is divided into an actor network and a critic network. The actor network outputs the mean and variance of a Gaussian distribution of its weight parameters. During deployment, to ensure the determinism of motion, the mean of the Gaussian distribution of the actor network's output weight parameters is used as the final action command. The actor-critic deep reinforcement learning framework also uses a critic network during training, denoted as... Used to evaluate the current state s t Perform the action that yields the greatest expected reward in the future. Its long-term value, expressed mathematically as: It receives input from state-action pairs and guides the updates of the actor network. After this velocity decision model has matured through training, the weight parameters of its actor network are solidified and saved, and then deployed into the control system of a real industrial robot, typically embedded in the high-level control layer of the robot's control system or a dedicated intelligent decision module.

[0156] Reference Figure 8 As shown, Figure 8 This is a second detailed flowchart of step S400 in the robot velocity planning method based on deep reinforcement learning provided in this embodiment of the invention. Step S400 includes, but is not limited to, steps S460 to S490. Specifically,

[0157] S460: The robot control system obtains the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body in real time through the state perception module. Based on the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint, it generates the current real state space and transmits the current real state space as input to the intelligent decision module.

[0158] S470: After receiving the current real state space, the intelligent decision-making model performs a fast forward inference calculation and outputs a normalized action vector.

[0159] S480: Perform inverse normalization on the motion vectors to obtain the motion space;

[0160] S490: Determine the optimal speed planning strategy for the robot body based on the current real state space and action space.

[0161] In some embodiments of the present invention, the robot control system acquires the current real state space and motion space of the robot body, and the intelligent decision-making module determines the optimal speed planning strategy for the robot body based on the current real state space and motion space. This includes: the robot control system acquires the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body in real time through the state perception module. This information constitutes the real state s at time t. t This data is then passed as input to the deployed intelligent decision-making model. The intelligent decision-making model receives the true state s at time t. t Then, a fast forward inference calculation is performed, instantly outputting a normalized action vector. For the output action vector Inverse normalization is performed to obtain the action space, and control commands with actual physical meaning are generated based on the action space. Finally, based on the current real state space and action space, the optimal velocity planning strategy for the robot body is determined.

[0162] In one embodiment of the present invention, if the action state is defined as the target angular velocity, then the target angular velocity of the j-th joint... The expression for the target angular velocity is:

[0163]

[0164] in, For the target angular velocity, For the j-th component of the action vector, Let be the physical maximum angular velocity of the j-th joint;

[0165] The robot generates a sequence of commands for the speed of each joint based on the target angular velocity, and sends the sequence of commands for each joint speed to the robot control system. The robot control system converts the sequence of commands for each joint speed into motor drive signals, thereby driving the robot body to move at the planned speed.

[0166] In some embodiments of the present invention, the deployed speed decision model directly outputs commands regarding the target speed or target acceleration, thus enabling real-time online speed planning. By periodically monitoring the robot's performance, when a performance degradation is detected, the speed decision model is fine-tuned or retrained offline using new data collected during actual operation to adapt to robot wear, load changes, or gradual changes in the working environment, thereby maintaining or further improving its performance and achieving a closed loop of continuous optimization. This combination of online inference and offline optimization enables the robot to quickly adapt to dynamic changes, achieving robust intelligent speed planning for the robot during actual operation.

[0167] This invention also provides a robot velocity planning system based on deep reinforcement learning, used to implement the robot velocity planning method based on deep reinforcement learning provided in the above embodiments. The robot velocity planning system based on deep reinforcement learning includes a robot body and a robot control system. The robot control system includes a velocity planning module, a state perception module, and an intelligent decision-making module. The intelligent decision-making module has a built-in intelligent decision-making model trained by deep reinforcement learning. The intelligent decision-making module is used to output the optimal velocity planning strategy output by the intelligent decision-making model. The velocity planning module is used to generate optimal joint velocity parameters based on the optimal velocity planning strategy output by the intelligent decision-making module. The state perception module includes state data for collecting the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body, as well as for receiving and transmitting the command execution status of each joint velocity.

[0168] Understandably, the robot body is the main component of the robot's velocity planning system based on deep reinforcement learning, executing specific tasks, and its motion is planned and managed by the robot control system. The robot control system is the core component of the robot's velocity planning system based on deep reinforcement learning, responsible for planning and managing the robot body's velocity, and includes the following three key modules:

[0169] Velocity planning module: This module includes a kinematic model. Based on the output of the intelligent decision-making module, the velocity planning module generates specific joint velocity parameters to guide the movement of the robot body.

[0170] State perception module: including data acquisition unit and execution feedback unit. The state perception module is used to collect the state data of the robot body (such as joint angles, angular velocity, end effector pose, etc.) and task information through the data acquisition unit, and to provide feedback on the execution status of speed commands through the execution feedback unit, so as to provide a basis for intelligent decision-making.

[0171] Intelligent decision-making module: includes an intelligent agent and has a built-in model trained by deep reinforcement learning. Based on the data provided by the state awareness module, it outputs the optimal speed planning strategy.

[0172] In one embodiment, the workflow of a robot velocity planning system based on deep reinforcement learning is as follows: (Refer to...) Figure 9As shown, the state perception module collects real-time state data of the robot body, including actual joint angles, joint angular velocities, end effector poses, and current task information, through the data acquisition unit. This data is then transmitted to the intelligent agent in the intelligent decision-making module via the execution feedback unit. Based on the data provided by the state perception module, the intelligent agent uses a built-in deep reinforcement learning model to calculate and output the optimal velocity planning strategy, which is then sent to the underlying controller of the robot control system. The velocity planning module receives the optimal velocity planning strategy output by the intelligent decision-making module and further generates specific velocity parameters for each joint based on the kinematic model. The robot body executes actions according to the velocity parameters generated by the velocity planning module. The state perception module monitors the execution status in real time and transmits feedback information to the intelligent decision-making module for dynamic adjustments.

[0173] Through deep reinforcement learning, the robot's speed planning system can dynamically adjust its speed planning strategy according to different tasks and environments, exhibiting strong adaptability. The goal of the deep reinforcement learning model is to maximize cumulative reward, enabling the robot's speed planning to output the optimal speed planning strategy, improving the robot's efficiency and accuracy in task execution. The state awareness module collects data in real time through the data acquisition unit and feeds back the execution status to the robot through the execution feedback unit. This allows the deep reinforcement learning-based robot speed planning system to quickly respond to environmental changes and task requirements, adjusting the speed planning in a timely manner to achieve a continuous optimization loop. This enables the robot to quickly adapt to dynamic changes, realizing robust intelligent speed planning in actual operation.

[0174] It should be understood that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0175] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0176] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention may also include the computer itself.

[0177] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0178] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A speed planning method for robots based on deep reinforcement learning, characterized in that, include: Construct a deep reinforcement learning environment, establish the kinematics and approximate dynamics model of the robot in the deep reinforcement learning environment, and define the state space and action space of the deep reinforcement learning. Provides a comprehensive reward function for velocity planning of agents; In the deep reinforcement learning environment, the agent is trained and optimized for speed planning based on the comprehensive reward function to obtain the optimal speed planning strategy. A speed decision model is generated based on the optimal speed planning strategy. The weight parameters of the speed decision model are solidified and saved. The speed decision model is then deployed to the intelligent decision module of the robot control system. The robot control system obtains the current real state space and motion space of the robot body. Based on the current real state space and motion space, the intelligent decision module determines the optimal speed planning strategy for the robot body. The robot body generates a sequence of instructions for the speed of each joint based on the optimal speed planning strategy of the robot body to drive the robot body to move.

2. The robot velocity planning method based on deep reinforcement learning according to claim 1, characterized in that, The process involves constructing a deep reinforcement learning environment, establishing a kinematic and approximate dynamic model of the robot within this environment, and defining the state space and action space for deep reinforcement learning, including: The open-source MuJoCo physics engine was selected as the simulation platform, and a deep reinforcement learning environment was built by combining it with the OpenAI Gym framework. A kinematic and approximate dynamic model of the robot is established in the deep reinforcement learning environment. The kinematic and approximate dynamic model of the robot includes the robot's DH parameters, mass, inertia, and joint limit information. The state space for deep reinforcement learning is defined based on the current angle and angular velocity of each joint of the robot, the error between the target path point and the current end effector position, the error between the target path point and the current end effector posture, and several key point information or curvature information of the future path segment. The action space for deep reinforcement learning is defined based on the target angular acceleration or target angular velocity of each joint of the robot in the next control cycle.

3. The speed planning method for robots based on deep reinforcement learning according to claim 1, characterized in that, The comprehensive reward function provided for velocity planning of the agent includes: Set the agent's task completion time reward, end effector trajectory tracking accuracy reward, motion stability reward, task completion time weight, end effector trajectory tracking accuracy weight, and motion stability weight; A comprehensive reward function is obtained by multiplying the task completion time reward by its weight, the end effector trajectory tracking accuracy reward by its weight, and the motion stability reward by its weight. The expression for the comprehensive reward function is as follows: Rx=w time r time,t +w path r path,t +w smooth r smooth,t Among them, R t For a comprehensive reward function, w time As the weight of task completion time, r time,t As a reward for completing the task in time, w path r is the weight for the trajectory tracking accuracy of the end effector. path,t As a reward for the end effector trajectory tracking accuracy, w smooth r represents the weight for motion stability. smooth,t Rewards for smoother movement.

4. The speed planning method for robots based on deep reinforcement learning according to claim 1, characterized in that, In the deep reinforcement learning environment, the agent is trained and optimized for velocity planning based on the comprehensive reward function to obtain the optimal velocity planning strategy, including: The actor-critic algorithm was chosen as the deep reinforcement learning algorithm for processing continuous action space to construct the corresponding neural network structure; In the deep reinforcement learning environment, the agent selects actions based on the current state and interacts with the environment to generate new states and immediate rewards. The effects of the agent's chosen action, the new state, and the immediate reward are evaluated using the comprehensive reward function. The parameters of the neural network are adjusted according to the update rules of the actor-critic algorithm until a speed planning strategy that maximizes the cumulative expected reward while satisfying various constraints is learned, or until the speed planning strategy performance converges or reaches the preset number of training rounds and performance indicators, thus obtaining the optimal speed planning strategy.

5. The robot velocity planning method based on deep reinforcement learning according to claim 4, characterized in that, The actor-critic algorithm includes an actor network and a critic network. In the deep reinforcement learning environment, the agent selects actions based on the current state, interacts with the environment to generate new states and immediate rewards, including: The actor network outputs an action based on the agent's current state. The action includes a first action or a second action, wherein the first action is generated by a random policy and the second action is generated by a deterministic policy. The actor network evaluates the value of the current state or the value of the current state-action pair; In the deep reinforcement learning environment, the agent begins to perform a task. At each time step, the agent observes the current state, obtains the target action through the actor network, performs the target action and obtains a new state, and returns an immediate reward. Based on the current state, the target action, the immediate reward, and the new state, an experience tuple is generated and stored in the experience replay buffer or used directly for online updates.

6. The velocity planning method for robots based on deep reinforcement learning according to claim 5, characterized in that, The process of adjusting the parameters of the neural network according to the update rules of the actor-critic algorithm until a speed planning strategy that maximizes the cumulative expected reward while satisfying various constraints is learned, or until the speed planning strategy performance converges or reaches a preset number of training rounds and performance indicators, to obtain the optimal speed planning strategy, includes: The loss function of the actor network weights is processed by gradient descent to obtain a minimized loss function, which is then used to update the parameters of the actor network and the critic network. Through interactive trial-and-error learning, the agent's speed planning strategy is continuously trained and optimized in order to maximize the cumulative expected reward in a complete task round, i.e., the objective function. The expression of the objective function is: Where J(θ) represents the objective function, θ represents the actor network weight parameters, τ represents a complete trajectory (s0, a0, r0, s1, ...), γ∈[0,1] is the discount factor, T represents the maximum step size of the round, and r t This represents the immediate reward the agent receives from the environment after performing an action at time step t. The speed planning strategy of the agent is continuously optimized and trained, and the comprehensive reward function is used to guide the agent to learn the correct behavior pattern from historical experience data until the performance of the speed planning strategy converges or reaches the preset number of training rounds and performance indicators.

7. The velocity planning method for robots based on deep reinforcement learning according to claim 5, characterized in that, The step of generating a speed decision model based on the optimal speed planning strategy, and then fixing and saving the weight parameters of the speed decision model, includes: A speed decision model is generated based on the optimal speed planning strategy, and the speed decision model is composed of a neural network based on an actor-critic deep reinforcement learning framework; The actor network generates the action with the greatest expected reward in the future based on the current state of the agent. The current state includes a vector of the robot's joint angles, angular velocities, and task information. The action with the greatest expected reward in the future includes the target angular velocity or target angular acceleration of each joint of the robot. The critic network evaluates the long-term value of performing the action that maximizes the expected future reward in the current state and guides the updating of the actor network. The weight parameters of the updated actor network are solidified and saved.

8. The speed planning method for robots based on deep reinforcement learning according to claim 7, characterized in that, The robot control system includes a state perception module, which acquires the robot's current real state space and motion space. The intelligent decision-making module, based on the current real state space and motion space, determines the optimal speed planning strategy for the robot, including: The robot control system obtains the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body in real time through the state perception module. Based on the actual joint angles, joint angular velocities, end effector poses, and current task information, it generates the current real state space and transmits the current real state space as input to the intelligent decision module. After receiving the current real state space, the intelligent decision-making model performs a fast forward reasoning calculation and outputs a normalized action vector. The action vectors are denormalized to obtain the action space; Based on the current real state space and the action space, the optimal speed planning strategy for the robot body is determined.

9. A velocity planning system for a robot based on deep reinforcement learning, used to implement the velocity planning method for a robot based on deep reinforcement learning as described in claim 8, characterized in that, include: The robot itself; The robot control system includes a speed planning module, a state perception module, and an intelligent decision-making module. The intelligent decision-making module has a built-in intelligent decision-making model trained by deep reinforcement learning. The intelligent decision-making module is used to output the optimal speed planning strategy output by the intelligent decision-making model. The speed planning module is used to generate the optimal speed parameters of each joint based on the optimal speed planning strategy output by the intelligent decision-making module. The state perception module includes state data for collecting the actual joint angles, joint angular velocities, end effector poses, and current task information of each joint of the robot body, as well as for receiving and transmitting the command execution status of each joint speed.

10. A computer device comprising a memory and a processor, characterized in that, When the processor executes a computer program stored in the memory, it performs the method as described in any one of claims 1 to 8.

Images