Training method and device for motion control strategy network of robotic arm
By generating an end-effector velocity vector that balances target approach and obstacle avoidance using a teacher controller as supervision for the student policy network, the problem of robots struggling to learn safe motion control in obstacle environments in existing technologies is solved, and the training effect of robots stably reaching the target position while avoiding obstacles is achieved.
Patent Information
- Application Number
- CN202611575677.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-12
- Publication Date
- 2026-11-10
AI Technical Summary
Existing robot motion control strategy networks struggle to effectively distinguish between passable and impassable paths in environments with obstacles, leading to robots easily entering collision zones, low training efficiency, and difficulty in learning safe obstacle avoidance motion control strategies.
The teacher controller generates an end-effector velocity vector that balances target approach and obstacle avoidance as the action supervision for the student policy network. By cooperating with the student policy network, the teacher controller uses privileged environment information to generate a safe end-effector velocity vector, which is then used as the training target for the student policy network, reducing environmental interaction and collision trial and error.
This improves the training effectiveness of the robot arm motion control strategy network, ensuring that the robot can stably reach the target position while avoiding obstacles, reducing training data loss due to collisions, and improving training efficiency.
Smart Images

Figure CN123060720A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotics, specifically to a method and apparatus for training a motion control strategy network for a robot arm. Background Technology
[0002] When a robot performs an end-effector task, it needs to generate control actions based on its current state and target position to gradually move the end-effector to the target location. In scenarios without obstacles, the robot can usually move in the direction from its current position to the target position; however, in scenarios with obstacles, the robot not only needs to approach the target position but also needs to avoid obstacles during its movement to prevent its trajectory from entering the collision zone.
[0003] In related technologies, robot motion control policy networks typically use a target position as the training objective and construct a reward function based on the positional error between the robot's current position and the target position. The smaller the positional error, the closer the robot's end-effector is to the target position. During training, the policy network continuously updates its parameters based on reward feedback to learn control actions that reduce positional errors. This type of method can guide the robot's end-effector to gradually approach the target position in barrier-free scenarios.
[0004] However, in scenarios with obstacles, positional error primarily reflects the distance between the robot's end effector and the target location, making it difficult to indicate whether the robot will collide with the obstacle while moving in a particular direction. For example, given two movement directions that both reduce positional error, one direction might bypass the obstacle and reach the target location, while the other might directly enter the area containing the obstacle. The reward signal generated solely based on positional error is insufficient to distinguish between traversable and impassable directions.
[0005] With the application of reinforcement learning and teacher-student distillation methods in robot motion control, some solutions are beginning to utilize privileged state information or complete state information available to the teacher side to assist in training the policy network on the student side. For example, CN120065751A discloses a robot motion control model training method based on deep reinforcement learning. This method uses a teacher encoder to encode the robot's privileged state information to obtain a teacher-side latent state representation, and uses a student encoder to encode the robot's historical motion state information to obtain a student-side latent state representation. During training, the student encoder is distilled by reducing the difference between the teacher-side and student-side latent state representations, and then the corresponding latent state representations are input into the policy network, which outputs the robot's motion actions. As another example, CN121361098A discloses a robot motion control policy network training method. This method extracts state representation features from the robot's historical state data, inputs these features into the motion control policy network to generate action policies, and uses a forward prediction model to predict the robot's state at multiple subsequent time steps. Based on the difference between the predicted and actual states and the reinforcement learning loss, the relevant network parameters are jointly updated.
[0006] The aforementioned schemes can utilize privileged state information, distillation of latent state representations, or prediction of future states to assist in training the policy network. However, their primary supervision targets are the latent state representations, predicted states, or reward values generated by environmental feedback. They do not directly provide supervision of the safe end-velocity motion, which simultaneously represents the target approach direction and obstacle avoidance direction. Therefore, in environments with obstacles, the student policy network still needs to learn, through interaction with the environment and random exploration, how to move while approaching the target location and avoiding obstacles.
[0007] When an obstacle exists between the robot's current position and the target position, the policy network may still directly approach the target position along the direction that reduces the position error, thus entering the collision zone where the obstacle is located. After a collision, the robot's motion trajectory may be interrupted, resulting in a reduction of effective training data in that trajectory that can be used for policy updates. In order to learn feasible obstacle avoidance paths, the policy network typically needs to repeatedly explore within a large state space and action space, and perform multiple collision trials for different obstacle positions and combinations.
[0008] In summary, existing training methods for robot motion control strategy networks mainly guide strategy updates based on the positional error between the robot's current end-effector position and the target position. They lack safe motion supervision signals that can simultaneously represent the target approach direction and obstacle avoidance direction, making it difficult to effectively distinguish between passable and impassable paths. Furthermore, the robot's trajectory is easily interrupted when it enters a collision zone. Therefore, it is difficult to ensure training efficiency while enabling the robot to learn a motion control strategy network that can both avoid obstacles and stably reach the target position. Summary of the Invention
[0009] Given that in related technologies, when a robotic arm performs an end-effector reaching task in an environment with obstacles, the strategy update is mainly guided by the positional error between the current position of the end-effector and the target position, which makes it difficult to directly reflect the collision risk corresponding to the current direction of movement. The student policy network needs to learn a safe direction of movement through a lot of environmental interaction and collision trial and error. This invention provides a training method and device for a robotic arm motion control policy network. The method generates an end-effector velocity that takes into account both target approach and obstacle avoidance through a teacher controller, and uses this end-effector velocity as the action supervision of the student policy network to improve the training effectiveness of the motion control policy network.
[0010] To achieve the above objectives, according to one aspect of the present invention, a method for training a robot arm motion control policy network is provided, comprising: The current state of the robot arm, the target position, and the privileged environment information are obtained as inputs to the teacher controller, so as to output the first end-effector velocity vector pointing to the target position and avoiding obstacles; The robot arm is controlled to move in the environment using the first end-effector velocity vector, and the current joint angle, current joint velocity, target position and corresponding first end-effector velocity vector of the robot arm during the movement process are constructed as a sample dataset; The current joint angle, current joint velocity, and target position in the sample dataset are used as inputs to the student policy network, which outputs a second end-effector velocity vector. The parameters are updated and trained based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain the motion control policy network.
[0011] The input to the student policy network may also include current environment information, which contains less information than the privileged environment information.
[0012] The privileged environment information includes at least one of obstacle geometry information, scene geometry information, and collision model information; the current environment information includes at least one of the distance information between the robot arm and the obstacle and the local distance field, wherein the local distance field is used to characterize the distance relationship between multiple spatial positions in the local space around the robot arm and the obstacle.
[0013] It can also obtain the current end-effector position corresponding to the current joint angle, current joint velocity, and target position in the sample dataset, and determine the relative displacement of the target position relative to the current end-effector position based on the current end-effector position and the target position; the input of the student policy network can include the current joint angle, current joint velocity, current end-effector position, target position, relative displacement, and current environmental information.
[0014] Specifically, when training the student policy network to update parameters, the current joint angle, current joint velocity, target position, and corresponding first end-effector velocity vector are obtained from the sample dataset. The current joint angle, current joint velocity, and target position are input into the student policy network to obtain the second end-effector velocity vector. The first end-effector velocity vector is used as the supervision target of the second end-effector velocity vector, and the parameters of the student policy network are updated according to the difference between the second end-effector velocity vector and the first end-effector velocity vector.
[0015] During training, the current joint angle, current joint velocity, and target position of the robot arm can be input into the student policy network to obtain a second end-effector velocity vector. The second end-effector velocity vector is used to control the robot arm's interaction with the environment. The motion result is determined based on the distance between the end-effector position before and after executing the second end-effector velocity vector and the target position. The safety judgment result is determined based on the spatial relationship between the robot arm and the obstacle. Feedback signals are generated based on the motion result and the safety judgment result to update the parameters of the student policy network.
[0016] Specifically, when the safety assessment result indicates that the robot arm has a collision risk, or the motion result indicates that the robot arm end effector is not moving toward the target position, the current state of the robot arm, the target position, and the privileged environment information under the current control cycle are obtained and input into the teacher controller to obtain a corrected end effector velocity vector for making the robot arm end effector avoid obstacles and move toward the target position.
[0017] Specifically, the current joint angle and current joint velocity corresponding to the time when the teacher controller generates the corrected end velocity vector can be obtained. The current joint angle, current joint velocity and target position are used as input data for new samples. The corrected end velocity vector is used as the corresponding supervision target to construct new samples. The new samples are added to the sample dataset. Based on the sample dataset after adding the new samples, the parameters of the student policy network are updated and trained.
[0018] In this process, after updating the parameters based on the sample dataset with newly added samples, the updated student policy network can be used to control the robot arm to interact with the environment again. If a collision risk occurs again or the end effector does not move toward the target position, the teacher controller generates a new corrected end effector velocity vector, constructs new samples based on the new corrected end effector velocity vector, and continues to train the student policy network based on the updated sample dataset.
[0019] The first and second end-effector velocity vectors can both include the linear velocity and angular velocity components of the robot arm's end effector. During training, the differences in linear velocity and angular velocity components between the second and first end-effector velocity vectors are determined, the training loss is determined based on the differences in linear velocity and angular velocity components, and the parameters of the student policy network are updated in the direction of reducing the training loss.
[0020] When the teacher controller generates the first end-effector velocity vector, it can determine the collision detection result between the robot arm and the obstacle based on the current state of the robot arm and the privileged environment information; determine the feasible movement direction that makes the end of the robot arm face the target position and avoid the obstacle based on the current state of the robot arm, the target position, the privileged environment information and the collision detection result; and then determine the first end-effector velocity vector based on the current state of the robot arm and the feasible movement direction.
[0021] According to another aspect of the present invention, a method for controlling the motion of a robot arm is also provided, comprising: acquiring the current joint angle, current joint velocity, and target position to be reached by the end effector of the robot arm; inputting the current joint angle, current joint velocity, and target position into a motion control strategy network trained by the above-described training method to obtain a motion control quantity for the end effector; and controlling the motion of the robot arm according to the motion control quantity.
[0022] Among them, the motion control policy network is a student policy network that completes parameter update training, and the motion control quantity is the end-effector velocity vector. During the deployment phase, the current joint angle, current joint velocity, and target position are input into the student policy network, and the student policy network outputs the end-effector velocity vector.
[0023] The current joint angle, current joint velocity, target position, and current environmental information are input into the motion control strategy network to obtain the motion control quantity of the end effector.
[0024] According to another aspect of the present invention, a training device for a robot arm motion control strategy network is also provided, comprising: a teacher control module, configured to acquire the current state, target position, and privileged environment information of the robot arm, and output a first end-effector velocity vector pointing to the target position and avoiding obstacles; a sample construction module, configured to control the robot arm to move in the environment using the first end-effector velocity vector, and construct a sample dataset by using the current joint angle, current joint velocity, target position, and corresponding first end-effector velocity vector during the movement process; and a network training module, configured to input the current joint angle, current joint velocity, and target position from the sample dataset into a student strategy network to obtain a second end-effector velocity vector, and perform parameter update training on the student strategy network based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain the motion control strategy network.
[0025] According to another aspect of the present invention, a robot arm motion control device is also provided, comprising: a state acquisition module for acquiring the current joint angle, current joint velocity, and target position to be reached by the end effector of the robot arm; a network inference module for inputting the current joint angle, current joint velocity, and target position into a motion control strategy network trained by the above-described training method to obtain a motion control quantity of the end effector; and a motion control module for controlling the movement of the robot arm according to the motion control quantity.
[0026] According to another aspect of the present invention, an electronic device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the training method of the above-described robot arm motion control strategy network or the above-described robot arm motion control method.
[0027] According to another aspect of the present invention, a robot arm is also provided, including a robot arm body and the aforementioned robot arm motion control device; the robot arm body includes an end effector and a joint drive mechanism for driving the robot arm joints; the robot arm motion control device is controlled to the joint drive mechanism, and is used to generate control commands based on the current joint angle, the current joint speed and the target position to be reached by the end effector, and send the control commands to the joint drive mechanism so that the actual end speed of the end effector follows the end speed vector generated by the motion control device.
[0028] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the training method of the above-described robot arm motion control strategy network or the above-described robot arm motion control method.
[0029] Compared with related technologies, the embodiments of the present invention have the following beneficial effects: This invention utilizes a teacher controller combined with privileged environmental information to generate a first end-effector velocity vector that enables the robot arm's end effector to move towards a target position while avoiding obstacles. This first end-effector velocity vector is then used as the motion supervision for the student policy network. Therefore, the student policy network can directly learn the robot arm's movement patterns in different states, moving towards the target position and avoiding obstacles. This reduces the reliance on repeated collisions and trial-and-error based solely on environmental feedback to explore obstacle avoidance directions, thus improving the training effectiveness of the motion control policy network.
[0030] After training, the student policy network is used as the motion control policy network for the actual control of the robotic arm. During the process of the robotic arm reaching the end effector, the motion control policy network can generate the corresponding end effector velocity vector based on the current state of the robotic arm and the target position. This enables the robotic arm to avoid obstacles according to the obstacle avoidance motion law trained during the movement towards the target position, thereby achieving autonomous obstacle avoidance during the task execution process and enabling the robotic arm to move safely towards the target position. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments are briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 A flowchart illustrating a training method for a robot arm motion control strategy network according to an embodiment of the present invention is shown.
[0033] Figure 2 The diagram illustrates a training process for updating the student policy network based on environmental interaction feedback in one embodiment of the present invention.
[0034] Figure 3 The student strategy training level curve is shown in one embodiment of the present invention.
[0035] Figure 4 The teacher strategy training level curve is shown in one embodiment of the present invention.
[0036] Figure 5 The termination failure rate curve of one embodiment of the present invention is shown.
[0037] Figure 6 A schematic diagram of the first motion state of a robot arm in a simulation training environment according to an embodiment of the present invention is shown.
[0038] Figure 7 A schematic diagram of the second motion state of a robot arm in a simulation training environment is shown in one embodiment of the present invention.
[0039] Figure 8 A flowchart illustrating a robot arm motion control method according to an embodiment of the present invention is shown.
[0040] Figure 9 This diagram illustrates the first motion state of a physical robot arm performing an end-effector task according to an embodiment of the present invention.
[0041] Figure 10 A schematic diagram of the second motion state of a physical robot arm performing an end-effector arrival task is shown in one embodiment of the present invention.
[0042] Figure 11 A schematic diagram of the third motion state of a physical robot arm performing an end-effector arrival task is shown in one embodiment of the present invention.
[0043] Figure 12 A schematic diagram of the fourth motion state of a physical robot arm performing an end-effector arrival task is shown in one embodiment of the present invention.
[0044] Figure 13 This diagram illustrates the fifth motion state of a physical robot arm performing an end-effector arrival task according to an embodiment of the present invention.
[0045] Figure 14 This diagram illustrates the sixth motion state of a physical robot arm performing an end-effector arrival task according to an embodiment of the present invention.
[0046] Figure 15 A structural block diagram of a training device for a robot arm motion control strategy network according to an embodiment of the present invention is shown.
[0047] Figure 16 A structural block diagram of a robot arm motion control device according to an embodiment of the present invention is shown.
[0048] Figure 17 A schematic diagram of the structural relationship of a robot arm according to an embodiment of the present invention is shown.
[0049] Figure 18 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown.
[0050] Figure 19 A schematic diagram of a robot platform according to an embodiment of the present invention is shown.
[0051] Figure 20 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown. Detailed Implementation
[0052] In the description of the embodiments of the present invention, the terms "first" and "second" are used only to distinguish different objects and are not used to indicate relative importance or quantitative relationship; "multiple" means at least two, unless otherwise expressly defined. A robot arm can also be called a mechanical arm, and the end effector can be a gripper, a tool end, or other actuating components that change spatial position and orientation with the joint movement of the robot arm.
[0053] The main technical terms involved in the embodiments of this invention are explained below. The following description is for the purpose of facilitating understanding of the subsequent implementation methods. The variables and data can be in appropriate data formats according to the structure of the robot arm, the type of task, and the training environment.
[0054] A robotic arm, also known as a mechanical arm, has multiple driveable joints and an end effector. A robotic arm can be a standalone mechanical arm or an upper limb mechanical arm within a robotic platform.
[0055] The current joint angle q represents the joint position or angle of each joint of the robot arm at the current control moment, and is used to characterize the current joint configuration.
[0056] Current joint velocity This represents the joint motion speed of each joint of the robot arm at the current control moment.
[0057] Current end position x ee This represents the current position of the end effector in the workspace. It can be obtained directly from the state of the robot arm, or determined from the current joint angles through kinematic relationships.
[0058] Current end attitude R ee This is the attitude information of the end effector at the current control moment, and is an optional input for the teacher side.
[0059] Target position x target The target location of the end effector specified for the current task.
[0060] relative displacement x rel The displacement relationship between the target position and the current end position can be represented as x. rel =x target -x ee .
[0061] Privileged environment information O refers to relatively complete environmental constraint information that can be used during the teacher-side training phase, including at least one of obstacle geometry information, scene geometry information, and collision model information.
[0062] Current environmental information d obsThe environmental information available to the student side has less information content compared to the privileged environmental information O in at least one aspect: information type, spatial coverage, or environmental geometric integrity. For example, the current environmental information may include at least one of the distance information between the robot arm and obstacles and the local distance field, but excludes all obstacle geometry information, scene geometry information, and collision model information used by the teacher controller for complete collision detection.
[0063] The teacher controller is a control module that generates a safe end-point speed under conditions that include privileged environment information.
[0064] First terminal velocity vector v T The end effector velocity vector output by the teacher controller is used to move the end effector toward the target position and avoid obstacles.
[0065] Teacher-side optional joint control quantity u T The joint space control quantity can be directly given by the teacher controller, and is an optional output in addition to the first end velocity vector.
[0066] The student policy network is a policy network that learns the velocity mapping relationship of the teacher-side end effector based on the robot arm's state and task information; in some implementations, at least one of the current environmental information, the current end effector position, and the relative displacement can also be used as additional inputs. The trained student policy network is used as a motion control policy network.
[0067] Second terminal velocity vector v S The terminal velocity vector output by the student policy network is compared with the first terminal velocity vector during the training phase and used as a reference value for terminal velocity during the deployment phase.
[0068] Student-side selectable joint speed The joint velocity control quantity can be obtained by mapping the second end-effector velocity vector through the kinematics of the robot arm.
[0069] The sample dataset D is a set of training samples consisting of student input states and corresponding teacher outputs, which can be represented as D={(s i ,v T,i )}.
[0070] The training loss L is used to characterize the difference between the student's output and the teacher's output. In one implementation, the learning objective is to reduce ||vS-vT|| or the square of it.
[0071] The feasible motion direction is the end effector motion direction determined by the teacher controller in combination with the current state of the robot arm, the target position, the privileged environment information and the collision detection results. It is used to make the end effector move toward the target position and avoid obstacles.
[0072] The motion result is used to indicate whether the end effector moves toward the target position after the second end velocity vector output by the student policy network is executed. It can be determined based on the change in distance between the end position and the target position before and after execution.
[0073] The safety assessment result indicates whether there is a collision risk in the current movement of the robot arm, and can be determined based on the spatial relationship between the robot arm and the obstacle.
[0074] The feedback signal is the training feedback formed based on the motion results and safety judgment results, which is used to further update the parameters of the student policy network during the interaction between the student policy network and the environment.
[0075] The corrected end-effector velocity vector refers to the safe end-effector velocity vector regenerated by the teacher controller based on the current state, target position, and privileged environment information when the current output of the student policy network poses a collision risk or fails to move the end effector toward the target position.
[0076] New samples refer to training samples formed when the teacher controller generates the corrected end-effector velocity vector, using the corresponding current joint angle, current joint velocity, and target position as input data, and the corrected end-effector velocity vector as the supervision target.
[0077] This embodiment employs a teacher controller and a student policy network to train the robot arm's motion control policy network. During the training phase, the teacher controller utilizes privileged environment information, including environmental constraints, to generate a safe end-effector velocity based on the robot arm's current state and task objective. The teacher controller controls the robot arm's movement and records the corresponding states and teacher outputs, forming the training samples required by the student policy network. The student policy network learns the correspondence between the robot arm's state, task information, and end-effector velocity based on these samples, and after training, it is used as the motion control policy network.
[0078] On the teacher's side, the input at a control moment can be represented as the current joint angle q and the current joint velocity. Current end position x ee Target position x target And privileged environment information O; when it is necessary to control the end effector attitude, the current end effector attitude R can also be input. ee The teacher controller outputs the first end velocity vector v. T The corresponding mapping relationship can be represented as: .
[0079] Wherein, the first end velocity vector v TThis is the safe end-effector velocity generated under environmental constraints. The first end-effector velocity vector can include the linear velocity and angular velocity components of the end effector. In addition to directly outputting the first end-effector velocity vector, the teacher controller can also output the joint space control quantity u corresponding to this end-effector motion. T , as an optional implementation.
[0080] On the student's side, in the basic implementation, the current joint angle q and the current joint velocity can be... and target position x target As input. In a further implementation, the current end position x can also be input. ee The relative displacement x of the target position with respect to the current end position rel and current environmental information d obs Among them, x rel =x target -x ee Current environmental information d obs The amount of information is less than the privileged environment information O on the teacher's side. The student policy network outputs the second terminal velocity vector v. S The corresponding mapping relationship can be represented as: .
[0081] Where, π θ Let θ represent the student policy network. The output of the student policy network is the second terminal velocity vector v. S The primary method is to obtain the joint space control quantities, such as joint velocities, by kinematic mapping based on the second end velocity vector.
[0082] Training data from both the teacher and student sides can be organized according to state-action pairs. Let 's' represent the robot state and task information used for student policy learning, and v... T This represents the first end velocity vector given by the teacher controller in the corresponding state, with v S Let v represent the second-terminal velocity vector predicted by the student policy network. The learning objective of the student policy network is to reduce v. S With v T The differences between them. The sample dataset can be represented as: D={(s i ,v T,i )}.
[0083] In one implementation that uses the squared error as the distillation objective, the parameter update objective can be expressed as: min θ E[||π θ (s)-v T || 2 ].
[0084] The aforementioned data relationships allow the safe end-effector velocity generated by the teacher controller to be directly used as motion supervision for the student policy network, eliminating the need for the student policy network to independently probe the motion direction based solely on the positional error between the end-effector and the target position. Through samples from different robot arm states and task conditions, the student policy network learns end-effector motion patterns that allow it to approach the target position while avoiding environmental constraints.
[0085] In the aforementioned teacher-student data relationship, state 's' is used to uniformly represent the state and task information of the robotic arm participating in learning on the student side. When using basic input, 's' can be composed of the current joint angle 'q' and the current joint velocity. and target position x target Composition; when using extended input, s can also include the current end position x. ee Relative displacement x rel And optional obstacle distance feature d obs Regardless of the input combination used, the state s in the same sample and the teacher output v T All correspond to the same robot instance and the same control moment; the student outputs v. S It is also based on this state s.
[0086] The teacher-side can optionally output the joint space control quantity u. T The joint velocities available on the student side The terminal velocity distillation relationship remains unchanged. The teacher controller directly outputs u. T At that time, it can be executed by the underlying joint interface; the student policy network outputs v S Subsequently, it can also be obtained through the kinematic relationships of the robotic arm. During training, the teacher's end velocity v can still be used. T As the student's terminal velocity v S The supervisory objective is to enable the teacher's environmental constraint information to participate in the student's strategy network training through action supervision, without requiring the input of complete privileged environmental information O into the student's strategy network.
[0087] Figure 1 A training process for a robot arm motion control strategy network is presented, which includes steps S101 to S104.
[0088] Step S101: Obtain the current state of the robot arm, the target position, and the privileged environment information as input to the teacher controller, and output the first end-effector velocity vector pointing to the target position and avoiding obstacles.
[0089] The current state represents the motion state of the robot arm at the current control moment, and may include at least the current joint angle and current joint velocity. When the teacher controller needs to directly use the end effector pose information, the current state may also include the current end effector position and current end effector orientation. The target position represents the position that the end effector is required to reach in the current task. Privileged environment information represents the environmental constraints available to the teacher.
[0090] Privileged environment information may include at least one of obstacle geometry information, scene geometry information, and collision model information. Obstacle geometry information may record the position, size, shape, or orientation of obstacles; scene geometry information may record the geometric boundaries within the robot arm's workspace; collision model information may represent the geometric relationships between the robot arm's links and environmental objects used for collision detection. The privileged environment information O is relative to the student's selectable current environment information d. obs It has a more complete environmental description.
[0091] In some embodiments, acquiring the current state of the robot arm, the target position, and privileged environment information as input to the teacher controller to output a first end-effector velocity vector pointing towards the target position and avoiding obstacles includes: determining the collision detection result between the robot arm and the obstacle based on the current state of the robot arm and the privileged environment information; determining a feasible movement direction that causes the end-effector of the robot arm to point towards the target position and avoid obstacles based on the current state of the robot arm, the target position, the privileged environment information, and the collision detection result; and then determining the first end-effector velocity vector based on the current state of the robot arm and the feasible movement direction.
[0092] Within a control cycle, the first end-effector velocity vector can be directly used as an end-effector velocity reference to control the robot arm's movement; alternatively, it can be first converted into a joint space control variable and then executed by the respective joint drive mechanisms. The conversion method used depends on the robot arm's underlying control interface and is not limited thereto.
[0093] Collision detection results can include at least one of the following: whether the robot arm has collided, whether it has entered a preset safe distance range, and the distance relationship between the robot arm and the obstacle. In some embodiments, the collision risk can be determined based on the minimum distance between each link of the robot arm and the obstacle and the preset safe distance; in a simulation training environment, the collision status or distance information output by the simulation environment can also be obtained. When determining a feasible motion direction, the target approach direction can be determined first based on the current end-effector position and the target position; when there is a collision risk in the target approach direction, the target approach direction is adjusted according to the position of the obstacle relative to the robot arm to obtain a feasible motion direction that avoids the obstacle and maintains the tendency to move towards the target position.
[0094] When determining collision detection results, the links and end effector of the robot arm can be considered as collision detection objects. For a given control moment, the positional relationship of each link in a unified coordinate system can be obtained based on the current joint angles. This positional relationship, combined with obstacle geometry information, determines the spatial distance between each link and the obstacle. When multiple obstacles exist, the distance between each collision detection object and each obstacle can be obtained separately, and the distance relationship most relevant to the current motion can be determined. The resulting collision detection results not only indicate whether contact has occurred but also reflect whether the robot arm has entered a preset safe distance range before actual contact occurs.
[0095] The preset safety distance can be set according to the structural dimensions, control cycle, and movement speed of the robot arm. When the robot arm's end effector speed is high, a relatively large safety distance can be used, allowing the teacher controller to change the movement direction before the robot arm approaches the obstacle. When the robot arm's end effector speed is low, the safety distance can be reduced while ensuring a safety margin. For the same training dataset, the definition of the safety distance is consistent with the selection of collision detection objects, ensuring that collision detection results in different samples have consistent meaning.
[0096] A feasible direction of motion can be determined based on the target approach direction. First, the basic direction of motion of the end effector toward the target position is obtained based on the current end effector position and the target position. Then, the safety of continuing motion along this direction is assessed in conjunction with collision detection results. When the basic direction of motion conflicts with the area where an obstacle is located, the basic direction of motion can be deflected according to the positional relationship of the obstacle relative to the end effector or related links, so that the adjusted direction moves away from the collision risk area. The adjusted direction still retains a component toward the target position, thus avoiding the end effector continuously deviating from the mission target simply to move away from the obstacle.
[0097] When further limiting the motion amplitude is required, the magnitude of the first end effector velocity vector can be determined jointly based on the distance from the end effector to the target position and the distance from the robot arm to nearby obstacles. A larger velocity amplitude can be used when the end effector is far from the target position and has a large safety margin; the velocity amplitude can be reduced when approaching the target position or obstacles. By determining the motion direction and velocity amplitude based on the target approach relationship and environmental safety relationship respectively, the first end effector velocity vector generated by the teacher controller can simultaneously reflect the task objective and environmental constraints within a control cycle, facilitating its direct use as motion supervision for the student policy network. When necessary, the first end effector velocity vector in adjacent control cycles can also be limited or smoothed.
[0098] The obstacle geometry, scene geometry, and collision model information used by the teacher controller can come from known scene models in the simulation environment or from complete environment modeling results obtainable during the training phase. Since the teacher can directly use the above information to make a safety judgment on the current direction of motion, the first end velocity vector already contains motion constraints on both target approach and obstacle avoidance before it is used as a supervision target. The student policy network does not need to rely solely on post-collision feedback to search for a safe direction of motion.
[0099] Step S102: Use the first end-effector velocity vector to control the movement of the robot arm in the environment, and construct a sample dataset by taking the current joint angle, current joint velocity, target position and corresponding first end-effector velocity vector of the robot arm during the movement process.
[0100] During the movement of the robot arm, the teacher controller continuously generates the first end-effector velocity vector according to the control cycle, and records the robot arm state and task information corresponding to the first end-effector velocity vector, thereby forming multiple training samples.
[0101] For the i-th sample, the current joint angle, current joint velocity, and target position can be used as the sample input s. i The basic components are the first end velocity vector v at the same control moment. T,i As a supervised target, if extended student input is used, the current end position, relative displacement, and current environmental information can also be recorded along with the sample. Multiple samples constitute the aforementioned sample dataset D.
[0102] When constructing samples, the current joint angle, current joint velocity, target position, and first end effector velocity vector all use data corresponding to the same control moment. After generating the first end effector velocity vector within the control cycle, the teacher controller can simultaneously record the joint state and task target at that control moment and establish the same sample record for this set of data. For multiple samples obtained from continuous control, each sample is saved according to its own control moment, and the robot state at the previous control moment is not combined with the teacher output at the next control moment.
[0103] When multiple robot arm instances are run simultaneously in the training environment, the samples of each instance can also include instance identifiers or be kept corresponding to each other by the data management process. A training batch can obtain samples from different robot instances and different control moments, but within a single sample, the student-side input and the teacher-side supervision target maintain a one-to-one correspondence. This avoids mismatches between the input state and the first end-effector velocity vector caused by parallel sampling or different control timings, ensuring that the mapping learned by the student policy network is consistent with the state when the teacher controller actually generates the safe end-effector velocity.
[0104] For training using extended input, the current end-effector position, relative displacement, and current environment information are recorded together with the corresponding current joint angle, current joint velocity, target position, and first end-effector velocity vector. Since these data are used to describe the robot arm state, task state, and local environment state at the same control moment, data can be read and arranged on a sample-by-sample basis before batch training, without needing to reassemble the corresponding data from different time series during the network training phase.
[0105] Step S103: Use the current joint angle, current joint velocity, and target position in the sample dataset as input to the student policy network to output the second end velocity vector.
[0106] In some implementations, the student policy network does not directly access the privileged environment information used by the teacher. Thus, after training, the student policy network can perform network inference solely based on the robot arm's own state and the task objective.
[0107] It is understandable that in an operating environment where the position and shape of obstacles are basically fixed during the training and deployment phases, the student policy network can complete network inference based solely on the robot arm's own state and the task objective after training. In this embodiment, an environmental acquisition device may not be required during the deployment phase.
[0108] In some embodiments, using the current joint angle, current joint velocity, and target position from the sample dataset as input to the student policy network includes: using the current joint angle, current joint velocity, target position, and current environmental information from the sample dataset as input to the student policy network; wherein the amount of information in the current environmental information is less than the amount of information in the privileged environmental information.
[0109] Specifically, the current joint angle, current joint velocity, target position, and current environmental information from the sample dataset can be used as inputs to the student policy network. The current environmental information reflects the motion control-related environmental state of the robot arm's current movement, and its information content is less than that of the privileged environmental information used by the teacher controller. The teacher controller can use relatively complete environmental constraint information to determine the first end-effector velocity vector, which balances target approach and obstacle avoidance, while the student policy network predicts the second end-effector velocity vector based only on the robot arm's state, task information, and relatively limited current environmental information. Thus, while preserving the differences between the teacher and student environmental information, the student policy network can distinguish different current local environmental states and learn corresponding safe end-effector velocity mappings for each environmental state. During training, the current environmental information, current joint angle, current joint velocity, target position, and corresponding first end-effector velocity vector use data from the same control time. During deployment, the current environmental information is provided to the motion control policy network according to the same data definition as during training to maintain consistency in the meaning of the network inputs.
[0110] Current environmental information can be obtained in different ways depending on the type of environment in which the robotic arm is located. During the simulation training phase, current environmental information can be provided directly by the simulation environment. For example, multiple query positions can be pre-set at the end effector of the robotic arm, one or more links, or around the robotic arm, and the distances between each query position and nearby obstacles can be obtained from the simulation environment. The current environmental information is then formed based on these distances. Alternatively, a preset local spatial range can be selected around the robotic arm, multiple spatial query positions can be set within this range, and the distances between each spatial query position and obstacles can be determined. These multiple distance values form local distance information describing the current local obstacle distribution.
[0111] The student side does not need to obtain all the environmental information used by the teacher side for the complete environmental constraint calculation. Instead, it can predict the second end-effector velocity vector based on relatively limited current environmental information, robot arm state, and task information.
[0112] It is understandable that in an operational environment where the position, number, or distribution of obstacles may change, using the current environmental information that the student side can obtain during the actual deployment process as input to the student policy network can improve the adaptability of the motion control policy network to changing obstacle environments. By providing the student policy network with current environmental information that contains less information than privileged environment information, the student policy network can obtain information related to the current local obstacle distribution without introducing complete obstacle geometry information, scene geometry information, and collision model information from the teacher side. This allows the second end velocity vector to be adjusted according to the corresponding local environmental relationship when the local environment changes, and reduces the dependence on complete environment modeling during the deployment phase.
[0113] In this embodiment, during the deployment phase, an environmental acquisition device is installed on the robot or robot arm to collect local environmental information around the robot arm; the environmental acquisition device can be at least one of an image sensor, a depth sensor, a distance sensor, and a lidar.
[0114] In some embodiments, privileged environment information may include at least one of obstacle geometry information, scene geometry information, and collision model information; current environment information may include at least one of distance information between the robot arm and the obstacle and a local distance field, wherein the local distance field is used to characterize the distance relationship between multiple spatial locations in the local space around the robot arm and the obstacle.
[0115] Among them, distance information can characterize the distance between the end effector or one or more links of the robot arm and a nearby obstacle; the local distance field is used to characterize the distance relationship between multiple spatial positions in the local space around the robot arm and the obstacle.
[0116] A local distance field can be used to represent the distribution of obstacles within a certain spatial range near a robotic arm. Multiple query locations can be selected around the end effector, the robotic arm itself, or a preset working area, and the distance from each query location to the nearest obstacle can be determined. These multiple distance values are then combined into local environmental features according to a preset order and input into the student policy network. This local distance field provides local spatial relationships directly related to the current motion; it does not require the reconstruction of the complete geometry of obstacles, nor does it require the provision of a complete scene model used by the teacher controller for collision calculations.
[0117] Before inputting the student policy network, distance values can be processed according to a pre-defined data range during training. For example, distances exceeding the perception range can be kept at an upper limit, ensuring that obstacles that are far away and have little impact on current control are represented consistently. When normalizing different distance values, the same normalization parameters are used in both the training and deployment phases. This allows the current environmental information to have a stable data scale across different control cycles, while maintaining its informational differences from the privileged environmental information on the teacher's side.
[0118] When the student side uses current environmental information, it does not change the relationship between the teacher controller and privileged environmental information in generating supervisory actions. The teacher controller can still use obstacle geometry, scene geometry, and collision model information to determine the first end-effector velocity vector, while the student policy network only uses the local distance relationships provided by the current environmental information for prediction. Through this input configuration, the student policy network can obtain information about the current local obstacle changes without needing to build the same complete environmental constraint model as the teacher side during the deployment phase.
[0119] In the presence of multiple nearby obstacles, the current environment information can simultaneously retain multiple distance relationships, rather than simply recording a single nearest distance. For example, the distances to the nearest obstacles can be recorded separately for the end effector and several pre-selected links, or multiple distance values for query positions can be retained in the local distance field. When the robot arm's movement causes a change in the nearest obstacle object, the features are still updated according to the pre-determined query object and its arrangement, enabling the student policy network to adjust the second end effector velocity vector based on changes in the current local environment.
[0120] The main difference between current environment information and privileged environment information lies in the fact that the student side only uses the local environment description required to complete network inference, while the teacher side can utilize a more complete obstacle geometry and collision model for safe motion calculations. During training, both can be acquired simultaneously in the same simulation environment, but are fed into the teacher controller and student policy network respectively; during deployment, only the current environment information needs to be provided according to the input definition of the student policy network. This allows the more complete environmental constraints from the teacher side to be transmitted to the student policy network through the first end velocity vector, while simultaneously controlling the range of environmental input required by the deployment side.
[0121] In some embodiments, the method further includes: obtaining the current end-effector position corresponding to the current joint angle, current joint velocity, and target position in the sample dataset, and determining the relative displacement of the target position relative to the current end-effector position based on the current end-effector position and the target position; using the current joint angle, current joint velocity, target position, and current environmental information in the sample dataset as input to the student policy network, including: using the current joint angle, current joint velocity, current end-effector position, target position, relative displacement, and current environmental information as input to the student policy network.
[0122] Relative displacement represents the spatial relationship from the current end position to the target position. When input together with the target position, the student policy network can simultaneously obtain information about the absolute task position and the relationship between the current end position and the target.
[0123] When the current end effector position and the target position are represented using the same coordinate system, the relative displacement can be directly obtained from the difference between their coordinates. The direction of the relative displacement represents the spatial direction of the end effector from the current position toward the target position, and the magnitude of the relative displacement corresponds to the distance from the end effector to the target position. If the target position is updated during task execution, the relative displacement can be redefined based on the new target position in each control cycle, ensuring that the relative displacement input to the student policy network always corresponds to the current task target.
[0124] When the current end-effector position, target position, and relative displacement are input into the student policy network, these three types of data serve different roles. The current end-effector position represents the absolute position of the end-effector in the workspace, the target position represents the position required by the task, and the relative displacement directly represents the relative spatial relationship between the two. When the number of samples is limited or the robotic arm has multiple different joint configurations, explicitly providing the relative displacement can reduce the burden on the network to infer the relative relationship between the end-effector and the target solely based on joint states, making the information related to the approach to the target in the training data more direct.
[0125] If the current end-effector position is calculated from the current joint angle using forward kinematics, then consistent robot kinematic parameters and coordinate system definitions are used during both the training and deployment phases. If the current end-effector position is directly provided by the state estimation module, the state estimation result can be recorded when constructing training samples, and data from the same source can be used during the deployment phase. By maintaining consistency in the data source, coordinate system, and arrangement of the current end-effector position and relative displacement, inconsistencies in network input semantics between training and deployment caused by changes in input definitions can be avoided.
[0126] In implementations where both the target position and the current end effector position are represented using the robot base coordinate system, the relative displacement can be directly expressed using a three-dimensional position difference vector. If the network input uses normalized data, the relative displacement can be normalized according to the scale determined during the training phase after it is obtained. For different robot arm instances, as long as the coordinate definition and data processing methods are consistent during the training and deployment phases, the relative displacement input can be formed according to the same data relationship, without requiring the network to resolve the complete environmental geometry during runtime.
[0127] Current environmental information can be obtained from distance sensors, depth sensors, or local environmental data provided by the training environment. Before inputting the data into the student policy network, the distance data can be normalized or truncated according to the range of the input data. The same data definition and arrangement are used in the training and deployment phases.
[0128] When constructing samples, the current environmental information, current joint angle, current joint velocity, and target position use data corresponding to the same control moment, so that the input obtained by the student policy network can correspond to the teacher's output at that control moment. By supplementing local distance relationships without introducing complete privileged environmental information, the student policy network can still obtain inputs related to the current local environment when the obstacle position changes, while preserving the information differences between the teacher and student sides.
[0129] The current end-effector position can be obtained directly from the robot arm's state estimation results, or it can be calculated using forward kinematics based on the current joint angles. When determining the relative displacement, the current end-effector position and the target position are represented in the same coordinate system, with the relative displacement x... rel =x target -x ee The training and deployment phases use the same data sources, coordinate systems, and data processing methods to maintain consistency in the meaning of student policy network input information.
[0130] For tasks that also require control of the end effector's attitude, the current end effector attitude or target attitude error information can be added to the input of the student policy network; for tasks that only require the end effector to reach the target position, attitude-related inputs can be omitted.
[0131] Step S104: Perform parameter update training based on the second end velocity vector and the corresponding first end velocity vector in the sample dataset to obtain the motion control policy network.
[0132] In some embodiments, the current joint angle, current joint velocity, and target position in the sample dataset are used as inputs to a student policy network to output a second end-effector velocity vector. The network is then trained by updating parameters based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain the motion control policy network. This includes: obtaining the current joint angle, current joint velocity, target position, and the first end-effector velocity vector corresponding to the current joint angle, current joint velocity, and target position from the sample dataset; inputting the current joint angle, current joint velocity, and target position into the student policy network to obtain the second end-effector velocity vector; using the first end-effector velocity vector as the supervision target of the second end-effector velocity vector; and updating the parameters of the student policy network based on the difference between the second end-effector velocity vector and the first end-effector velocity vector.
[0133] Through the above-mentioned supervised training, the student policy network gradually approaches the safe end velocity given by the teacher controller under the same or corresponding conditions.
[0134] In some embodiments, both the first end-effector velocity vector and the second end-effector velocity vector include linear velocity components and angular velocity components of the robot arm end-effector; parameter update training is performed based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain a motion control policy network, including: determining the difference in linear velocity components between the linear velocity components of the second end-effector velocity vector and the corresponding linear velocity components of the first end-effector velocity vector, and determining the difference in angular velocity components between the angular velocity components of the second end-effector velocity vector and the corresponding angular velocity components of the first end-effector velocity vector, and determining the training loss of the student policy network based on the differences in linear velocity components and angular velocity components; updating the parameters of the student policy network in the direction of reducing the training loss to obtain the motion control policy network.
[0135] L=λ v ||v S,lin -v T,lin || 2 +λ ω ||ω S -ω T || 2 .
[0136] Where, λ v and λ ω These represent the weights for the linear velocity and angular velocity components, respectively. Both can be set according to the requirements of end-effector position and attitude control in the current task. By updating the student policy network parameters in the direction of reducing training loss, the student policy network can simultaneously learn the linear and angular velocity relationships given by the teacher controller.
[0137] In a single parameter update, a set of samples can be obtained from the sample dataset D. For each sample, the current joint angle, current joint velocity, and target position are read, along with the corresponding first end-effector velocity vector. The input data is then fed into the student policy network to obtain the second end-effector velocity vector. The first end-effector velocity vector from the same sample is then used as the supervision target for the second end-effector velocity vector. Thus, the student outputs and teacher outputs are compared according to the sample correspondence, avoiding mismatches in end-effector velocity data at different control moments or under different robot states.
[0138] When updating parameters for a training batch, the student policy network first extracts the current joint angle, current joint velocity, target position, and optional current end-effector position, relative displacement, and current environment information from each sample according to the same field order, based on the input definition of the student policy network. Then, the corresponding first end-effector velocity vector in the same batch is used as supervision data. After the student policy network completes the forward computation of the batch input, it obtains the second end-effector velocity vector corresponding to each sample and compares it with the first end-effector velocity vector according to the sample number.
[0139] When both the first and second end-velocity vectors contain linear and angular velocity components, the corresponding components can be compared separately in the same coordinate system. The linear velocity component can include velocity components in the three translational directions, and the angular velocity component can include angular velocity components around the corresponding coordinate axes. If the numerical scales of different components differ significantly, a consistent scaling method can be used during the training data preprocessing stage, allowing the differences in linear and angular velocity components to participate in the training loss calculation according to preset weights.
[0140] Weights λv and λω are used to adjust the relative impact of linear velocity learning errors and angular velocity learning errors on a single parameter update. When the end-effector task primarily focuses on position changes, the weights corresponding to the linear velocity components can be relatively larger; when end-effector attitude tracking is also required, the relative impact of the angular velocity components can be increased. Regardless of the weight configuration used, the weight definitions used in the same training session remain consistent, ensuring that changes in training loss reflect the overall difference between student and teacher outputs.
[0141] After updating the student policy network parameters using the training loss, the next training batch can be obtained from the sample dataset. Since the teacher output, as the supervision target, already includes the relationship between target approach and obstacle avoidance, the student policy network gradually learns the safe end-effector velocity mapping for different robot states during multiple parameter updates. Compared to adjusting only based on sparse results after task success or collision, this supervision based on the corresponding end-effector velocity vector can directly provide the motion direction and velocity information that should be output in the current state.
[0142] The training loss can also take other forms that can represent the difference between the first and second end velocity vectors, depending on the actual training needs.
[0143] The sample dataset can be sampled according to training batches, and the student-side input and teacher-side supervision target in the same training sample are kept in correspondence.
[0144] The student policy network, after parameter update training, serves as the motion control policy network. During the training phase, the teacher controller provides safety action supervision using privileged environment information; during the deployment phase, the trained student policy network is used for network inference, and the teacher controller no longer needs to be run.
[0145] Figure 1 In steps S101 to S104, the robot arm state, task information, teacher output, and student output used are kept in correspondence according to the training samples, so that the student policy network output is compared with the teacher output recorded in the same sample.
[0146] Figure 1 In the training process shown, privileged environment information is located in the teacher-side input branch. The student policy network does not receive privileged environment information O when using basic input; when using extended input, it can receive a smaller amount of information d. obs , but d obs The complete obstacle geometry, scene geometry, and collision model still belong to different information levels. After training, the resulting motion control policy network retains the state-end velocity mapping relationship formed through parameter updates, and does not require re-executing the complete environmental constraint calculation in step S101 during the deployment phase.
[0147] With the above data organization method, the first end-effector velocity vector is not used only as a single control command, but rather forms repeatable supervisory data together with the corresponding state during the robot arm's movement. For different joint configurations and different target positions, corresponding samples can be generated separately, enabling the student policy network to learn the safe end-effector velocity given by the teacher controller in multiple states. In an implementation where the student policy network also receives current environmental information, corresponding samples can also be generated for different local environmental constraints, allowing the student policy network to learn the corresponding safe end-effector velocity in conjunction with the current environmental information.
[0148] exist Figure 1 Based on the basic training process shown, in some embodiments, the student policy network can be further updated using the motion results and safety judgment results obtained from the interaction between the student policy network and the environment. Figure 2 A training process for updating a student policy network based on environmental interaction feedback is shown, including steps S201 to S205.
[0149] Step S201: During the training process, the current joint angle, current joint velocity, and target position of the robot arm are input into the student policy network to obtain the second end-effector velocity vector.
[0150] In step S201, the student policy network can adopt... Figure 1 The basic input format is shown. For student policy networks trained with extended inputs such as current environment information, current end-effector position, or relative displacement, additional inputs can also be provided according to the corresponding training data definitions, but these extended inputs are not... Figure 2 The necessary limitations of the environmental interaction feedback update process shown.
[0151] Step S202: Use the second end-effector velocity vector to control the robot arm to interact with the environment.
[0152] Within a control cycle, the second end-effector velocity vector can be used as an end-effector velocity reference, or it can be converted into a joint space control quantity that the robot arm's underlying control interface can execute, so that the robot arm can generate actual motion based on the current output of the student policy network.
[0153] Step S203: Based on the distances between the end position of the robot arm before and after executing the second end velocity vector and the target position, determine the motion result used to characterize whether the end of the robot arm is moving toward the target position.
[0154] The motion result can be determined based on the change in distance before and after execution. When the distance after execution is less than the distance before execution, it can be used as evidence that the end effector has a tendency to move towards the target position; when the distance does not decrease, it can be determined that the target approach in the current control cycle is insufficient, or the distance changes of multiple consecutive control cycles can be combined to further determine whether the end effector is moving towards the target position as a whole. For situations requiring short-term lateral movement or short-term movement away from the target position, such as avoiding obstacles, the motion result can be determined based on the distance change trend of multiple consecutive control cycles.
[0155] Step S204: Based on the spatial relationship between the robot arm and the obstacle, determine the safety assessment result to characterize whether there is a collision risk to the robot arm. The safety assessment result can be determined based on the collision detection results between each link and the obstacle, the minimum distance, or other safe distance indicators.
[0156] Step S205: A feedback signal is generated based on the motion result and the safety judgment result, and the parameters of the student policy network are updated based on the feedback signal to update the motion control policy network.
[0157] When generating feedback signals, the motion results and safety judgment results can be retained separately and then used together for parameter updates, or they can be combined according to preset rules to form a joint feedback quantity. The feedback signals maintain a correspondence with the robot arm state and the second end effector velocity vector of the corresponding control cycle, so as to adjust the network parameters based on the actual motion results generated by the current output of the student policy network.
[0158] For safety assessment results, a safe distance condition can be set before an actual collision occurs. When the minimum distance between the robot arm and the obstacle is less than the preset safe distance, a collision risk can be determined even before actual contact occurs, allowing the student policy network to receive safety feedback in advance when approaching the obstacle.
[0159] The motion outcome and safety determination result can be determined based on a single control cycle or based on state changes over multiple consecutive control cycles. Using multiple consecutive control cycles for judgment can reduce the impact of short-term obstacle avoidance motion or state fluctuations in a single control cycle on the feedback result.
[0160] Through the above-mentioned environmental interaction and feedback updates, in addition to receiving action supervision from the teacher's end velocity, the student policy network can further adjust the network parameters based on the actual execution results of its own output in the environment, so that the parameter updates take into account both the target approach situation and the collision risk situation.
[0161] Figure 2 The steps shown can be repeated during the training of the student policy network. After each parameter update, the robot arm state can be reacquired in subsequent control cycles, and the updated student policy network will output a new second end-effector velocity vector. Whether to continue training can be determined based on preset training iterations, training loss, or training termination conditions such as evaluating task completion.
[0162] In some embodiments, during the aforementioned environmental interaction, if the safety assessment result indicates a collision risk to the robot arm, or the motion result indicates that the robot arm's end effector is not moving towards the target position, the current state, target position, and privileged environment information of the robot arm in the current control cycle can be obtained. This information is then input into the teacher controller to obtain a corrected end effector velocity vector. The corrected end effector velocity vector is used to enable the robot arm's end effector to avoid obstacles and move towards the target position.
[0163] The current state of the robotic arm used by the teacher for correction corresponds to the same control phase as the student output that triggered the correction. For example, after determining that there is a collision risk or insufficient target approach, the current joint angle, current joint velocity, target position, and privileged environment information available to the teacher can be saved for that control cycle. This allows the teacher controller to regenerate a safe end-effector velocity based on the current problematic state of the robotic arm in the student policy network.
[0164] When the teacher controller generates the corrected end velocity vector, it can continue to use the aforementioned safe motion determination method. If the triggering cause is collision risk, the motion direction can be adjusted to avoid the corresponding obstacle; if the triggering cause is insufficient target approach, the motion direction with a target approach trend can be re-determined; when both situations exist simultaneously, a corrected end velocity vector that takes into account both obstacle avoidance and target approach can be generated.
[0165] In some embodiments, the method further includes: after the teacher controller generates the corrected end-effector velocity vector, obtaining the current joint angle and current joint velocity corresponding to the generation of the corrected end-effector velocity vector from the current state of the robot arm, using the current joint angle, current joint velocity, and target position as input data for the new sample, and using the corrected end-effector velocity vector as the supervision target corresponding to the input data to construct the new sample; adding the new sample to the sample dataset; and continuing to update and train the student policy network based on the sample dataset after adding the new sample.
[0166] If the student policy network adopts extended input, the new sample can also synchronously record the current end position, relative displacement and current environment information, so that the new sample maintains the same input data structure as the original sample.
[0167] The newly added samples correspond to the robot arm's state and teacher's corrective output when the student policy network encounters collision risks or insufficient target approach during environmental interactions. Adding these samples to the dataset allows subsequent training to supplement teacher action supervision for states where problems have been exposed in the student policy network, without needing to change the original form of the supervision objective.
[0168] In some embodiments, after updating and training the student policy network based on the sample dataset with added new samples, the updated student policy network can be used to control the robot arm to interact with the environment again. If there is a risk of collision during the robot arm's interaction with the environment again, or if the robot arm's end effector does not move toward the target position, the teacher controller generates a new corrected end effector velocity vector based on the robot arm's current state, target position, and privileged environment information. New new samples are constructed based on the new corrected end effector velocity vector, added to the sample dataset, and the student policy network is further updated and trained based on the updated sample dataset.
[0169] During the above iterations, each teacher correction generates a corrected end-effector velocity vector for the robot arm state corresponding to the current round of environmental interaction, ensuring that the input data in the new samples correspond to the supervised target. Once the training termination condition is met, the iterations can be ended and the motion control strategy network parameters can be fixed.
[0170] The aforementioned teacher corrections, construction of new samples, and re-interactive training all fall under the category of... Figure 2 The further implementation methods based on the environmental interaction feedback update shown can be adopted sequentially according to training needs. Figure 2 It is only used to illustrate the process of updating student policy network parameters by involving motion results and safety judgment results.
[0171] Apart from handling cases with dependencies, the aforementioned implementation methods, such as current environmental information input, current end position and relative displacement input, teacher end velocity supervision, environmental interactive feedback, teacher correction, and velocity component loss, can be adopted individually or in combination according to the actual training configuration.
[0172] Therefore, additional technical features in different implementations are not required to be necessary steps in the same training process at the same time; when using a certain additional technical feature, training can be completed according to the data input, parameter update or teacher control relationship of the corresponding implementation.
[0173] Training metrics related to student strategies, teacher strategies, and training termination failures can be recorded during the training process. Figures 3 to 5 The recorded results show how the corresponding training metrics change with the number of training iterations.
[0174] Figure 3 This shows how the student's strategy training level changes with the number of training iterations, where the horizontal axis represents the number of training iterations and the vertical axis represents the student's strategy training level. Figure 3 It can be seen that the students' strategy training level was in a low range in the early stage of training, then gradually increased as the training progressed, and tended to stabilize in the later stage of training. Figure 3 The curves shown are used to represent the changing trends of student strategy-related training indicators during the training process, and are not used to limit the specific number of training rounds, training termination conditions, or the values that the corresponding indicators must reach for the student strategy network.
[0175] Figure 4 This shows how the teacher's strategy training level changes with the number of training iterations, where the horizontal axis represents the number of training iterations and the vertical axis represents the teacher's strategy training level. Figure 4 It can be seen that the overall level of teachers' strategy training increases with the number of training iterations, and tends to stabilize in the later stages of training. Figure 4The curves shown represent the changes in training metrics related to the teacher's policy. The teacher controller can generate a first end-effector velocity vector based on the current state of the robot arm, the target position, and privileged environment information. This first end-effector velocity vector can serve as a supervisory target for the student policy network during parameter update training.
[0176] Figure 5 The diagram shows how Termination / fail changes with the number of training iterations, where the horizontal axis represents the number of training iterations and the vertical axis represents the Termination / fail values recorded during the training process. Figure 5 This metric is labeled as the termination failure rate, with recorded values ranging from -1 to 0. Here, Termination / fail is a signed recorded value used in the training program to reflect the situation related to termination failures and is not interpreted according to the failure probability usually expressed as a percentage. Figure 5 It can be seen that the Termination / fail record value gradually changes from close to -1 to 0 in the early stage of training, and remains near 0 in the later stage of training.
[0177] Figures 3 to 5 Each curve corresponds to a different training metric, and the ordinate of each curve has a different meaning. Therefore, the ordinate values of different curves are not directly comparable. The curves described above are used to illustrate how the corresponding training metrics change during the training process, and are not used to limit the specific number of training iterations, the training termination threshold, or the values that each training metric must reach.
[0178] Figure 6 and Figure 7 The different training states in the parallel simulation training environment are shown. Multiple robot arm instances can run simultaneously in the simulation environment, each instance having its own current joint state, target position, and environmental information.
[0179] For each robotic arm instance, the teacher controller generates a first end-effector velocity vector based on the instance's current state, target position, and privileged environment information, forming a corresponding training sample. Training samples generated from multiple instances can be aggregated into a sample dataset for training the student policy network. Figure 6 and Figure 7 The robot arm poses in the image are used to represent examples under different training states.
[0180] After training, the student policy network is used as a motion control policy network for online control. Figure 8 A motion control process for a robotic arm during the deployment phase is shown, including steps S801 to S803.
[0181] Step S801: Obtain the current joint angle, current joint speed, and target position to be reached by the robot arm's end effector.
[0182] The current joint angle and current joint velocity can be provided by the robot arm's own joint state detection unit; the target position is given by the current robot task. For continuously changing tasks, the target position can be updated in different control cycles.
[0183] Step S802: Input the current joint angle, current joint velocity, and target position into the motion control strategy network to obtain the motion control quantities of the end effector. The motion control strategy network can be trained using the method described above for training the robot arm's motion control strategy network.
[0184] Motion control quantities characterize the motion control results that the robot arm needs to execute under its current state and task objective. The specific data format of the motion control quantities can be determined based on the robot arm's control architecture and underlying control interface. Motion control quantities can be motion reference quantities that need to be further converted into control commands, or they can be direct control commands used to control the robot arm's movement. Motion reference quantities can include the end effector velocity vector of the robot arm's end effector. In other words, under different control architectures, the motion control policy network can output motion reference quantities characterizing the desired motion state of the end effector, or it can directly output control commands that can be used to control the robot arm's movement.
[0185] In some embodiments, the motion control quantity is the end-effector velocity vector, and the motion control policy network is a student policy network that has completed parameter update training; the motion control quantity of the end-effector is obtained by inputting the current joint angle, current joint velocity, and target position into the motion control policy network, including: a deployment phase in which the current joint angle, current joint velocity, and target position are input into the student policy network, and the student policy network outputs the end-effector velocity vector of the current control cycle based on the current joint angle, current joint velocity, and target position.
[0186] exist Figure 8 In the basic deployment implementation shown, the student policy network no longer obtains the privileged environment information used by the teacher controller during training.
[0187] The current joint angles and current joint velocities used during the deployment phase have the same meaning as the corresponding data in the training samples. The robot arm control system can read the joint state detection results in each control cycle and input the data of the current moment into the network. The target position can be provided by the upper-level task module, trajectory planning module, or user-defined task objective; regardless of the task source, the target position is represented according to the coordinate definition used in the training phase before being input into the motion control strategy network.
[0188] In some embodiments, the method further includes: acquiring the current environmental information of the robot arm, and inputting the current joint angle, current joint velocity, target position and current environmental information into the motion control strategy network to obtain the end effector velocity vector.
[0189] The current environment information on the deployment side uses the same data definition and arrangement as that in the training phase. For example, if the current environment information during training uses multiple preset distance features, the distance features are obtained during deployment according to the same query object and arrangement order. For networks that also use the current end position or relative displacement as input during training, the deployment side can also provide corresponding features according to the same data definition.
[0190] The terminal velocity vector in the deployment phase corresponds to the second terminal velocity vector in the training phase, and its output does not need to be compared with the first terminal velocity vector of the teacher controller. The student policy network directly completes network inference based on the current input to obtain the terminal velocity reference value for the current control cycle.
[0191] Step S803: Control the movement of the robot arm according to the motion control quantity.
[0192] In some implementations, the motion control quantity can be a motion reference quantity. For example, the motion control quantity can be the end effector velocity vector of a robot arm end effector, which may include at least one of the linear velocity component and the angular velocity component of the end effector.
[0193] In this implementation, during the training phase, the teacher controller generates a first end-effector velocity vector based on the current state of the robot arm, the target position, and privileged environment information. The student policy network outputs a second end-effector velocity vector based on input data such as the current joint angles, current joint velocities, and target position of the robot arm. The first end-effector velocity vector is used as the supervisory target for the second end-effector velocity vector, and the student policy network is updated with parameters based on the difference between the first and second end-effector velocity vectors, enabling the student policy network to learn the robot arm end-effector motion pattern determined by the teacher controller.
[0194] After training, the student policy network is used as the motion control policy network. During deployment, the current joint angles, current joint velocities, and target positions of the robot arm are input into the motion control policy network to obtain the end-effector velocity vector as the motion control quantity. If the robot arm's underlying control interface cannot directly receive the end-effector velocity vector, it can be converted into a control command matching the underlying control interface based on the robot arm's kinematics and the interface itself, and the robot arm's movement can be controlled according to the control command. If the underlying control interface can directly receive the end-effector velocity vector, it can be used as a corresponding velocity control command to control the robot arm's movement.
[0195] In another embodiment, the motion control quantity can be directly a control command for controlling the movement of the robot arm. In this embodiment, after inputting the current joint angle, current joint velocity, and target position of the robot arm into the motion control strategy network, the motion control strategy network directly outputs a control command that matches the underlying control interface of the robot arm. Since the output of the motion control strategy network itself can be used to control the movement of the robot arm, there is no need to perform the processing of generating control commands from motion reference quantities such as the end effector velocity vector; the movement of the robot arm can be controlled according to the control commands output by the motion control strategy network.
[0196] Therefore, in this embodiment of the invention, the motion control quantity is used to uniformly represent the data related to the motion control of the robot arm output by the motion control strategy network. When the motion control quantity is a motion reference quantity such as an end-effector velocity vector, it can be determined whether further control conversion is needed based on the robot arm's underlying control interface; when the motion control quantity itself is a control command, the robot arm's movement can be directly controlled based on the motion control quantity. The main difference between different implementations lies in the output form of the motion control strategy network and whether the output needs further control conversion.
[0197] In implementations where the motion control quantity is the end-effector velocity vector, the end-effector velocity vector can be converted into joint velocity control quantities based on the kinematic relationship of the robot arm. For example, the end-effector velocity vector can be mapped to the joint velocities corresponding to each joint using the Jacobian matrix corresponding to the current joint configuration of the robot arm; alternatively, the pseudo-inverse of the Jacobian matrix or damped least squares can be used to determine the corresponding joint velocities.
[0198] After determining the corresponding joint control quantities based on the end-effector velocity vector, the joint control quantities can be limited according to the joint velocity range, acceleration range, or joint position range of the robot arm. Furthermore, the joint control quantities corresponding to adjacent control cycles can be smoothed as needed to ensure that the generated control quantities match the actual motion capabilities of the robot arm.
[0199] During continuous control, after the motion control quantity corresponding to one control cycle is executed, the joint position and joint velocity of the robot arm change. In the next control cycle, the current joint angle and current joint velocity of the robot arm are reacquired and combined with the target position to be re-input into the motion control policy network to obtain new motion control quantities. This forms a closed-loop control process of state acquisition, network inference, and robot arm motion.
[0200] When the motion control variable is the end-effector velocity vector, the motion control strategy network redetermines the end-effector velocity vector in each control cycle based on the updated robot arm state. When the motion control variable is directly the control command, the motion control strategy network redetermines the corresponding control command in each control cycle based on the updated robot arm state. Therefore, the output of the motion control strategy network can be continuously updated as the actual motion state of the robot arm changes, without needing to generate a complete motion trajectory from the starting position to the target position all at once.
[0201] The deployment phase does not require the teacher controller to continue running, nor does it require supervised comparison between the teacher output and student output from the training phase. After training, the motion control policy network completes network inference based on the robot arm state and task information available during the deployment phase. When current environmental information is used as input to the student policy network during training, the deployment phase can also provide the motion control policy network with the same data definition as the training phase.
[0202] Steps S801 to S803 can be repeatedly executed according to the control cycle of the robot arm. As each joint of the robot arm moves, the next control cycle reacquires the current joint angle and current joint speed, and then calls the motion control strategy network according to the updated state, until the end effector reaches the target area specified by the current task or the target of the upper-level task switching.
[0203] When the end-effector velocity vector includes linear velocity and angular velocity components, the control command can simultaneously address changes in end-effector position and attitude. If the robot arm's underlying interface uses joint-space commands, the end-effector velocity vector can be converted into target motion quantities for each joint based on the kinematic relationships under the current joint configuration. After execution, the joint drive mechanism generates new current joint angles and current joint velocities for the next control cycle to continue network inference.
[0204] The training phase uses the same combination of student inputs, and the deployment phase configures the network inputs according to the same data definition. For example, the base network uses q... and x target During the deployment phase, the same three types of data are used; if the trained network also contains x ee xrel Or d obs The deployment side also provides corresponding features. Input dimensions from different training versions cannot be arbitrarily mixed during deployment to ensure that the meaning of the input during network inference remains consistent with the training samples.
[0205] Figures 9 to 14 The diagram illustrates multiple consecutive motion states of the robotic arm's end effector as it reaches its task after the trained motion control policy network is deployed on a physical robot platform. The end effector of the robotic arm is located within a desktop workspace, on which scene objects such as bottle-shaped objects and yellow objects are placed. Figures 9 to 14 These can correspond to different control moments during the continuous execution of the robot arm's end effector to reach the task, illustrating the continuous deployment and control process of the motion control strategy network on a physical robot platform.
[0206] Figure 9 This illustrates the first motion state of a robotic arm as it reaches its task endpoint. In the current control cycle, the current joint angles, current joint velocities, and the target position to be reached by the end effector are acquired. This information is then input into a trained motion control policy network, which outputs the end effector velocity vector corresponding to the current control cycle. Based on the velocity vector, corresponding control commands are generated, and the robotic arm is moved according to these commands.
[0207] As control commands are executed, the joint configuration of the robot arm and the position of the end effector change. Figure 10 The diagram illustrates the second motion state of the robotic arm as it reaches its end effector during the task execution process. In the next control cycle, the updated joint angles and velocities of the robotic arm are reacquired and input into the motion control policy network along with the target position. The motion control policy network then regenerates the end effector velocity vector based on the updated robotic arm state. Thus, during continuous motion, each control cycle can re-perform network inference based on the current state of the robotic arm, instead of directly using the end effector velocity vector generated in the previous control cycle.
[0208] Figure 11 This illustrates the third motion state of the robotic arm as it reaches its end effector during the task execution process. (Relative to...) Figure 9 and Figure 10 In the state shown, the end effector of the robot arm moves further, and the position and velocity of each joint of the robot arm change accordingly. In this control cycle, the motion control policy network outputs the corresponding end effector velocity vector based on the reacquired current joint angle, current joint velocity, and target position, and generates control commands based on the end effector velocity vector to control the robot arm to continue to execute the current end effector arrival task.
[0209] Figure 12 This illustrates the fourth motion state of the robotic arm's end effector as it reaches the task. As the robotic arm continues to move, the end effector relative to... Figure 11 The position shown changes further. In this control cycle, the current joint angle and current joint velocity of the robot arm are acquired again, and the corresponding end effector velocity vector is re-determined based on the current joint angle, current joint velocity, and target position, so that the end effector continues to move in the motion direction corresponding to the current task.
[0210] Figure 13 This illustrates the fifth motion state of the robotic arm as it reaches its end effector during the task execution process. (Relative to...) Figure 12 As shown, the robot arm's joint configuration continues to adjust, and the end effector moves further towards the target area corresponding to the current task. During this process, it can continue to follow... Figure 8 The process shown periodically executes state acquisition, motion control strategy network inference, control command generation, and robot arm movement, so that the input and output of the motion control strategy network are continuously updated with the actual movement state of the robot arm.
[0211] Figure 14 The diagram illustrates the sixth motion state of the robotic arm during the process of its end effector reaching the task. At this point, the end effector, under the continuous control of the aforementioned multiple control cycles, further approaches the target area corresponding to the current task. The robotic arm can continue to perform motion according to the aforementioned closed-loop control method until the end effector reaches the preset target area or meets the termination condition corresponding to the current end effector reaching the task.
[0212] Figures 9 to 14 The various motion states shown correspond to different time points in the continuous control process of the robot arm. Figures 9 to 14 As shown in the continuous states, even with the constantly changing joint configuration of the robot arm and the position of the end effector, the motion control strategy network can generate the corresponding end-effector velocity vector based on the robot arm state reacquired in each control cycle, and complete the end-effector arrival task by continuously executing multiple control cycles. The specific joint configuration of the robot arm, the positions of various scene objects on the table, and the specific settings of the target position in the figure are only used to illustrate one task scenario and do not constitute a limitation on the structure of the robot arm, the workspace, or the way the target position is set.
[0213] Corresponding to the aforementioned training methods, Figure 15 A training device 1500 for a robot arm motion control strategy network is shown. The device includes a teacher control module 1501, a sample construction module 1502, and a network training module 1503.
[0214] The teacher control module 1501 is used to acquire the current state of the robot arm, the target position, and privileged environment information, and output a first end-effector velocity vector pointing towards the target position and avoiding obstacles based on the above information. The teacher control module 1501 can also perform collision detection, determination of feasible motion directions, and correction of end-effector velocity vector generation.
[0215] The sample construction module 1502 is used to control the robot arm's movement in the environment using the first end-effector velocity vector, and to record the robot arm's current joint angle, current joint velocity, target position, and corresponding first end-effector velocity vector during the movement process, in order to construct a sample dataset. When a teacher correction is triggered, the sample construction module 1502 can also construct a new sample from the current state and the corrected end-effector velocity vector and add it to the sample dataset.
[0216] The network training module 1503 is used to input the corresponding input data from the sample dataset into the student policy network to obtain the second end velocity vector, and update the student policy network parameters according to the difference between the second end velocity vector and the corresponding first end velocity vector. Figure 2 The process of generating feedback signals based on motion results and safety judgment results and updating student policy network parameters can also be performed by the network training module 1503.
[0217] When employing the aforementioned extended training method, the network training module 1503 can also use at least one of the current environmental information, the current end position, and the relative displacement as input to the student policy network, and calculate the differences in linear velocity components and angular velocity components respectively. In a further teacher correction implementation, the teacher control module 1501 can generate a corrected end velocity vector, the sample construction module 1502 can form new samples based on this vector, and the network training module 1503 can continue to update the student policy network parameters based on the sample dataset after adding the new samples. Thus, Figure 15 Each of the functional modules shown can implement the corresponding processing in the aforementioned different training implementation methods.
[0218] Through data coordination among the teacher control module 1501, sample construction module 1502, and network training module 1503, the safe end-effector velocity generated by the teacher controller can form training samples together with the corresponding robot arm state and task information, serving as the supervised target for the current output of the student policy network. For collision risks or insufficient target approach states encountered by the student policy network during environmental interactions, the three modules can further perform teacher correction, new sample construction, and parameter updates, thereby enabling the training device to cover the aforementioned basic distillation training and correction training processes.
[0219] The 1600-type motion control device for the robotic arm on the deployment side Figure 16As shown, it includes a state acquisition module 1601, a network inference module 1602, and a motion control module 1603.
[0220] The state acquisition module 1601 is used to acquire the current joint angle, current joint velocity, and target position to be reached by the robot arm's end effector; the network inference module 1602 is used to input the current joint angle, current joint velocity, and target position into the motion control strategy network to obtain the motion control quantity of the end effector; the motion control strategy network is trained using the aforementioned robot arm motion control strategy network training method; the motion control module 1603 is used to control the robot arm's movement based on the motion control quantity. These modules can be implemented by different program modules in the same processor, or they can be implemented by different control units.
[0221] In one control cycle, the state acquisition module 1601 first obtains the current joint state and target position. The network inference module 1602 then obtains the motion control quantity based on this data. The motion control module 1603 applies the motion control quantity to the joint drive mechanism. After the joint moves, the state acquisition module 1601 re-acquires the updated current joint angle and current joint velocity in the next control cycle. As each module executes continuously according to the above data flow, the motion control quantity output by the motion control strategy network can be continuously converted into the actual movement of the robot arm.
[0222] Through the continuous cooperation of the state acquisition module 1601, the network inference module 1602, and the motion control module 1603, corresponding motion control quantities can be obtained based on the current state of the robot arm and the task objective, and the robot arm can be controlled to move according to the motion control quantities. When the motion control quantity is an end-effector velocity vector, the end-effector velocity vector can be further converted into a control command that matches the underlying control interface; when the motion control quantity itself is a control command, the robot arm can be directly controlled to move according to the control command.
[0223] exist Figure 17 In one embodiment shown, the motion control quantity output by the motion control strategy network is the end velocity vector. Figure 17 A structural relationship of a robotic arm 1700 is illustrated. The robotic arm 1700 includes a robotic arm body 1710 and a motion control device 1720. The robotic arm body 1710 includes a joint drive mechanism 1711 and an end effector 1712. The joint drive mechanism 1711 is used to drive one or more joints of the robotic arm to change the position and orientation of the end effector 1712. In another embodiment, the motion control quantity output by the motion control strategy network can be directly a control command. The motion control device 1720 can provide the control command to the joint drive mechanism 1711 so that the joint drive mechanism 1711 drives the joint movement of the robotic arm.
[0224] The motion control device 1720 may include a processor 1721 and a memory 1722. The memory 1722 stores a trained motion control strategy network and corresponding control program. The motion control device 1720 acquires the current joint angle, current joint velocity, and target position of the end effector 1712 of the robot arm, obtains the end effector velocity vector using the motion control strategy network, and generates control commands based on the end effector velocity vector.
[0225] The motion control device 1720 is connected to the joint drive mechanism 1711 and sends control commands to the joint drive mechanism 1711. The joint drive mechanism 1711 drives the robot arm joints to move according to the control commands, causing the actual end effector velocity of the end effector 1712 to follow the end effector velocity vector output by the motion control strategy network. The updated joint state after the joint movement can be used for network inference in the next control cycle.
[0226] In this structure, the robot arm body 1710 is used to perform actual mechanical movements, and the motion control device 1720 is used to perform state processing, network inference, and control command generation. The processor 1721 can read the motion control strategy network parameters stored in the memory 1722, and perform network inference in each control cycle using the current joint angle, current joint speed, and target position. After the generated control commands are sent to the joint drive mechanism 1711, the joint drive mechanism 1711 drives the corresponding joint to move, thereby changing the actual end effector speed of the end effector 1712.
[0227] The joint state after movement can be fed back to the motion control device 1720 as input for the next control cycle. Thus, a data interaction relationship based on joint state and control commands is established between the motion control device 1720 and the robot arm body 1710. This structure does not require a teacher controller to be set up in the online control loop of the robot arm 1700; the trained motion control strategy network can be directly used for the deployment control of the robot arm 1700.
[0228] In another embodiment, the motion control quantity output by the motion control strategy network can be directly a control command. The motion control device 1720 can provide the control command to the joint drive mechanism 1711 so that the joint drive mechanism 1711 can drive the robot arm joint to move.
[0229] In another embodiment, the robotic arm may include a processor and a memory. The memory stores a computer program, which, when executed by the processor, enables the processor to perform the aforementioned training method for the robotic arm motion control strategy network or the robotic arm motion control method. The processor and memory may be located in the control unit of the robotic arm.
[0230] Figure 18 An example structure of an electronic device 1800 is shown. The electronic device 1800 includes a processor 1801 and a memory 1802, the memory 1802 storing a computer program. When the processor 1801 executes the computer program, it can execute the aforementioned training method for the robot arm motion control strategy network or the robot arm motion control method.
[0231] The electronic device 1800 can be located in the training device or in the control unit of the robot arm or robot platform. The memory 1802 can also store at least one of the following: motion control strategy network parameters, training samples, and state data used during the control process.
[0232] When the electronic device 1800 is used in the training phase, the memory 1802 can store the teacher controller's related programs, training samples, and student policy network parameters. The processor 1801 executes the program to complete the generation of the first end-effector velocity vector, sample construction, and updating of student policy network parameters. When the electronic device 1800 is used in the deployment phase, it can load only the trained motion control policy network and control program, and perform network inference and control command generation based on the current state and target position of the robot arm. Both application methods can use the same type of processor and memory structure, or they can be set in the training device and the robot arm control device respectively.
[0233] Figure 19 An example of a robotic platform for a robotic arm is shown. In the illustrated robotic platform, one or both upper limbs of the platform can be used as the aforementioned robotic arm, and the aforementioned motion control method can be executed based on the current joint angle, current joint velocity, and target position of the respective upper limb.
[0234] The robotic arm in this embodiment can also be an independent robotic arm, a collaborative robotic arm, or a robotic arm mounted on another robotic platform. The specific shape of the robotic platform, the number of degrees of freedom, and the form of the end effector do not affect the motion control process of the aforementioned teacher-student training relationship and deployment phase.
[0235] For a robotic arm mounted on a humanoid robot or other robotic platform, the aforementioned current joint angle and current joint velocity can be taken from the corresponding upper limb joint, and the target position of the end effector can be determined according to the current task of the robotic platform. The teacher-student training process can establish corresponding training data for this upper limb mechanical structure, and during deployment, the trained motion control strategy network can be applied to the same type of mechanical structure. The remaining components of the robotic platform can operate according to their own control methods, without changing the training and control relationship for the end effector motion of the robotic arm in this embodiment.
[0236] Figure 20A computer-readable storage medium 2000 is shown. The computer-readable storage medium 2000 stores a computer program, which, when executed by a processor, can implement the aforementioned training method for the robot arm motion control strategy network or the robot arm motion control method.
[0237] The computer-readable storage medium 2000 may be a storage medium capable of storing computer programs in a non-transitory manner. The computer program may include instructions for data acquisition, teacher control, sample construction, and student policy network parameter updates during the training phase, or instructions for state acquisition, network inference, control instruction generation, and robot arm motion control during the deployment phase.
[0238] The training instructions in the computer program can be organized according to the processing relationship of steps S101 to S104 or steps S201 to S205, or corresponding program modules can be set according to the aforementioned teacher correction, new sample construction, and iterative training implementation methods. When used for deployment control, the computer program can sequentially complete state acquisition, motion control strategy network inference, and robot arm motion control according to steps S801 to S803; when the motion control quantity is a motion reference quantity that needs further conversion, it can also perform the processing of generating control instructions based on the motion reference quantity.
[0239] The above embodiments are used to illustrate the technical solution of the present invention. The specific data formats of the teacher-side input, student-side input, and training loss can be set according to the structure of the robot arm and the needs of the task; while maintaining the teacher-side using relatively complete environmental constraint information to generate a safe end-effector velocity, and the student-side using corresponding state and task information to learn the mapping relationship of the end-effector velocity, the conversion method of the end-effector velocity to the joint control quantity, the training data organization method, and the control execution method can all be adjusted according to the actual implementation.
Claims
1. A method for training a motion control strategy network for a robotic arm, characterized in that, include: The current state of the robot arm, the target position, and the privileged environment information are obtained as inputs to the teacher controller to output a first end-effector velocity vector that points to the target position and avoids obstacles. The privileged environment information is the environmental constraint information that the teacher can obtain during the training phase. The robot arm is controlled to move in the environment using the first end-effector velocity vector, and the current joint angle, current joint velocity, target position and corresponding first end-effector velocity vector of the robot arm during the movement process are constructed as a sample dataset; The current joint angle, current joint velocity, and target position in the sample dataset are used as inputs to the student policy network to output a second end-effector velocity vector. The parameters are updated and trained based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain the motion control policy network.
2. The training method for the robot arm motion control strategy network according to claim 1, characterized in that, The step of using the current joint angle, current joint velocity, and target position from the sample dataset as input to the student policy network includes: The current joint angle, current joint velocity, target position, and current environmental information in the sample dataset are used as inputs to the student policy network; The amount of information in the current environment information is less than the amount of information in the privileged environment information.
3. The training method for the robot arm motion control strategy network according to claim 2, characterized in that, The privileged environment information includes at least one of obstacle geometry information, scene geometry information, and collision model information; the current environment information includes at least one of distance information between the robot arm and the obstacle and a local distance field, wherein the local distance field is used to characterize the distance relationship between multiple spatial positions in the local space around the robot arm and the obstacle.
4. The training method for the robot arm motion control strategy network according to claim 2, characterized in that, The method further includes: Obtain the current end position corresponding to the current joint angle, current joint velocity and target position in the sample dataset, and determine the relative displacement of the target position relative to the current end position based on the current end position and the target position; The step of using the current joint angle, current joint velocity, target position, and current environmental information from the sample dataset as input to the student policy network includes: The current joint angle, the current joint velocity, the current end position, the target position, the relative displacement, and the current environmental information are used as inputs to the student policy network.
5. The training method for the robot arm motion control strategy network according to claim 1, characterized in that, The process involves using the current joint angle, current joint velocity, and target position from the sample dataset as input to the student policy network, outputting a second end-effector velocity vector, and performing parameter update training based on the second end-effector velocity vector and the corresponding first end-effector velocity vector from the sample dataset to obtain the motion control policy network, including: Obtain the current joint angle, current joint velocity, target position, and a first end-effector velocity vector corresponding to the current joint angle, current joint velocity, and target position from the sample dataset; The current joint angle, the current joint velocity, and the target position are input into the student policy network to obtain the second end velocity vector; The first end velocity vector is used as the supervision target of the second end velocity vector; The parameters of the student policy network are updated based on the difference between the second end velocity vector and the first end velocity vector to obtain the motion control policy network.
6. The training method for the robot arm motion control strategy network according to claim 1, characterized in that, The method further includes: During training, the current joint angle, current joint velocity, and target position of the robot arm are input into the student policy network to obtain the second end-effector velocity vector; The second end-effector velocity vector is used to control the robot arm's interaction with the environment; Based on the distances between the end position of the robot arm before and after executing the second end velocity vector and the target position, a motion result is determined to characterize whether the end of the robot arm is moving toward the target position. Based on the spatial relationship between the robot arm and the obstacle, a safety assessment result is determined to characterize whether there is a collision risk to the robot arm; Feedback signals are generated based on the motion results and the safety determination results, and the parameters of the student policy network are updated based on the feedback signals to update the motion control policy network.
7. The training method for the robot arm motion control strategy network according to claim 6, characterized in that, The method further includes: If the safety assessment result indicates that the robot arm is at risk of collision, or if the motion result indicates that the end effector of the robot arm is not moving toward the target position, then the current state of the robot arm, the target position, and the privileged environment information in the current control cycle are obtained. The current state of the robot arm, the target position, and the privileged environment information are input into the teacher controller to obtain a corrected end-effector velocity vector. The corrected end-effector velocity vector is used to enable the robot arm end to avoid obstacles and move toward the target position.
8. The training method for the robot arm motion control strategy network according to claim 7, characterized in that, The method further includes: Obtain the current joint angle and current joint velocity of the robot arm when the teacher controller generates the corrected end effector velocity vector from the current state of the robot arm; The new sample is constructed by using the current joint angle, the current joint velocity, and the target position as input data for the new sample, and using the corrected end velocity vector as the supervision target corresponding to the input data. Add the new sample to the sample dataset; Based on the sample dataset after adding the new samples, the student policy network continues to be trained with updated parameters.
9. The training method for the robot arm motion control strategy network according to claim 8, characterized in that, The method further includes: After updating and training the student policy network based on the sample dataset with the added samples, the robot arm is controlled to interact with the environment again using the updated student policy network. If the robotic arm is at risk of collision during its next interaction with the environment, or if the end effector of the robotic arm does not move toward the target position, the teacher controller generates a new corrected end effector velocity vector based on the current state of the robotic arm, the target position, and the privileged environment information. New samples are constructed based on the new corrected terminal velocity vector, and these new samples are added to the sample dataset. The student policy network is then trained with updated parameters based on the updated sample dataset.
10. The training method for the robot arm motion control strategy network according to claim 1, characterized in that, Both the first and second end-effector velocity vectors include linear velocity and angular velocity components at the end of the robot arm; the step of updating and training parameters based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain the motion control policy network includes: The difference in linear velocity components between the linear velocity component of the second end velocity vector and the corresponding linear velocity component of the first end velocity vector is determined, and the difference in angular velocity components between the angular velocity component of the second end velocity vector and the corresponding angular velocity component of the first end velocity vector is determined. The training loss of the student policy network is determined based on the difference in linear velocity components and the difference in angular velocity components. The parameters of the student policy network are updated in the direction of reducing the training loss to obtain the motion control policy network.
11. The training method for the robot arm motion control strategy network according to claim 1, characterized in that, The process of acquiring the current state of the robot arm, the target position, and the privileged environment information as input to the teacher controller, and outputting a first end-effector velocity vector pointing towards the target position and avoiding obstacles, includes: Based on the current state of the robotic arm and the privileged environment information, the collision detection result between the robotic arm and the obstacle is determined; Based on the current state of the robot arm, the target position, the privileged environment information, and the collision detection results, a feasible movement direction is determined to make the end effector of the robot arm move toward the target position while avoiding obstacles. The first end effector velocity vector is determined based on the current state of the robot arm and the feasible direction of motion.
12. A method for controlling the motion of a robotic arm, characterized in that, include: Obtain the current joint angle, current joint velocity, and target position to be reached by the robot arm's end effector; The current joint angle, the current joint velocity, and the target position are input into the motion control strategy network to obtain the motion control parameters for the end effector. The motion control strategy network is trained using the training method for the robot arm motion control strategy network according to any one of claims 1 to 11; The robot arm moves according to the motion control parameters.
13. The robot arm motion control method according to claim 12, characterized in that, The motion control strategy network is a student strategy network that completes parameter update training, and the motion control quantity is the end velocity vector; The step of inputting the current joint angle, the current joint velocity, and the target position into the motion control strategy network to obtain the motion control quantity of the end effector includes: During the deployment phase, the current joint angle, the current joint velocity, and the target position are input into the student policy network; The student policy network outputs the end-effector velocity vector based on the current joint angle, the current joint velocity, and the target position.
14. The robot arm motion control method according to claim 12, characterized in that, The method further includes: Obtain the current environmental information of the robotic arm; The step of inputting the current joint angle, the current joint velocity, and the target position into the motion control strategy network to obtain the motion control quantity of the end effector includes: The current joint angle, the current joint velocity, the target position, and the current environmental information are input into the motion control strategy network to obtain the motion control quantity of the end effector.
15. A training device for a robot arm motion control strategy network, characterized in that, include: The teacher control module is used to acquire the current state of the robot arm, the target position, and the privileged environment information, and output the first end-effector velocity vector pointing to the target position and avoiding obstacles. The privileged environment information is the environmental constraint information that the teacher can obtain during the training phase. The sample construction module is used to control the robot arm to move in the environment using the first end-effector velocity vector, and to construct a sample dataset by the current joint angle, current joint velocity, target position and corresponding first end-effector velocity vector of the robot arm during the movement process. The network training module is used to input the current joint angle, current joint velocity and target position from the sample dataset into the student policy network to obtain the second end-effector velocity vector, and to perform parameter update training on the student policy network based on the second end-effector velocity vector and the corresponding first end-effector velocity vector in the sample dataset to obtain the motion control policy network.
16. A motion control device for a robotic arm, characterized in that, include: The status acquisition module is used to acquire the current joint angle, current joint velocity, and target position to be reached by the robot arm's end effector. The network inference module is used to input the current joint angle, the current joint velocity and the target position into the motion control strategy network to obtain the motion control quantity of the end effector. The motion control strategy network is trained by the training method of the robot arm motion control strategy network according to any one of claims 1 to 11. The motion control module is used to control the movement of the robot arm according to the motion control parameters.
17. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the training method of the robot arm motion control strategy network according to any one of claims 1 to 11, or the robot arm motion control method according to any one of claims 12 to 14.
18. A robotic arm, characterized in that, Includes the robot arm body and the robot arm motion control device as described in claim 16; The robot arm body includes an end effector and a joint drive mechanism for driving the movement of the robot arm joints; The robot arm motion control device is connected to the joint drive mechanism and is used to send control commands to the joint drive mechanism; The joint drive mechanism is used to drive the robot arm joints to move according to the control command, so that the actual end effector velocity follows the end effector velocity vector generated by the robot arm motion control device.
19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the robot arm motion control strategy network according to any one of claims 1 to 11, or the robot arm motion control method according to any one of claims 12 to 14.
Citation Information
Patent Citations
Robot motion control model training method and device based on deep reinforcement learning
CN120065751A
Robot motion control strategy network training method and device based on deep reinforcement learning, robot motion control method and device, equipment, robot and storage medium
CN121361098A
