Training method and device of control policy network for controlling robot double-arm end, equipment, robot and medium based on reinforcement learning

Abstract

The present disclosure provides a training method and device of a control policy network for controlling the end of a robot double arm based on reinforcement learning, an equipment, a robot and a medium, relating to the technical field of robots. The method comprises: based on the target pose data and the current state data for training the control policy, calculating the position error and the orientation error of the first arm end, and the position error and the orientation error of the second arm end; calculating the first reward term according to the position error of the first arm end and the position error of the second arm end respectively, and calculating the second reward term according to the orientation error of the first arm end and the orientation error of the second arm end respectively, the first reward term and the second reward term are both constructed by an exponential function; based on the product of the first reward term and the second reward term plus a constant term, calculating the comprehensive reward value and taking the comprehensive reward value as the training signal to train the control policy network. The method provided by the present disclosure can avoid the problem of significant decline in control accuracy on one side caused by resource competition.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and to a training method, apparatus, device, robot, and medium for a control strategy network for controlling the end effectors of a robot's two arms based on reinforcement learning. Background Technology

[0002] In humanoid robot control scenarios, achieving high-precision trajectory tracking of the end effectors (such as the wrists) is a crucial prerequisite for various complex tasks, such as grasping, handling, and interaction. With the development of deep reinforcement learning technology, more and more control strategy training methods are being applied to robot end effector control to improve the adaptability and robustness of the control system.

[0003] The paper "Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulator" (Kaiwen Jiang et al., December 4, 2024) proposes a deep reinforcement learning-based control method for the end effector of a wheeled quadruped robot. This method only performs 6D posture tracking control for a single end effector and does not establish an error balancing mechanism between the two end effectors. Applying this method leads to the robot prioritizing the coordination of control tasks for a single end effector when performing complex tasks, failing to balance the tracking accuracy of both end effectors. This results in the weakening or neglect of the other end effector's motion execution, leading to uncoordinated movements between the two end effectors.

[0004] In some existing technologies, such as the Chinese patent application with publication number CN119141551A, the position information and direction error of the two ends are introduced into the state definition. However, although multiple corresponding reward function terms are constructed in this scheme, only weighted or individual term excitation is used. When the error is large or small, the reward structure in this scheme may suffer from gradient explosion. At the same time, although the scheme includes left and right end actuators, it lacks a mechanism to control the resource competition between the left and right ends, making it difficult to ensure the tracking accuracy of both ends at the same time. This can easily lead to a decrease in control accuracy on one side, resulting in uncoordinated actions of the two ends. Summary of the Invention

[0005] This disclosure provides a training method, apparatus, device, robot, and medium for a control strategy network based on reinforcement learning for controlling the end effectors of a robot's two arms. It can automatically coordinate the tracking accuracy between the first and second arms, avoiding the problem of a significant decrease in control accuracy on one side due to resource contention.

[0006] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0007] According to a first aspect of this disclosure, a reinforcement learning-based method for training a control policy network for a robot's dual-arm end effector is provided, comprising:

[0008] Acquire target pose data for training the control strategy. The target pose data includes the target position and target orientation at the end of the first arm and the target position and target orientation at the end of the second arm.

[0009] Based on the target attitude data and the current state data, calculate the position error and orientation error of the first arm end and the position error and orientation error of the second arm end respectively;

[0010] The first reward item is calculated based on the position error of the first arm end and the position error of the second arm end, respectively. The second reward item is calculated based on the orientation error of the first arm end and the orientation error of the second arm end, respectively. Both the first reward item and the second reward item are constructed using exponential functions.

[0011] The comprehensive reward value is calculated based on the product of the first reward term, the second reward term, and the constant term. This comprehensive reward value is then used as the training signal to train the control policy network.

[0012] In one exemplary embodiment of this disclosure, the position error and orientation error of the first arm end-effector and the position error and orientation error of the second arm end-effector are calculated based on target attitude data and current state data, including:

[0013] Acquire current status data, which includes the actual position and orientation of the first arm end cap in the current control cycle, and the actual position and orientation of the second arm end cap in the current control cycle;

[0014] The position error of the first arm end is determined based on the Euclidean distance between the target position and the actual position corresponding to the end of the first arm. The position error of the second arm end is determined based on the Euclidean distance between the target position and the actual position corresponding to the end of the second arm.

[0015] The first rotation difference matrix is ​​determined based on the target orientation and the actual orientation of the end of the first arm, and the second rotation difference matrix is ​​determined based on the target orientation and the actual orientation of the end of the second arm.

[0016] The orientation error of the end of the first arm is determined based on the first rotation difference matrix, and the orientation error of the end of the second arm is determined based on the second rotation difference matrix.

[0017] In one exemplary embodiment of this disclosure, determining a first rotation difference matrix based on the target orientation and actual orientation of the first arm's end, and determining a second rotation difference matrix based on the target orientation and actual orientation of the second arm's end, includes:

[0018] A first target rotation matrix is ​​determined based on the target orientation at the end of the first arm, and a first actual rotation matrix is ​​determined based on the actual orientation at the end of the first arm; a second target rotation matrix is ​​determined based on the target orientation at the end of the second arm, and a second actual rotation matrix is ​​determined based on the actual orientation at the end of the second arm.

[0019] Perform transpose on the first target rotation matrix to obtain its inverse, and perform transpose on the second target rotation matrix to obtain its inverse.

[0020] Multiply the inverse of the first target rotation matrix with the first actual rotation matrix to obtain the first rotation difference matrix;

[0021] Multiply the inverse of the second target rotation matrix with the second actual rotation matrix to obtain the second rotation difference matrix.

[0022] In one exemplary embodiment of this disclosure, determining the orientation error of the first arm end based on a first rotation difference matrix and determining the orientation error of the second arm end based on a second rotation difference matrix includes:

[0023] Logarithmic transformation is performed on the first rotation difference matrix to obtain the orientation error vector of the end of the first arm; logarithmic transformation is performed on the second rotation difference matrix to obtain the orientation error vector of the end of the second arm.

[0024] The Euclidean norm corresponding to the orientation error vector at the end of the first arm is determined as the orientation error at the end of the first arm; the Euclidean norm corresponding to the orientation error vector at the end of the second arm is determined as the orientation error at the end of the second arm.

[0025] In an exemplary embodiment of this disclosure, the calculation of a first reward item based on the positional error of the first arm's end and the positional error of the second arm's end includes:

[0026] The first squared error term is obtained by squaring the position error at the end of the first arm, and the second squared error term is obtained by squaring the position error at the end of the second arm.

[0027] The first squared error term and the second squared error term are weighted and averaged to obtain the joint position error term.

[0028] An exponential function operation is performed on the joint position error term to obtain the first reward term.

[0029] In one exemplary embodiment of this disclosure, a second reward is calculated based on the orientation error at the end of the first arm and the orientation error at the end of the second arm, respectively, including:

[0030] The orientation error at the end of the first arm is squared to obtain the first orientation error term; the orientation error at the end of the second arm is squared to obtain the second orientation error term.

[0031] The first orientation error term and the second orientation error term are weighted and averaged to obtain the joint orientation error term.

[0032] An exponential function operation is performed on the joint orientation error term to obtain the second reward term.

[0033] In one exemplary embodiment of this disclosure, calculating the comprehensive reward value based on the product of the first reward term and the second reward term plus a constant term includes: according to:

[0034]

[0035] Calculate the overall reward value;

[0036] in, This indicates the first reward item. denoted as the second reward item; C represents a constant.

[0037] In one exemplary embodiment of this disclosure, the value of the constant is 0.9 to 1.1.

[0038] In one exemplary embodiment of this disclosure, a control policy network is trained using a comprehensive reward value as a training signal, including:

[0039] The current state data is input into the control strategy network to determine the action output variable corresponding to the current state data and the probability value corresponding to the action output variable.

[0040] Based on the comprehensive reward value and probability value, construct the reinforcement loss function corresponding to the control policy network;

[0041] Based on the reinforcement loss function, the trainable parameters in the control policy network are updated through backpropagation.

[0042] In one exemplary embodiment of this disclosure, determining the action output variable corresponding to the current state data and the probability value corresponding to the action output variable includes:

[0043] The current state data is input into the control policy network to determine the action probability distribution corresponding to the current state data;

[0044] Among the multiple candidate actions corresponding to the action probability distribution, select one candidate action as the action output variable for the current control cycle;

[0045] Obtain the probability value of the action output variable in the action probability distribution.

[0046] In one exemplary embodiment of this disclosure, a reinforcement loss function corresponding to the control policy network is constructed based on the comprehensive reward value and probability value, including:

[0047] The difference between the comprehensive reward value and the preset moving average reward value is defined as the reward difference;

[0048] Determine the sum of the probability value and the preset disturbance factor, and perform a logarithmic transformation on the sum to obtain the logarithmic probability value;

[0049] Multiply the reward difference by the logarithmic probability value to obtain the first loss term;

[0050] A regularization term is constructed based on the gradient norm of the trainable parameters in the control policy network.

[0051] The first loss term and the regularization term are weighted and combined to obtain the reinforcement loss function.

[0052] According to a second aspect of this disclosure, a reinforcement learning-based control method for the end effector of a robot dual arm is provided, comprising:

[0053] Acquire target attitude data, which includes the target position and orientation at the end of the first arm, and the target position and orientation at the end of the second arm;

[0054] The target pose data is input into a pre-trained control policy network to obtain the control policy;

[0055] The pre-trained control policy network is obtained according to the training method of the aforementioned reinforcement learning-based control policy network for controlling the end effector of the robot's two arms.

[0056] According to a third aspect of this disclosure, a training apparatus for a control policy network for controlling the end effector of a robot's dual arms based on reinforcement learning is provided, comprising:

[0057] The first acquisition module is used to acquire target pose data for training the control strategy. The target pose data includes the target position and target orientation at the end of the first arm and the target position and target orientation at the end of the second arm.

[0058] The calculation module is used to calculate the position error and orientation error of the first arm end and the position error and orientation error of the second arm end based on the target attitude data and the current state data.

[0059] The reward module is used to calculate the first reward item based on the position error of the first arm end and the position error of the second arm end, and to calculate the second reward item based on the orientation error of the first arm end and the orientation error of the second arm end, respectively. Both the first reward item and the second reward item are constructed using exponential functions.

[0060] The training module is used to calculate the comprehensive reward value based on the product of the first reward term and the second reward term plus a constant term, and use the comprehensive reward value as the training signal to train the control policy network.

[0061] According to a fourth aspect of this disclosure, a reinforcement learning-based control device for the end effector of a robot dual arm is provided, comprising:

[0062] The second acquisition module is used to acquire target attitude data, which includes the target position and target orientation at the end of the first arm, and the target position and target orientation at the end of the second arm.

[0063] The control strategy module is used to input the target posture data into the pre-trained control strategy network to obtain the control strategy; wherein, the pre-trained control strategy network is obtained according to any of the above-mentioned reinforcement learning-based training methods for the control strategy network for controlling the end effector of the robot's two arms.

[0064] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0065] Processor; and

[0066] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0067] According to a sixth aspect of this disclosure, a robot is provided, comprising:

[0068] Processor; and

[0069] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0070] In one exemplary embodiment of this disclosure, the robot includes any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, cleaning robots, transport robots, mobile robots, and robotic arms.

[0071] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions, which, when invoked by a robot's processor, cause the robot to perform the method as described in the above embodiments.

[0072] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0073] This disclosure constructs a first reward term corresponding to the position error of the first and second arm ends and a second reward term corresponding to the orientation error during the training of the control policy network. This enables simultaneous accuracy tracking of the first and second arms during training, avoiding the problem of decreased tracking accuracy of the other arm due to training bias towards single-arm data. Furthermore, a comprehensive reward value is obtained by multiplying the first and second reward terms with a constant term. This achieves the goal of using position error as the main reward term and attitude error as an adjustment factor to balance the priority of tracking accuracy for the left and right ends, avoiding significant decrease in control accuracy on one side due to resource competition and preventing uncoordinated movements of both arms during task execution. Simultaneously, the exponential function used to determine the reward term and the product coupling mechanism used to determine the comprehensive reward value provide more sensitive reward feedback when the error is small, and avoid reward gradient explosion when the error is large, ensuring training stability. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A system architecture diagram of a reinforcement learning-based method for training a control policy network for controlling the end effector of a robot's dual arms is shown in an embodiment of this disclosure.

[0076] Figure 2 A flowchart illustrating a training method for a control policy network for controlling the end effector of a robot's dual arms based on reinforcement learning, as shown in an embodiment of this disclosure, is presented.

[0077] Figure 3 The illustration shows a scenario in which a dual-arm robot according to an embodiment of this disclosure performs different target tasks in a simulation environment.

[0078] Figure 4 A flowchart illustrating a method for calculating the position and orientation errors of the ends of the two arms according to an embodiment of this disclosure is shown.

[0079] Figure 5 A flowchart illustrating a method for determining the rotation difference matrix according to an embodiment of this disclosure is shown.

[0080] Figure 6A flowchart illustrating a method for determining orientation error according to an embodiment of this disclosure is shown.

[0081] Figure 7 A flowchart illustrating a method for training a control strategy using a comprehensive reward value as a training signal, as shown in an embodiment of this disclosure, is illustrated.

[0082] Figure 8 A flowchart illustrating a reinforcement learning-based control method for controlling the end effectors of a robot's dual arms is shown in an embodiment of this disclosure.

[0083] Figure 9 A block diagram of a training apparatus for a control policy network for controlling the end effector of a robot's two arms, based on reinforcement learning, is shown in an embodiment of this disclosure.

[0084] Figure 10 A block diagram of a reinforcement learning-based control device for controlling the end effectors of a robot's two arms is shown in an embodiment of this disclosure.

[0085] Figure 11 A schematic diagram of a humanoid robot according to an embodiment of this disclosure is shown.

[0086] Figure 12 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0087] Figure 13 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0088] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0089] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained:

[0090] Deep Reinforcement Learning (DRL) is a machine learning method that combines reinforcement learning decision-making mechanisms with deep neural networks. By enabling an agent to interact with the environment in the training environment, the neural network parameters are iteratively updated based on the reward signals from the environment, thereby learning the mapping relationship from state to action.

[0091] Robot Dual-Arm Ends: Refers to the mounting reference point (e.g., wrist position) of the end effectors in a robot dual-arm system, specifically described in this disclosure as the first arm end effector and the second arm end effector. This end effector serves as a reference point for cooperative trajectory tracking in three-dimensional space, and its definition is independent of the specific structure of the end effector (e.g., a dexterous hand or gripper). This disclosure applies both to the end effector point of the robotic arm when no actuator is installed and to the specific position of the end effector itself after installation.

[0092] Control Policy Network (CPN): This refers to a neural network model built based on deep reinforcement learning. The CPN can take the current state data of the two arm end-effectors and the target pose data as input, and output corresponding action probability distributions or control commands to drive the two arm end-effectors to collaboratively approach the target state.

[0093] Target Pose Data: refers to the set of ideal states that the robot's end arms should reach in the task space, including two dimensions: target position and target orientation.

[0094] Comprehensive Reward Value: This refers to the core scalar signal used to evaluate the quality of the strategy during training. In this disclosure, the comprehensive reward value is calculated by product coupling of the first reward term corresponding to the position error and the second reward term corresponding to the orientation error. This mechanism aims to establish the priority of position accuracy and use orientation accuracy as an adjustment factor to balance the competition for resources between the two arms.

[0095] Exponential function construction refers to the use of an exponential nonlinear mapping method when calculating the reward term. This disclosure utilizes an exponential function to map the position error or orientation error to the (0,1] interval. This construction method makes the reward function highly sensitive when the error is small, while the gradient is flat when the error is large, which can effectively prevent the gradient explosion problem in the early stage of training.

[0096] Rotation matrix: An orthogonal matrix used to describe the rotational transformation of a rigid body in three-dimensional Euclidean space. In this disclosure, the first rotation difference matrix and the second rotation difference matrix are determined by calculating the product of the inverse of the target rotation matrix and the actual rotation matrix, and then the orientation error of the ends of the two arms is quantified by combining a logarithmic map.

[0097] Reinforcement Loss Function: This refers to the objective function used to guide the updating of parameters in the control policy network. In this disclosure, the function is a weighted combination of the product of the combined reward value and the log probability value of the action, along with a regularization term. This loss function is minimized through the backpropagation algorithm, thereby optimizing the output probability distribution of the policy network.

[0098] Figure 1 A system architecture diagram is shown that can be applied to the training method of the reinforcement learning-based control policy network for controlling the end effector of a robot's dual arms, as described in the embodiments of this disclosure. Figure 1 As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104. The terminal device 101 can be a desktop computer, laptop computer, smartphone, or tablet computer, etc. The terminal device 101 serves as an interactive interface, providing visualization functions to display information such as the target posture data, current state data, and position errors of the first and second arm ends of the robot 102. It also supports sending control strategies to the robot 102.

[0099] Robot 102 is equipped with various sensors to collect its current state data and can execute control strategies under the control of server 104 to complete corresponding training actions or tasks. Server 104 is deployed with a deep reinforcement learning-based training framework. During training, server 104 receives target posture data and current state data from robot 102, calculates the position and orientation errors of the first and second arm end caps respectively, and inputs them into the in-situ motion control strategy network. Combining the output motion strategy with the parameters of the control strategy network based on the first and second reward items, server 104 uses a deep reinforcement learning algorithm to optimize the network parameters. Server 104 also supports multi-concurrent training task management, task scheduling, and policy transfer.

[0100] After training, server 104 can deploy the trained control strategy network to robot 102, enabling it to autonomously reason and execute action strategies in actual operation. Terminal device 101 can also be used to call and deploy control strategies, monitor robot motion status, or remotely schedule and control.

[0101] Network 103 serves as the medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. It should be understood that... Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0102] This disclosure provides an exemplary embodiment of a method for training a control policy network for controlling the end effectors of a robot's two arms based on reinforcement learning. The control policy network can receive target posture data for training, including the target position and orientation of the first and second arm end effectors, as well as real-time state data of the robot in the current control cycle, including the actual position and orientation of the first and second arm end effectors. During training, the control policy network focuses on the coordinated control of the two arm end effectors. By calculating the position and orientation errors of the first and second arm end effectors respectively, and constructing a first and a second reward term respectively, it can automatically coordinate resource allocation between the two arm end effectors and balance the tracking accuracy of both arms. In the policy training phase, this embodiment calculates a comprehensive reward value by multiplying the first and second reward terms by a constant term, and uses this as a training signal to iteratively optimize the network parameters using a deep reinforcement learning algorithm. This training method does not rely on the traditional linear weighted reward structure, avoiding the gradient explosion problem under extreme error conditions. Instead, it uses a product coupling mechanism to set the position error as the main reward and the posture error as the adjustment factor, achieving high-precision and coordinated trajectory tracking of the two arm end-effectors. This alleviates the problems of significant decrease in control accuracy on one side due to resource competition and uncoordinated movements of the two arm end-effectors. In the example implementations of this disclosure, deep reinforcement learning includes, but is not limited to, algorithms such as PPO (Proximal Policy Optimization), DDPG (Deep Deterministic Policy Gradient), and SAC (Soft Actor-Critic). This disclosure does not limit the specific algorithm type of deep reinforcement learning.

[0103] In some example implementations, references Figure 2 As shown, the training method for the control policy network for controlling the end effector of a robot's dual arms based on reinforcement learning includes steps S201-S204:

[0104] Step S201: Obtain target pose data for training the control strategy. The target pose data includes the target position and target orientation at the end of the first arm and the target position and target orientation at the end of the second arm.

[0105] As an example, such as Figure 3 As shown, Figure 3The illustrations depict scenarios where a dual-arm robot performs different target tasks. Each scenario includes two distinct task objectives, each with strict requirements for the target position and orientation of the end effectors. The end effectors represent the robotic arm's end point and are independent of specific end effectors such as dexterous hands, grippers, or suction cups. The target position characterizes the three-dimensional spatial position the end effectors should reach, while the target orientation represents the spatial attitude the end effectors should maintain after reaching the target position. This orientation can be represented using rotation matrices, quaternions, or Euler angles, indicating the rotation angle relationship of the end effectors relative to a reference coordinate system. This reference coordinate system is typically a fixed coordinate system, such as the robot's base coordinate system (world coordinate system) or a specific task platform coordinate system, ensuring the uniqueness and determinism of the target orientation.

[0106] Step S202: Based on the target attitude data and the current state data, calculate the position error and orientation error of the first arm end and the position error and orientation error of the second arm end, respectively.

[0107] The current status data may include the current position of the end of the first arm and the end of the second arm, respectively, and the current orientation of the end of the first arm and the end of the second arm, respectively.

[0108] The positional error of each end point can be calculated based on the Euclidean distance, using the target position and current position of each end point. The orientation error of each end point can be determined based on the target orientation and current orientation. For example, the orientation error can be obtained by converting the target orientation and current orientation into direction cosine matrices respectively, and then calculating the rotation angle between them. Alternatively, it can be obtained using the formula for calculating the angle between unit quaternions. For example, if using direction cosine matrices, the product of the transpose of the current matrix and the target matrix can be calculated first to obtain a relative rotation matrix indicating the difference between the two. Then, the minimum rotation angle between the two can be calculated based on the trace of this relative rotation matrix. Alternatively, if using unit quaternions, the minimum rotation angle between the two can be calculated using the inverse cosine function by calculating the absolute value of the dot product of the target quaternion and the current quaternion.

[0109] Step S203: Calculate the first reward item based on the position error of the first arm end and the position error of the second arm end, respectively, and calculate the second reward item based on the orientation error of the first arm end and the orientation error of the second arm end, respectively. Both the first reward item and the second reward item are constructed using exponential functions.

[0110] The first reward term can be calculated based on the position errors of the first and second arm endpoints, and the second reward term can be calculated based on the orientation errors of the second arm endpoints. Specifically, when the error is extremely large, the reward term calculated by the exponential function approaches zero and has a gentle gradient, avoiding the gradient explosion problem caused by excessive error in the initial training phase and ensuring training stability. Simultaneously, when the position error is small, it has a larger gradient, resulting in more sensitive reward feedback, thereby guiding the policy network to perform fine-tuning and significantly improving the final tracking accuracy. Therefore, the first and second reward terms constructed based on the exponential function ensure that the reward terms during training possess both stability and sensitivity.

[0111] Step S204: Calculate the comprehensive reward value based on the product of the first reward term and the second reward term plus the constant term, and use the comprehensive reward value as the training signal to train the control policy network.

[0112] The first and second reward items can be standardized separately to ensure dimensional consistency and prevent any one indicator from dominating subsequent calculations due to its large numerical range. After standardization, a non-negative constant term can be introduced into the second reward item. This constant term can be used to adjust the weight of orientation accuracy in the reward construction. The adjusted second reward item can then be multiplied with the first reward item to obtain the comprehensive reward value.

[0113] After obtaining the comprehensive reward value, this value can be used as an immediate reward signal in the reinforcement learning process to perform backpropagation and parameter updates on the current policy network. During training, actual action trajectories can be continuously sampled, and position and orientation errors can be calculated based on the target pose and actual state at each step. Then, corresponding first and second reward terms can be constructed, and the comprehensive reward value can be calculated using the above method to continuously optimize the output performance of the control policy network.

[0114] The comprehensive reward value constructed in the above manner, by using positional reward as the dominant multiplier, prioritizes the accuracy of the target position at the end of the arms in the control policy network. This allows for dynamic adjustment of resource allocation priorities in the end-of-arm control task. Furthermore, fine-tuning of the orientation error is performed, avoiding motion incoordination issues caused by resource contention. This approach exhibits good training stability and is suitable for policy optimization during long-term iterations.

[0115] In some example implementations, based on the target attitude data and the current state data, the position error and orientation error of the first arm end-effector and the position error and orientation error of the second arm end-effector are calculated respectively, including steps S401-S404:

[0116] Step S401: Obtain current status data, which includes the actual position and orientation of the first arm end in the current control cycle, and the actual position and orientation of the second arm end in the current control cycle.

[0117] The robot's current state data can be obtained, including the actual position and orientation of the first arm's end effector and the second arm's end effector in the current control cycle. As an example, the actual position and orientation of both arm end effectors relative to a reference coordinate system can be obtained by acquiring the robot's joint encoder data in real time.

[0118] Step S402: Determine the position error of the first arm end based on the Euclidean distance between the target position and the actual position corresponding to the end of the first arm, and determine the position error of the second arm end based on the Euclidean distance between the target position and the actual position corresponding to the end of the second arm.

[0119] The target position at the end of the first arm can be compared with the current actual position to calculate the Euclidean distance. This Euclidean distance reflects the distance between the target position and the actual position, thus measuring the degree of spatial deviation of the first arm's end. Similarly, the target position at the end of the second arm can be compared with the actual position to obtain the position error of the second arm's end.

[0120] Step S403: Determine the first rotation difference matrix based on the target orientation and actual orientation of the end of the first arm, and determine the second rotation difference matrix based on the target orientation and actual orientation of the end of the second arm.

[0121] To calculate the orientation error, the target orientation and the actual orientation can first be unified in the same coordinate system, such as the robot's base coordinate system. Then, it can be converted into the form of rotation matrices, constructing the target rotation matrix and the actual rotation matrix for each arm respectively. Based on the orthogonality of rotation matrices, i.e., the inverse of a matrix is ​​equal to its transpose, a rotation difference matrix describing the attitude difference can be constructed through matrix operations, namely the first rotation difference matrix and the second rotation difference matrix. As an example, the rotation difference matrix can be obtained by calculating the matrix product of the transpose of the actual orientation matrix and the target orientation matrix. This rotation difference matrix physically describes the relative rotation required to transform from the actual attitude to the target attitude. It eliminates the influence of the unified base coordinate system, characterizes the deviation between the two in spatial attitude, and implicitly contains the equivalent rotation axis and rotation angle required to adjust the end effector from the current orientation to the target orientation.

[0122] Step S404: Determine the orientation error of the end of the first arm based on the first rotation difference matrix, and determine the orientation error of the end of the second arm based on the second rotation difference matrix.

[0123] For each rotation difference matrix, a matrix logarithmic transformation can be performed on the matrix to extract the orientation error vector. The Euclidean norm of the orientation error vectors corresponding to the first and second arms can be calculated separately. This norm is the final scalar value of the orientation error.

[0124] Using the above method, position and orientation errors can be calculated for the first and second arm ends based on target attitude data and current state data, providing accurate error information for the construction of the reward function of the subsequent control strategy.

[0125] In some example implementations, Figure 5 A flowchart illustrating the method for determining the rotation difference matrix is ​​shown, which can be performed as follows: steps S501-S503:

[0126] Step S501: Determine the first target rotation matrix based on the target orientation of the end of the first arm, and determine the first actual rotation matrix based on the actual orientation of the end of the first arm; determine the second target rotation matrix based on the target orientation of the end of the second arm, and determine the second actual rotation matrix based on the actual orientation of the end of the second arm.

[0127] As an example, for each end of an arm, the target orientation and the actual orientation corresponding to that end can be processed using a coordinate system to ensure spatial consistency in subsequent calculations.

[0128] The first and second arm ends can be transformed separately. The rotation matrix of the target orientation of the first arm end after transformation can be determined as the first target rotation matrix, and the rotation matrix of the actual orientation of the first arm end after transformation can be determined as the first actual rotation matrix. Similarly, for the second arm end, the rotation matrix of the target orientation of the second arm end after transformation can be determined as the second target rotation matrix, and the rotation matrix of the actual orientation of the second arm end after transformation can be determined as the second actual rotation matrix. For example, it can be converted to... The rotation matrix.

[0129] Step S502: Perform transpose processing on the first target rotation matrix to obtain the inverse of the first target rotation matrix, and perform transpose processing on the second target rotation matrix to obtain the inverse of the second target rotation matrix.

[0130] Since both the first and second target rotation matrices are rotation matrices in three-dimensional space and belong to the special orthogonal group SO(3), their inverses are equal to their transposes. Furthermore, utilizing the orthogonality property of rotation matrices—that is, the inverse of a matrix is ​​equal to its transpose—a more computationally efficient transpose operation can be used to invert the rotation matrices. Specifically, the first target rotation matrix can be transposed to obtain its inverse; the second target rotation matrix can be transposed to obtain its inverse.

[0131] Step S503: Multiply the inverse of the first target rotation matrix with the first actual rotation matrix to obtain the first rotation difference matrix; multiply the inverse of the second target rotation matrix with the second actual rotation matrix to obtain the second rotation difference matrix.

[0132] For the end of the first arm, the inverse of the first target rotation matrix can be multiplied by the first actual rotation matrix, and the resulting matrix is ​​the first rotation difference matrix. This first rotation difference matrix can be used to represent the relative rotation deviation required for the end of the first arm to transform from the target orientation to the actual orientation. For the end of the second arm, the inverse of the second target rotation matrix can be multiplied by the second actual rotation matrix to obtain the second rotation difference matrix, which can be used to represent the relative rotation deviation required for the end of the second arm to transform from the target orientation to the actual orientation.

[0133] Since the target orientation and the actual orientation may exist in the form of quaternions or Euler angles, this transpose-multiplication method avoids the numerical singularity and computational instability that may be caused by subtracting Euler angles when calculating attitude error. It also avoids the interference of quaternion symbol ambiguity on error calculation, and has good versatility and stability.

[0134] In some example implementations, such as Figure 6 As shown, the orientation error of the first arm end is determined according to the first rotation difference matrix, and the orientation error of the second arm end is determined according to the second rotation difference matrix, as shown in steps S601-S602:

[0135] Step S601: Perform logarithmic transformation on the first rotation difference matrix to obtain the orientation error vector at the end of the first arm; perform logarithmic transformation on the second rotation difference matrix to obtain the orientation error vector at the end of the second arm.

[0136] A logarithmic transformation can be performed on the first rotation difference matrix corresponding to the end of the first arm. Since rotation matrices are special orthogonal matrices, this first rotation difference matrix can be mapped to an antisymmetric matrix. This antisymmetric matrix can then be uniquely mapped to a preset dimension, such as a three-dimensional vector, as the orientation error vector of the first arm's end. Similarly, a logarithmic transformation can be performed on the second rotation difference matrix corresponding to the end of the second arm, mapping it to an antisymmetric matrix. This mapped antisymmetric matrix can then be further mapped to obtain the orientation error vector of the second arm's end. This orientation error vector can be used to describe the rotation direction and magnitude corresponding to the actual orientation in the current control cycle relative to the target orientation. As an example, for any three-dimensional rotation difference matrix... The corresponding rotation difference matrix can be calculated by performing a logarithmic transformation on it using the following formula:

[0137]

[0138] in, Let be the rotation angle, satisfying , For antisymmetric matrices, for The sum of all elements on the main diagonal of the matrix.

[0139] After obtaining the corresponding rotation difference matrix, the inverse operation of the antisymmetric operator can be used to map the corresponding rotation difference matrix into a three-dimensional column vector, which is the corresponding orientation error vector.

[0140] Step S602: The Euclidean norm corresponding to the orientation error vector at the end of the first arm is determined as the orientation error at the end of the first arm; the Euclidean norm corresponding to the orientation error vector at the end of the second arm is determined as the orientation error at the end of the second arm.

[0141] The Euclidean norm of each orientation error vector can be calculated separately. This Euclidean norm reflects the length of the orientation error vector in three-dimensional space, that is, the magnitude of the rotation angle. Therefore, the Euclidean norm of the orientation error vector at the end of the first arm can be determined as the orientation error at the end of the first arm; and the Euclidean norm of the orientation error vector at the end of the second arm can be determined as the orientation error at the end of the second arm.

[0142] The orientation error calculated in this way not only has good geometric interpretation, but is also numerically stable and computationally efficient. It can be used as a differentiable orientation deviation index in multiple processes such as subsequent reward function design and control strategy training.

[0143] In some example implementations, a first reward term is calculated based on the position errors at the ends of the first and second arms, respectively. This includes: squaring the position errors at the ends of the first and second arms to obtain a first squared error term, and squaring the position errors at the ends of the second and second arms to obtain a second squared error term; performing a weighted average of the first and second squared error terms to obtain a joint position error term; and performing an exponential function operation on the joint position error term to obtain the first reward term. As an example, the first reward term can be calculated using the following formula:

[0144]

[0145] in, This indicates the first reward item. This indicates the positional error at the end of the first arm. This indicates the positional error at the end of the second arm. This indicates the weight at the end of the first arm. This indicates the weight at the end of the second arm. This represents the preset position error sensitivity coefficient, the value of which can be determined according to the accuracy requirements of the specific task, or it can be determined by experience, such as based on the standard deviation corresponding to the position error of the sample. It can also be dynamically adjusted during the calculation process; this disclosure does not impose any special limitations. For example, when the range of the end-position error is 0–0.2 meters, It can be taken as 0.01 to 0.05; under different task spatial scales or control accuracy requirements, It can also be scaled synchronously according to the change ratio of the error dimension to ensure that the reward function has stable gradient change characteristics under different training scenarios. Weights and Different values ​​can be set according to the task requirements; this disclosure does not impose any special restrictions.

[0146] This approach avoids gradient abruptness or oscillation near the zero point of linear error. Due to the nonlinear nature of the exponential function, a high value for the first reward term can only be obtained when the weighted sum term within the exponent remains small. If the positional error on either side of the two arms is large, the exponent will increase significantly after square amplification, thus lowering the value of the entire first reward term. This forces the control policy network to optimize the errors at both arms simultaneously, preventing the sacrifice of one side's accuracy for a high score in the first reward term. This effectively avoids overfitting of a single arm or incoordination of the two arms' movements. At the same time, the introduction of the exponential function strictly limits the range of the reward term to the (0,1) interval, effectively preventing gradient explosion in the early stages of training. By adjusting the sensitivity coefficient, the same reward framework can be adapted to tasks with different accuracy requirements.

[0147] In some example implementations, the second reward term is calculated based on the orientation errors at the ends of the first and second arms, respectively. This includes: squaring the orientation error at the end of the first arm to obtain a first orientation error term; squaring the orientation error at the end of the second arm to obtain a second orientation error term; performing a weighted average of the first and second orientation error terms to obtain a joint orientation error term; and performing an exponential function operation on the joint orientation error term to obtain the second reward term. As an example, the second reward term can be calculated using the following formula:

[0148]

[0149] in, This indicates the second reward item. This indicates the positional error at the end of the first arm. This indicates the positional error at the end of the second arm. This indicates the weight at the end of the first arm. This indicates the weight at the end of the second arm. This represents a preset orientation error sensitivity coefficient, the value of which can be determined according to the accuracy requirements of the specific task, or it can be determined by experience, such as based on the standard deviation corresponding to the position error of the sample. It can also be dynamically adjusted during the calculation process; this disclosure does not impose any special limitations. For example, when the end-point orientation error... When expressed in radians and the value range is 0 to π, It can be taken as 0.05 to 0.2, depending on the different task spatial scales or control accuracy requirements. It can also be scaled synchronously according to the change ratio of the error dimension to ensure that the reward function has stable gradient change characteristics in different training scenarios; weights and Different values ​​can be set according to the task requirements; this disclosure does not impose any special restrictions.

[0150] This approach avoids gradient abruptness or oscillation near the zero point of linear error, preventing the control policy network from sacrificing accuracy on one side to obtain a high score in the second reward item. This effectively avoids overfitting of one arm or incoordination of movements between the two arms. At the same time, it can effectively prevent gradient explosion in the early stage of training and adapt to tasks with different accuracy requirements.

[0151] In some example implementations, the comprehensive reward value is calculated based on the product of the first reward item, the second reward item, and a constant term. This comprehensive reward value can be calculated using the following formula:

[0152]

[0153] in, This indicates the first reward item. denoted as the second reward item; C represents a constant.

[0154] Unlike the traditional method of constructing a dual-arm reward function using a weighted summation approach, this disclosure employs a product-coupled approach to construct a comprehensive reward function. This ensures that when the position error at either end is large, the overall reward value will decrease. Specifically, when the position error at the end of either arm exceeds a corresponding preset threshold, its corresponding position reward item... Approaching zero, at which point the other arm's end faces the reward item. The value and the overall reward value are kept at a low level, thus forming a constraint effect during training. This product coupling structure can avoid the situation of sacrificing the control accuracy of one end to improve the reward of the other end, which is something that traditional linear weighted reward structures cannot achieve. It can effectively prevent resource competition and control bias problems in the process of dual-arm collaborative control.

[0155] In some example implementations, the value of the aforementioned constant can be a positive real number used to adjust the bias of the orientation reward term. Its value ranges from 0.9 to 1.1, and in a preferred embodiment, the constant term is 1. This value can appropriately amplify the magnitude of the overall reward value without introducing additional training bias, preventing excessive bias from weakening the policy's responsiveness to accuracy improvement and enhancing the reward function's ability to perform multi-task collaborative optimization.

[0156] In some example implementations, such as Figure 7 As shown, the method of training a control strategy using the comprehensive reward value as a training signal may include steps S701-S703:

[0157] Step S701: Input the current state data into the control strategy network to determine the action output variable corresponding to the current state data and the probability value corresponding to the action output variable.

[0158] The process involves inputting real-time state data of the robot's current control cycle, including the actual position and orientation of the end effectors, into the control policy network to be trained. The output layer of this network maps the action probability distribution of the action space in the current state. Random sampling can be performed from this action probability distribution, selecting a specific action from multiple candidate actions as the action output variable for the current control cycle, and then controlling the robot to execute that action. Simultaneously, the probability value corresponding to the selected action output variable can be calculated based on this action probability distribution; this probability value reflects the policy network's confidence in the current decision.

[0159] Step S702: Based on the comprehensive reward value and probability value, construct the reinforcement loss function corresponding to the control policy network.

[0160] After obtaining the action probability value and the combined reward value calculated after actually executing the action, a reinforcement learning loss function is constructed for controlling policy training. For example, a policy gradient-based method can be used to construct the reinforcement loss function. The reinforcement loss function can be constructed using the following formula:

[0161]

[0162] in, This represents the loss value corresponding to the enhancement loss function. Indicates the expected value. ( () represents the probability value corresponding to the action policy. Indicates action strategy, Indicates the robot's current state. This represents the current total reward value.

[0163] Step S703: Based on the reinforcement loss function, update the trainable parameters in the control policy network through backpropagation.

[0164] The loss value corresponding to the reinforcement loss function can be obtained, and optimization algorithms such as gradient descent can be used to backpropagate and iteratively update the trainable parameters in the control policy network. This training process can be repeated over multiple training cycles, each cycle including state sampling, action decision-making, action execution, reward evaluation, and network update. As the number of training cycles increases, the control policy network will gradually learn action strategies that enable precise coordinated control of the two arm ends under different states.

[0165] By using the comprehensive reward value as the training signal, we can ensure that both positional accuracy and orientation accuracy are considered during training, and learn more natural and coordinated action sequences. At the same time, since the comprehensive reward value adopts an exponential function structure and is associated with the action probability through a product, the vanishing gradient phenomenon can be effectively avoided, and the effectiveness of the policy gradient signal in the early stage of training can be improved.

[0166] In some example implementations, in step S701 above, the current state data can be input to the control strategy network to determine the action probability distribution corresponding to the current state data; among the multiple candidate actions corresponding to the action probability distribution, one candidate action is selected as the action output variable of the current control cycle; and the probability value of the action output variable in the action probability distribution is obtained.

[0167] A control policy network can output a set of action probability distributions that match the current state data. This action probability distribution is typically represented as a vector, where each dimension corresponds to a candidate action and its probability of selection in that state. These candidate actions can be a predefined set of discrete actions in the robot's action space, such as standard angle combinations of joints, or action samples parameterized from continuous actions, such as angular velocity, torque, or trajectory control parameters.

[0168] From this action probability distribution, based on a preset action sampling strategy (such as random sampling or maximum probability sampling), a specific action can be selected as the action output variable to be executed in the current control cycle. This action output variable directly determines the robot's behavior in the next moment, such as performing a movement, adjusting posture, or performing a grasping or releasing action. Simultaneously, the probability value corresponding to this action output variable in the aforementioned action probability distribution is extracted. This probability value is typically in logarithmic probability form. This probability value can be used to subsequently construct a reinforcement learning loss function to measure the degree of preference of the current control policy network for the action output variable.

[0169] The above method enables the control policy network to explore control policies probabilistically, which can effectively prevent the control policy from getting stuck in local optima. At the same time, by explicitly obtaining the probability value corresponding to the action and combining it with the comprehensive reward value, the control policy network is guided to update in the direction of high reward and high probability, thereby improving the accuracy of network training.

[0170] In some example implementations, in step S702 above, the difference between the comprehensive reward value and the preset moving average reward value can be determined as the reward difference; the sum of the probability value and the preset perturbation factor is determined, and the sum is logarithmically transformed to obtain the logarithmic probability value; the reward difference is multiplied by the logarithmic probability value to obtain the first loss term; a regularization term is constructed based on the gradient norm corresponding to the trainable parameters in the control policy network; the first loss term and the regularization term are weighted and merged to obtain the reinforcement loss function.

[0171] A preset moving average reward value can be obtained, which records the average value of rewards during historical training. The overall reward value can be calculated by subtracting the moving average reward value from the overall reward value, and the resulting difference is determined as the reward difference. This reward difference can be used to evaluate the performance of the current action compared to the historical average.

[0172] The probability value of the current action output variable in the action probability distribution can be obtained. The sum of this probability value and a preset disturbance factor can be determined, and the logarithmic probability value can be obtained by performing a logarithmic transformation on the sum of the obtained probability value and the preset disturbance factor. Here, the disturbance factor is an extremely small positive number, such as 1e-6; a larger value of the logarithmic probability value indicates that the control strategy is more inclined towards this action. Since directly taking the logarithm of an extremely small probability value may lead to numerical overflow during calculation, this smoothing method is used to obtain the logarithmic probability value.

[0173] The calculated reward difference can be multiplied by the logarithmic probability value to obtain the first loss term. This first loss term represents the objective direction and strength of policy optimization, that is, it characterizes the tendency to increase or decrease the probability of an action by adjusting parameters under the current policy. Specifically, when the reward difference is positive, the gradient direction of the loss function will drive the parameter updates of the control policy network, thereby increasing the logarithmic probability of the action being selected in the future; when the reward difference is negative, it decreases the probability of the action.

[0174] The gradient norm, such as the L2 norm, corresponding to the trainable parameters of each layer in the control policy network can be calculated, and a regularization term can be constructed based on this gradient norm. This regularization term can prevent gradient explosion during the training process of the control policy network, or prevent policy oscillation caused by excessive parameter update amplitude, thereby improving the robustness of training.

[0175] The first loss term and the regularization term are weighted and combined according to preset weight coefficients to obtain the reinforcement loss function. The control policy network is then optimized using this reinforcement loss function, and the trainable parameters in the network are updated using the gradient backpropagation mechanism, thereby gradually improving the performance of the policy in complex control tasks.

[0176] The following will provide a detailed description of the reinforcement learning-based control method for the end effector of a robot's dual arms in this example embodiment. Figure 8 As shown, the reinforcement learning-based control method for the end effector of a robot's dual arms includes steps S801-S802:

[0177] Step S801: Obtain target attitude data, which includes the target position and target orientation at the end of the first arm, and the target position and target orientation at the end of the second arm.

[0178] Target posture data can be acquired, which may include the target position and target orientation of the first arm's end effector, as well as the target position and target orientation of the second arm's end effector. The target position can represent the three-dimensional spatial position in the reference coordinate system that the robot's two arm end effectors should reach, and the target orientation can represent the spatial posture state that the robot's two arm end effectors should maintain after reaching the target position. Specifically, the target position represents the three-dimensional spatial position in the reference coordinate system that the two arm end effectors should reach, and the target orientation represents the spatial posture state that the two arm end effectors should maintain after reaching the target position.

[0179] Step S802: Input the target pose data into the pre-trained control policy network to obtain the control policy.

[0180] After acquiring the target posture data, this data, along with the robot's current state data, can be encoded and input into a pre-trained control strategy network model to generate the action strategy for the current control cycle. The current state data can include the current positions and orientations of the first and second arm end-effectors, respectively. The pre-trained control strategy network is a deep, reinforced neural network model with pre-initialized parameters. The weights and biases within this network have been optimized to convergence in previous training steps. It can perform forward propagation calculations on the encoded input vector, and through nonlinear mapping of multiple neurons, output the control strategy for the current state. This control strategy can include end-effector velocity commands, joint control commands, and action probability distributions, the specifics of which can be determined based on actual interface requirements; this disclosure does not impose any special limitations. This approach fully leverages the powerful nonlinear fitting and generalization capabilities of pre-trained networks, enabling the rapid generation of highly coupled and coordinated two-hand control strategies in multi-task or complex action scenarios, such as hand-to-hand transfer and collaborative handling. This avoids the incoordination issues caused by independent planning of the two arms in traditional methods, and possesses good versatility and real-time performance.

[0181] In an exemplary embodiment of this disclosure, a training apparatus for a control policy network for controlling the end effectors of a robot's dual arms, based on reinforcement learning, is also provided. (See reference...) Figure 9 As shown, the training device 900 for a control policy network based on reinforcement learning for controlling the end effectors of a robot's dual arms includes a first acquisition module 901, a calculation module 902, a reward module 903, and a training module 904, wherein:

[0182] The first acquisition module 901 is used to acquire target posture data for training the control strategy. The target posture data includes the target position and target orientation at the end of the first arm and the target position and target orientation at the end of the second arm.

[0183] Calculation module 902 is used to calculate the position error and orientation error of the first arm end and the position error and orientation error of the second arm end based on the target posture data and the current state data, respectively.

[0184] The reward module 903 is used to calculate a first reward item based on the position error of the first arm end and the position error of the second arm end, and to calculate a second reward item based on the orientation error of the first arm end and the orientation error of the second arm end, respectively. Both the first reward item and the second reward item are constructed using an exponential function.

[0185] The training module 904 is used to calculate a comprehensive reward value based on the product of the first reward term and the second reward term plus a constant term, and use the comprehensive reward value as a training signal to train the control strategy network.

[0186] The specific details of each module in the training device for the control policy network of the robot's dual arms based on reinforcement learning have been described in detail in the corresponding training method for the control policy network of the robot's dual arms based on reinforcement learning, so they will not be repeated here.

[0187] In an exemplary embodiment of this disclosure, a reinforcement learning-based control device for the end effector of a robotic dual arm is also provided. (See reference...) Figure 10 The reinforcement learning-based control device 1000 for the end effector of a robot includes a second acquisition module 1001 and a control strategy module 1002, wherein:

[0188] The second acquisition module 1001 is used to acquire target attitude data, the target attitude data including the target position and target orientation at the end of the first arm, and the target position and target orientation at the end of the second arm;

[0189] The control strategy module 1002 is used to input the target posture data into a pre-trained control strategy network to obtain a control strategy; wherein, the pre-trained control strategy network is obtained according to a reinforcement learning-based training method for a control strategy network for controlling the end effector of a robot's two arms.

[0190] The specific details of each module in the aforementioned reinforcement learning-based control device for robot dual-arm end effectors have been described in detail in the corresponding reinforcement learning-based control method for robot dual-arm end effectors, so they will not be repeated here.

[0191] In an exemplary embodiment of this disclosure, a robot is also provided, which includes any one of legged robots, quadruped robots, bipedal robots, wheeled robots, wheel-legged robots, quadrupedal robots, humanoid robots, cleaning robots, transport robots, mobile robots, and robotic arms. Of course, other types of mobile robots may also be included, and this disclosure does not limit them.

[0192] refer to Figure 11 As shown, a schematic diagram of another type of robot for which the motion control method of the robot in multi-terrain environments, as described in the embodiments of this disclosure, can be applied is illustrated. Figure 11 It can be seen that this type of robot is a humanoid robot.

[0193] refer to Figure 12 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1200 includes a processor 1201 and a memory 1202. The memory 1202 stores computer-readable instructions, which, when executed by the processor 1201, implement the method of this disclosure.

[0194] In an exemplary embodiment of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a robot's processor, cause the robot to perform the method described in the embodiments.

[0195] refer to Figure 13 As shown, a program product 1300 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0196] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0197] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this disclosure and are not restrictive. Although this disclosure has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this disclosure. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A training method of a control policy network for controlling an end of a robot dual-arm based on reinforcement learning, characterized by, The method comprises: acquiring target posture data for training a control strategy, the target posture data comprising target positions and target orientations of first and second arm ends; based on the target posture data and current state data, calculating position errors and orientation errors of the first and second arm ends respectively; calculating a first reward term according to the position error of the first arm end and the position error of the second arm end respectively, and calculating a second reward term according to the orientation error of the first arm end and the orientation error of the second arm end respectively, the first and second reward terms being constructed by an exponential function; based on the product of the first reward term and the second reward term plus a constant term, calculating a comprehensive reward value and taking the comprehensive reward value as a training signal to train a control strategy network.

2. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 1, wherein, The method comprises: acquiring the current state data, the current state data comprising actual positions and actual orientations of the first and second arm ends in a current control cycle; determining the position error of the first arm end according to the Euclidean distance between the target position and the actual position corresponding to the first arm end, and determining the position error of the second arm end according to the Euclidean distance between the target position and the actual position corresponding to the second arm end; determining a first rotation difference matrix according to the target orientation and the actual orientation of the first arm end, and determining a second rotation difference matrix according to the target orientation and the actual orientation of the second arm end; determining the orientation error of the first arm end according to the first rotation difference matrix, and determining the orientation error of the second arm end according to the second rotation difference matrix.

3. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 2, wherein, The method comprises: determining a first target rotation matrix according to the target orientation of the first arm end, and determining a first actual rotation matrix according to the actual orientation of the first arm end; determining a second target rotation matrix according to the target orientation of the second arm end, and determining a second actual rotation matrix according to the actual orientation of the second arm end; performing transpose processing on the first target rotation matrix to obtain an inverse of the first target rotation matrix, and performing transpose processing on the second target rotation matrix to obtain an inverse of the second target rotation matrix; matrix-multiplying the inverse of the first target rotation matrix and the first actual rotation matrix to obtain the first rotation difference matrix; matrix-multiplying the inverse of the second target rotation matrix and the second actual rotation matrix to obtain the second rotation difference matrix.

4. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 2, wherein, The method comprises: determining the orientation error of the first arm end according to the first rotation difference matrix, and determining the orientation error of the second arm end according to the second rotation difference matrix. Logarithmic transformation is performed on the first rotation difference matrix to obtain an orientation error vector of the first arm end, and logarithmic transformation is performed on the second rotation difference matrix to obtain an orientation error vector of the second arm end; A Euclidean norm corresponding to the orientation error vector of the first arm end is determined as an orientation error of the first arm end, and a Euclidean norm corresponding to the orientation error vector of the second arm end is determined as an orientation error of the second arm end.

5. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 1, wherein, The first reward item is calculated according to the position error of the first arm end and the position error of the second arm end, respectively, including: A square operation is performed on the position error of the first arm end to obtain a first square error term, and a square operation is performed on the position error of the second arm end to obtain a second square error term; A weighted average operation is performed on the first square error term and the second square error term to obtain a joint position error term; An exponential function operation is performed on the joint position error term to obtain the first reward item.

6. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 1, wherein, The second reward item is calculated according to the orientation error of the first arm end and the orientation error of the second arm end, respectively, including: A square operation is performed on the orientation error of the first arm end to obtain a first orientation error term, and a square operation is performed on the orientation error of the second arm end to obtain a second orientation error term; A weighted average operation is performed on the first orientation error term and the second orientation error term to obtain a joint orientation error term; An exponential function operation is performed on the joint orientation error term to obtain the second reward item.

7. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 1, wherein, The comprehensive reward value is calculated based on the product of the first reward item, the second reward item and a constant term, including: The comprehensive reward value is calculated; wherein represents the first reward term, represents the second reward term; C represents the constant.

8. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 7, wherein, The constant is 0.9-1.

1.

9. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 1, wherein, The comprehensive reward value is used as a training signal to train the control policy network, including: The current state data is input into the control policy network to determine an action output variable corresponding to the current state data and a probability value corresponding to the action output variable; Based on the comprehensive reward value and the probability value, a reinforcement loss function corresponding to the control policy network is constructed; Based on the reinforcement loss function, the trainable parameters in the control policy network are updated through back propagation.

10. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 9, wherein, The action output variable corresponding to the current state data and the probability value corresponding to the action output variable are determined, including: The current state data is input into the control policy network to determine an action probability distribution corresponding to the current state data; In a plurality of candidate actions corresponding to the action probability distribution, a candidate action is selected as an action output variable of a current control period; The probability value of the action output variable in the action probability distribution is obtained.

11. The training method of a control policy network for controlling an end of a robot double arm based on reinforcement learning according to claim 9, wherein, Based on the comprehensive reward value and the probability value, a reinforcement loss function corresponding to the control policy network is constructed, including: A difference between the comprehensive reward value and a preset sliding average reward value is determined as a reward difference; The sum of the probability value and a preset disturbance factor is determined, and logarithmic transformation is performed on the sum to obtain a logarithmic probability value; The reward difference and the logarithmic probability value are multiplied to obtain a first loss term; constructing a regularization term based on a gradient norm corresponding to a trainable parameter in the control policy network; combining the first loss term and the regularization term by weighting, to obtain the reinforced loss function.

12. A control method for a robot dual-arm end based on reinforcement learning, characterized by, The method comprises: obtaining target pose data, the target pose data comprising target position and target orientation of a first arm end, and target position and target orientation of a second arm end; inputting the target pose data into a pre-trained control policy network to obtain a control policy; wherein the pre-trained control policy network is obtained according to the training method of the control policy network for controlling robot double-arm ends based on reinforcement learning in any one of claims 1-11.

13. A training apparatus of a control policy network for controlling an end of a robot dual-arm based on reinforcement learning, characterized by, The method comprises: a first obtaining module configured to obtain target pose data for training a control policy, the target pose data comprising target position and target orientation of a first arm end, and target position and target orientation of a second arm end; a calculating module configured to calculate position error and orientation error of the first arm end, and position error and orientation error of the second arm end, based on the target pose data and current state data, respectively; a reward module configured to calculate a first reward term according to the position error of the first arm end and the position error of the second arm end, respectively, and to calculate a second reward term according to the orientation error of the first arm end and the orientation error of the second arm end, respectively, the first reward term and the second reward term being constructed by an exponential function; a training module configured to calculate a comprehensive reward value based on the product of the first reward term and the second reward term plus a constant term, and to train a control policy network using the comprehensive reward value as a training signal.

14. A control device for the end of a robot double arm based on reinforcement learning, characterized by, The method comprises: a second obtaining module configured to obtain target pose data, the target pose data comprising target position and target orientation of a first arm end, and target position and target orientation of a second arm end; a control policy module configured to input the target pose data into a pre-trained control policy network to obtain a control policy, wherein the pre-trained control policy network is obtained according to the training method of the control policy network for controlling robot double-arm ends based on reinforcement learning in any one of claims 1-11.

15. An electronic device, comprising: The method comprises: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method in any one of claims 1-12.

16. A robot, characterized in that The method comprises: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method in any one of claims 1-12.

17. The robot of claim 16, wherein, The robot comprises a mobile robot.

18. The robot of claim 16, wherein, The robot comprises a humanoid robot.

19. The robot of claim 16, wherein, The robot comprises one of a legged robot and a wheeled robot.

20. The robot of claim 16, wherein, The robot comprises a wheeled-legged robot.

21. The robot of claim 16, wherein, The robot comprises one of a quadruped robot, a biped robot, and a four-wheeled-legged robot.

22. A computer-readable storage medium, characterized in that, The computer readable storage medium has computer program code instructions stored thereon, the computer program code instructions being invoked by the processor of the robot to cause the robot to perform the method in any one of claims 1-12.

Images