Robot motion control strategy network training method and device based on imitation learning

Abstract

The present disclosure provides a robot motion control strategy network training method and device based on imitation learning, and relates to the technical field of sensors and robots. The method comprises: performing mirror processing on part of a plurality of original samples to obtain a plurality of mirror samples; mixing the plurality of mirror samples and the original samples that have not been mirror processed to form a mixed sample set for training a discriminator, so as to force the discriminator to learn symmetrical data distribution features; inputting the mixed sample set and a generated sample into the discriminator to obtain a reward signal; and feeding back the reward signal to a motion control strategy network, so that the subsequent generated actions of the motion control strategy network are closer to the features of the mixed sample set; wherein the original sample is human motion capture data, and the generated sample is a sample obtained by outputting a predicted action according to the current observation state by the motion control strategy network. The present disclosure can effectively accelerate the convergence speed of the strategy network and improve the training efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of sensors and robotics, and relates to a robot motion control strategy network training method and device based on imitation learning. BACKGROUND

[0002] Robot imitation learning technology enables a robot to master complex motor skills by observing and imitating demonstration data of an expert (such as a human demonstrator or an expert strategy). However, in the actual data collection process, due to the inherent physiological habits of the demonstrator, the calibration residual of the motion capture device, or the precision limitation of the motion reorientation algorithm, the original training samples may exhibit significant asymmetry in the motion space. Further, the control strategy learned by the robot produces systematic motion asymmetry.

[0003] To solve the above-mentioned asymmetry problem, one of the traditional solutions is data augmentation, that is, offline flipping expansion of the original data set before training or random flipping of samples during training to increase the amount of data; however, this will exponentially increase the volume of the training data set, resulting in a significant increase in memory occupation and a significant prolongation of the training time of each round; and in high-dimensional continuous control tasks, simply increasing the number of samples may cause the training to fail to converge. Therefore, the mirror loss (Mirror Loss) method is currently more commonly used, that is, a penalty term is added to the loss function to forcibly constrain the policy network to output mirror actions for mirror states.

[0004] However, the inventors have found in practice that the mirror loss method needs to construct auxiliary mirror state pairs during the forward propagation process of the network and perform complex Euclidean distance calculations to solve the penalty term, which not only increases the computational power overhead of a single step of training, but when the original sample distribution itself is severely skewed, relying solely on the penalty term to forcibly correct will cause the optimization goal of the policy network to conflict, making it difficult to eliminate the deviation from the source of the data distribution, and also causing problems such as slow model convergence and low training efficiency. SUMMARY

[0005] The present disclosure provides a robot motion control strategy network training method and device based on imitation learning, which can effectively accelerate the convergence speed of the policy network and improve the training efficiency.

[0006] Additional aspects and advantages of the present disclosure will be set forth in part in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a robot motion control strategy network training method based on imitation learning is provided, comprising:

[0008] mirror processing part of the plurality of original samples to obtain a plurality of mirror samples;

[0009] The multiple mirrored samples and the remaining original samples that have not been mirrored are mixed to form a mixed sample set for training the discriminator, so as to force the discriminator to learn symmetric data distribution features;

[0010] The mixed sample set and the generated sample are input into the discriminator to obtain a reward signal;

[0011] The reward signal is fed back to the motion control policy network so that the subsequent actions generated by the motion control policy network are closer to the features of the mixed sample set.

[0012] The original samples are human motion capture data, and the generated samples are samples obtained by the motion control strategy network based on the current observation state and the predicted actions output.

[0013] In one exemplary embodiment of this disclosure, inputting the mixed sample set and the generated sample into the discriminator to obtain a reward signal includes:

[0014] Obtain the generation discrimination probability output by the discriminator for the generated sample; the generation discrimination probability characterizes the degree to which the discriminator determines that the generated sample conforms to the features of the mixed sample set;

[0015] The generation discrimination probability is converted into the reward signal using a preset reward mapping function; wherein, the higher the generation discrimination probability, the larger the value of the converted reward signal.

[0016] In one exemplary embodiment of this disclosure, converting the generation discrimination probability into the reward signal using a preset reward mapping function includes:

[0017] calculate The value of is used as the reward signal; where log represents the logarithmic operation and D represents the generation discrimination probability.

[0018] In one exemplary embodiment of this disclosure, the step of converting the generation discrimination probability into the reward signal using a preset reward mapping function further includes:

[0019] In response to the value of the reward signal exceeding a preset reward truncation threshold, the value of the reward signal is corrected to the preset reward truncation threshold.

[0020] In one exemplary embodiment of this disclosure, the method further includes:

[0021] Obtain the true discrimination probability output by the discriminator for the mixed sample set;

[0022] construct an adversarial loss function for measuring deviation of the real discrimination probability and the generated discrimination probability from a target value;

[0023] update parameters of the discriminator based on the adversarial loss function.

[0024] In an exemplary embodiment of the present disclosure, the adversarial loss function is:

[0025]

[0026] wherein, denotes an adversarial loss, denotes a mathematical expectation, denotes the real discrimination probability, denotes the generated discrimination probability.

[0027] In an exemplary embodiment of the present disclosure, the constructing an adversarial loss function further includes:

[0028] interpolating and sampling between the mixed sample set and the generated sample to obtain an intermediate sample;

[0029] calculating a gradient norm of the discriminator with respect to the intermediate sample;

[0030] calculating a square of a difference between the gradient norm and a preset target constant to obtain a gradient penalty term, and adding the gradient penalty term to the adversarial loss function.

[0031] In an exemplary embodiment of the present disclosure, the feeding back the reward signal to the motion control strategy network so that the motion control strategy network generates subsequent actions closer to features of the mixed sample set includes:

[0032] feeding back the reward signal to the robot motion control strategy network;

[0033] calculating a parameter gradient direction that maximizes an expected value of the reward signal;

[0034] adjusting a weight parameter of the robot motion control strategy network along the parameter gradient direction so that the motion control strategy network generates subsequent actions closer to features of the mixed sample set.

[0035] In an exemplary embodiment of the present disclosure, the mirroring processing on part of the plurality of original samples includes:

[0036] generating a random number for each of the original samples in a current training batch;

[0037] comparing the random number with a preset threshold to obtain a comparison result, and taking the comparison result as a mirror selection marker of each of the original samples;

[0038] extracting a mirror sample subset from the training batch based on the mirror selection marker;

[0039] generating mirror state data after performing the mirror processing on each of the original samples in the mirror sample subset and writing back the mirror sample subset.

[0040] In an exemplary embodiment of the present disclosure, the method further comprises:

[0041] obtaining a set of mirror discrimination scores output by the discriminator on the mirror sample subset, and a set of original discrimination scores output by the discriminator on the part of the original samples that is not subjected to mirror processing;

[0042] calculating a first statistical value based on the set of mirror discrimination scores, calculating a second statistical value based on the set of original discrimination scores, and calculating a bias indicator based on the first statistical value and the second statistical value;

[0043] comparing the bias indicator with a preset bias threshold, and updating the preset threshold of the next training batch according to the comparison result.

[0044] In an exemplary embodiment of the present disclosure, the comparison of the random number with the preset threshold to obtain a comparison result comprises: the random number has a value range of 0-1, and the preset threshold is 0.5.

[0045] In an exemplary embodiment of the present disclosure, the mirror processing on part of the plurality of original samples comprises:

[0046] the mirror processing is performed on 35-65% of the plurality of original samples; or the mirror processing is performed on 45-55% of the plurality of original samples; or the mirror processing is performed on 49-51% of the plurality of original samples.

[0047] In an exemplary embodiment of the present disclosure, each of the original samples comprises state data and action data corresponding to the state data; the state data comprises joint state information and linear velocity and angular velocity; and the mirror processing on part of the plurality of original samples comprises:

[0048] the state mirror processing comprises: exchanging left and right limb joint state information, and performing mirror flipping on the linear velocity and the angular velocity;

[0049] Performing action mirroring processing on action data corresponding to the state data after the state mirroring processing; the action mirroring processing comprises: rearranging the element positions of the action vector according to a preset symmetric index relationship.

[0050] In an example embodiment of the present disclosure, the mirroring flipping of the linear velocity and the angular velocity comprises:

[0051] Obtaining the coordinate axis components of the linear velocity in the robot body coordinate system, and taking the negative value of the Y coordinate axis component;

[0052] Obtaining the roll angular velocity component and the yaw angular velocity component of the angular velocity in the robot body coordinate system, and taking the negative value of the roll angular velocity component and the yaw angular velocity component respectively.

[0053] In an example embodiment of the present disclosure, the mirroring flipping of the linear velocity and the angular velocity information comprises:

[0054] Obtaining the attitude data corresponding to the state data to determine the coordinate transformation matrix;

[0055] Transforming the linear velocity and the angular velocity to the body coordinate system by using the coordinate transformation matrix to obtain the body system velocity vector;

[0056] Calling a preset mirroring flipping matrix and performing matrix multiplication with the body system velocity vector to obtain a mirrored body system velocity vector;

[0057] Transforming the mirrored body system velocity vector back to the original coordinate system by using the inverse matrix of the coordinate transformation matrix to obtain the mirrored flipped linear velocity and the mirrored flipped angular velocity.

[0058] In an example embodiment of the present disclosure, the exchanging of the left and right limb joint state information comprises:

[0059] Establishing a symmetric mapping table of robot joints, the symmetric mapping table containing a one-to-one mapping relationship between the index of each joint of the left limb and the index of the corresponding joint of the right limb;

[0060] Traversing the joint state vector in the state data;

[0061] According to the symmetric mapping table, exchanging the joint state vector of the left limb joint and the joint state vector of the corresponding right limb joint.

[0062] In an example embodiment of the present disclosure, the rearranging of the element positions of the action vector according to the preset symmetric index relationship comprises:

[0063] obtain an action dimension definition table consistent with the action vector dimension, and generate a left-right symmetric action dimension pair list based on the action dimension definition table;

[0064] generate the preset symmetric index relationship based on the left-right symmetric action dimension pair list;

[0065] perform index rearrangement on the action vector using the preset symmetric index relationship to obtain a rearranged action vector.

[0066] According to a second aspect of the present disclosure, a robot motion control method based on imitation learning is provided, comprising:

[0067] obtain a current observation state of a robot and input it into a pre-trained motion control policy network to output an action policy for controlling robot motion based on the motion control policy network;

[0068] The motion control policy network is obtained according to the above robot motion control policy network training method based on imitation learning.

[0069] According to a third aspect of the present disclosure, a robot motion control policy network training device based on imitation learning is provided, comprising:

[0070] a sample mirroring module for mirroring part of a plurality of original samples to obtain a plurality of mirrored samples;

[0071] a sample mixing module for mixing the plurality of mirrored samples and the original samples that have not been mirrored to form a mixed sample set for training a discriminator, so as to force the discriminator to learn symmetric data distribution features;

[0072] a reward signal generation module for inputting the mixed sample set and a generated sample into the discriminator to obtain a reward signal;

[0073] a policy network updating module for feeding back the reward signal to a motion control policy network, so that the action generated by the motion control policy network subsequently is closer to the features of the mixed sample set;

[0074] The original sample is human motion capture data, and the generated sample is a sample obtained by outputting a predicted action by the motion control policy network based on a current observation state.

[0075] According to a fourth aspect of the present disclosure, a robot motion control device based on imitation learning is provided, comprising:

[0076] a data acquisition module for acquiring current state environment data of a robot and inputting it into a pre-trained motion control policy network;

[0077] A control policy module is configured to output a motion control policy network to control the motion of the robot.

[0078] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0079] A processor and a memory, the memory storing computer readable instructions which, when executed by the processor, implement the method of the above embodiments.

[0080] According to a sixth aspect of the present disclosure, a robot is provided, comprising:

[0081] A processor and a memory, the memory storing computer readable instructions which, when executed by the processor, implement the method of the above embodiments.

[0082] In an exemplary embodiment of the present disclosure, the robot is any one of a legged robot, a quadruped robot, a biped robot, a wheeled robot, a wheeled-legged robot, a four-wheeled-legged robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot, and a robotic arm.

[0083] According to a seventh aspect of the present disclosure, a computer readable storage medium is provided, the computer readable storage medium storing computer program code instructions, when the computer program code instructions are invoked by a processor, causing the processor to execute the method of the above embodiments.

[0084] According to the above technical solutions, the present disclosure has at least one of the following advantages and positive effects:

[0085] The method for training a robot motion control strategy network based on imitation learning in the example embodiment of the present disclosure forces the discriminator to observe reference data with uniform spatial distribution in the training process by performing mirror flipping and rearrangement of the execution state and action of part of the samples, and inputting the samples after mirror processing and the original samples into the discriminator, thereby constructing a reward signal with symmetry, and further ensuring that the discriminator can give high reward feedback when the policy network explores the correct symmetric action, avoiding the reward distribution misplacement problem caused by the inherent bias of the discriminator (such as only identifying one-sided expert features), and fundamentally ensuring the effective acquisition of symmetric motion skills in robot imitation learning. At the same time, the example embodiment of the present disclosure realizes symmetry correction at the data source level, thereby eliminating the gradient conflict that may be caused by the inconsistency between geometric constraints and task objectives without the need to additionally construct mirror state pairs or calculate complex Euclidean distance penalty terms in the network backpropagation process. This not only reduces the computational power overhead of single-step training, but also enables the policy network to focus on maximizing the discriminator reward, thereby significantly accelerating the convergence speed of the policy network and improving the training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0087] Figure 1 A system architecture diagram to which the robot motion control method based on imitation learning in the embodiments of the present disclosure can be applied is shown.

[0088] Figure 2 A flowchart showing the robot motion control strategy network training method based on imitation learning in the embodiments of the present disclosure is shown.

[0089] Figure 3 A flowchart showing another robot motion control strategy network training method based on imitation learning in the embodiments of the present disclosure is shown.

[0090] Figure 4 A flowchart showing the step of inputting the mixed sample set and the generated sample into the discriminator to obtain the reward signal in the embodiments of the present disclosure is shown.

[0091] Figure 5 A flowchart showing the step of mirror processing part of the multiple original samples in the embodiments of the present disclosure is shown.

[0092] Figure 6A flowchart showing the mirroring processing steps for part of the plurality of original samples in the embodiments of the present disclosure is shown.

[0093] Figure 7 A flowchart showing the robot motion control method based on imitation learning in the embodiments of the present disclosure is shown.

[0094] Figure 8 A block diagram of a robot motion control strategy network training device based on imitation learning in the embodiments of the present disclosure is shown.

[0095] Figure 9 A block diagram of a robot motion control device based on imitation learning in the embodiments of the present disclosure is shown.

[0096] Figure 10 A schematic diagram of a type of robot to which the robot motion control method based on imitation learning in the embodiments of the present disclosure can be applied is shown.

[0097] Figure 11 A schematic diagram of another type of robot to which the robot motion control method based on imitation learning in the embodiments of the present disclosure can be applied is shown.

[0098] Figure 12 A structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present disclosure is shown.

[0099] Figure 13 A schematic diagram of a computer-readable storage medium in the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0100] In the present disclosure description, the terms "first" and "second" are used only for description, and do not indicate relative importance or imply the number of technical features. Therefore, the "first" and "second" features can explicitly or implicitly include at least one of the features. The meaning of "plurality" is at least two, unless otherwise explicitly limited.

[0101] First, the related terms involved in the example embodiments of the present disclosure are explained and described:

[0102] Robot: a programmable mechanism with certain autonomous ability to perform motion, operation or positioning. The autonomous ability refers to the ability to perform the intended task without human intervention based on the current state and perception information.

[0103] Original sample: refers to expert demonstration data collected from a human motion capture system, and each original sample includes state data and action data corresponding to the state data.

[0104] State data: refers to a set of information describing the physical state of the robot at a certain time, at least including joint state information and linear and angular velocities; wherein the joint state information includes at least one of the angle, angular velocity or torque of each joint.

[0105] Action data: refers to a control instruction vector corresponding to the state data, used to drive each actuator of the robot, the dimension of which is consistent with the degree of freedom of the robot, and each dimension corresponds to the output target of an actuator.

[0106] Motion control strategy network: refers to a neural network model used to generate an action strategy according to the current observed state, which is trained by imitation learning, and the goal is to make the output action distribution approximate the expert behavior characteristics.

[0107] Mirror processing: refers to the operation of performing left-right symmetric transformation on the original sample, including performing state mirror processing on the state data and performing action mirror processing on the corresponding action data to generate mirror samples with consistent semantics.

[0108] State mirror processing: refers to a transformation operation on the state data according to the left-right symmetry of the humanoid robot body structure, specifically including exchanging joint state information corresponding to left and right limbs, and performing mirror flipping on linear and angular velocities.

[0109] Action mirror processing: refers to rearranging the element positions in the action vector according to a predetermined symmetric index relationship, so that the mirrored action instruction matches the mirrored body configuration.

[0110] Linear velocity: refers to the translational velocity vector of the robot's center of mass or reference point in space.

[0111] Angular velocity: refers to the rate vector of the robot's rotation around the spatial coordinate axis.

[0112] Mirror sample: refers to a sample obtained after mirror processing; mirror sample subset refers to a set of original samples selected and completed mirror processing in the current training batch.

[0113] Mixed sample set: refers to a training data set composed of multiple mirror samples and the remaining original samples that have not been mirror processed, used to train the discriminator to learn the symmetric data distribution characteristics.

[0114] Generated sample: refers to the state-action pair obtained by executing the predicted action output by the motion control strategy network according to the current observed state in the simulation environment or actual system.

[0115] Discriminator: refers to a neural network model used to distinguish whether the input sample belongs to the mixed sample set (real) or is generated by the motion control strategy network (fake), and its output is the discrimination probability.

[0116] generated discriminative probability: refers to the discriminative score output by the discriminator for the generated sample, representing the degree to which the generated sample is judged to conform to the characteristics of the mixed sample set.

[0117] real discriminative probability: refers to the discriminative score output by the discriminator for the sample in the mixed sample set.

[0118] reward signal: refers to a scalar value obtained by converting the generated discriminative probability through a reward mapping function, used to feed back to the motion control policy network to guide its parameter update.

[0119] reward mapping function: refers to a function that maps the generated discriminative probability to the reward signal, designed so that the higher the generated discriminative probability, the larger the reward signal.

[0120] preset reward cutoff threshold: refers to an upper limit parameter for limiting the maximum value of the reward signal; when the calculated reward signal exceeds the threshold, it is corrected to the threshold to improve training stability.

[0121] adversarial loss function: refers to the objective function used to train the discriminator, used to measure the discriminative error of the discriminator for the mixed sample set and the generated sample.

[0122] intermediate sample: refers to a sample obtained by interpolating the sample in the mixed sample set and the generated sample, used to calculate the gradient penalty term.

[0123] gradient penalty term: refers to a regularization term constructed based on the square of the difference between the gradient norm output by the discriminator for the intermediate sample and a preset target constant, used to stabilize the adversarial training process.

[0124] first statistical value: refers to a statistical quantity calculated for the set of discriminative scores corresponding to the mirror image sample subset, which can be the mean, median or weighted average.

[0125] second statistical value: refers to a statistical quantity calculated for the set of discriminative scores corresponding to the original sample that has not been mirror processed.

[0126] bias indicator: refers to a difference measure calculated based on the first statistical value and the second statistical value, used to reflect whether the discriminator has a systematic preference or rejection for the mirror image sample.

[0127] preset threshold: refers to a comparison benchmark used to determine whether to perform mirror processing on a certain original sample; in the training process, a random number generated for each sample is compared with the threshold to determine the mirror selection marker.

[0128] symmetric mapping table: refers to a data structure containing a one-to-one mapping relationship between the indices of each joint of the left limb and the corresponding joint indices of the right limb, used to guide the exchange of joint state information.

[0129] Action dimension definition table: refers to a metadata table describing each dimension in the action vector and its corresponding physical actuator and position attribute (such as "left hip flexion" and "right knee abduction").

[0130] Left-right symmetric action dimension pair list: refers to a paired index list generated based on the action dimension definition table, each pair containing a left actuator dimension and a corresponding right actuator dimension.

[0131] Predefined symmetric index relationship: refers to the mapping rule derived from the left-right symmetric action dimension pair list, used to perform index rearrangement on the action vector.

[0132] Body coordinate system: refers to a local coordinate system established with the robot's torso as the reference, usually defining the X-axis pointing forward, the Y-axis pointing left, and the Z-axis vertically upward, used to uniformly describe the mirror image flipping rule of the velocity quantity.

[0133] Figure 1 A system architecture diagram showing the application of the robot motion control strategy network training method based on imitation learning in the embodiments of the present disclosure is shown. As shown in Figure 1 The system architecture 100 can include a terminal device 101, a robot 102, a network 103, and a server 104. The terminal device 101 includes but is not limited to desktop computers, portable computers, smartphones, and tablet computers, etc. The terminal device 101 is used to provide a human-computer interaction interface, support users to set training parameters (such as mirror sample ratio, training rounds, discriminator structure, etc.), monitor the training process or view the training results, at the same time, can issue task control instructions and receive motion state feedback from the server 104 or the robot 102, and cooperates with the server 104 to complete the visualization analysis and management of expert demonstration data.

[0134] The robot 102 is equipped with various body perception sensors, such as inertial measurement units (IMU), joint angle encoders, force / torque sensors, etc., for real-time acquisition of joint state, linear velocity, angular velocity, and contact force sensor data during the execution of motion tasks, to determine the current motion state; in addition, it can also be equipped with environmental perception sensors, such as cameras, laser radars or depth cameras, etc., for acquiring external environmental information, to assist in completing motion decision-making in complex scenarios.

[0135] The server 104 is deployed with a robot motion control strategy training module based on imitation learning, which is configured to: receive raw demonstration data collected by a human demonstrator through a motion capture system as raw samples; mirror process part of the raw samples to obtain a plurality of mirror samples; mix the plurality of mirror samples with the raw samples that are not mirror processed to form a mixed sample set for training the discriminator; input the mixed sample set and the generated sample into the discriminator to obtain a reward signal; and feed back the reward signal to the motion control strategy network, so that the subsequent generated actions of the motion control strategy network are closer to the features of the mixed sample set. After the training is completed, the server 104 can deploy the trained motion control strategy model to the robot 102 for real-time generation of motion control instructions with high symmetry and high robustness in actual operation.

[0136] The network 103 is a medium for providing a communication link between the terminal device 101, the robot 102 and the server 104, and can include various connection types, such as wired, wireless communication links or optical fiber cables, etc., to ensure stable and efficient data interaction between the terminal device 101, the robot 102 and the server 104, including expert demonstration data uploading, training intermediate result synchronization, model parameter distribution and robot state feedback, etc.

[0137] Through the cooperative operation of the components in the system architecture 100, an efficient symmetry correction training mechanism that does not require additional mirror loss terms and does not rely on offline data expansion can be achieved, so that the robot can eliminate motion asymmetry caused by human demonstration habits or motion capture errors from the source of data distribution in the imitation learning process, thereby improving the generalization ability and motion stability of the strategy, and effectively avoiding the risk of gait imbalance, falling, etc. caused by unilateral preference.

[0138] It should be understood that Figure 1 The number and type of terminal devices, robots, networks and servers in the system architecture 100 are only illustrative. According to actual application requirements, the system architecture 100 can include any number and any type of terminal devices, robots, networks and servers.

[0139] The example embodiments of the present disclosure provide a robot motion control strategy network training method based on imitation learning, which can include the following steps S201 to S204, as shown in Figure 2

[0140] Step S201, mirror processing part of the plurality of raw samples to obtain a plurality of mirror samples;

[0141] Step S202, mix the plurality of mirror samples and the raw samples that are not mirror processed to form a mixed sample set for training the discriminator, so as to force the discriminator to learn symmetric data distribution features;​

[0142] Step S203, inputting the mixed sample set and the generated sample into the discriminator to obtain a reward signal;

[0143] Step S204, feeding back the reward signal to the motion control strategy network, so that the motion control strategy network generates subsequent actions closer to the characteristics of the mixed sample set; wherein the original sample is human motion capture data, and the generated sample is a sample obtained by the motion control strategy network outputting a predicted action according to the current observation state.

[0144] The method for training the motion control strategy network of the robot based on imitation learning provided by the example embodiments of the present disclosure performs mirror flipping and rearrangement of the states and actions of part of the samples, and inputs the samples after mirror processing and the original samples into the discriminator after mixing, so that the discriminator observes uniformly distributed reference data in the training process, thereby constructing a symmetric reward signal, and further ensuring that the discriminator can give high reward feedback when the strategy network explores the correct symmetric action, avoiding the reward distribution misalignment problem caused by the inherent bias of the discriminator (such as only identifying one-sided expert characteristics), and fundamentally ensuring the effective learning of symmetric motion skills in robot imitation learning. At the same time, the example embodiments of the present disclosure achieve symmetry correction at the data source level, so that there is no need to additionally construct mirror state pairs or calculate complex Euclidean distance penalty terms in the network backpropagation process, eliminating the gradient conflict that may be caused by the inconsistency between geometric constraints and task targets. This not only reduces the computational power overhead of single-step training, but also enables the strategy network to focus on maximizing the discriminator reward, thereby significantly accelerating the convergence speed of the strategy network and improving the training efficiency.

[0145] Moreover, due to the systematic motion deviation problem of the original data caused by the performer's physiological habits (such as right-limb tendency), calibration errors of the motion capture device, and bone reset calculation errors, by implementing online real-time spatial mirror processing (including state mirroring and action mirroring) on part of the samples in each batch of strategy training, the motion control strategy network is forced to equally process the left and right observation inputs and action outputs. This corrects the systematic preference of the strategy network for a particular direction from the data source, significantly improves the physical symmetry and dynamic balance ability of the robot in the actual motion process, and effectively avoids the side slip or turning deviation caused by asymmetric data. In addition, compared with the traditional mirror loss method, the method for training the motion control strategy network of the robot based on imitation learning provided by the example embodiments of the present disclosure does not need to artificially construct symmetric action pairs, nor does it need to perform additional action prediction calculations under mirror observation, so the data utilization rate is improved by about 100%, while ensuring symmetry, significantly accelerating the training convergence speed and reducing the computational overhead.

[0146] Below, the robot motion control strategy network training method based on imitation learning in the present example embodiment will be described in detail.

[0147] In step S201, a part of the plurality of original samples is mirror-processed to obtain a plurality of mirror samples.

[0148] In some embodiments, the original sample can include state observation information and corresponding action instruction of the current moment, and the state observation information can at least contain the angles, angular velocities, linear velocities and trunk postures of each joint of the left and right limbs and other proprioceptive quantities; the action instruction can be a control signal such as joint torque or target position corresponding to the state observation information.

[0149] In the embodiments of the present disclosure, the mirror processing can include two sub-steps of state mirror processing and action mirror processing.

[0150] In the state mirror processing, left-right symmetric rearrangement can be performed on the state observation information in the original sample: the state variables of the left joints (such as left hip, left knee, left ankle, etc.) are exchanged with the state variables of the corresponding right joints (such as right hip, right knee, right ankle, etc.); at the same time, the sign of the direction-related vector (such as linear velocity, angular velocity, etc.) is flipped, for example, the velocity component along the Y axis (assuming the left-right direction) is taken as negative to maintain physical consistency. Through the above operation, the mirrored state is semantically equivalent to the original state about the symmetry of the sagittal plane of the body.

[0151] In the action mirror processing, index rearrangement matching the state mirror can be performed on the corresponding action instruction in the original sample: according to the preset joint symmetry mapping relationship (for example, the i-th dimension of the action vector corresponds to a joint of the left leg, and the j-th dimension of the mirrored position corresponds to the corresponding joint of the right leg), the elements in the action vector are rearranged according to the symmetry rule, and the sign of the action component related to the direction (such as the transverse component of the joint torque) is adjusted to ensure that the mirrored action is physically executable and compatible with the mirrored state.

[0152] In some embodiments, the mirror processing of a part of the plurality of original samples includes: mirror processing of 35-65% of the plurality of original samples; or mirror processing of 45-55% of the plurality of original samples; or mirror processing of approximately 50% of the plurality of original samples.

[0153] Among them, approximately 50%, that is, within the preset range of 50%, the preset range includes 50%. For example, approximately 50% can be 49%-52%, or 49.9%-50.1%, or 50%, etc., without limitation.

[0154] It can be understood that the proportion range of 35-65% can effectively introduce symmetry prior while retaining sufficient original behavior characteristics, avoiding insufficient learning of symmetric actions due to too few mirror samples, or diluting the dominance of original expert behavior due to too many mirror samples. While the narrower proportion interval of 45-55% can be applied to walking or running tasks with higher symmetry requirements, it can statistically approximate the balance of left and right action distribution, thereby guiding the policy network to output unbiased biped control instructions.

[0155] In some embodiments, the mirror processing can be performed on part of the original sample set, where the part of the original sample set can be randomly selected according to a preset proportion. For example, in each training batch, 50% of the N original samples included in the current batch are randomly selected for mirror processing, and the remaining 50% remain unchanged. The proportion can be adjusted according to the task complexity or data skew, for example, the value range is 30% to 70%.

[0156] In some embodiments, mirror processing can be performed online during training, i.e., without pre-generating and storing mirror samples, but determining whether to trigger mirror operation in real time after reading original samples each time. This approach avoids the storage overhead caused by offline data expansion, while ensuring the dynamic diversity of training data.

[0157] Through the above mirror processing, the original sample and its mirror sample together form a set of symmetric data pairs in the motion semantics, thereby enhancing the symmetry of the training set from the source of data distribution without introducing additional loss terms or artificial annotations, and laying a foundation for subsequent discriminator learning of unbiased symmetric reward signals.

[0158] In step S202, the plurality of mirror samples and the remaining original samples not subjected to mirror processing are mixed to form a mixed sample set for training the discriminator, so as to force the discriminator to learn symmetric data distribution characteristics.

[0159] In some embodiments, each sample in the mixed sample set can include state observation information and corresponding action instructions, and the structure remains consistent with the original demonstration data, and the state and action of part of the samples have been mirror transformed according to the preset symmetry rule.

[0160] In some specific implementations, the mixing operation can be dynamically performed within each training batch. For example, in a batch including N original samples, first, a part of the samples are randomly selected according to a preset proportion (such as 50%) for mirror processing to generate corresponding mirror samples; then, the part of the mirror samples and the remaining original samples not subjected to processing are combined to form a mixed sample set with a size of N. This mixing strategy ensures that the discriminator receives left-right symmetric motion patterns during each forward propagation, thereby statistically approximating the ideal symmetric data distribution.

[0161] In some embodiments, the mixing ratio can be adaptively adjusted. For example, when it is detected that there is a significant left-right asymmetry in the motion control strategy network output (e.g., the left leg swing amplitude is consistently greater than the right leg), the proportion of mirror samples can be temporarily increased (e.g., to 60% or 70%) to enhance the correction strength for the skewed distribution; conversely, after the strategy tends to be symmetric, it can be gradually restored to the balanced ratio to maintain the stability of the training.

[0162] It should be understood that by providing the discriminator with semantically mirrored positive samples (i.e., the mixed sample set), it cannot rely on the specific motion preference of a single limb to distinguish between expert data and generated data, and must learn a discriminant that is invariant to left-right symmetric structure, which can force the discriminator to learn the symmetric data distribution characteristics. In other words, if a motion sequence and its mirrored version are both judged to be real, it means that the discriminator has internalized the symmetry prior, and thus can provide an unbiased reward signal for the policy network.

[0163] In some embodiments, the mixed sample set can be used as the positive sample input for the discriminator, while the negative sample is composed of the data pairs of the current generated actions and states by the policy network. The discriminator outputs a discrimination probability by comparing the distribution difference between the positive and negative samples, and calculates the reward value accordingly. Since the positive samples themselves already have symmetry, even if the policy network initially tends to unilateral action, the mirrored action generated by it can also obtain a similar high reward, which can guide the policy to gradually converge to the symmetric solution.

[0164] In other embodiments, the mixed sample set can also be used in a multi-task training scenario. For example, in the same training stage, some tasks focus on gait symmetry (e.g., walking, running, etc.), while others allow asymmetric behavior (e.g., kicking a ball, single-leg support, etc.). At this time, different mixing strategies can be configured for different tasks, and the mixing mechanism of this step can be enabled only in tasks that require symmetry, in order to balance task specificity and universality.

[0165] In addition, the mixing operation can be completed in the data loading stage, the pre-processing stage before training, or in real time inside the training loop, and the specific implementation form can be flexibly selected according to the computing resources and real-time requirements, which is not limited.

[0166] In step S203, the mixed sample set and the generated sample are input into the discriminator, so as to obtain the reward signal.

[0167] Among them, the mixed sample set can be used as the positive sample for the discriminator to distinguish real behavior, while the generated sample can be used as the negative sample for the discriminator to distinguish generated behavior.

[0168] In some embodiments, the mixed sample set and the generated samples can both be organized in the form of (state, action) data pairs and have the same dimensional structure to ensure that the discriminator can make effective comparisons.

[0169] In some implementations, the discriminator can be a trainable neural network model, taking a single (state, action) sample as input and outputting the probability value of that sample originating from expert teaching data. In each training iteration, multiple samples from the mixed sample set can be concatenated with or alternately fed into the discriminator along with an equal number of generated samples for forward propagation. Based on the discriminator's output probabilities for the two classes of samples, a discriminative loss (such as binary cross-entropy loss) is calculated, and the discriminator parameters are updated in reverse based on this loss, gradually improving the discriminator's distinguishing ability. Simultaneously, the discriminative output corresponding to the generated samples is used as a reward signal, fed back to the motion control policy network to guide its policy optimization.

[0170] In some embodiments, the reward signal can be directly taken from the output probability value of the discriminator on the generated sample. For example, if the discriminator determines that the probability of a generated sample “looking like expert behavior” is 0.85, then this value is used as the immediate reward at that time step; the policy network can maximize the cumulative expectation of this reward through reinforcement learning algorithms (such as PPO, SAC, etc.), thereby driving its output action to continuously approach the symmetrical motion pattern reflected by the mixed sample set.

[0171] In other embodiments, to improve training stability, the original discrimination output can be transformed before being used as the reward signal. For example, the logarithmic transformation of the following formula can be used:

[0172]

[0173] in, For the discriminator to compare state and action data pairs The output probability, This serves as the final reward signal. This form is widely used in frameworks such as generative adversarial imitation learning, and helps alleviate the vanishing gradient problem caused by overconfidence in the early training stages.

[0174] It should be understood that since the mixed sample set itself contains semantically symmetrical mirror and non-mirror samples, the discriminator is forced to learn a discrimination criterion that is invariant to left-right limb switching during training. Therefore, when the policy network generates a reasonable but left-biased action, its corresponding right-side mirror action can also obtain a similarly high reward, thus avoiding the reward bias problem caused by the natural bias of expert data (such as right-handedness) in traditional methods.

[0175] In some embodiments, the construction process of the generated sample can include: receiving a current environment observation (such as joint angle, angular velocity, torso posture, etc.) through the motion control policy network, outputting a predicted action; the action is combined with the current state to form a generated sample, and the next state is obtained after the generated sample is executed in the simulation environment, forming a complete trajectory segment. Each (state, action) data pair in the trajectory segment can be input into the discriminator as an independent generated sample.

[0176] In other embodiments, to improve the efficiency of the sample, a playback buffer mechanism can be used to store historical generated samples, and in subsequent training, the historical generated samples are jointly sampled with the current mixed sample set to input the discriminator, so as to enhance the data diversity of the discriminant training.

[0177] It can be understood that the structure of the above discriminator can adopt a multilayer perceptron (MLP), a graph neural network (GNN), or a time series model (such as LSTM (Long Short-Term Memory) or Transformer), and the specific selection can depend on the complexity of the state-action space and the task characteristics.

[0178] In addition, the calculation method of the reward signal can also be extended to advanced mechanisms based on integrated discrimination, attention weighting, or multi-scale discrimination, and there is no limitation.

[0179] In step S204, the reward signal is fed back to the motion control policy network, so that the actions generated by the motion control policy network subsequently are closer to the characteristics of the mixed sample set; wherein the original sample is human motion capture data, and the generated sample is a sample obtained by the motion control policy network outputting a predicted action according to the current observation state.

[0180] In some embodiments, the motion control policy network can not directly fit the expert action, but implicitly learn the distribution characteristics of the expert behavior by maximizing the reward signal provided by the discriminator.

[0181] In some embodiments, the reward signal can be derived from the discrimination result of the discriminator on the generated sample. Specifically, when the motion control policy network generates an action under a state observation, and forms a generated sample by combining the action with the state, the generated sample is input into the discriminator; the discriminator can judge whether the generated sample is like expert behavior based on the learning of the mixed sample set (containing original human motion capture data and its mirror version) on the generated sample, and output a confidence score. The confidence score is fed back to the motion control policy network as an immediate reward to guide the parameter update of the motion control policy network.

[0182] The closeness of the generated actions to the mixed sample set is positively correlated with the reward. That is, the closer the generated actions are to the symmetric and natural movement patterns embodied in the mixed sample set, the higher the reward obtained; otherwise, the reward is lower.

[0183] In some embodiments, the reward signal can be fed back to the motion control policy network to make the subsequent actions generated by the motion control policy network closer to the characteristics of the mixed sample set, including: feeding the reward signal to the robot motion control policy network; calculating the parameter gradient direction that maximizes the expected value of the reward signal; and adjusting the weight parameters of the robot motion control policy network along the parameter gradient direction to make the subsequent actions generated by the motion control policy network closer to the characteristics of the mixed sample set.

[0184] Specifically, after the reward signal is fed back to the robot motion control policy network, the motion control policy network can generate a series of actions based on the current parameters and execute the series of actions in a simulated or real environment to obtain the corresponding state transition sequence and the reward signal at each step. Reinforcement learning algorithms such as proximal policy optimization (PPO), soft actor-critic (SAC), etc. can be used to process these interaction data, estimate the influence of policy parameters on the expected value of the reward, and calculate the parameter gradient direction that can improve the future reward sum. This gradient direction reflects how to adjust the network weights to make the policy more inclined to generate high-reward actions, i.e., closer to the expert behavior characteristics embodied in the mixed sample set.

[0185] After obtaining the gradient direction, the optimizer can update the weight parameters of the motion control policy network along this direction. It can be understood that as the training iteration proceeds, the policy network gradually adjusts its internal representation so that the actions output by the policy network under the same or similar observation states increasingly conform to the symmetric movement distribution in the mixed sample set. Since the mixed sample set itself integrates the original human motion capture data and its mirror version, the policy network can naturally learn the left-right balanced movement skills under this guidance, avoiding the asymmetric control problem caused by the unilateral preference of the original data.

[0186] In some embodiments, to improve training stability, the gradient can be clipped, normalized, or a trust region constraint can be introduced to prevent performance degradation caused by excessive parameter update amplitude. In other embodiments, an experience replay mechanism can also be combined to reuse historical trajectory data to improve sample efficiency, or a multi-step return estimate can be used to more accurately evaluate the long-term value of actions.

[0187] It should be understood that the above method of updating the policy based on the expected maximization of the reward signal realizes end-to-end policy training without explicit action labels by converting the adversarial discrimination result into a differentiable optimization target, while ensuring that the learned skills have good symmetry and generalization ability.

[0188] In some embodiments, to improve training stability, the original discriminative output can be smoothed or the historical average value can be used as the final reward, so as to avoid the situation that the policy network receives a signal with sharp fluctuations due to insufficient discriminative ability of the discriminator in the early stage.

[0189] In addition, the policy gradient type algorithm (such as proximal policy optimization, soft actor-critic, etc.) can be combined to optimize the motion control policy network end to end, so that the motion control policy network gradually improves the cumulative reward in long-term interaction, thereby stably approaching the overall statistical characteristics of the expert demonstration data.

[0190] It should be understood that, since the mixed sample set contains both the original human demonstration data and the semantic mirror version thereof, the overall presents a left-right symmetric data distribution. Therefore, when the motion control policy network attempts to generate an action biased to one side, the corresponding symmetric action can also obtain a similar high reward, and the policy will be effectively motivated to learn the truly symmetric motion skill. This mechanism enables the policy network to internalize the symmetry prior from the reward signal itself without relying on an additional symmetry constraint loss term, and fundamentally avoids the policy bias caused by the unilateral preference (such as right-handed leg habit) of the original motion capture data.

[0191] In some embodiments, the construction process of the generated sample can include: in the simulation environment, the motion control policy network receives the current robot state as input and outputs a predicted action; after the predicted action is applied to the simulation model, the execution result is recorded, and the data pair of (state, action) is stored in the experience pool as a generated sample. The generated sample can be used to calculate the reward and participate in policy update. This process can be performed in online interaction or offline playback mode, depending on the adopted reinforcement learning framework.

[0192] In other embodiments, to improve sample utilization efficiency, multiple time steps of generated samples can be grouped into a trajectory segment, and the cumulative reward is calculated in units of the entire trajectory, so as to better capture the temporal dependence between actions.

[0193] In addition, an exploration mechanism (such as action noise injection or entropy maximization) can be introduced to encourage the policy to try diverse actions in the early stage of network training, so as to prevent premature convergence to a suboptimal solution.

[0194] In some embodiments, as Figure 3As shown, a plurality of original samples 301 are obtained, which can be derived from a human motion capture system, each of which can include state data and action data corresponding to the state data; step 302 is performed on part of the original samples 301, i.e., mirror processing is performed on part of the original samples to obtain mirror samples; the mirror samples are mixed with the original samples in the plurality of original samples 301 which have not been subjected to mirror processing to form a mixed sample set 303; in the training process, the motion control strategy network 304 outputs a predicted action according to the current observation state to form a generated sample 305; the mixed sample set 303 and the generated sample 305 are jointly input into the discriminator 306, the authenticity of the generated sample is judged by the discriminator 306, and the corresponding discrimination probability is output, the reward signal 307 is calculated based on the discrimination probability, and the reward signal 308 is fed back to the motion control strategy network 304 for updating the network parameters of the motion control strategy network 304, so that the action strategy generated by the motion control strategy network 304 gradually approaches the expert behavior, thereby completing the training of the robot motion control strategy.

[0195] In some example embodiments, with reference to Figure 4 As shown, the mixed sample set and the generated sample are input into the discriminator to obtain a reward signal, including the following steps S401 to S402:

[0196] Step S401, obtaining a generated discrimination probability output by the discriminator for the generated sample; the generated discrimination probability represents the degree to which the discriminator determines that the generated sample meets the characteristics of the mixed sample set.

[0197] It can be understood that the higher the value of the generated discrimination probability (for example, between 0 and 1 or between 0 and 100), the more the discriminator considers that the generated sample has motion characteristics and symmetry structure consistent with the expert demonstration data, i.e., the more it "looks like real human demonstration behavior"; on the contrary, the lower the value, the farther the generated sample deviates from the expert behavior pattern, which can be manifested as unnatural, asymmetric or physically infeasible action. The generated discrimination probability is the core basis for constructing the reward signal, and can directly reflect the quality and degree of simulation of the action output by the current motion control strategy network.

[0198] In some embodiments, the generated sample is input into a discriminator model that has been trained or is being jointly trained, and the authenticity of the generated sample is evaluated by the discriminator to obtain a generated discrimination probability output by the generated sample.

[0199] Specifically, the discriminator can determine whether the generated sample is close to the real expert behavior in terms of statistical characteristics, motion style, and symmetry, etc. based on its learning experience on the mixed sample set (i.e. the expert data set containing the original human motion capture data and its mirror sample), and output a discrimination probability. The higher the probability value of the discrimination probability is, the more the discriminator tends to think that the generated sample is derived from the expert demonstration data.

[0200] In step S402, the generated discrimination probability is converted into a reward signal by using a preset reward mapping function. The higher the generated discrimination probability is, the larger the value of the converted reward signal is.

[0201] The reward mapping function is used to convert the probability value output by the discriminator into a scalar reward suitable for policy optimization. The design of the mapping function can satisfy the monotonic increasing characteristic, i.e. the stronger the reward signal obtained is when the generated sample is closer to the distribution characteristics of the mixed sample set, thereby effectively guiding the policy network to approach the expert behavior. The mapping mode can include directly using the discrimination probability itself as the reward, or using a logarithmic transformation form to enhance the gradient signal in the early stage of training.

[0202] In some embodiments, converting the generated discrimination probability into a reward signal by using a preset reward mapping function includes: calculating the value of log(D) as the reward signal; where log represents a logarithmic operation, and D represents the generated discrimination probability.

[0203] It can be understood that, by using log(D) as the reward mapping function, the discrimination result of the discriminator can be converted into a reinforcement learning reward signal with good gradient characteristics. When the quality of the generated sample is low and the value of D is small, approaches 1, tends to 0, and the reward obtained by the policy network is low; when the generated sample gradually approaches the expert behavior and the value of D increases, rapidly decreases, significantly rises, thereby giving the policy network stronger positive incentives. Such a nonlinear mapping relationship can provide sufficient gradient for the policy that has not yet converged to continuously improve the action output. In some embodiments, converting the generated discrimination probability into a reward signal by using a preset reward mapping function further includes: in response to the value of the reward signal exceeding a preset reward cutoff threshold, correcting the value of the reward signal to the preset reward cutoff threshold.

[0204]

[0205] ​​wherein the preset reward clipping threshold is a pre-set positive number representing the maximum value allowed for the single-step reward. When the calculated value of the reward signal exceeds the preset reward clipping threshold, the value of the reward signal can be automatically clipped (i.e., corrected) to the preset reward clipping threshold itself; if not, the value of the reward signal is retained. Through the clipping operation, the excessive influence of extreme high rewards on policy updating can be effectively suppressed, making the training process more smooth and controllable.

[0206] In some embodiments, the preset reward clipping threshold can be dynamically adjusted according to the task complexity and the training stage. For example, in the early stage of training, the policy has not yet stabilized, and a higher clipping threshold can be set to retain sufficient exploration incentive; while in the later stage of training, to improve the action accuracy and robustness, the threshold can be appropriately reduced to avoid the strategy sacrificing overall motion coordination for pursuing individual ultra-high rewards.

[0207] In addition, the preset reward clipping threshold can also be offline optimized through the performance of the policy on the validation set to balance the training efficiency and the final performance.

[0208] It can be understood that soft clipping, exponential decay or other smoothing clipping strategies can also be used to adjust the value of the reward signal, which is not limited herein.

[0209] In some embodiments, to avoid unstable training, the discriminant probability can be smoothed before mapping, for example, by using a sliding average or a clipping threshold to limit the influence of extreme values.

[0210] In other embodiments, the reward mapping function can be dynamically adjusted according to the training stage. For example, in the early stage of training, to encourage exploration, a loose mapping relationship can be used so that moderate discriminant probability can also obtain an observable reward; while in the later stage of training, to improve the accuracy, a steeper mapping curve can be used, only high confidence generated samples are given significant rewards, thus promoting the fine convergence of the policy.

[0211] It should be understood that since the mixed sample set itself has a left-right symmetric data distribution, the "real" standard learned by the discriminator naturally contains a symmetry constraint. Therefore, when the motion control policy network generates a reasonable but left-biased action, if its right mirror version also obtains a high discriminant probability, both will obtain a high reward after mapping, thus prompting the policy network to actively learn a symmetric motion strategy, rather than simply copying the unilateral habit in the original motion capture data.

[0212] In some embodiments, the reward signal described above can be used in policy gradient calculation in reinforcement learning algorithm as the basis for policy network parameter update. The motion control policy network can gradually adjust its internal parameters by constantly trying new actions and receiving feedback rewards from the discriminator, so that the action sequence output by the policy network approximates the expert behavior characteristics embodied in the mixed sample set in the overall distribution.

[0213] It can be understood that the specific form of the reward mapping function can be flexibly designed according to the task complexity, the training stability requirement, and the reinforcement learning framework adopted, and is not limited herein. For example, a temperature coefficient can be introduced to adjust the reward sensitivity, or an adaptive mapping mechanism can be constructed in combination with historical discrimination results.

[0214] Further, in some embodiments, the method further comprises: obtaining a real discrimination probability output by the discriminator for the mixed sample set; constructing an adversarial loss function, the adversarial loss function being used to measure the deviation of the real discrimination probability and the generated discrimination probability from a target value; and updating the parameters of the discriminator based on the adversarial loss function.

[0215] The mixed sample set can be input to the discriminator, and the discriminator outputs a discrimination result, i.e., a real discrimination probability, for each sample in the mixed sample set. The value of the real discrimination probability reflects the confidence of the discriminator that the sample is derived from the real expert behavior. By monitoring the real discrimination probability, the recognition ability of the discriminator for the expert data can be evaluated, and a supervision signal can be provided for subsequent adversarial training.

[0216] It can be understood that the adversarial loss function is used to guide the discriminator to accurately distinguish between real samples and generated samples. Specifically, for real samples in the mixed sample set, it is expected that their discrimination probability is close to a preset high target value (for example, 1); for the generated samples generated by the policy network, it is expected that their discrimination probability is close to a low target value (for example, 0).

[0217] In some embodiments, the adversarial loss function can be the following formula:

[0218]

[0219] wherein, represents the adversarial loss, represents the mathematical expectation, represents the real discrimination probability, represents the generated discrimination probability.

[0220] In some embodiments, the numerical value of the adversarial loss function can be calculated in each training iteration, and the gradient of the loss with respect to each parameter of the discriminator can be calculated by a backpropagation algorithm, and then the parameters of the discriminator can be adjusted by an optimizer. Through this process, the discriminator can gradually improve its ability to distinguish between real expert behavior and policy-generated behavior, thereby providing a more discriminative and stable reward signal for the motion control policy network.

[0221] Notably, the training of the discriminator and the training of the policy network are alternated or synchronized, and both co-evolve in the adversarial process: the stronger the discriminator, the more accurate the reward signal; the better the policy network, the more realistic the generated samples, which in turn drive the discriminator to further improve.

[0222] In some embodiments, to prevent the discriminator from converging too quickly and preventing the policy network from effectively learning, the update frequency of the discriminator can be limited, such as performing discriminator updates only every few policy updates, or introducing a gradient penalty term in the loss function to enhance training stability.

[0223] In other embodiments, the real discrimination probability can also be used to monitor the training state. If the real discrimination probability continues to approach 1 and the generated discrimination probability continues to approach 0 for a long time, it may indicate that the discriminator is too strong or the policy network has not been fully explored, in which case the learning rate, reward mapping method, or mixed sample proportion can be dynamically adjusted to maintain training balance.

[0224] It can be understood that by explicitly modeling the difference between the real and generated distributions and driving the discriminator optimization with an adversarial loss, the reliability and effectiveness of the reward signal can be ensured, thereby laying the foundation for the policy network to learn symmetric, natural, and robust motor skills.

[0225] Further, in some embodiments, constructing the adversarial loss function further includes: interpolating and sampling between the mixed sample set and the generated sample to obtain an intermediate sample; calculating the gradient norm of the discriminator with respect to the intermediate sample; calculating the square of the difference between the gradient norm and a predetermined target constant to obtain a gradient penalty term, and adding the gradient penalty term to the adversarial loss function.

[0226] To enhance the stability of the discriminator training and meet the Lipschitz continuity constraint, an interpolation sampling method can be used to construct intermediate samples between the two types of samples. Specifically, for a real sample in the mixed sample set and a corresponding generated sample in the generated sample set, linear interpolation is performed according to a random weight to generate a new sample point between the two. The intermediate sample is located on the line connecting the real distribution and the generated distribution in the input space, and is used to detect the response characteristics of the discriminator in this region.

[0227] After obtaining the intermediate sample, it is input into the discriminator, and the gradient of the discriminator's output with respect to the intermediate sample input is calculated, which is the rate of change of the discrimination score relative to the data pair of input state and action. Subsequently, the norm (usually the L2 norm) of the gradient vector is calculated to quantify the discriminator's sensitivity in that local region.

[0228] Understandably, in an ideal situation, if the discriminator satisfies the Lipschitz constraint, its gradient norm should be bounded, and it will not produce drastic output fluctuations due to small input perturbations.

[0229] In some implementations, a preset target constant can be set to 1, representing the desired ideal value of the gradient norm. The difference between the actually calculated gradient norm and this preset target constant is squared to form a gradient penalty term. This penalty term is added to the original adversarial loss function as a regularization term in the discriminator's parameter updates. By minimizing this penalty term, abnormal amplification of the discriminator's gradient can be effectively suppressed, preventing it from becoming too "sharp" or overfitting during training, thereby improving the overall training convergence and the quality of generated samples.

[0230] In some embodiments, the weights for interpolation sampling can be randomly sampled from a uniform distribution in each iteration to ensure that intermediate samples cover the entire connection path between the real and generated distributions. In other embodiments, the gradient penalty term is enabled only in a subset of training batches to balance computational overhead with training stability.

[0231] Understandably, by interpolating between the mixed sample set and the generated samples and introducing a gradient norm-based penalty term, the Lipschitz continuity of the discriminator is effectively constrained, preventing it from becoming unstable due to oversensitivity. This technique significantly improves the convergence and robustness of the adversarial training process, making the reward signal output by the discriminator smoother and more reliable. This helps the motion control policy network learn symmetrical and natural expert motion skills more stably and efficiently, while reducing the risk of pattern collapse or training oscillations.

[0232] In some example implementations, reference is made to Figure 5 As shown, mirroring is performed on portions of multiple original samples, including the following steps S501 to S504:

[0233] Step S501: Generate random numbers for each original sample in the current training batch.

[0234] During each training batch, a random number is generated independently for each original sample within that batch. This random number can follow a uniform distribution and ranges from 0 to 1, and is used to subsequently determine whether to perform mirroring processing on that sample.

[0235] Step S502, compare the random number with the preset threshold to obtain a comparison result, and use the comparison result as a mirror selection mark of each original sample.

[0236] Specifically, a preset threshold (e.g. 0.5) can be set. If the random number corresponding to an original sample is less than or equal to the preset threshold, it is determined that the original sample should be selected for mirror processing, and the mirror selection mark thereof is set to "yes"; otherwise, the mark is "no" and the original sample is kept unchanged. Through this mechanism, a controllable probability can be used to dynamically select part of the samples for mirror operation in each training batch, so as to achieve a mirror ratio of about 50%. The threshold can also be adjusted according to the task requirements to change the proportion of mirror samples.

[0237] In some embodiments, comparing the random number with the preset threshold to obtain a comparison result includes: the random number has a value range of 0-1, and the preset threshold is 0.5.

[0238] Step S503, extracting a mirror sample subset from the training batch based on the mirror selection mark.

[0239] The mirror selection marks of all original samples in the current batch are traversed, and the samples with the mark "yes" are screened out to form a mirror sample subset. The samples in the mirror sample subset will be subjected to mirror transformation at the semantic level in the next step, while the remaining unselected samples are kept in the original form and used to construct a mixed sample set together with the mirror samples.

[0240] Step S504, performing mirror processing on each original sample in the mirror sample subset to generate mirror state data and write back to the mirror sample subset.

[0241] In some embodiments, the mirror processing is not simply a coordinate flip, but a semantic consistent rearrangement and sign adjustment of the state and action according to the left-right symmetry of the humanoid robot body structure.

[0242] For example, the state variables of the joints of the left leg are exchanged with the corresponding joints of the right leg, and the sign of the direction-related components such as transverse velocity and angular velocity is reversed to ensure that the mirrored state is physically reasonable and symmetric with the original motion semantics.

[0243] After the mirror transformation is completed, the generated mirror state data (including the mirrored state and the corresponding mirror action) will replace the original data in the original subset to form the final mirror sample subset.

[0244] It can be understood that the above mirror selection mechanism is executed online during the training process, without the need to pre-expand the data set, which not only saves storage overhead, but also ensures the diversity of each batch of data.

[0245] In addition, since whether to mirror is determined by a random number, all samples have equal opportunities to be mirrored in long-term training, so that the policy network learns a symmetric motion representation in a statistically unbiased manner. Those skilled in the art can adjust the preset threshold, random number distribution or mirroring rule according to actual needs, and construct symmetric training data by randomly and dynamically selecting and semantically mirroring part of the original samples, which is not limited herein.

[0246] Further, in some embodiments, the above method further comprises: obtaining a set of mirror discrimination scores output by the discriminator for the subset of mirrored samples, and a set of original discrimination scores output by the discriminator for the part of the original samples that is not mirrored; calculating a first statistical value based on the set of mirror discrimination scores, calculating a second statistical value based on the set of original discrimination scores, and calculating a bias indicator based on the first statistical value and the second statistical value; comparing the bias indicator with a preset bias threshold, and updating the preset threshold for the next training batch according to the comparison result.

[0247] The first statistical value and the second statistical value can be the average, median or combination of maximum and minimum of the respective set, for reflecting the difference in overall discrimination tendency of the discriminator for the mirrored samples and the original samples. The bias indicator is used to measure whether the current mirroring processing causes the mixed sample set to be asymmetrically biased in distribution. For example, if the mirrored samples are generally judged to be lower scores, it may indicate that the mirroring operation introduces an unreasonable state, or the mirroring proportion is imbalanced.

[0248] Specifically, the subset of mirrored samples is input to the discriminator, and the discriminator outputs a discrimination score for each sample in the subset of mirrored samples to obtain a set of mirror discrimination scores; the part of the original samples that is not mirrored is input to the discriminator, and the discriminator outputs a discrimination score for the part of the original samples that is not mirrored to obtain a set of original discrimination scores.

[0249] In some embodiments, the bias indicator can be compared with the preset bias threshold, and the preset threshold for mirroring selection in the next training batch can be updated according to the comparison result.

[0250] For example, if the bias indicator is greater than the preset bias threshold, it indicates that there is a significant inconsistency in the discrimination response between the mirrored samples and the original samples, and in this case the preset threshold for the next training batch can be reduced so that fewer samples are selected for mirroring; if the bias indicator is less than or equal to the preset bias threshold, the threshold can be maintained or appropriately increased to enhance the learning of symmetric motion patterns by the policy network.

[0251] In some embodiments, the preset bias threshold can be pre-set according to the task complexity, or can be dynamically adjusted during training. For example, a larger bias tolerance is allowed at the beginning of training to promote exploration, and the threshold is gradually tightened to improve the accuracy of action symmetry as the training converges.

[0252] The skilled in the art should understand that by monitoring the discrimination difference of the discriminator between the mirror image and the original sample in real time, the mirror image ratio is dynamically adjusted, which helps to ensure the statistical consistency of the mixed sample set in the bilateral symmetry, and avoid the motion control strategy network from learning the unilateral preference behavior due to data bias.

[0253] In some example embodiments, each original sample includes state data and action data corresponding to the state data, the state data including joint state information and linear velocity and angular velocity; as Figure 6 As shown, the mirror image processing on part of the plurality of original samples can include the following steps S601 to S602:

[0254] Step S601, performing state mirror image processing on part of the original samples; the state mirror image processing includes: exchanging left and right limb joint state information, and performing mirror image flipping on linear velocity and angular velocity.

[0255] The action data is a control instruction corresponding to the state data, which can represent the target output of each actuator in the form of a vector.

[0256] Specifically, the state mirror image processing can include: according to the symmetrical structure of the left and right limbs of the robot, the joint state information corresponding to the left and right limbs in the state data can be exchanged; at the same time, mirror image flipping is performed on the linear velocity and angular velocity in the overall motion state.

[0257] In some embodiments, exchanging left and right limb joint state information includes: establishing a symmetrical mapping table of robot joints, the symmetrical mapping table containing a one-to-one mapping relationship between the index of each joint of the left limb and the index of the corresponding joint of the right limb; traversing the joint state vector in the state data; according to the symmetrical mapping table, the joint state vector of the left limb joint and the joint state vector of the corresponding right limb joint are interchanged.

[0258] The joint state vector can include state quantities such as the angle, angular velocity or torque of each joint, arranged in a predetermined order to form a fixed-dimensional vector.

[0259] The symmetrical mapping table can be predefined according to the mechanical structure of the humanoid robot, for example, the left hip joint, the left knee joint and the left ankle joint are respectively indexed and corresponded to the right hip joint, the right knee joint and the right ankle joint.

[0260] When performing mirror image processing, the mapping table can be read to sequentially locate the position of the left joint in the state vector and the position of its right counterpart joint, and exchange the values corresponding to the two.

[0261] It can be understood that in this way, the mirrored state data is ensured to maintain left-right symmetric semantic consistency at the joint level, providing physically reasonable input for subsequent policy training.

[0262] In some embodiments, the linear velocity and angular velocity information are mirrored; including: obtaining attitude data corresponding to the state data to determine a coordinate transformation matrix; transforming the linear velocity and angular velocity to a body coordinate system using the coordinate transformation matrix to obtain a body system velocity vector; calling a preset mirror flip matrix and the body system velocity vector to perform matrix multiplication to obtain a mirrored body system velocity vector; and transforming the mirrored body system velocity vector back to the original coordinate system using the inverse matrix of the coordinate transformation matrix to obtain the mirrored linear velocity and the mirrored angular velocity.

[0263] The attitude data is used to represent the orientation of the humanoid robot's torso or reference link in the original coordinate system (such as the world coordinate system), which can be provided by an inertial measurement unit, a visual positioning system, or a pose sensor in a simulation environment. The coordinate transformation matrix constructed based on the attitude data can accurately convert the linear velocity and angular velocity in the global coordinate system to the body coordinate system with the robot body as the reference. In this coordinate system, the front-back, left-right, and up-down directions have clear physical definitions, which facilitates the implementation of mirroring operations.

[0264] In the body coordinate system, the preset mirror flip matrix can be designed to flip the sign of the velocity component in a specific direction. For example, the lateral linear velocity (usually corresponding to the body y-axis) and the angular velocity around the vertical axis (yaw angular velocity, corresponding to the z-axis) are negated, while the longitudinal linear velocity (x-axis), vertical linear velocity (z-axis), and pitch, roll angular velocities remain unchanged. This mirror flip matrix, by multiplying with the body system velocity vector, realizes the velocity mirroring consistent with human motion symmetry.

[0265] Subsequently, the mirrored body system velocity vector is remapped back to the original coordinate system using the inverse matrix of the coordinate transformation matrix, thereby obtaining the mirrored and flipped linear velocity and angular velocity in the original reference system, which are semantically correct and physically consistent. This processing method ensures that the mirror result accurately reflects the dynamic characteristics of left-right symmetric motion regardless of the orientation of the original state data, avoiding mirror distortion caused by coordinate system dependence.

[0266] In other embodiments, if the original state data is directly represented in the body coordinate system, the coordinate transformation step can be omitted, and the mirror flip matrix can be directly applied for processing. Those skilled in the art should understand that the above mirror flip process can be adaptively adjusted according to the actual sensor configuration, coordinate system convention, or robot structure, and the exemplary embodiments in this example are not limited thereto.

[0267] In some embodiments, the lateral component (such as lateral velocity or angular velocity around the vertical axis) in the overall motion state can be sign-inverted, while the longitudinal and vertical direction components remain unchanged, to maintain physical consistency.

[0268] In some other embodiments, performing the mirror flipping on the linear velocity and the angular velocity includes: obtaining coordinate axis components of the linear velocity in the robot body coordinate system, and taking the negative value of the Y coordinate axis component; obtaining roll angular velocity components and yaw angular velocity components of the angular velocity in the robot body coordinate system, and taking the negative values of the roll angular velocity components and the yaw angular velocity components, respectively.

[0269] In the robot body coordinate system, the linear velocity can be decomposed into components along the X, Y, and Z directions, corresponding to forward / backward, lateral movement, and lifting motion, respectively. When performing the mirror operation, only the linear velocity component in the Y-axis direction can be taken as negative to reflect the left-right symmetric flipping, while the X-axis and Z-axis components remain unchanged.

[0270] For the angular velocity, it generally includes roll angular velocity (roll) around the X-axis, pitch angular velocity (pitch) around the Y-axis, and yaw angular velocity (yaw) around the Z-axis in the body coordinate system. In the mirror processing, the roll angular velocity and the yaw angular velocity have left-right asymmetry: when the body is flipped left and right, both the roll direction and the turning direction are reversed, so their components need to be taken as negative; while the pitch angular velocity describes the forward and backward pitching motion, which remains unchanged under left-right mirroring, so it is not processed.

[0271] In this way, without complex coordinate transformation or matrix operation, the mirror flipping of the linear velocity and the angular velocity can be efficiently and accurately completed in the body coordinate system, ensuring that the generated mirror state data is reasonable in dynamics and conforms to the physical law of left-right symmetric motion of humanoid robots.

[0272] In some embodiments, if the Y-axis of the robot body coordinate system is defined as pointing to the right, the mirror logic is adjusted accordingly, but the core principle remains unchanged: only the lateral motion and the rotational components related to left-right symmetry can be sign-inverted.

[0273] Step S602, performing action mirror processing on the action data corresponding to the state data after the state mirror processing; the action mirror processing includes: rearranging the element positions of the action vector according to the preset symmetric index relationship.

[0274] Specifically, the action mirroring processing can include: rearranging element positions in the action vector according to a preset symmetric index relationship. The symmetric index relationship defines the corresponding positions of the left and right actuators in the action vector, for example, the indexes of the left hip joint driver and the right hip joint driver are interchanged. Through the rearrangement, it can be ensured that the mirrored action instruction matches the mirrored body configuration, thereby generating a semantically correct and dynamically feasible mirrored sample.

[0275] The symmetric index relationship can be pre-configured based on the robot hardware topology and stored as a mapping table, which is called in real time during the training process. For a robot containing asymmetric actuators, only the symmetric part can be rearranged, and the remaining dimensions remain unchanged.

[0276] In some embodiments, the rearranging element positions in the action vector according to a preset symmetric index relationship includes: obtaining an action dimension definition table consistent with the dimension of the action vector, and generating a left-right symmetric action dimension pair list based on the action dimension definition table; generating a preset symmetric index relationship based on the left-right symmetric action dimension pair list; and performing index rearrangement on the action vector using the preset symmetric index relationship to obtain a rearranged action vector.

[0277] The action vector is used to represent the control instructions of each actuator (such as joint motor) of the robot, and each dimension corresponds to a specific driving degree of freedom. The action dimension definition table records the physical actuator corresponding to each dimension in the action vector and its position attribute on the robot body (such as "left hip abduction", "right knee flexion", etc.), which is pre-configured according to the robot hardware topology in the system initialization stage, and strictly aligned with the action vector.

[0278] Based on the definition table, actuator pairs with left-right symmetric relationship can be identified, such as "left hip flexion" and "right hip flexion", "left ankle inversion" and "right ankle inversion", etc., thereby constructing a left-right symmetric action dimension pair list. The list explicitly indicates which dimensions should be interchanged in position when mirroring in the form of paired indexes.

[0279] In some embodiments, the action dimension definition table can be automatically loaded with the robot model, supporting quick adaptation of different numbers and layouts of joints.

[0280] Subsequently, the preset symmetric index relationship can be generated according to the list, and the preset symmetric index relationship can be represented as a mapping array, where the i-th element indicates that the i-th dimension in the original action vector should be placed in the new position after mirroring. When performing action mirroring processing, the original action vector can be rearranged according to the index relationship: for the dimensions belonging to the symmetric pair, exchange their values; for the dimensions located on the body midline or without symmetric pairing (such as the trunk pitch joint), keep the original position unchanged.

[0281] Through the above mechanism, it can be ensured that the mirrored action instruction and the mirrored body configuration are strictly matched at the execution level, avoiding movement imbalance or dynamics conflict caused by left-right instruction misplacement. The method does not rely on specific numerical modification, and only through index rearrangement, the symmetric conversion of action semantics is realized, which has the advantages of strong universality, efficient implementation and flexible adaptation.

[0282] It can be understood that the above mirror processing method ensures the joint consistency of state and action under mirror transformation, avoiding policy learning confusion caused by mismatch. In addition, the mirror operation can be applied as needed in each training batch without modifying the original expert data, realizing flexible and efficient data enhancement. The present exemplary embodiments are not limited thereto.

[0283] The present exemplary embodiments also provide a robot motion control method based on imitation learning, comprising the following steps: obtaining the current observation state of the robot and inputting it into a pre-trained motion control policy network, and outputting the action policy for controlling the robot motion based on the motion control policy network; wherein the motion control policy network is obtained according to the above-mentioned robot motion control policy network training method based on imitation learning.

[0284] The motion control policy network is optimized by a mixed sample set (containing original human motion capture data and its mirror samples) and an adversarial discrimination mechanism in the training stage, and has learned symmetric, natural and physically feasible motion skills. In the deployment stage, the motion control policy network can be used as a deterministic or random policy function, receiving the current observation state as input and outputting the corresponding action policy, such as the target torque, target speed or target position of each joint.

[0285] In specific implementation, the action policy is sent to the underlying execution controller to drive each actuator of the humanoid robot to complete the corresponding action, thereby realizing walking, turning, crossing obstacles and other motion behaviors. Since the motion control policy network introduces mirror enhancement, adversarial reward and symmetry constraint mechanism in the training process, the action policy output by it has good left-right balance and environmental adaptability, and can maintain stable motion performance even under unobserved terrain or disturbance conditions.

[0286] In some embodiments, the pre-trained motion control policy network can be deployed in the embedded computing unit of the robot body, or can be run on a remote server and issue control instructions through a low-latency communication link. In other embodiments, an online fine-tuning mechanism can be combined at runtime to update the policy network with actual interaction data for further improvement of task specificity.

[0287] In the example embodiment, the current observation state of the robot is input into the trained pre-trained motion control strategy network to directly output an action strategy for controlling the motion of the robot, thereby realizing efficient, stable and good symmetry real-time motion control. Since the strategy network combines the original human motion capture data and semantically correct mirror samples in the training stage, and optimizes the reward signal through the means of adversarial discrimination mechanism and gradient penalty, the learned strategy not only highly approximates the expert behavior, but also naturally has the balance and coordination of left and right limb motion, effectively avoiding the problem of gait asymmetry or yaw caused by unilateral preference of the original data. At the same time, thanks to the dynamic adjustment of the mirror ratio, the consistency processing of the state-action joint mirror and the statistical balance design of the mixed sample set in the training process, the strategy exhibits stronger generalization ability in the network deployment stage, and can maintain stable walking performance under unseen terrain or disturbance conditions. In addition, this method does not need to solve complex optimization problems or rely on accurate dynamics models, and can generate high-quality action instructions through forward inference, significantly reducing the computational overhead and meeting the real-time control requirements of embedded platforms.

[0288] Reference Figure 7 Fig. 1 shows a schematic diagram of a robot motion control method based on imitation learning in an example embodiment. In Figure 7 After the motion control strategy network 304 obtained by the above training method is deployed to the robot, the robot performs the regular motion task according to the preset motion strategy. During the operation of the robot, the current observation state 701 of the robot can be obtained and input into the motion control strategy network 304, and the motion control strategy network 304 outputs an action strategy 703 for controlling the motion of the robot, and drives the joint actuators of the robot to perform corresponding actions based on the action strategy 703.

[0289] The motion control strategy network 304 uses the original human motion capture samples and the mirror samples generated by semantic mirror processing in the training stage, and jointly optimizes the adversarial discrimination mechanism and symmetry constraint, so that the output action strategy has good symmetry and coordination in the motion of the left and right limbs. Moreover, due to the introduction of the mirror ratio dynamic adjustment and the state-action joint mirror consistency guarantee mechanism in the training process, the strategy network can effectively avoid the motion skew caused by the unilateral preference of the expert data in actual operation, and significantly improve the motion stability and generalization ability of the humanoid robot in unknown environments or disturbance conditions.

[0290] In the example embodiment of the present disclosure, a robot motion control strategy network training device based on imitation learning is also provided. Referring to Figure 8As shown, the robot motion control strategy network training device 800 based on imitation learning includes a sample mirroring module 801, a sample mixing module 802, a reward signal generation module 803, and a policy network updating module 804, wherein:

[0291] The sample mirroring module 801 is configured to mirror part of the plurality of original samples to obtain a plurality of mirrored samples.

[0292] The sample mixing module 802 is configured to mix the plurality of mirrored samples and the original samples that have not been mirrored to form a mixed sample set for training the discriminator, so as to force the discriminator to learn the symmetric data distribution characteristics.

[0293] The reward signal generation module 803 is configured to input the mixed sample set and the generated sample into the discriminator to obtain a reward signal.

[0294] The policy network updating module 804 is configured to feed back the reward signal to the motion control strategy network, so that the subsequent generated action of the motion control strategy network is closer to the characteristics of the mixed sample set.

[0295] The original sample is human motion capture data, and the generated sample is a sample obtained by outputting a predicted action according to the current observation state according to the motion control strategy network.

[0296] The specific details of each module in the above robot motion control strategy network training device based on imitation learning have been described in detail in the corresponding robot motion control strategy network training method based on imitation learning, and thus will not be described here.

[0297] In the example embodiments of the present disclosure, a robot motion control device based on imitation learning is also provided. Referring to Figure 9 As shown, the robot motion control device 900 based on imitation learning includes a data acquisition module 901 and a control policy module 902, wherein:

[0298] The data acquisition module 901 is configured to acquire the current state environment data of the robot and input it into the pre-trained motion control strategy network.

[0299] The control policy module 902 is configured to output an action policy for controlling the motion of the robot based on the motion control strategy network, wherein the motion control strategy network is obtained according to the above robot motion control strategy network training method based on imitation learning.

[0300] The specific details of each module in the above robot motion control device based on imitation learning have been described in detail in the corresponding robot motion control method based on imitation learning, and thus will not be described here.

[0301] In the example embodiments of the present disclosure, a robot is also provided, the humanoid robot comprising a processor and a memory, the memory having stored thereon computer readable instructions which, when executed by the processor, implement the above method. The robot can be any one of a foot-type robot, a quadruped robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot, and a robotic arm. Of course, it can also be other types of mobile robots, which are not limited by the present disclosure.

[0302] Reference is made to Figure 10 As shown in FIG. 1, a schematic diagram of a type of robot to which the robot motion control method based on imitation learning in the embodiments of the present disclosure can be applied is shown as 1001.

[0303] Reference is made to Figure 11 As shown in FIG. 2, a schematic diagram of another type of robot to which the robot motion control method based on imitation learning in the embodiments of the present disclosure can be applied is shown as 2001. Figure 11 It can be seen that the type of robot is a humanoid robot.

[0304] Reference is made to Figure 12 As shown in FIG. 12, an electronic device capable of implementing the above method is also provided. The electronic device 1200 comprises a processor 1201 and a memory 1202, the memory 1202 having stored thereon computer readable instructions which, when executed by the processor 1201, implement the method in the embodiments of the present disclosure.

[0305] In the example embodiments of the present disclosure, a computer readable storage medium having stored thereon computer program code instructions which, when invoked by a processor of a robot, cause the robot to perform the method as described in the embodiments is also provided.

[0306] Reference is made to Figure 13 As shown in FIG. 13, a program product 1300 for implementing the above method according to the embodiments of the present disclosure is described, which can take the form of a portable compact disc read-only memory (CD-ROM) and comprise program code, and can be run on a terminal device such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, the readable storage medium can be any tangible medium containing or storing a program which can be used by or in conjunction with an instruction execution system, apparatus or device.

[0307] Those skilled in the art can clearly understand, through the description of the above examples, that the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

[0308] Finally, the above preferred embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail, those skilled in the art should understand that changes can be made to the form and details thereof without departing from the scope defined by the claims of the present application. The sizes of the drawings are not related to the actual size, and the actual size can be arbitrarily changed.

Claims

1. A method for training a robot motion control policy network based on imitation learning, characterized in that, include: Multiple mirrored samples are obtained by mirroring parts of multiple original samples. The multiple mirrored samples and the remaining original samples that have not been mirrored are mixed to form a mixed sample set for training the discriminator, so as to force the discriminator to learn symmetric data distribution features; The mixed sample set and the generated sample are input into the discriminator to obtain a reward signal; The reward signal is fed back to the motion control policy network so that the subsequent actions generated by the motion control policy network are closer to the features of the mixed sample set. The original samples are human motion capture data, and the generated samples are samples obtained by the motion control strategy network based on the current observation state and the predicted actions output.

2. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The step of inputting the mixed sample set and the generated sample into the discriminator to obtain a reward signal includes: Obtain the generation discrimination probability output by the discriminator for the generated sample; the generation discrimination probability characterizes the degree to which the discriminator determines that the generated sample conforms to the features of the mixed sample set; The generation discrimination probability is converted into the reward signal using a preset reward mapping function; wherein, the higher the generation discrimination probability, the larger the value of the converted reward signal.

3. The robot motion control strategy network training method based on imitation learning according to claim 2, characterized in that, The step of converting the generated discrimination probability into the reward signal using a preset reward mapping function includes: calculate The value of is used as the reward signal; where log represents the logarithmic operation and D represents the generation discrimination probability.

4. The method for training a robot motion control strategy network based on imitation learning according to claim 2, characterized in that, The step of converting the generated discrimination probability into the reward signal using a preset reward mapping function further includes: In response to the value of the reward signal exceeding a preset reward truncation threshold, the value of the reward signal is corrected to the preset reward truncation threshold.

5. The method for training a robot motion control strategy network based on imitation learning according to claim 2, characterized in that, The method further includes: Obtain the true discrimination probability output by the discriminator for the mixed sample set; Construct an adversarial loss function, which measures the deviations of the true discrimination probability and the generated discrimination probability from their respective target values. The parameters of the discriminator are updated based on the adversarial loss function.

6. The method for training a robot motion control strategy network based on imitation learning according to claim 5, characterized in that, The adversarial loss function is: in, Indicating resistance to loss, Represents the mathematical expectation. This represents the true discrimination probability. This represents the generated discrimination probability.

7. The method for training a robot motion control strategy network based on imitation learning according to claim 5, characterized in that, The construction of the adversarial loss function also includes: Interpolation sampling is performed between the mixed sample set and the generated samples to obtain intermediate samples; Calculate the gradient norm of the discriminator with respect to the intermediate samples; The gradient penalty term is obtained by calculating the square of the difference between the gradient norm and the preset target constant, and then the gradient penalty term is added to the adversarial loss function.

8. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The step of feeding the reward signal back to the motion control policy network so that the subsequent actions generated by the motion control policy network are closer to the features of the mixed sample set includes: The reward signal is fed back to the robot motion control strategy network; Calculate the parameter gradient direction that maximizes the expected value of the reward signal; The weight parameters of the robot motion control strategy network are adjusted along the direction of the parameter gradient so that the actions subsequently generated by the motion control strategy network are closer to the features of the mixed sample set.

9. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The mirroring process of a portion of multiple original samples includes: Generate random numbers for each of the original samples in the current training batch; The random number is compared with a preset threshold to obtain a comparison result, and the comparison result is used as a mirror selection mark for each of the original samples; Based on the mirror selection marker, extract a subset of mirror samples from the training batch; After performing the mirroring process on each original sample in the mirrored sample subset, mirrored state data is generated and written back to the mirrored sample subset.

10. The method for training a robot motion control strategy network based on imitation learning according to claim 9, characterized in that, The method further includes: Obtain the set of mirror discrimination scores output by the discriminator for the mirrored sample subset, and the set of original discrimination scores output for the unmirrored portion of the original sample; A first statistical value is calculated based on the mirror discriminant score set, a second statistical value is calculated based on the original discriminant score set, and a bias index is calculated based on the first statistical value and the second statistical value. The bias metric is compared with a preset bias threshold, and the preset threshold for the next training batch is updated based on the comparison result.

11. The method for training a robot motion control strategy network based on imitation learning according to claim 9, characterized in that, The step of comparing the random number with a preset threshold to obtain a comparison result includes: the random number has a value range of 0 to 1, and the preset threshold is 0.

5.

12. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, Mirroring portions of multiple original samples, including: Mirroring is performed on 35-65% of the plurality of original samples; or, mirroring is performed on 45-55% of the plurality of original samples; or, mirroring is performed on 49-51% of the plurality of original samples.

13. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, Each of the original samples includes state data and motion data corresponding to the state data; the state data includes joint state information as well as linear velocity and angular velocity; Mirroring portions of multiple original samples, including: A state mirroring process is performed on a portion of the original samples; the state mirroring process includes: exchanging the state information of the left and right limb joints, and performing a mirror flip on the linear velocity and angular velocity; Perform action mirroring processing on the action data corresponding to the state data after state mirroring processing; the action mirroring processing includes rearranging the element positions of the action vector according to a preset symmetric index relationship.

14. The method for training a robot motion control strategy network based on imitation learning according to claim 13, characterized in that, The mirror flipping of the linear velocity and angular velocity includes: Obtain the coordinate axis components of the linear velocity in the robot body coordinate system, and take the negative value of the Y coordinate axis component; Obtain the rolling angular velocity component and the yaw angular velocity component in the robot body coordinate system, and take negative values ​​for the rolling angular velocity component and the yaw angular velocity component respectively.

15. The method for training a robot motion control strategy network based on imitation learning according to claim 13, characterized in that, The mirroring and flipping of the linear velocity and angular velocity information includes: Obtain the attitude data corresponding to the state data to determine the coordinate transformation matrix; The linear velocity and angular velocity are transformed to the body coordinate system using the coordinate transformation matrix to obtain the body velocity vector; The mirrored machine system velocity vector is obtained by performing matrix multiplication with the preset mirror flip matrix and the machine system velocity vector. The inverse of the coordinate transformation matrix is ​​used to transform the velocity vector of the mirror machine system back to the original coordinate system to obtain the linear velocity and angular velocity after mirror flipping.

16. The method for training a robot motion control strategy network based on imitation learning according to claim 13, characterized in that, The exchange of left and right limb joint status information includes: Establish a symmetric mapping table for robot joints, wherein the symmetric mapping table contains a one-to-one mapping relationship between the joint indices of the left limb and the corresponding joint indices of the right limb; Traverse the joint state vectors in the state data; According to the symmetric mapping table, the joint state vector of the left limb is interchanged with the corresponding joint state vector of the right limb.

17. The method for training a robot motion control policy network based on imitation learning according to claim 13, characterized in that, The rearrangement of the element positions of the action vector according to a preset symmetric indexing relationship includes: Obtain an action dimension definition table that is consistent with the action vector dimension, and generate a list of left and right symmetrical action dimension pairs based on the action dimension definition table; Based on the aforementioned left-right symmetrical action dimensions, a preset symmetrical index relationship is generated for the list; The action vector is rearranged by performing index rearrangement on the action vector using the preset symmetric index relationship.

18. A robot motion control method based on imitation learning, characterized in that, include: The current observation state of the robot is obtained and input into the pre-trained motion control strategy network. Based on the motion control strategy network, the action strategy for controlling the robot's movement is output. The motion control policy network is obtained by the robot motion control policy network training method based on imitation learning according to any one of claims 1 to 17.

19. A robot motion control strategy network training device based on imitation learning, characterized in that, include: The sample mirroring module is used to mirror parts of multiple original samples to obtain multiple mirrored samples; The sample mixing module is used to mix the multiple mirrored samples with the remaining original samples that have not been mirrored to form a mixed sample set for training the discriminator, so as to force the discriminator to learn symmetric data distribution features; A reward signal generation module is used to input the mixed sample set and the generated sample into the discriminator to obtain a reward signal; The policy network update module is used to feed the reward signal back to the motion control policy network so that the subsequent actions generated by the motion control policy network are closer to the features of the mixed sample set. The original samples are human motion capture data, and the generated samples are samples obtained by the motion control strategy network based on the current observation state and the predicted actions output.

20. A robot motion control device based on imitation learning, characterized in that, include: The data acquisition module is used to acquire the robot's current state environment data and input it into the pre-trained motion control policy network; A control strategy module is used to output an action strategy for controlling the robot's motion based on the motion control strategy network; wherein the motion control strategy network is obtained by the robot motion control strategy network training method based on imitation learning according to any one of claims 1 to 17.

21. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 18.

22. A robot, characterized in that, include: A processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the method of claim 18.

23. The robot according to claim 22, characterized in that, The robots include mobile robots.

24. The robot according to claim 22, characterized in that, The robots include humanoid robots.

25. The robot according to claim 22, characterized in that, The robot includes either a legged robot or a wheeled robot.

26. The robot according to claim 22, characterized in that, The robot includes a wheeled robot.

27. The robot according to claim 22, characterized in that, The robot includes any one of quadruped robots, bipedal robots, and four-wheeled robots.

28. The robot according to claim 22, characterized in that, The robot includes either a cleaning robot or a transport robot.

29. The robot according to claim 22, characterized in that, The robot includes a robotic arm.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by the robot's processor, cause the robot to perform the method as described in any one of claims 1 to 18.

Images