Robot motion control strategy network training method based on imitation learning, robot motion control method, device, equipment, robot and storage medium

Abstract

The present disclosure provides a robot motion control strategy network training method based on imitation learning, a robot motion control method, device, equipment, robot and storage medium, and relates to the technical field of robots. The present disclosure constructs multiple independent style discriminators, respectively trains them using targeted reference motion data, effectively decouples the dependency relationship of different motion styles in the feature space, thereby avoiding the feature competition problem caused by the use of shared networks in related technologies, enabling the discriminators to focus on single style feature extraction, and even in the scene where the task space highly coincides, the action details can also be accurately distinguished. According to the current state, the target style discriminator is dynamically locked, and the score of the target discriminator is used as the only imitation learning reward value, so that the policy network can provide a directional, single and clear optimization gradient during the training process, and the policy network can learn a clear style switching boundary.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robotics, and relates to a robot motion control strategy network training method based on imitation learning, a robot motion control method, device, equipment, robot and storage medium. BACKGROUND

[0002] In the training scene of robot motion control, it is often necessary to make a set of control strategies present multiple distinguishable action styles. In the training of robot action generation based on imitation learning, the common practice is to introduce a discriminator to evaluate the style consistency between the actions output by the strategy network and the reference trajectory, so that the strategy network learns actions with clear styles such as walking, running, and limping.

[0003] For example, in Chinese patent application No. CN114779661A, in order to enable the robot to master multiple different action styles, a classifier is added to the network structure, and a shared hidden layer parameter is used to try to enable the robot to master multiple expert strategies in a unified algorithm framework, so as to realize the differentiation and imitation of different action styles.

[0004] However, the inventors have found in practice that this method of relying on a single or shared network discriminator to guide action style learning has defects. Specifically, since some different action styles (such as walking and running) have a high degree of overlap in task space and state features, the shared discriminator network may have difficulty providing unique directional reward feedback for actions with ambiguous boundaries during training. This ambiguity in evaluation signals results in the control strategy learned by the robot being unable to learn clear style switching boundaries. Further, when the robot is actually performing a task and is in an action transition phase, the control instructions output by the control strategy may fluctuate between different style features, resulting in unstable phenomena such as gait disorder, high-frequency jitter, or transition stiffness in the robot. SUMMARY

[0005] The present disclosure provides a robot motion control strategy network training method based on imitation learning, a robot motion control method, device, equipment, robot and storage medium, which can effectively suppress unstable phenomena such as gait disorder, high-frequency jitter, or transition stiffness caused by the fluctuation of control instructions output by the control strategy between different style features.

[0006] Additional aspects and advantages of the present disclosure will be set forth in part in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a robot motion control strategy network training method based on imitation learning is provided, comprising:

[0008] a plurality of style discriminators are constructed, each of which is trained based on reference action data of a corresponding target style, and is used to evaluate the consistency between the output action of the motion control policy network and the target style;

[0009] a target style discriminator corresponding to the current state of the robot is determined, and a score result of the target style discriminator on the current state and the current action of the robot is taken as an imitation learning reward value at the current time;

[0010] The motion control policy network is updated based on the imitation learning reward value to guide the motion control policy network to learn action features of the corresponding style.

[0011] Optionally, the above determining a target style discriminator corresponding to the current state of the robot, and taking a score result of the target style discriminator on the current state and the current action of the robot as an imitation learning reward value at the current time, comprises:

[0012] a style transition buffer is obtained;

[0013] when the current state is in the style transition buffer, it is determined that both target style discriminators corresponding to the current state are hybrid style discriminators, and a weight coefficient of each hybrid style discriminator is determined;

[0014] score results of the two hybrid style discriminators on the current state and the current action are respectively calculated;

[0015] the score results are weighted and summed according to the weight coefficients of the two hybrid style discriminators to obtain an imitation learning reward value at the current time.

[0016] Optionally, the above determining that both target style discriminators corresponding to the current state are hybrid style discriminators and determining a weight coefficient of each hybrid style discriminator, comprises:

[0017] a target style index corresponding to the current state is obtained;

[0018] a main style discriminator is determined based on the target style index;

[0019] a secondary style discriminator adjacent to the main style discriminator is determined based on the style transition buffer;

[0020] the main style discriminator and the secondary style discriminator are determined as hybrid style discriminators, and a weight coefficient assigned to the main style discriminator and the secondary style discriminator is calculated based on the current state.

[0021] Optionally, the above determining a weight coefficient of each hybrid style discriminator, comprises:

[0022] a lower threshold value, an upper threshold value of the style transition buffer, and a current linear velocity scalar of the robot corresponding to the current state are obtained.

[0023] a first difference value is calculated according to the current linear velocity scalar and the lower threshold value, and a second difference value is calculated according to the upper threshold value and the lower threshold value;

[0024] a first non-normalized weight is calculated according to a ratio of the first difference value and the second difference value, and a complementary weight of the first non-normalized weight is taken as a second non-normalized weight;

[0025] the first non-normalized weight and the second non-normalized weight are normalized to obtain weight coefficients of the mixed style discriminators.

[0026] Optionally, the style transition buffer is obtained by:

[0027] a time difference of the current linear velocity scalar of the robot corresponding to the current state is calculated to obtain a speed change amount, and a buffer width parameter is calculated based on the speed change amount;

[0028] the lower threshold value and the upper threshold value of the buffer are generated according to the buffer width parameter;

[0029] the buffer formed by the lower threshold value and the upper threshold value is determined as the style transition buffer.

[0030] Optionally, the plurality of style discriminators are constructed by:

[0031] a positive sample state-action pair is sampled from the reference action data of the target style, and a negative sample state-action pair is sampled from the reference action data of the non-target style;

[0032] the positive sample state-action pair and the negative sample state-action pair are input into the discriminator network structure to be trained to output a score result;

[0033] a discriminator loss is calculated based on the score result and a target label, and parameters of the discriminator network structure are updated to obtain a style discriminator corresponding to the target style, the target label including a label of the positive sample and a label of the negative sample.

[0034] Optionally, the discriminator loss is calculated based on the score result and the target label by:

[0035] the discriminator loss is calculated by:

[0036]

[0037] wherein, is the discriminator loss; is an adversarial classification loss calculated based on the score result and the label of the positive sample and the label of the negative sample; is a preset penalty coefficient; is an interpolation sample; an output score of the discriminator network structure for the interpolated sample; a gradient of the output score with respect to the interpolated sample; denotes a norm regular term; denotes an expectation of a distribution of the interpolated sample .

[0038] Optionally, the plurality of style discriminators are constructed, including:

[0039] a state-action encoding sub-network is constructed for each target style, and the state-action encoding sub-networks are configured with network parameters independent of each other;

[0040] a style score sub-network is constructed for each target style, and the style score sub-networks are configured with network parameters independent of each other;

[0041] the state-action encoding sub-network corresponding to the same target style is connected with the style score sub-network to obtain a style discriminator corresponding to the target style.

[0042] Optionally, the target style discriminator corresponding to the current state of the robot is determined, including:

[0043] a state quantity is extracted from the current state of the robot;

[0044] a target style index corresponding to the state quantity is determined according to a preset state quantity-style index mapping rule;

[0045] the target style discriminator is selected from the plurality of style discriminators based on the target style index.

[0046] Optionally, the current state of the robot includes at least one or more of a linear velocity, an angular velocity of the robot, and a slope of a terrain where the robot is located.

[0047] Optionally, the score result of the target style discriminator for the current state and the current action of the robot is taken as an imitation learning reward value at a current time, including:

[0048] a state history sequence and an action history sequence of the robot containing N latest time steps are obtained; wherein N is an integer greater than 2;

[0049] a state-action feature tensor is constructed based on the state history sequence and the action history sequence;

[0050] the state-action feature tensor is input into the target style discriminator to output a discrimination log value as the score result;

[0051] the discrimination log value is monotonically transformed to obtain the imitation learning reward value.

[0052] Optionally, the constructing the state-action feature tensor based on the state history sequence and the action history sequence comprises:

[0053] concatenating the state history sequence according to time steps to obtain a state sequence tensor;

[0054] concatenating the action history sequence according to time steps to obtain an action sequence tensor;

[0055] concatenating the state sequence tensor and the action sequence tensor in a feature dimension to obtain the state-action feature tensor.

[0056] Optionally, the updating the motion control policy network based on the imitation learning reward value comprises:

[0057] performing time-reverse accumulation on the imitation learning reward value according to a preset discount factor to generate a return sequence;

[0058] inputting the current state into a value network to output a state value, and calculating an advantage sequence based on the return sequence and the state value;

[0059] calculating a policy loss based on the advantage sequence and an action probability output by the motion control policy network, and performing back propagation on the policy loss to update parameters of the motion control policy network.

[0060] Optionally, the performing back propagation on the policy loss to update the parameters of the motion control policy network comprises:

[0061] generating a discriminator gating mask vector based on the target style discriminator;

[0062] element-wise multiplying the discriminator gating mask vector and a score result output by each style discriminator to obtain a gated score result;

[0063] performing gradient blocking processing on components in the gated score result whose mask is zero to obtain a blocked gated score result;

[0064] using the blocked gated score result and the policy loss together for back propagation to update the parameters of the motion control policy network.

[0065] According to a second aspect of the present disclosure, a robot motion control method based on imitation learning is provided, comprising:

[0066] obtaining current state data of a robot;

[0067] inputting the current state data into a pre-trained motion control policy network, and outputting an action policy for controlling robot motion based on the motion control policy network;

[0068] The motion control strategy network is obtained according to the robot motion control strategy network training method based on imitation learning in the first aspect.

[0069] According to a third aspect of the present disclosure, a robot motion control strategy network training apparatus based on imitation learning is provided, comprising:

[0070] a discriminator construction module configured to construct a plurality of style discriminators, each of which is trained based on reference action data of a corresponding target style and used to evaluate the consistency between the output action of the motion control strategy network and the target style;

[0071] a reward value calculation module configured to determine a target style discriminator corresponding to the current state of the robot, and take the scoring result of the target style discriminator on the current state and the current action of the robot as the imitation learning reward value at the current time;

[0072] a strategy network updating module configured to perform parameter updating on the motion control strategy network based on the imitation learning reward value, so as to guide the motion control strategy network to learn the action features of the corresponding style.

[0073] According to a fourth aspect of the present disclosure, a robot motion control apparatus based on imitation learning is provided, comprising:

[0074] a data acquisition module configured to acquire current state data of the robot;

[0075] a control strategy module configured to input the current state data into the pre-trained motion control strategy network, and output an action strategy for controlling the motion of the robot based on the motion control strategy network;

[0076] The motion control strategy network is obtained according to the robot motion control strategy network training method based on imitation learning in the first aspect.

[0077] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0078] a processor; and

[0079] a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of the first aspect or the second aspect.

[0080] According to a sixth aspect of the present disclosure, a robot is provided, comprising:

[0081] a processor; and

[0082] a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of the second aspect.

[0083] Optionally, the robot includes any one of a foot robot, a quadruped robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot, and a robotic arm.

[0084] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method of the first aspect or the second aspect is implemented.

[0085] According to the technical solution, the present disclosure has at least one of the following advantages and positive effects:

[0086] The robot motion control strategy network training method based on imitation learning in the exemplary embodiments of the present disclosure trains multiple independent style discriminators using targeted reference motion data, effectively decouples the dependency relationship of different motion styles in the feature space, and further avoids the feature competition problem caused by the use of a shared network in the related art, so that the discriminator can focus on single style feature extraction. Even in the scene where the task spaces of walking and running are highly overlapped, the motion details can be accurately distinguished.

[0087] On this basis, the target style discriminator is dynamically locked according to the current state, and the score of the target discriminator is used as the only imitation learning reward value, so that the optimization gradient of the policy network is provided with directionality, single and explicit in the training process. The evaluation signal ambiguity at the style boundary is fundamentally eliminated, and the policy network can learn a clear style switching boundary.

[0088] Further, the motion control strategy network can output stable motion instructions according to the explicit control strategy during the motion transition phase after the robot is actually deployed, effectively suppressing unstable phenomena such as gait disorder, high-frequency jitter, or transition stiffness caused by decision swings. BRIEF DESCRIPTION OF DRAWINGS

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0090] Figure 1 is a system architecture diagram provided by an embodiment of the present disclosure;

[0091] Figure 2 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0092] Figure 3 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0093] Figure 4 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0094] Figure 5 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0095] Figure 6 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0096] Figure 7 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0097] Figure 8 is a flowchart of a robot motion control strategy network training method based on imitation learning provided by an embodiment of the present disclosure;

[0098] Figure 9 is a flowchart of a robot motion control method based on imitation learning provided by an embodiment of the present disclosure;

[0099] Figure 10 is a structural schematic diagram of a robot motion control strategy network training device based on imitation learning provided by an embodiment of the present disclosure;

[0100] Figure 11 is a structural schematic diagram of a robot motion control device based on imitation learning provided by an embodiment of the present disclosure;

[0101] Figure 12 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure;

[0102] Figure 13 is a schematic diagram of a first type of robot provided by an embodiment of the present disclosure;

[0103] Figure 14 is a schematic diagram of a second type of robot provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0104] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the schemes of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0105] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other different manners from those described herein; obviously, the embodiments described in the specification are only a part of the embodiments of the present disclosure, and not all the embodiments.

[0106] It should be understood that each step recorded in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0107] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between or among the entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a... " does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0108] It should be noted that the modification of "one" and "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0109] In order to better understand the inventive concept of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be described below in conjunction with exemplary embodiments.

[0110] First, the related terms involved in the embodiments of the present disclosure are explained and described.

[0111] Style can be understood as action style, that is, action performance with representative characteristics or appearance.

[0112] The style discriminator can be understood as a network or model for identifying action style.

[0113] The state in the state-action pair can be understood as the ontology state data and the external state data perceived by the robot.

[0114] Positive sample state-action pair can be understood as a state-action pair composed of the state and action of the robot that matches the reference action data of the target style.

[0115] Negative sample state-action pair can be understood as a state-action pair composed of the state and action of the robot that does not match the reference action data of the target style.

[0116] Discriminator loss can be understood as a joint loss function for measuring the discriminant ability of the discriminator network to distinguish target style actions from non-target style actions and the stability of its scoring function.

[0117] Adversarial classification loss can be understood as a loss function constructed based on the difference between the scoring results of the discriminator output and the labels of positive samples and negative samples.

[0118] Interpolation sample can be understood as a transition sample between target style action and non-target style action.

[0119] State history sequence can be understood as a state sequence composed of state data collected by the robot in chronological order within a plurality of time steps before the current time step.

[0120] Action history sequence can be understood as an action sequence composed of control actions actually executed by the robot or output by the policy network in chronological order within the time range corresponding to the state history sequence.

[0121] Discrimination logarithmic value can be understood as a value for representing the degree of consistency of the state-action feature tensor with the target style.

[0122] Style transition buffer can be understood as a transition state region set in the critical state interval of action style switching.

[0123] Figure 1 A system architecture diagram showing the imitation learning-based robot motion control policy network training method and imitation learning-based robot motion control method in the embodiments of the present disclosure is shown.

[0124] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a robot 102, a network 103 and a server 104. The terminal device 101 includes but is not limited to desktop computers, portable computers, smartphones and tablet computers, etc. The terminal device 101 is configured with a graphical user interface, which can provide visualization functions to display the running state, motion trajectory, etc. of the robot 102, and can also be used to configure training parameters, control the training process and monitor the model training state, and support sending motion control instructions to the robot 102.

[0125] The robot 102 is equipped with various sensors such as inertial measurement devices, joint position sensors, speed sensors, angle sensors, etc. for real-time acquisition of state data during movement.

[0126] The server 104 is deployed with a training module for implementing the robot motion control policy network training method based on imitation learning, which can train the motion control policy network. After training is completed, the server 104 can deploy the trained motion control policy network to the robot 102, so that the robot 102 can output the action policy for controlling the robot motion based on the motion control policy network.

[0127] The network 103 is used to provide a communication link medium between the terminal device 101, the robot 102 and the server 104. The network 103 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc. It should be understood that, Figure 1 The number and type of terminal devices, robots, networks and servers in the system architecture 100 are only illustrative. According to the needs of implementation, there can be any number and any type of terminal devices, robots, networks and servers.

[0128] Through the cooperative operation of each part in the system architecture 100, multiple independent style discriminators can be constructed, and targeted reference action data can be trained respectively, effectively decoupling the dependence of different action styles in the feature space, and then avoiding the feature competition problem caused by the use of shared networks in related technologies, so that the discriminator can focus on single style feature extraction. Even in the scene where the task spaces of walking and running are highly overlapped, the action details can also be accurately distinguished. On this basis, the target style discriminator is dynamically locked according to the current state, and the score of the target discriminator is used as the only imitation learning reward value, so that the policy network can provide a directional, single and clear optimization gradient during the training process, which fundamentally eliminates the evaluation signal ambiguity at the style boundary, and the policy network can learn a clear style switching boundary. Further, after the motion control policy network is actually deployed on the robot, stable action instructions can be output according to the clear control strategy during the action transition phase, effectively suppressing unstable phenomena such as gait disorder, high-frequency jitter or transition stiffness caused by decision swing.

[0129] The embodiment of the present disclosure provides a robot motion control policy network training method based on imitation learning, which can include the following steps 210 to 230: Figure 2 The embodiment of the present disclosure provides a robot motion control policy network training method based on imitation learning, which can include the following steps 210 to 230:

[0130] Step 210, a plurality of style discriminators are constructed, each of which is trained based on reference action data of a corresponding target style, and is used to evaluate the consistency of the output action of the motion control strategy network with the target style.

[0131] Step 220, determining the target style discriminator corresponding to the current state of the robot, and taking the scoring result of the target style discriminator on the current state and the current action of the robot as the imitation learning reward value at the current time.

[0132] Step 230, updating the parameters of the strategy network based on the imitation learning reward value to guide the strategy network to learn the action features of the corresponding style.

[0133] In the embodiments of the present disclosure, the robot motion control strategy network training method based on imitation learning constructs a plurality of independent style discriminators, respectively trains them with targeted reference action data, effectively decouples the dependency relationship of different action styles in the feature space, and further avoids the feature competition problem caused by the use of shared networks in related technologies, so that the discriminator can focus on single style feature extraction. Even in the scene where the task spaces of walking and running are highly overlapped, the action details can also be accurately distinguished. On this basis, the target style discriminator is dynamically locked according to the current state, and the score of the target discriminator is taken as the only imitation learning reward value, so that the strategy network can be provided with a directional single and clear optimization gradient during the training process. The evaluation signal ambiguity at the style boundary is fundamentally eliminated, and the strategy network can learn a clear style switching boundary. Further, after the motion control strategy network is actually deployed on the robot, it can output stable action instructions according to the clear control strategy in the action transition phase, effectively suppressing the unstable phenomena such as gait disorder, high-frequency jitter or transition stiffness caused by decision swing.

[0134] In the following, the robot motion control strategy network training method based on imitation learning in the embodiments of the present disclosure will be described in detail.

[0135] In step 210, a plurality of style discriminators are constructed, each of which is trained based on reference action data of a corresponding target style, and is used to evaluate the consistency of the output action of the motion control strategy network with the target style.

[0136] Wherein, the style can be understood as the action style, that is, the action performance with representative characteristics or appearance. For example, the action style can include walking, running, limping and other action performances with clear characteristics.

[0137] The style discriminator can be understood as a network or model for discriminating the action style.

[0138] In the embodiments of the present disclosure, a plurality of style discriminators can be constructed, each of which is trained based on reference action data of a target style corresponding to the style discriminator, and each of which can be used to evaluate the consistency of the output action of the motion control strategy network with the target style corresponding to the style discriminator.

[0139] That is, each style discriminator can discriminate one action style, and one style discriminator can be trained based on reference action data of a target style corresponding to the style discriminator, and the style discriminator can be used to evaluate the consistency of the output action of the motion control strategy network with the target style corresponding to the style discriminator.

[0140] The reference action data of the target style can be understood as standard action data of the target style.

[0141] In some embodiments, the above-mentioned construction of a plurality of style discriminators can be performed Figure 3 The provided robot motion control strategy network training method based on imitation learning can include the following steps: Figure 3 As shown in the figure, the method can include the following steps:

[0142] Step 310, sampling a positive sample state-action pair from the reference action data of the target style, and sampling a negative sample state-action pair from the reference action data of a non-target style.

[0143] The state in the state-action pair can be understood as the body state data and the external state data perceived by the robot. For example, the body state data can include joint position, joint speed, body linear speed, angular speed, attitude angle, contact phase, support foot information, etc. The external state data can include terrain data, etc.

[0144] The action in the state-action pair can be understood as the control action performed by the robot in a certain state.

[0145] The positive sample state-action pair can be understood as a state-action pair composed of the state and action of the robot matching the reference action data of the target style.

[0146] The negative sample state-action pair can be understood as a state-action pair composed of the state and action of the robot not matching the reference action data of the target style.

[0147] Step 320, inputting the positive sample state-action pair and the negative sample state-action pair into the discriminator network structure to be trained to output a score result.

[0148] In the embodiments of the present disclosure, the positive sample state-action pair and the negative sample state-action pair can be input into a discriminator network structure to be trained, and the positive sample state-action pair and the negative sample state-action pair are scored based on the discriminator network structure to obtain a score result.

[0149] The score result can represent a confidence degree of the state-action pair belonging to the target style or a consistency degree with the reference action distribution of the target style.

[0150] The non-target style can be understood as not being the target style or not belonging to the target style.

[0151] In step 330, a discriminator loss is calculated based on the score result and a target label, and parameters of the discriminator network structure are updated to obtain a style discriminator corresponding to the target style. The target label includes a label of the positive sample and a label of the negative sample.

[0152] In the embodiments of the present disclosure, based on the score result output by the discriminator network for the positive sample state-action pair and the negative sample state-action pair, and in combination with the label information of the positive sample and the negative sample, a discriminator loss used to represent the ability of the discriminator to distinguish the target style action from the non-target style action is calculated. The discriminator loss is back propagated to update the parameters of the discriminator network structure, so that the updated discriminator outputs a higher score in the state-action region corresponding to the target style reference action and outputs a lower score in the non-target style action region, thereby obtaining a style discriminator corresponding to the target style. That is, the supervision signal that the positive sample “should be like the target style” and the negative sample “should not be like the target style” can be used to constrain the score distribution output by the discriminator, so that the discriminator gives a high score in the target style state-action region and a low score in the non-target style region. Therefore, the parameters of the discriminator network structure are continuously adjusted through back propagation, and finally a special style discriminator sensitive only to the “target style” is obtained.

[0153] For example, the label semantics of the positive sample can be “belonging to the target style”, and the label of the positive sample can be represented by “1”.

[0154] The label semantics of the negative sample can be “not belonging to the target style”, and the label of the negative sample can be represented by “0”.

[0155] In some embodiments, the calculation of the discriminator loss based on the score result and the target label can be calculated by the following calculation method:

[0156]

[0157] wherein, is the discriminator loss; is an adversarial classification loss calculated based on the score result, the label of the positive sample, and the label of the negative sample. is a preset penalty coefficient; is an interpolation sample; is an output score of the discriminator network structure on the interpolation sample; is a gradient of the output score on the interpolation sample; represents is a norm regularization term; represents an expectation of the distribution of the interpolation sample .

[0158] In the above calculation method, based on the score results output by the discriminator network on the positive sample state-action pair and the negative sample state-action pair, the label information of the positive sample and the negative sample are combined to calculate the adversarial classification loss for constraining the discrimination ability of the discriminator; at the same time, the interpolation sample is constructed between the positive sample and the negative sample, and the gradient norm of the output score of the discriminator on the interpolation sample with respect to the input is regularized to constrain the change amplitude of the score function of the discriminator in the state-action space; the adversarial classification loss and the gradient norm regularization term are weighted and summed according to a preset penalty coefficient to obtain the discriminator loss, and the discriminator network structure parameters are updated based on the discriminator loss.

[0159] The discriminator loss can be understood as a joint loss function for measuring the discrimination ability of the discriminator network on the target style action and the non-target style action and the stability of the score function thereof, which guides the update of the discriminator network structure parameters by simultaneously constraining the matching relationship between the output score of the discriminator and the sample style label, and constraining the change amplitude of the score function in the state-action space. The discriminator loss is used to represent the comprehensive performance of the discriminator network in distinguishing the target style action from the non-target style action, which constrains the discrimination ability of the discriminator by constructing an adversarial classification loss based on the score results output by the discriminator and the style labels of the positive sample and the negative sample, and limits the change amplitude of the score function in the state-action space by regularizing the gradient norm of the output score of the discriminator with respect to the input; the adversarial classification loss and the regularization loss are combined according to a preset penalty coefficient to obtain the discriminator loss for updating the discriminator network structure parameters.

[0160] Adversarial classification loss can be understood as a loss function built based on the difference between the discriminator's output rating and the labels of positive and negative samples. It constrains the discriminator to output higher ratings for target style state-action pairs and lower ratings for non-target style state-action pairs, thereby enhancing the discriminator's ability to distinguish target style actions. Adversarial classification loss characterizes the difference between the discriminator's output rating and the true style labels of the samples. It applies opposite optimization constraints to positive sample state-action pairs from the target style and negative sample state-action pairs from the non-target style, causing the discriminator to output higher ratings in the target style state-action region and lower ratings in the non-target style region, thus enhancing the discriminator's ability to distinguish target style actions.

[0161] Interpolated samples can be understood as transitional samples between target style actions and non-target style actions.

[0162] The gradient of the output score with respect to the interpolated samples can be understood as a measure of the degree of change in the score result when there are small changes in state-action.

[0163] Norm regularization can constrain the shape of a function, forcing the discriminator scoring function to be neither too steep nor too flat in the state-action space.

[0164] Therefore, multiple style discriminators can be built accurately and quickly.

[0165] In some embodiments, the above-described construction of multiple style discriminators can be executed. Figure 4 This provides a method for training robot motion control policy networks based on imitation learning, such as... Figure 4 As shown, the method may include the following steps:

[0166] Step 410: Construct a state-action encoding subnetwork for each target style and configure independent network parameters for each state-action encoding subnetwork.

[0167] Step 420: Construct a style scoring subnetwork for each target style and configure independent network parameters for each style scoring subnetwork.

[0168] Step 430: Connect the state action encoding subnetwork and the style scoring subnetwork corresponding to the same target style to obtain the style discriminator corresponding to the target style.

[0169] The state-action encoding subnetwork can be understood as a network used to extract joint features from the robot's raw state data and action data to generate a state-action feature representation that characterizes the robot's motion behavior.

[0170] The style score sub-network can be understood as a network for outputting a score result for representing the consistency degree of the current state-action pair with the target style based on the state-action feature representation.

[0171] For each target style, the robot state and action can be first mapped to a style-related feature by the state-action encoding sub-network, and then the feature is input into the style score sub-network corresponding to the target style to output a score value for representing the consistency degree of the state-action with the target style; thereby forming a style discriminator for the target style. The state-action encoding sub-network and the style score sub-network corresponding to the same target style are connected to form a style discriminator for the target style, thereby realizing independent discrimination of actions of different target styles.

[0172] Thus, by decoupling the "state-action feature extraction" and the "style consistency score" into two functionally explicit sub-networks, the style discriminator can not only stably extract the motion features related to the style, but also form independent and controllable scoring functions for different target styles, thereby avoiding mutual interference of different style features in the same network and improving the accuracy of the style discriminator construction.

[0173] In step 220, a target style discriminator corresponding to the current state of the robot is determined, and the score result of the target style discriminator on the current state and the current action of the robot is taken as the imitation learning reward value at the current time.

[0174] In the embodiments of the present disclosure, at each time step, the target action style corresponding to the current state of the robot can be determined, and the target style discriminator corresponding to the target action style is selected from the plurality of style discriminators; the output score of the target style discriminator on the current state and the current action is taken as the imitation learning reward value at the time step, which is used to guide the motion control policy network to generate a control action conforming to the target action style.

[0175] For example, the current state of the robot can include at least one or more of the linear velocity, the angular velocity of the robot, and the slope of the terrain where the robot is located.

[0176] In some embodiments, the above determination of the target style discriminator corresponding to the current state of the robot can be performed Figure 5 Provided is a method for training a robot motion control policy network based on imitation learning, as shown in Figure 5 The method can include the following steps:

[0177] Step 510, extracting a state quantity from the current state of the robot.

[0178] The state quantity can be understood as one or more physical quantities or characteristic quantities extracted from the current state of the robot, used to reflect the motion of the robot, and used as a basis for determining the target action style.

[0179] Step 520, determining the target style index corresponding to the state quantity according to the preset state quantity-style index mapping rule.

[0180] The state quantity and the style index have a mapping relationship, and the style index and the style discriminator have a mapping relationship.

[0181] The target style index can be understood as an index used to identify the target action style corresponding to the state quantity of the robot in a plurality of predefined action styles.

[0182] Step 530, selecting a target style discriminator from a plurality of style discriminators based on the target style index.

[0183] The target style discriminator corresponding to the target style index can be determined as the target style discriminator.

[0184] Therefore, the target style discriminator corresponding to the current state can be accurately and quickly determined according to the target style index corresponding to the state quantity in the current state.

[0185] In some embodiments, the above-mentioned score result of the target style discriminator on the current state and the current action of the robot as the imitation learning reward value at the current time can be executed Figure 6 The provided robot motion control strategy network training method based on imitation learning can include the following steps: Figure 6

[0186] Step 610, obtaining a state history sequence and an action history sequence of the robot containing the last N time steps; wherein N is an integer greater than 2.

[0187] The state history sequence can be understood as a state sequence formed by the state data collected by the robot in the continuous time steps before the current time step in chronological order.

[0188] The action history sequence can be understood as an action sequence formed by the control actions actually executed by the robot or output by the policy network in the time range corresponding to the state history sequence in chronological order.

[0189] Step 620, constructing a state-action feature tensor based on the state history sequence and the action history sequence.

[0190] In some embodiments, the above-mentioned construction of the state-action feature tensor based on the state history sequence and the action history sequence can include steps 6201-6203: ​

[0191] Step 6201, concatenating the state history sequence by time steps to obtain a state sequence tensor.

[0192] Step 6202, concatenating the action history sequence by time steps to obtain an action sequence tensor.

[0193] Step 6203, concatenating the state sequence tensor and the action sequence tensor in the feature dimension to obtain a state-action feature tensor.

[0194] The state-action feature tensor can be understood as a unified numerical representation of the continuous motion behavior of the robot in the last N time steps.

[0195] Step 630, inputting the state-action feature tensor into the target style discriminator to output a discriminant numerical value as a score result.

[0196] In the embodiments of the present disclosure, the state-action feature tensor constructed by the state history sequence and the action history sequence can be input into the target style discriminator corresponding to the current state, the state-action feature tensor is discriminated by the target style discriminator, a discriminant numerical value used to represent the consistency degree of the state-action feature tensor and the target style reference action distribution is output, and the discriminant numerical value is taken as the score result of the target style discriminator.

[0197] The discriminant numerical value can be understood as a value used to represent the consistency degree of the state-action feature tensor and the target style.

[0198] Step 640, monotonically transforming the discriminant numerical value to obtain an imitation learning reward value.

[0199] The discriminant numerical value is the original discriminant result output by the target style discriminator for the state-action feature tensor, and the numerical range and scale thereof are not directly applicable to the policy network training; therefore, a monotonically transformed discriminant numerical value is obtained by performing a monotonically transformed discriminant numerical value on the discriminant numerical value, which maintains the numerical size relationship unchanged, so as to obtain a reward value used for imitation learning, so that the state-action behavior with a higher consistency degree with the target style corresponds to a greater reward value, thereby guiding the motion control policy network to learn the action mode consistent with the target style.

[0200] Thus, by acquiring the state history sequence and the action history sequence containing a plurality of recent time steps, and constructing the state-action feature tensor based on the history sequence, the style discrimination is based on the time sequence features of the continuous motion behavior of the robot, thereby effectively suppressing the influence of single-frame noise on the discrimination result. Further, the state-action feature tensor is input into the target style discriminator to output a discrimination pair value, and a monotonic transformation is performed on the discrimination pair value to generate an imitation learning reward value, thereby obtaining a numerical stable and continuously controllable reward signal while keeping the relative order of the style discrimination unchanged, thereby improving the stability, style consistency and smoothness during action style switching of the motion control policy network training process.

[0201] In some embodiments, the above determining the target style discriminator corresponding to the current state of the robot, taking the score result of the target style discriminator on the current state and the current action of the robot as the imitation learning reward value at the current time, can be performed Figure 7 The provided robot motion control policy network training method based on imitation learning can include the following steps. Figure 7 As shown in the figure, the method can include the following steps:

[0202] Step 710, acquiring a style transition buffer.

[0203] The style transition buffer can be understood as a transition state region set in the critical state region of action style switching, which is used to smoothly blend multiple adjacent action styles in this region to avoid discontinuous reward signals or control policy jitter caused by sudden style judgment at the style switching boundary.

[0204] In some embodiments, the above acquiring the style transition buffer can include S11-S13:

[0205] S11, calculating the time difference of the current linear velocity scalar of the robot corresponding to the current state to obtain a velocity change amount, and calculating a buffer width parameter based on the velocity change amount.

[0206] The current linear velocity scalar can be understood as the linear velocity of the robot at the current time. For example, the current linear velocity scalar can be walking 0.5 m / s, running 2.0 m / s.

[0207] The time difference can be understood as approximating the change trend of a certain physical quantity with the difference between adjacent time steps.

[0208] The time difference of the current linear velocity scalar of the robot corresponding to the current state is calculated to obtain a velocity change amount, that is, the difference between the linear velocity scalar at the current time step and the linear velocity scalar at the previous time step, which is the velocity change amount.

[0209] The buffer width parameter can be understood as a span parameter of the style transition interval on the speed axis. The buffer width parameter is used to represent the "width and narrowness" of the style transition interval, which is adaptively adjusted according to the time variation of the linear speed of the robot: the more drastic the speed variation, the wider the buffer; the more smooth the speed variation, the narrower the buffer.

[0210] By time-differencing the current linear speed scalar of the robot at the current time step and the linear speed scalar at the last time step, the speed variation is obtained, and the width parameter of the style transition buffer is calculated based on the speed variation through a monotonic mapping relationship, so that a larger buffer width corresponds to a more drastic speed variation, and a smaller buffer width corresponds to a more smooth speed variation.

[0211] For example, the buffer width parameter can be 0.6 m / s.

[0212] S12, respectively generating a lower limit threshold of the buffer and an upper limit threshold of the buffer according to the buffer width parameter.

[0213] In the embodiments of the present disclosure, a preset style switching center threshold can be obtained, and then based on the style switching center threshold and the buffer width parameter, the lower limit threshold of the buffer and the upper limit threshold of the buffer are generated.

[0214] The style switching center threshold is used to represent the reference benchmark value of the switching of two adjacent action styles in a certain key state quantity dimension. The threshold is used as the center point of the style transition buffer, and is used to generate the corresponding upper and lower limits of the buffer. For example, the style switching center threshold represents the typical position of the transition from style A to style B. The style switching center threshold can be preset according to the design parameters of the robot or task experience.

[0215] For example, the style switching center threshold of walking to running is 1.5 m / s, the buffer width parameter is 0.6 m / s, the half width parameter of the buffer is 0.3 m / s, the lower limit threshold of the buffer is 1.2 m / s (1.5 m / s-0.3 m / s=1.2 m / s), and the upper limit threshold of the buffer is 1.8 m / s (1.5 m / s+0.3 m / s=1.8 m / s).

[0216] S13, determining the buffer composed of the lower limit threshold and the upper limit threshold as the style transition buffer.

[0217] For example, if the lower limit threshold of the buffer is 1.2 m / s and the upper limit threshold of the buffer is 1.8 m / s, the style transition buffer of walking to running is 1.2 m / s~1.8 m / s.

[0218] Step 720, when the current state is in the style transition buffer, determining that the two target style discriminators corresponding to the current state are both hybrid style discriminators, and determining the weight coefficients of each hybrid style discriminator.

[0219] The style transition buffer is located between two adjacent styles. When the current state is in the style transition buffer, two styles on both sides of the style transition buffer, style A and style B, are determined, the target style discriminator corresponding to style A is determined as a hybrid style discriminator, the target style discriminator corresponding to style B is also determined as a hybrid style discriminator, two hybrid style discriminators are obtained, and then the weight coefficients of each hybrid style discriminator are calculated according to the position of the current state in the style transition buffer.

[0220] The weight coefficient of the hybrid style discriminator can be used to represent the relative membership degree of the current state of the robot in the style transition buffer to the adjacent action style, which is determined according to the position of the current state in the style transition buffer and continuously changes with the change of the current state; the weight coefficient is used to weight the scoring results of the corresponding hybrid style discriminator to realize the smooth transition of the imitation learning reward between different action styles.

[0221] For example, assuming that the lower limit of the style transition buffer is biased towards the walking style and the upper limit is biased towards the running style, if the current state is close to the lower limit of the style transition buffer, the weight of the walking discriminator is large and the weight of the running discriminator is small; if the current state is close to the upper limit of the style transition buffer, the weight of the running discriminator is large and the weight of the walking discriminator is small.

[0222] Therefore, when the current state of the robot falls into the style transition buffer between different action styles, instead of selecting a single target style discriminator, the style discriminators corresponding to the two target styles adjacent to the style transition buffer are selected, and both of the two style discriminators are determined as hybrid style discriminators; at the same time, according to the position of the current state in the style transition buffer, weight coefficients are assigned to the two hybrid style discriminators for subsequent weighted fusion of the discriminator scoring results, so as to realize the smooth transition of the action style.

[0223] In some embodiments, the above-mentioned determination that the two target style discriminators corresponding to the current state are both hybrid style discriminators and the determination of the weight coefficients of each hybrid style discriminator can include S21-S24:

[0224] S21, obtaining a target style index corresponding to the current state.

[0225] The style index corresponding to the state quantity in the current state can be determined as the target style index.

[0226] S22, determining a main style discriminator based on the target style index.

[0227] The style discriminator corresponding to the target style index can be used as the main style discriminator.

[0228] S23, determining the auxiliary style discriminator adjacent to the main style discriminator based on the style transition buffer.

[0229] S24, determining the main style discriminator and the auxiliary style discriminator as the hybrid style discriminator, and calculating the weight coefficient assigned to the main style discriminator and the auxiliary style discriminator based on the current state.

[0230] In some embodiments, the above-mentioned determination of the weight coefficient of each hybrid style discriminator can include S31-S34:

[0231] S31, obtaining the lower threshold value, the upper threshold value of the style transition buffer, and the current linear velocity scalar of the robot corresponding to the current state.

[0232] S32, calculating a first difference value according to the current linear velocity scalar and the lower threshold value, and calculating a second difference value according to the upper threshold value and the lower threshold value.

[0233] The first difference value is the difference between the current linear velocity scalar and the lower threshold value.

[0234] The second difference value is the difference between the upper threshold value and the lower threshold value.

[0235] S33, calculating a first non-normalized weight according to the ratio of the first difference value and the second difference value, and taking the complementary weight of the first non-normalized weight as a second non-normalized weight.

[0236] The first non-normalized weight is the ratio of the first difference value and the second difference value.

[0237] For example, if the first non-normalized weight is 0.3, the second non-normalized weight is 0.7.

[0238] S34, normalizing the first non-normalized weight and the second non-normalized weight to obtain the weight coefficient of each hybrid style discriminator.

[0239] The normalization of the first non-normalized weight and the second non-normalized weight unifies the scaling of the first non-normalized weight and the second non-normalized weight to the same scale, so that it can not only reflect the relative importance of the two styles, but also meet the numerical constraint that can be directly used for weighted fusion, thereby obtaining the weight coefficient of the hybrid style discriminator used for reward calculation.

[0240] Step 730, respectively calculating the score results of the two hybrid style discriminators on the current state and the current action.

[0241] Step 740, weighting and summing the score results according to the weight coefficients of the two mixed style discriminators to obtain the imitation learning reward value at the current time.

[0242] For example, the score result a of the mixed style discriminator A for the current state and the current action is calculated; the score result b of the mixed style discriminator B for the current state and the current action is calculated; the product k1 of the score result a and the weight coefficient of the mixed style discriminator A is calculated, the product k2 of the score result b and the weight coefficient of the mixed style discriminator B is calculated, and the sum of k1 and k2 is taken as the imitation learning reward value at the current time.

[0243] Therefore, by setting the style transition buffer and determining the style discriminators corresponding to adjacent action styles as mixed style discriminators in the buffer, assigning weight coefficients to the mixed style discriminators according to the current state, and weighting and fusing the score results of each mixed style discriminator, a continuous and smooth imitation learning reward value is generated, thereby effectively avoiding the policy jitter problem caused by the sudden change of the reward signal at the style switching boundary, and significantly improving the stability and naturalness of the action style switching of the motion control policy network in the multi-style training and execution process.

[0244] In step 230, the motion control policy network is updated based on the imitation learning reward value to guide the motion control policy network to learn the action features corresponding to the style.

[0245] The policy loss function for optimizing the motion control policy network can be constructed based on the imitation learning reward value output by the target style discriminator, and the parameters of the motion control policy network are updated by performing back propagation on the policy loss function, so that the action that can obtain a higher imitation learning reward value under the same or similar state condition is given a higher output probability in the subsequent training process, thereby guiding the motion control policy network to gradually learn and internalize the action features corresponding to the target style.

[0246] The action features include but are not limited to step frequency, step length, action rhythm, joint coordination relationship, contact phase mode, acceleration and deceleration mode, etc.

[0247] In some embodiments of the present disclosure, the above parameter updating of the policy network based on the imitation learning reward value to guide the policy network to learn the action features corresponding to the style can be performed Figure 8 A flowchart of a robot motion control policy network training method based on imitation learning is shown, as shown in FIG. 8, the method can include the following steps: Figure 8

[0248] Step 810, accumulating the imitation learning reward value in reverse according to time with a preset discount factor to generate a return sequence.

[0249] ​In the training process, for each time step, a corresponding imitation learning reward value is obtained by the target style discriminator or the mixed style discriminator. In one training trajectory, the imitation learning reward values are reversely accumulated in time sequence according to a preset discount factor to generate a reward sequence.

[0250] The reward sequence can be understood as a sequence of reward values obtained by reversely accumulating the imitation learning reward values of each time step and the time steps after the time step as the starting point in a training trajectory according to a preset discount factor. The reward sequence can be used to represent the long-term cumulative reward that can be obtained by performing the current action in the corresponding state.

[0251] In step 820, the current state is input into the value network to output a state value, and an advantage sequence is calculated based on the reward sequence and the state value.

[0252] In the embodiment of the present disclosure, after obtaining the reward sequence, the current state of the corresponding time step is input into the value network to output the state value corresponding to the current state, and then the advantage sequence is calculated based on the reward sequence and the state value.

[0253] The state value can be used to estimate the expected long-term reward in the current state.

[0254] The advantage sequence can be used to represent the performance of the current action relative to the average policy.

[0255] In step 830, a policy loss is calculated based on the advantage sequence and the action probability output by the motion control policy network, and the policy loss is back propagated to update the parameters of the motion control policy network.

[0256] In the embodiment of the present disclosure, after the advantage sequence is calculated, the advantage sequence is combined with the action probability output by the motion control policy network in the corresponding state to construct a policy loss function for optimizing the policy network.

[0257] Specifically, for each time step, a policy loss term is calculated according to the action probability output by the policy network and the corresponding advantage value, and the policy loss is accumulated or expected in the time dimension to obtain the overall policy loss. Then, the policy loss is back propagated to update the parameters of the motion control policy network, so that the action that can obtain a higher imitation learning reward value in the same or similar state condition has a higher output probability in the subsequent training process, thereby gradually guiding the motion control policy network to learn the motion control policy conforming to the target action style.

[0258] Therefore, by the method of the above embodiment, the imitation learning reward generated based on the style discriminator can be effectively integrated into the reinforcement learning training process, so that the policy network gradually internalizes the time sequence features and control rules of the target action style while ensuring training stability, and can significantly reduce policy jitter and improve the naturalness and consistency of action generation, especially in the multi-style switching and style transition stage.

[0259] In some embodiments, the above performing back propagation on the policy loss to update the parameters of the motion control policy network can include S41-S44:

[0260] S41, generating a discriminator gate mask vector based on the target style discriminator.

[0261] In the embodiments of the present disclosure, at each time step, the state quantity can be extracted according to the current state of the robot, and the target style index corresponding to the current state can be determined according to the preset state quantity-style mapping rule. After obtaining the target style discriminator, a discriminator gate mask vector with a length of M is constructed, and each dimension element in the gate mask vector corresponds to a style discriminator, which is used to indicate whether the style discriminator participates in scoring at the current time step.

[0262] The discriminator gate mask vector can be understood as a control vector corresponding to the style discriminator one by one, and the discriminator gate mask vector is used to indicate whether the style discriminator corresponding to the mask vector participates in scoring and reward calculation at the current time step.

[0263] S42, multiplying the discriminator gate mask vector and the scoring results output by each style discriminator element by element to obtain a gated scoring result.

[0264] In the embodiments of the present disclosure, the discriminator gate mask vector and the scoring results output by the plurality of style discriminators at the current time step can be multiplied element by element, so that the style discriminator scores irrelevant to the current target style or mixed style are suppressed, while the style discriminator scores corresponding to the current target style or mixed style are retained or scaled by weight, thereby obtaining a gated scoring result for subsequent reward calculation.

[0265] S43, performing gradient blocking processing on the components in the gated scoring result whose mask is zero to obtain a blocked gated scoring result.

[0266] In the embodiments of the present disclosure, the gradient blocking processing can be performed on the components in the gated scoring result whose corresponding gate mask is zero, so that in the back propagation process, the style discriminator and its parameters corresponding to the component do not participate in the gradient update at the current time step, thereby avoiding the gradient interference of the non-target style discriminator on the policy network training process, and obtaining a blocked gated scoring result for subsequent reward calculation and policy update.

[0267] S44, the blocked gating score result is used together with the policy loss for back propagation to update the parameters of the motion control policy network.

[0268] In the embodiments of the present disclosure, in the process of updating the policy network parameters, the policy loss constructed based on the reward sequence and the advantage sequence is jointly included in the back propagation calculation with the gating score result after the gating and gradient blocking processing, so that the update of the policy network parameters is not only driven by the task reward optimization goal, but also constrained by the current target style discriminator score result, thereby ensuring the stability of the training while guiding the motion control policy network to learn the action features consistent with the current target style.

[0269] Therefore, by generating the discriminator gating mask vector based on the target style index, and using the gating mask vector to perform element-by-element gating and gradient blocking processing on the score results of each style discriminator, only the discriminator score result corresponding to the current target style participates in the update of the policy network parameters in the back propagation process, thereby effectively avoiding the gradient interference of the non-target style discriminators on the training process. Meanwhile, the blocked gating score result is used together with the policy loss for back propagation, so that the update of the policy network parameters meets the task optimization goal while being constrained by the target style, guiding the motion control policy network to stably learn the action features consistent with the target style.

[0270] The embodiments of the present disclosure provide a robot motion control method based on imitation learning, which can include the following steps: Figure 9 A flowchart of a robot motion control method based on imitation learning is shown, which can include the following steps:

[0271] Step 910, obtaining current state data of the robot.

[0272] Step 920, inputting the current state data into the pre-trained motion control policy network, and outputting an action policy for controlling the motion of the robot based on the motion control policy network.

[0273] The motion control policy network is obtained according to the robot motion control policy network training method based on imitation learning of any of the above embodiments.

[0274] The embodiments of the present disclosure effectively decouple the dependency relationship of different action styles in the feature space by constructing multiple independent style discriminators and training them respectively using targeted reference action data, thereby avoiding the feature competition problem caused by the shared network in the related art, enabling the discriminators to focus on single style feature extraction, and accurately distinguishing action details even in the scene where tasks such as walking and running are highly overlapped. On this basis, the target style discriminator is dynamically locked according to the current state, and the score of the target discriminator is used as the only imitation learning reward value, so that the policy network can be provided with a directional, single and clear optimization gradient during the training process, fundamentally eliminating the evaluation signal ambiguity at the style boundary, and enabling the policy network to learn a clear style switching boundary. Further, the motion control policy network can output stable action instructions according to the clear control policy during the action transition phase after the robot is actually deployed, effectively suppressing unstable phenomena such as gait disorder, high-frequency jitter or transition stiffness caused by decision swings.

[0275] The embodiments of the present disclosure provide a robot motion control policy network training device based on imitation learning, which comprises Figure 10 A structural schematic diagram of a robot motion control policy network training device based on imitation learning is shown. As shown in Figure 10 The robot motion control policy network training device based on imitation learning 1000 comprises:

[0276] The discriminator construction module 1010 is configured to construct multiple style discriminators, each of which is trained based on reference action data of a corresponding target style and used to evaluate the consistency of the output action of the motion control policy network with the target style.

[0277] The reward value calculation module 1020 is configured to determine the target style discriminator corresponding to the current state of the robot, and take the score result of the target style discriminator on the current state and the current action of the robot as the imitation learning reward value at the current time.

[0278] The policy network updating module 1030 is configured to update the parameters of the motion control policy network based on the imitation learning reward value, so as to guide the motion control policy network to learn the action features of the corresponding style.

[0279] The robot motion control policy network training device based on imitation learning provided by the embodiments of the present disclosure can implement the robot motion control policy network training method based on imitation learning of any of the above embodiments, and has similar implementation modes and beneficial effects, which will not be described here again.

[0280] The embodiments of the present disclosure provide a robot motion control device based on imitation learning, which comprises Figure 11A structural schematic diagram of a robot motion control device based on imitation learning is shown. As shown in Figure 11 The robot motion control device based on imitation learning 1100 includes:

[0281] A data acquisition module 1110 configured to acquire current state data of the robot.

[0282] A control strategy module 1120 configured to input the current state data into a pre-trained motion control strategy network, and output an action strategy for controlling the motion of the robot based on the motion control strategy network.

[0283] The motion control strategy network is obtained according to the robot motion control strategy network training method based on imitation learning of any one of the above embodiments.

[0284] The robot motion control device based on imitation learning provided by the embodiments of the present disclosure can implement the robot motion control method based on imitation learning of any one of the above embodiments, and has similar implementation manners and beneficial effects, which will not be described here again.

[0285] The embodiments of the present disclosure also provide an electronic device including a processor and a memory, wherein the memory stores a computer program which, when executed by the processor, can implement the method of any one of the above embodiments, and has similar implementation manners and beneficial effects, which will not be described here again.

[0286] The electronic device in the embodiments of the present disclosure can be understood as any device with processing and computing capabilities, which can include but is not limited to mobile terminals such as smartphones, notebook computers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, and fixed electronic devices such as digital TVs, desktop computers, and smart home devices.

[0287] Figure 12 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure is shown in Figure 12 As shown in the figure, the electronic device 1200 can include a processor 1210 and a memory 1220, wherein the memory 1220 stores a computer program 1221 which, when executed by the processor 1210, can implement the method provided by any one of the above embodiments, and has similar implementation manners and beneficial effects, which will not be described here again.

[0288] Of course, in order to simplify, Figure 12Only some of the components of the electronic device 1200 related to the present application are shown, and components such as a bus, an input / output interface, an input device, and an output device are omitted. In addition, the electronic device 1200 can include any other appropriate components according to a specific application.

[0289] The embodiments of the present disclosure further provide a robot, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot motion control method based on imitation learning described above can be implemented, and the implementation manner and beneficial effects are similar, which will not be described here in detail.

[0290] The robot can include any one of a foot-type robot, a quadruped robot, a biped robot, a wheeled robot, a wheel-foot robot, a four-wheel-foot robot, a humanoid robot, a cleaning robot, a transport robot, a mobile robot, and a robotic arm. Figures 13-14 As shown in the figure, Figure 13 is a schematic diagram of a first type of robot provided by the embodiments of the present disclosure, Figure 13 The first type of robot 1301 is shown. Figure 14 is a schematic diagram of a second type of robot provided by the embodiments of the present disclosure.

[0291] The embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method of any of the above embodiments can be implemented, and the implementation manner and beneficial effects are similar, which will not be described here in detail.

[0292] The above computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0293] The computer program above can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server.

[0294] The above description is only preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0295] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments.

[0296] The above description is only a specific implementation of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a robot motion control policy network based on imitation learning, characterized in that, include: Multiple style discriminators are constructed, each of which is trained based on reference action data of the corresponding target style, and is used to evaluate the consistency between the output action of the motion control policy network and the target style. Determine the target style discriminator corresponding to the robot's current state, and use the score result of the target style discriminator on the current state and the robot's current action as the imitation learning reward value at the current moment; The motion control policy network is updated with parameters based on the imitation learning reward value to guide it in learning the movement features of the corresponding style.

2. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The step of determining the target style discriminator corresponding to the robot's current state, and using the score result of the target style discriminator for the current state and the robot's current action as the imitation learning reward value at the current moment, includes: Get the style transition buffer; When the current state is in the style transition buffer, it is determined that both target style discriminators corresponding to the current state are hybrid style discriminators and the weight coefficients of each hybrid style discriminator are determined. Calculate the scores of the current state and the current action for the two hybrid style discriminators respectively; The scoring results are weighted and summed according to the weight coefficients of the two hybrid style discriminators to obtain the imitation learning reward value at the current moment.

3. The robot motion control strategy network training method based on imitation learning according to claim 2, characterized in that, The determination that both target style discriminators corresponding to the current state are hybrid style discriminators and the determination of the weight coefficients of each hybrid style discriminator include: Obtain the target style index corresponding to the current state; The primary style discriminator is determined based on the target style index; The auxiliary style discriminator adjacent to the primary style discriminator is determined based on the style transition buffer. The primary style discriminator and the secondary style discriminator are determined as the hybrid style discriminator, and weight coefficients assigned to the primary style discriminator and the secondary style discriminator are calculated based on the current state.

4. The method for training a robot motion control strategy network based on imitation learning according to claim 2, characterized in that, Determining the weight coefficients of each of the hybrid style discriminators includes: Obtain the lower threshold and upper threshold of the style transition buffer, as well as the current linear velocity scalar of the robot corresponding to the current state; A first difference is calculated based on the current linear velocity scalar and the lower threshold, and a second difference is calculated based on the upper threshold and the lower threshold. The first unnormalized weight is calculated based on the ratio of the first difference to the second difference, and the complementary weight of the first unnormalized weight is used as the second unnormalized weight. The first unnormalized weights and the second unnormalized weights are normalized to obtain the weight coefficients of each of the hybrid style discriminators.

5. The method for training a robot motion control strategy network based on imitation learning according to claim 2, characterized in that, The process of obtaining the style transition buffer includes: Calculate the time difference of the robot's current linear velocity scalar corresponding to the current state to obtain the velocity change, and calculate the buffer width parameter based on the velocity change; The lower limit threshold and the upper limit threshold of the buffer are generated based on the buffer width parameter. The buffer formed by the lower threshold and the upper threshold is defined as the style transition buffer.

6. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The construction of multiple style discriminators includes: Positive sample state-action pairs are sampled from the reference action data of the target style, and negative sample state-action pairs are sampled from the reference action data of the non-target style. The positive sample state-action pair and the negative sample state-action pair are input into the discriminator network structure to be trained to output the scoring result; Based on the scoring results and target labels, the discriminator loss is calculated and the parameters of the discriminator network structure are updated to obtain the style discriminator corresponding to the target style. The target labels include labels for positive samples and labels for negative samples.

7. The robot motion control strategy network training method based on imitation learning according to claim 6, characterized in that, The calculation of the discriminator loss based on the scoring result and the target label includes: The discriminator loss is calculated using the following method: in, The loss of the discriminator; The adversarial classification loss is calculated based on the scoring results and the labels of the positive and negative samples. Preset penalty coefficient; For interpolation samples; The discriminator network structure outputs a score for the interpolated sample; The gradient of the output score with respect to the interpolated sample; express Norm regularization term; Indicates the distribution of the interpolated samples The expectation.

8. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The construction of multiple style discriminators includes: For each target style, a state-action encoding subnetwork is constructed, and each state-action encoding subnetwork is configured with independent network parameters. For each target style, a style scoring subnetwork is constructed, and each style scoring subnetwork is configured with independent network parameters. The style discriminator corresponding to the target style is obtained by connecting the state action encoding subnetwork corresponding to the same target style with the style scoring subnetwork.

9. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The target style discriminator that determines the current state of the robot includes: Extract state variables from the current state of the robot; According to the preset state quantity-style index mapping rules, the target style index corresponding to the state quantity is determined; The target style discriminator is selected from the plurality of style discriminators based on the target style index.

10. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The robot's current state includes at least one or more of the robot's linear velocity, angular velocity, and the slope of the terrain in which the robot is located.

11. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The step of using the target style discriminator's score for the current state and the robot's current action as the imitation learning reward value at the current moment includes: Obtain the state history sequence and action history sequence of the robot, which contain the most recent N time steps; where N is an integer greater than 2. Construct a state-action feature tensor based on the state history sequence and the action history sequence; The state-action feature tensor is input into the target style discriminator to output a discrimination log value as the scoring result; The discriminant log value is monotonically transformed to obtain the imitation learning reward value.

12. The method for training a robot motion control strategy network based on imitation learning according to claim 11, characterized in that, The construction of the state-action feature tensor based on the state history sequence and the action history sequence includes: The state history sequence is concatenated step-by-step to obtain the state sequence tensor; The action sequence tensor is obtained by concatenating the action history sequence according to time steps; The state sequence tensor and the action sequence tensor are concatenated along the feature dimension to obtain the state-action feature tensor.

13. The method for training a robot motion control strategy network based on imitation learning according to claim 1, characterized in that, The step of updating the parameters of the motion control policy network based on the imitation learning reward value includes: The imitation learning reward value is accumulated in reverse over time according to a preset discount factor to generate a reward sequence; The current state is input into the value network to output a state value, and an advantage sequence is calculated based on the reward sequence and the state value. The policy loss is calculated based on the dominant sequence and the action probability output by the motion control policy network, and backpropagation is performed on the policy loss to update the parameters of the motion control policy network.

14. The method for training a robot motion control strategy network based on imitation learning according to claim 13, characterized in that, The backpropagation of the policy loss to update the parameters of the motion control policy network includes: Generate a discriminator gate mask vector based on the style index of the target style discriminator; The gated mask vector of the discriminator is multiplied element by element with the scoring results output by each style discriminator to obtain the gated scoring result; Gradient blocking is performed on the components with zero mask in the gated scoring result to obtain the blocked gated scoring result; The blocked gating score and the policy loss are used together in the backpropagation to update the parameters of the motion control policy network.

15. A robot motion control method based on imitation learning, characterized in that, include: Obtain the robot's current state data; The current state data is input into a pre-trained motion control strategy network, and an action strategy for controlling the robot's movement is output based on the motion control strategy network. The motion control policy network is obtained by the robot motion control policy network training method based on imitation learning according to any one of claims 1 to 14.

16. A robot motion control strategy network training device based on imitation learning, characterized in that, include: A discriminator construction module is used to construct multiple style discriminators, each of which is trained based on reference action data of the corresponding target style, and is used to evaluate the consistency between the output action of the motion control policy network and the target style. The reward value calculation module is used to determine the target style discriminator corresponding to the robot's current state, and use the score result of the target style discriminator on the current state and the robot's current action as the imitation learning reward value at the current moment. The policy network update module is used to update the parameters of the motion control policy network based on the imitation learning reward value, so as to guide the motion control policy network to learn the action features of the corresponding style.

17. A robot motion control device based on imitation learning, characterized in that, include: The data acquisition module is used to acquire the robot's current state data; The control strategy module is used to input the current state data into a pre-trained motion control strategy network and output an action strategy to control the robot's movement based on the motion control strategy network. The motion control policy network is obtained by the robot motion control policy network training method based on imitation learning according to any one of claims 1 to 14.

18. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 15.

19. A robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method of claim 15.

20. The robot according to claim 19, characterized in that, The robots include mobile robots.

21. The robot according to claim 19, characterized in that, The robots include humanoid robots.

22. The robot according to claim 19, characterized in that, The robot includes either a legged robot or a wheeled robot.

23. The robot according to claim 19, characterized in that, The robot includes a wheeled robot.

24. The robot according to claim 19, characterized in that, The robot includes any one of quadruped robots, bipedal robots, and four-wheeled robots.

25. The robot according to claim 19, characterized in that, The robot includes either a cleaning robot or a transport robot.

26. The robot according to claim 19, characterized in that, The robot includes a robotic arm.

27. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1 to 15.

Images