Robot control model training method and device, computer readable storage medium and electronic equipment

Abstract

Embodiments of the present application provide a robot control model training method and device, a computer readable storage medium and an electronic device. The method comprises: generating a driving control instruction based on an observed motion sample through a preset policy network; outputting a motion style reward corresponding to the driving control instruction through a preset discriminator; determining a state value score of the observed motion sample based on the motion style reward and a driving control reward of the driving control instruction through a preset value network; updating the preset policy network based on the state value score through a preset objective function, and taking the updated preset policy network as a robot control model. The technical solution of the embodiments of the present application can constrain the robot control model in advance based on preset expert data across motion modalities, so that the robot control model generates a driving control instruction under the constraint of cross-modal motion style, thereby improving the motion fluency, motion safety and control robustness of a multi-motion modality robot.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a method, apparatus, computer-readable storage medium, and electronic device for training a robot control model. Background Technology

[0002] With the continuous development of science and technology, people's demand for robots in different scenarios continues to grow.

[0003] In related technologies, robots are typically configured with multiple motion modes to meet the work requirements in different scenarios, such as legged motion mode, wheeled motion mode, and flight motion mode. However, when controlling robots with multiple motion modes, finite state machines are often used to control the robot to discretely switch between different motion modes according to environmental conditions.

[0004] However, the switching of finite state machines relies on hard threshold judgments, and there are abrupt changes in control commands at the modal boundaries, which leads to discontinuous motion trajectories of the robot, which can easily cause the robot to lose balance or fall, seriously endangering motion safety. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a method, apparatus, computer-readable storage medium, and electronic device for training a robot control model.

[0006] According to one aspect of the embodiments of this application, a method for training a robot control model is provided, comprising: generating drive control commands based on observed motion samples through a preset policy network; outputting motion style rewards corresponding to the drive control commands through a preset discriminator, wherein the motion style rewards are authenticity evaluations output by the preset discriminator after matching the drive control commands with preset expert data, and the preset expert data includes benchmark motion samples of at least two motion modes; determining a state value score of the observed motion samples based on the motion style rewards and drive control rewards of the drive control commands through a preset value network; updating the preset policy network based on the state value scores through a preset objective function, and using the updated preset policy network as a robot control model.

[0007] According to one aspect of the embodiments of this application, a training device for a robot control model is provided, comprising: a policy network unit configured to generate drive control commands based on observed motion samples through a preset policy network; a discriminator unit configured to output a motion style reward corresponding to the drive control command through the preset discriminator, wherein the motion style reward is a authenticity evaluation output by the preset discriminator after matching the drive control command with preset expert data, and the preset expert data includes benchmark motion samples of at least two motion modes; a value network unit configured to determine a state value score of the observed motion sample based on the motion style reward and the drive control reward of the drive control command through a preset value network; and a model generation unit configured to update the preset policy network based on the state value score through a preset objective function, and use the updated preset policy network as a robot control model.

[0008] In some embodiments of this application, based on the foregoing scheme, the discriminator unit is further configured to: acquire the current motion rate corresponding to the observed motion sample; determine a reward fusion coefficient based on the current motion rate; adjust the motion style reward and the driving control reward based on the reward fusion coefficient, so that the preset value network determines the state value score of the observed motion sample based on the adjusted motion style reward and driving control reward, wherein the adjustment direction of the motion style reward under the reward fusion coefficient is positively correlated with the magnitude of the current motion rate, and the adjustment direction of the driving control reward under the reward fusion coefficient is negatively correlated with the magnitude of the current motion rate.

[0009] In some embodiments of this application, based on the foregoing scheme, the discriminator unit is further configured to: after determining the state value score of the observed motion sample based on the motion style reward and the driving control reward of the driving control instruction through a preset value network, determine the target motion mode information in the driving control instruction; and calculate the driving control reward of the driving control instruction according to the preset driving control reward algorithm corresponding to the target motion mode information.

[0010] In some embodiments of this application, based on the foregoing scheme, the formula for the preset discriminator to output the sports style reward is as follows: ; in, Rewards for the aforementioned sports style. For the discriminator network D in the preset discriminator, based on the current robot state and the next robot state The input is the authenticity score of the output, which is used to characterize the current robot state. To the next robot state The degree of matching between the motion changes and the corresponding motion change distribution in the preset expert data.

[0011] In some embodiments of this application, based on the foregoing scheme, the formula for the preset value network to output the state value score is as follows: ; in, Rate the state value; For For the preset strategy network of parameters, parameters Includes the weights and biases in the preset policy network; Rewards for athletic style and drive control rewards sum; To be in a given current robot state Under the conditions, according to the preset strategy network After the output drive control command is executed, the current robot state will be changed. All subsequent rewards After applying the corresponding discount factor Accumulate expectations after the discount.

[0012] In some embodiments of this application, based on the foregoing scheme, the preset objective function includes a near-end strategy optimization algorithm; The near-end strategy optimization algorithm includes the following calculation expression: ; Where L is the total value of the objective function. The strategy loss value. Value loss The entropy reward value, For discriminator loss, For gradient penalty loss, The value loss coefficient, The entropy reward coefficient, This represents the discriminator loss coefficient.

[0013] In some embodiments of this application, based on the foregoing scheme, the strategy network unit is further configured to: determine the observed motion sample from the motion database corresponding to the robot to be controlled before generating the drive control command based on the observed motion sample through the preset strategy network, wherein the motion database includes motion samples of the robot to be controlled in different speed domains.

[0014] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform a training method for a robot control model as described in the above embodiments.

[0015] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the robot control model training method as described in the above embodiments.

[0016] In the technical solution of this application embodiment, a preset policy network first generates driving control commands based on observed motion samples. Then, a preset discriminator outputs the motion style reward corresponding to the driving control command. Next, a preset value network determines the state value score of the observed motion sample based on the motion style reward and the driving control reward of the driving control command. Finally, a preset objective function updates the preset policy network based on the state value score, and the updated preset policy network is used as the robot control model. Since the preset discriminator performs style evaluation and constraints on the robot control model in advance based on preset expert data across motion modalities, the robot control model can achieve a smooth transition between different motion modalities under the constraints of cross-modal motion style when generating driving control commands. This avoids defects such as sudden changes in control commands, discontinuous motion trajectories, and imbalances and falls caused by hard threshold switching, thereby improving the motion smoothness, motion safety, and control robustness of multi-modal robots. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a flowchart illustrating a method for training a robot control model, as shown in an exemplary embodiment of this application. Figure 2 This is a block diagram illustrating a training apparatus for a robot control model, as shown in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0019] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0020] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0021] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0022] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0023] The technical solution of this application proposes a method for training a robot control model, specifically referring to... Figure 1 As shown. The method includes at least steps S110 to S140, which are described in detail below: In step S110, drive control commands are generated based on observed motion samples through a preset strategy network.

[0024] The preset policy network is a neural network that controls the robot's movement, such as an MLP (Multi-Layer Perceptron) or CNN (Convolutional Neural Network). Specifically, in one example, the preset policy network can use a three-layer fully connected MLP as its network architecture. The hidden layers can use ELU (Exponential Linear Unit) as the activation function, and the output layer can output the mean vector of the action, which is then used as a control command to control the robot's movement.

[0025] In another example, the preset policy network can use a one-dimensional convolutional neural network as its network architecture. The hidden layers can use ELU (Exponential Linear Unit) as the activation function, and the output layer can output the action mean vector.

[0026] In some embodiments of this application, the observed motion sample may include the current robot state of the robot to be controlled (e.g., IMU (Inertial Measurement Unit) angular velocity, gravity vector, leg joint velocity, wheel angular velocity, flight thruster thrust, etc.) and motion task instructions (e.g., task speed, task target, task direction, etc.).

[0027] Correspondingly, during the process of the preset strategy network generating drive control commands based on observed motion samples, after the preset strategy network obtains the observed motion samples as input, the preset strategy network can output drive control commands for controlling the robot to be controlled based on the current robot state and motion task commands in the observed motion samples. The drive control commands include parameters such as joint direction, joint torque, wheel speed, and flight thruster thrust.

[0028] In some embodiments of this application, the observed motion sample may further include the historical robot state corresponding to the current robot state of the robot to be controlled. The historical robot state may be the robot state of the robot to be controlled within a preset historical period before the current robot state (e.g., the robot state within the past 0.5 seconds), or the robot state within a preset number of historical frames before the current robot state (the robot state within the past 4 frames), and there is no limitation herein.

[0029] Correspondingly, once the preset strategy network obtains the observed motion samples as input, it can output drive control commands based on the current robot state, historical robot state, and motion task commands in the observed motion samples, thereby improving the robustness of the drive control commands output by the preset strategy network.

[0030] Secondly, in some embodiments of this application, during the process of generating drive control commands based on observed motion samples through a preset strategy network, the observed motion samples can first undergo dimensional stacking processing, and then the dimension-stacked observed motion samples are used as input to the preset strategy network. Specifically, this dimensional stacking processing involves stacking and splicing the current robot state, historical robot state, and motion task commands in the observed motion samples in chronological order along the feature dimension to form a composite observed sample as input. This improves the preset strategy network's ability to perceive motion trends and the robustness of command generation by fusing states and task commands from multiple moments.

[0031] In addition, in some embodiments of this application, in order to obtain observed motion samples, before generating drive control commands based on the observed motion samples through the preset strategy network, the observed motion samples can be determined from the motion database corresponding to the robot to be controlled. The motion database includes motion samples of the robot to be controlled in different speed domains, so that the preset strategy network can learn the mapping relationship between speed changes and motion style from the observed motion samples, avoiding the problem of control failure due to sudden changes in speed domain or single training samples.

[0032] For example, motion samples of the robot to be controlled can be recorded in a simulation environment or on a real platform at slow speeds [0.2, 0.8] m / s, medium speeds [0.9, 2.0] m / s, and high speeds [2.1, 4.0] m / s, and a motion database can be built based on the recorded motion samples.

[0033] In some examples, the number of samples in the motion database for different speed domains can be flexibly adjusted as needed. For example, the ratio of the number of samples in the slow, medium, and high speed domains in the database is 4:4:2. This ensures the accuracy of motion in the commonly used low-to-medium speed range while also taking into account the learning of high-speed motion capabilities. It also avoids the problem of the preset policy network becoming aggressive in the commonly used speed domain due to an excessive proportion of motion samples in the high-speed domain.

[0034] In addition, it should be noted that, in order to increase the number of motion samples, in some embodiments of this application, during the construction of the motion database, the original motion samples obtained from the recording can be mirrored to obtain mirrored motion samples that are physically equivalent to each original motion sample, thereby doubling the effective motion samples in each velocity domain in the motion database, which in turn facilitates the improvement of the efficiency of subsequent training convergence.

[0035] In step S120, the motion style reward corresponding to the drive control command is output through a preset discriminator.

[0036] The preset discriminator is a neural network that determines whether the motion changes generated by the drive control command conform to a preset motion style. For example, the AMP (Adversarial Motion Priors) discriminator.

[0037] Specifically, the pre-defined discriminator can use a three-layer fully connected MLP (Multi-Layer Perceptron) as the network architecture. The hidden layers can use ELU (Exponential Linear Unit) as the activation function, and the output layer can use a linear output.

[0038] The motion style reward is a authenticity evaluation output by a preset discriminator based on the matching of drive control commands and preset expert data. The preset expert data includes benchmark motion samples of at least two motion modes.

[0039] In some examples, in addition to being systematically recorded in a simulation environment or on a real platform, the preset expert data can also directly use motion samples of the robot to be controlled in the motion database as benchmark motion samples in the preset expert data.

[0040] In some embodiments of this application, during the process of outputting the motion style reward corresponding to the drive control command through a preset discriminator, the method by which the preset discriminator outputs the motion style reward can refer to the following formula: ; in, Rewards for athletic style To pre-determine the discriminator network D in the discriminator, based on the current robot state and the next robot state The input is the authenticity score, which is used to characterize the current state of the robot. To the next robot state The degree of matching between the motion changes and the corresponding motion change distribution in the preset expert data.

[0041] In the formula above, when there is a high similarity between the motion changes generated by the drive control command and the motion change distribution corresponding to the preset expert data, Approaching the maximum value of 1. Conversely, when the motion changes generated by the drive control command deviate from the motion change distribution corresponding to the preset expert data, Approaching 0.

[0042] In step S130, the state value score of the observed motion sample is determined by a preset value network based on the motion style reward and the driving control reward of the driving control command.

[0043] Among them, the preset value network is a neural network that determines the value of the current robot state for the robot to be controlled to complete multi-motion mode control.

[0044] The drive control reward of the drive control command is used to quantify the degree of matching between the state of the robot to be controlled and the desired state after the drive control command is executed.

[0045] In some embodiments of this application, in order to determine the driving control reward of the driving control command, the target motion mode information in the driving control command can be determined first before determining the state value score of the observed motion sample based on the motion style reward and the driving control reward of the driving control command through a preset value network. Then, the driving control reward of the driving control command is calculated according to the preset driving control reward algorithm corresponding to the target motion mode information. This allows for adaptive selection of the algorithm based on the target motion mode information, so that the driving control reward under different motion modes can more accurately reflect the control expectation and constraints under that motion mode.

[0046] For example, when the target motion mode information in the drive control command is a legged motion mode, the drive control reward of the drive control command can be calculated according to the preset drive control reward algorithm corresponding to the legged motion mode. Specifically, the following formula can be used as a reference: ; in, For speed tracking rewards, , The current speed in the current robot state. To achieve the desired speed in the drive control command, For speed tracking tolerance band; For highly trackable rewards, , This represents the current height of the robot in its current state. To maintain a high level of expectation for the robot to be controlled, For high tracking tolerance; For attitude tracking rewards, , For the roll angular velocity, It is the pitch angular velocity; As a penalty for foot slippage, , Let i be the velocity vector of the i-th foot. Let be the contact force vector of the i-th foot; To provide sufficient contact with rewards, , The reward for the contact phase is -1 if the swing phase detects foot-to-ground contact or the foot of the support phase does not contact the ground; otherwise, it is 0. To impact punishment, For foot contact force, The preset maximum impact force threshold for the robot to be controlled; Penalty for joint acceleration, , This refers to the acceleration of the leg joint.

[0047] When the target motion mode information in the drive control command is a wheel motion mode, the drive control reward of the drive control command can be calculated with reference to the following formula: ; in, For wheel speed smoothing rewards, , Let be the angular velocity of the wheels of the robot to be controlled at time t. Let be the angular velocity of the wheels of the robot to be controlled at time t-1; As a reward for rotating ground contact, , The angular velocity of the wheel, The radius of the wheel's rolling radius, Let be the horizontal velocity vector of the robot's axle relative to the ground.

[0048] When the target motion mode information in the drive control command is the flight motion mode, the drive control reward of the drive control command can be calculated with reference to the following formula: ; in, For thrust tracking rewards, , This represents the current thrust vector of the flight thruster in the robot to be controlled. The desired thrust vector in the drive control command; For flight attitude tracking rewards, , The current roll angle of the robot to be controlled. The current pitch angle of the robot to be controlled. The current roll angular velocity of the robot to be controlled. The current pitch angular velocity of the robot to be controlled; As a penalty for flight energy consumption, , Let be the power consumption of the i-th thruster.

[0049] When the target motion mode information in the drive control command is the wheel-leg motion mode, the drive control reward of the drive control command can be calculated with reference to the following formula: ; in, Rewards for wheeled leg coordination , Let be the current phase angle of the wheel on the i-th leg. Let be the swing phase angle of the i-th leg.

[0050] When the target motion mode information in the drive control command is the flying leg motion mode, the drive control reward of the drive control command can be calculated with reference to the following formula: ; in, Rewards for coordinated flying kicks. ; As a landing cushioning bonus, , The impact force of the foot landing; For takeoff coordination rewards, , The thrust is generated when the feet leave the ground. This provides the lift-off thrust for the flight propulsion system. As a reward for the handover upon arrival, , For the thrust of the flight propulsion system, The current gravitational force acting on the robot to be controlled.

[0051] When the target motion mode information in the drive control command is the flywheel motion mode, the drive control reward of the drive control command can be calculated with reference to the following formula: ; in, To coordinate rewards for flywheels, ; Penalty for matching ground-to-air speed. , Let V be the horizontal velocity vector of the robot to be controlled relative to the ground. As a penalty for sliding on the ground, , The tangential linear velocity of the wheel.

[0052] When the target motion mode information in the drive control command is a full-modal motion mode, the drive control reward of the drive control command can be calculated with reference to the following formula: ; in, For cross-modal pose coordination rewards, , The attitude control torque generated by the leg joints of the robot being controlled. The torque generated by the wheels to control the robot's attitude. The attitude control torque generated by the flight thruster for the robot under control. This refers to the desired attitude control torque for the robot to be controlled.

[0053] It should be noted that each reward parameter in the above reward calculation formula can be configured with a corresponding weight coefficient according to actual needs, so as to balance the differences in the dimensions and importance between different reward items.

[0054] For example, after configuring weight coefficients for each reward item in the drive control reward of the wheel-leg motion mode, the calculation method for the drive control reward of the wheel-leg motion mode is updated to the following formula: ; in, =1, =1, =0.5, =-0.25, =-0.01, , , , .

[0055] In the embodiments of this application, the method for presetting the value network output state value score can refer to the following formula: ; in, Rate the state value; For For the preset strategy network of parameters, parameters Includes weights and biases in a pre-defined policy network; Rewards for athletic style and drive control rewards sum; To be in a given current robot state Under the conditions, according to the preset strategy network After the output drive control command is executed, the current robot state will be changed. All subsequent rewards After applying the corresponding discount factor Accumulate expectations after the discount.

[0056] In some embodiments of this application, during the process of determining the state value score of the observed motion sample based on the motion style reward and the drive control reward of the drive control command through the preset value network, the current motion rate corresponding to the observed motion sample is first obtained, then the reward fusion coefficient is determined based on the current motion rate, and then the motion style reward and drive control reward are adjusted based on the reward fusion coefficient, so that the preset value network determines the state value score of the observed motion sample based on the adjusted motion style reward and drive control reward. Thus, by configuring the reward fusion mechanism between task execution and motion style at different motion rates, the preset value network can effectively take into account both task execution efficiency and motion style naturalness when outputting the state value score.

[0057] Specifically, the adjustment direction of motion style reward under the reward fusion coefficient is positively correlated with the magnitude of the current motion speed, while the adjustment direction of drive control reward under the reward fusion coefficient is negatively correlated with the magnitude of the current motion speed. Based on this, under low-speed or stationary conditions, the proportion of drive control reward in the state value score can be increased to prioritize the accuracy and stability of task execution. Simultaneously, under high-speed motion conditions, the proportion of motion style reward in the state value score can be increased to prioritize the naturalness and smoothness of motion posture, preventing the robot under control from experiencing motion distortion or loss of stability.

[0058] For example, the method of adjusting the motion style reward and drive control reward using the reward fusion coefficient can refer to the following formula: ; in, The total reward is the sum of the sporty performance reward and the drive control reward. To reward the fusion coefficient, , To preset the high-speed limit fusion coefficient, To preset the low-speed limit fusion coefficient, The velocity attenuation coefficient, This represents the current speed of motion.

[0059] In one example =0.65, , Based on this, The maximum value of 0.85 is taken at the current speed of motion, which is 0 m / s. Furthermore, It decreases exponentially to 0.65 as the current speed of motion increases. Additionally, It also shifts from the current motion speed of 0.77 m / s to an intermediate value of 0.75, achieving a smooth and continuous transition between the respective influences of motion style reward and drive control reward at different motion speeds.

[0060] In step S140, the preset policy network is updated based on the state value score using a preset objective function, and the updated preset policy network is used as the robot control model.

[0061] The preset objective function can be a near-end policy optimization algorithm, a regular policy gradient algorithm, a synchronous parallel policy gradient reinforcement learning (Advantage Actor-Critic) algorithm, or a trust region policy optimization (TGR) algorithm, etc. No restrictions are imposed here.

[0062] For example, given that the preset objective function is the near-end policy optimization algorithm, the near-end policy optimization algorithm can be defined as the following computational expression: ; Where L is the total value of the objective function, The strategy loss value. Value loss, The entropy reward value, For discriminator loss, For gradient penalty loss, The value loss coefficient, The entropy reward coefficient, This represents the discriminator loss coefficient.

[0063] In some embodiments of this application, in order to determine You can refer to the following expression: ; in, For the preset policy network The objective function is used to optimize the preset policy network. Parameters in ; It is a probability ratio, which represents the ratio of the probability that the current policy network and the old policy network will select the same drive control command under the observed motion sample. The value of the dominance function represents the degree to which the execution of drive control commands relative to the average value is superior to the observed motion sample. Used to express probability ratios Limited to Within the corresponding range of changes, This is the cutting factor; Used from and Take the minimum value between them; Used to accumulate and average the clipping loss over all time points.

[0064] Specifically, this probability ratio The calculation process can be referred to this calculation expression: .

[0065] The advantage function value The calculation process can be referred to the following calculation expression: ; in, Estimate the corresponding decay factor for the generalized advantage; This is the time offset; From The deviation of the value estimate from the next time step; for example, at the current time t, then... .

[0066] In some embodiments of this application, in order to determine You can refer to the following expression: ; in, The discriminator network D in the preset discriminator is based on the current robot state from the preset expert data. and the next robot state The output is a score that reflects the authenticity of the input. To preset the discriminator network D in the discriminator to drive the current robot state in the control commands. and the next robot state The output is rated as authentic.

[0067] In some embodiments of this application, in order to determine You can refer to the following expression: ; in, The gradient penalty coefficient is... Preset expert data and drive control commands Random interpolation samples between , It follows a distribution in the range [0,1]. For parameters Perform differential calculations, that is, calculate partial derivatives. .

[0068] In the embodiments of this application, after determining the state value score of the observed motion sample, the preset policy network can be updated based on the state value score through a preset objective function, and the updated preset policy network is used as the robot control model.

[0069] For example, under the condition that the preset objective function is a proximal policy optimization algorithm, the process of updating the preset policy network based on the state value score through the preset objective function involves first calculating the value estimation deviation of the preset policy network under the observed motion samples, then determining the dominance function value based on the value estimation deviation, and then determining the probability ratio between the policy network to be updated and the preset policy network when selecting the same driving control command. The initial parameters of the policy network to be updated are consistent with the parameters of the preset control policy network. Then, the process is performed through the objective function... The initial parameters in the policy network to be updated are adjusted based on the advantage function value and the probability ratio, i.e., according to... The way to treat the initial parameters in the updated policy network. To be updated, among which, For parameters Perform differential calculations, that is, calculate partial derivatives. This leads to the updated preset policy network.

[0070] In some embodiments of this application, after updating the preset strategy network, the updated preset strategy network can be directly used as the robot control model to control the robot based on the robot control model, thereby improving the smoothness of the robot's movement.

[0071] In some embodiments of this application, it can also be determined whether the updated preset policy network meets preset update conditions. These preset update conditions include the preset policy network reaching a preset number of iterations and the preset objective function of the preset policy network converging to a preset range. If the determination is yes, the updated preset policy network is used as the robot control model. If the determination is no, the updated preset policy network is used as a new preset policy network, and the above process is repeated to update it further until the updated preset policy network meets the preset update conditions, thereby further improving the smoothness of the robot control model's motion when controlling the robot.

[0072] Additionally, it should be noted that in some embodiments of this application, after using the preset policy network as the robot control model, the robot control model (including normalized parameters, network structure, network weights, etc.) can be exported as a pre-formatted encapsulated deployment file, such as the ONNX (Open Neural Network Exchange) format, so that the control end only needs to call the standard application programming interface (API) to achieve efficient cross-platform and cross-language deployment and ensure that the inference behavior of the deployment end is consistent with that of the training end.

[0073] Through the above implementation method, a preset policy network first generates driving control commands based on observed motion samples. Then, a preset discriminator outputs the motion style reward corresponding to the driving control command. Next, a preset value network determines the state value score of the observed motion sample based on the motion style reward and the driving control reward of the driving control command. Finally, a preset objective function updates the preset policy network based on the state value score, and the updated preset policy network is used as the robot control model. Because the preset discriminator performs style evaluation and constraints on the robot control model in advance based on preset expert data across motion modalities, the robot control model can achieve a smooth transition between different motion modalities under the constraints of cross-modal motion style when generating driving control commands. This avoids defects such as sudden changes in control commands, discontinuous motion trajectories, and imbalances and falls caused by hard threshold switching, thereby improving the motion smoothness, motion safety, and control robustness of multi-modal robots.

[0074] The following describes an architectural embodiment of this application, which can be used to execute the robot control model training method in the above embodiments of this application. For details not disclosed in the architectural embodiments of this application, please refer to the embodiments of the robot control model training method described above.

[0075] Figure 2 A block diagram of a training architecture 100 for a robot control model according to an embodiment of this application is shown.

[0076] Reference Figure 2As shown, a training architecture 100 for a robot control model according to an embodiment of this application includes: a policy network unit 110 configured to generate drive control commands based on observed motion samples through a preset policy network; a discriminator unit 120 configured to output motion style rewards corresponding to the drive control commands through a preset discriminator, wherein the motion style reward is a authenticity evaluation output by the preset discriminator after matching the drive control commands with preset expert data, and the preset expert data includes benchmark motion samples of at least two motion modes; a value network unit 130 configured to determine the state value score of the observed motion samples based on the motion style reward and the drive control reward of the drive control commands through a preset value network; and a model generation unit 140 configured to update the preset policy network based on the state value score through a preset objective function, and use the updated preset policy network as the robot control model.

[0077] In some embodiments of this application, based on the foregoing scheme, the discriminator unit 120 is further configured to: acquire the current motion rate corresponding to the observed motion sample; determine the reward fusion coefficient based on the current motion rate; and adjust the motion style reward and driving control reward based on the reward fusion coefficient, so that the preset value network determines the state value score of the observed motion sample based on the adjusted motion style reward and driving control reward, wherein the adjustment direction of the motion style reward under the reward fusion coefficient is positively correlated with the magnitude of the current motion rate, and the adjustment direction of the driving control reward under the reward fusion coefficient is negatively correlated with the magnitude of the current motion rate.

[0078] In some embodiments of this application, based on the foregoing scheme, the discriminator unit 120 is further configured to: determine the target motion mode information in the driving control instruction after determining the state value score of the observed motion sample based on the motion style reward and the driving control reward of the driving control instruction through a preset value network; and calculate the driving control reward of the driving control instruction according to the preset driving control reward algorithm corresponding to the target motion mode information.

[0079] In some embodiments of this application, based on the foregoing scheme, the formula for the discriminator to output the motion style reward is as follows: ; in, Rewards for athletic style To pre-determine the discriminator network D in the discriminator, based on the current robot state and the next robot state The input is the authenticity score, which is used to characterize the current state of the robot. To the next robot state The degree of matching between the motion changes and the corresponding motion change distribution in the preset expert data.

[0080] In some embodiments of this application, based on the foregoing scheme, the formula for the preset value network output state value score is as follows: ; in, Rate the state value; For For the preset strategy network of parameters, parameters Includes weights and biases in a pre-defined policy network; Rewards for athletic style and drive control rewards sum; To be in a given current robot state Under the conditions, according to the preset strategy network After the output drive control command is executed, the current robot state will be changed. All subsequent rewards After applying the corresponding discount factor Accumulate expectations after the discount.

[0081] In some embodiments of this application, based on the foregoing scheme, the preset objective function includes a near-end strategy optimization algorithm; The near-end policy optimization algorithm includes the following calculation expression: ; Where L is the total value of the objective function. The strategy loss value. Value loss The entropy reward value, For discriminator loss, For gradient penalty loss, The value loss coefficient, The entropy reward coefficient, This represents the discriminator loss coefficient.

[0082] In some embodiments of this application, based on the foregoing scheme, the strategy network unit 110 is further configured to: determine the observed motion samples from the motion database corresponding to the robot to be controlled before generating drive control commands based on the observed motion samples through the preset strategy network, wherein the motion database includes motion samples of the robot to be controlled in different speed domains.

[0083] It should be noted that the training architecture 100 of the robot control model provided in the above embodiments and the training method of the robot control model provided in the above embodiments belong to the same concept. The specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0084] Embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the training method for the robot control model as described above.

[0085] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0086] It should be noted that, Figure 3 The computer system 200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0087] like Figure 3 As shown, the computer system 200 includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 202 or programs loaded from storage portion 208 into Random Access Memory (RAM) 203, such as performing the methods described in the above embodiments. The RAM 203 also stores various programs and data required for system operation. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An Input / Output (I / O) interface 205 is also connected to the bus 204.

[0088] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0089] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs various functions defined in the system of this application.

[0090] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0093] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0094] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0095] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0096] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0097] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for training a robot control model, characterized in that, The method includes: Drive control commands are generated based on observed motion samples through a preset strategy network; wherein, the preset strategy network includes an MLP network and a CNN network; The motion style reward corresponding to the drive control command is output by a preset discriminator. The motion style reward is a authenticity evaluation output by the preset discriminator after matching the drive control command and preset expert data. The preset expert data includes benchmark motion samples of at least two motion modes. The state value score of the observed motion sample is determined by a preset value network based on the motion style reward and the drive control reward of the drive control command. The step of determining the state value score of the observed motion sample based on the motion style reward and the drive control reward of the drive control command through a preset value network includes: Obtain the current motion rate corresponding to the observed motion sample; The reward fusion coefficient is determined based on the current movement rate; The motion style reward and the driving control reward are adjusted based on the reward fusion coefficient, so that the preset value network determines the state value score of the observed motion sample based on the adjusted motion style reward and driving control reward. The adjustment direction of the motion style reward under the reward fusion coefficient is positively correlated with the magnitude of the current motion rate, and the adjustment direction of the driving control reward under the reward fusion coefficient is negatively correlated with the magnitude of the current motion rate. The preset policy network is updated based on the state value score using a preset objective function, and the updated preset policy network is used as the robot control model. The preset objective function includes a near-end strategy optimization algorithm; The near-end strategy optimization algorithm includes the following calculation expression: ； Where L is the total value of the objective function, The strategy loss value. Value loss, The entropy reward value, For discriminator loss, For gradient penalty loss, The value loss coefficient, The entropy reward coefficient, This represents the discriminator loss coefficient.

2. The method according to claim 1, characterized in that, Before determining the state value score of the observed motion sample based on the motion style reward and the drive control reward of the drive control command through a preset value network, the method further includes: Determine the target motion mode information in the drive control command; The drive control reward of the drive control command is calculated based on the preset drive control reward algorithm corresponding to the target motion mode information.

3. The method according to claim 1, characterized in that, The formula for the motion style reward output by the preset discriminator is as follows: ； in, Rewards for the aforementioned sports style. The discriminator network D in the preset discriminator is based on the current robot state. and the next robot state The input is the authenticity score of the output, which is used to characterize the current state of the robot. To the next robot state The degree of matching between the motion changes and the corresponding motion change distribution in the preset expert data.

4. The method according to any one of claims 1-3, characterized in that, The formula for the state value score output by the preset value network is as follows: ； in, Score the value of the state; For A preset strategy network for parameters; Rewards for athletic style and drive control rewards sum; To be in a given current robot state Under the conditions, according to the preset strategy network After the output drive control command is executed, the current robot state will be changed. All subsequent rewards After applying the corresponding discount factor Accumulate expectations after the discount.

5. The method according to claim 1, characterized in that, Before generating drive control commands based on observed motion samples through a preset strategy network, the method further includes: The observed motion sample is determined from the motion database corresponding to the robot to be controlled, wherein the motion database includes motion samples of the robot to be controlled in different speed domains.

6. A training device for a robot control model, characterized in that, include: The strategy network unit is configured to generate driving control commands based on observed motion samples through a preset strategy network; wherein, the preset strategy network includes an MLP network and a CNN network; The discriminator unit is configured to output a motion style reward corresponding to the drive control command through a preset discriminator, wherein the motion style reward is a authenticity evaluation output by the preset discriminator after matching the drive control command and preset expert data, and the preset expert data includes benchmark motion samples of at least two motion modes; The value network unit is configured to determine the state value score of the observed motion sample based on the motion style reward and the drive control reward of the drive control command through a preset value network; The step of determining the state value score of the observed motion sample based on the motion style reward and the drive control reward of the drive control command through a preset value network includes: Obtain the current motion rate corresponding to the observed motion sample; The reward fusion coefficient is determined based on the current movement rate; The motion style reward and the driving control reward are adjusted based on the reward fusion coefficient, so that the preset value network determines the state value score of the observed motion sample based on the adjusted motion style reward and driving control reward. The adjustment direction of the motion style reward under the reward fusion coefficient is positively correlated with the magnitude of the current motion rate, and the adjustment direction of the driving control reward under the reward fusion coefficient is negatively correlated with the magnitude of the current motion rate. The model generation unit is configured to update the preset policy network based on the state value score through a preset objective function, and use the updated preset policy network as the robot control model. The preset objective function includes a near-end strategy optimization algorithm; The near-end strategy optimization algorithm includes the following calculation expression: ； Where L is the total value of the objective function, The strategy loss value. Value loss, The entropy reward value, For discriminator loss, For gradient penalty loss, The value loss coefficient, The entropy reward coefficient, This represents the discriminator loss coefficient.

7. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the training method for the robot control model according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the training method for the robot control model as described in any one of claims 1 to 5.

Images