Robot control model training method, device, equipment and product

Abstract

The application relates to the technical field of robot control, and discloses a robot control model training method, device, equipment and product, the method comprising: acquiring training samples for training a robot control model, the training samples comprising a plurality of time-series trajectory data of the robot; inputting the trajectory data into a policy model to obtain control instructions that the robot needs to execute at a next moment based on the trajectory data at an ending moment of the trajectory data; simulating execution of the control instructions by the robot to obtain corresponding simulation execution results; determining action similarity between the simulation execution results and human execution actions by a discriminator network, wherein the human execution actions are actions performed by a human at the next moment based on the trajectory data at the ending moment of the trajectory data; and updating the discriminator network and the policy model based on the action similarity corresponding to the plurality of trajectory data. Thus, the convergence speed of model training is effectively improved, and the model training efficiency is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to robot control model training methods, devices, equipment, and products. Background Technology

[0002] In practical applications, humanoid robots may be in various initial states, such as sitting up, prone, or lying flat, due to external collisions or task interruptions. The robot's getting-up actions based on these different initial states differ significantly in force application logic and limb coordination patterns. Designing a separate getting-up scheme for each initial state not only requires maintaining multiple sets of motion parameters, leading to a bloated control system, but also reduces the robot's operational reliability due to adaptation issues during state transitions. In ground-based getting-up scenarios with multiple motion types, model-based control methods mainly rely on preset motion state judgment rules, making it difficult to flexibly adapt to the dynamic switching between different initial postures such as sitting, prone, and lying down. For example, when the robot is in a relatively ambiguous transitional state, such as a semi-sitting / semi-prone position, it cannot accurately identify the current working condition and call the corresponding motion planning logic, resulting in abrupt motion transitions or even failures. Furthermore, due to the significant differences in the mechanical characteristics of different getting-up actions, the model needs to design complex constraints for each action, which is not only costly to develop but also difficult to handle unpredictable posture changes.

[0003] To address this, related technologies employ learning-based methods to effectively reduce reliance on pre-modeling. However, in multi-action type learning, explicit tracking reward functions are typically required for different actions, and a uniform reward function struggles to accommodate the learning needs of various actions. For example, a reward weight effective for one type of action might mislead another, causing slow learning or policy shifts in some actions, ultimately affecting the overall balance of standing ability. Therefore, learning-based methods also face significant technical bottlenecks. Summary of the Invention

[0004] This invention provides a robot control model training method, apparatus, equipment, and product to solve the technical bottleneck problem of controlling robots solely based on models or learning.

[0005] In a first aspect, the present invention provides a method for training a robot control model, the method comprising: Obtain training samples for training the robot control model. The training samples include multiple time-series trajectory data of the robot. By inputting the trajectory data into the strategy model, the control instructions that the robot needs to execute at the next moment after the trajectory data ends are obtained. The simulation robot executes control commands, and the corresponding simulation results are obtained. The discriminator network is used to determine the similarity between the simulation execution result and the action performed by a real person. The action performed by the real person is the action performed by the real person based on the trajectory data at the next moment after the end of the trajectory data. The discriminator network and policy model are updated based on the action similarity corresponding to multiple trajectory data.

[0006] This invention discloses a robot control model training method. After obtaining control commands from training samples, it simulates the robot's execution of these commands. A discriminator network directly determines the similarity between the robot's actions and those of a human. Based on this similarity, the discriminator network and policy model are updated. Therefore, during model training, implicit rewards are only needed for each simulation result based on the action similarity obtained from the discriminator, eliminating the need for designing and configuring explicit reward functions for different actions. This achieves high adaptability to the multiple action types included in the training samples while effectively avoiding the design difficulty and model complexity associated with designing explicit reward functions for multiple action types. Furthermore, it avoids the adaptation difficulty and bias caused by multi-action type identification and reward function adaptation. Moreover, updating the discriminator network and policy model through action similarity effectively improves the convergence speed of model training and significantly enhances model training efficiency.

[0007] In some alternative implementations, trajectory data is input into the policy model, including: Multiple trajectory data points are input into the strategy model according to a set frequency set; The set of frequencies is used to indicate the sample training ratio of each trajectory in the current training round; The sample training ratio is the ratio of the number of times each trajectory data is input to the policy model to the total number of times all trajectory data is input to the policy model in the current training round.

[0008] In some optional implementations, updating the discriminator network and the policy model includes: The gradient parameters of the policy model are updated based on the action completion degree and the action similarity corresponding to multiple trajectory data.

[0009] In some optional implementations, after obtaining the corresponding simulation execution results, the method further includes: Based on the simulation results, calculate the success rate of the robot completing the action to be performed in the next moment based on trajectory data of different states; If the success rate is greater than the set success ratio threshold, the auxiliary pulling force value configured for the trajectory data of the robot in the corresponding state will be reduced according to the set reduction rules.

[0010] This invention provides a robot control model training method that dynamically adjusts sample sampling probabilities based on simulation execution results, effectively avoiding model training imbalance caused by a single strategy. Furthermore, by employing a sampling probability distribution inversely proportional to the action completion rate, the method rapidly increases the training frequency of the model on trajectory data with lower completion rates, effectively improving the model's convergence speed and significantly enhancing its adaptability to multiple action types.

[0011] In some optional implementations, the frequency set for inputting multiple trajectory data into the strategy model is adjusted based on the degree of action completion, including: Reduce the training ratio of trajectory data with high action completion rate; The corresponding training ratio of trajectory data with low motion completion rate is increased.

[0012] In some alternative implementations, the simulated robot's execution of control commands includes: In each round of model training, corresponding auxiliary pulling force values ​​are configured for the robot's trajectory data in different states. When the corresponding auxiliary pulling force values ​​are applied to the robot, the robot's execution of control commands is simulated.

[0013] In some optional implementations, after obtaining the corresponding simulation execution results, the method further includes: Based on the simulation results, the success rate of the robot completing the action to be performed in the next moment based on trajectory data of different states is calculated. Adjust the auxiliary pulling force value according to the success rate and the trajectory data of the robot in the corresponding state.

[0014] Secondly, the present invention provides a robot control model training device, the device comprising: The acquisition module is used to acquire training samples for training the robot control model. The training samples include multiple time-series trajectory data of the robot. The instruction generation module is used to input trajectory data into the strategy model to obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends. The simulation module is used to simulate the robot's execution of control commands and obtain the corresponding simulation execution results; The discrimination module is used to determine the similarity between the simulation execution result and the action performed by a real person through a discriminator network. The action performed by the real person is the action performed by the real person based on the trajectory data at the next moment after the end of the trajectory data. The model update module is used to update the discriminator network and policy model based on the action similarity corresponding to multiple trajectory data.

[0015] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the robot control model training method of the first aspect or any corresponding embodiment described above.

[0016] Fourthly, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the robot control model training method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the first process of a robot control model training method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a second process for training a robot control model according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating a specific application example of the robot control model training method according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a robot control model training device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0021] The robot control model training method of the present invention can be applied to train the autonomous recovery process control method of robots in different environments and postures in the following application scenarios.

[0022] Specifically, humanoid robots need to recover fully autonomously in different environments and postures.

[0023] For example, in home care scenarios, robots may fall due to accidents or external forces in homes, on slippery surfaces, in complex environments with obstacles, or in public places, resulting in an uncontrollable state when they fall. In such cases, the robot needs to be able to autonomously recover and stand up from any initial position, whether lying on its back, side, or prone. In the industrial and logistics sectors, robots are prone to losing balance when carrying heavy objects, moving on uneven ground, working on temporary scaffolding, or performing special operations. They are also prone to falling in hazardous environments or complex terrains. Similarly, robots need to be able to quickly rise from any initial position (lying on its back, side, or prone) and continue performing tasks in complex terrains and environments to reduce production line downtime, minimize human intervention in hazardous environments, and ensure the continuity of robot tasks. Furthermore, when performing charging actions, the robot needs to be able to autonomously sit down and stand up based on its current state, completing the charging and recovery process without intervention. This effectively improves the efficiency of charging and other maintenance work, ensuring process reliability and continuity. Furthermore, in fields such as human-machine collaboration, robots may fall in any state during stage performances, exhibition demonstrations, and adversarial training. In such cases, robots need to be able to adapt to multiple action states, multiple skills, and multiple styles to recover and move normally, achieving a level of motion completion that is more similar to that of a real person.

[0024] For the above-mentioned and similar application scenarios, embodiments of the present invention provide a robot control model training method, device and equipment, medium and product to solve the technical bottleneck problem of controlling robots solely based on models or solely based on learning.

[0025] According to an embodiment of the present invention, a robot control model training method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] This embodiment provides a robot control model training method, which can be used for the aforementioned robot. Figure 1 This is a flowchart of a robot control model training method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain training samples for training the robot control model. The training samples include multiple time-series trajectory data of the robot.

[0027] In some alternative implementations, the robot control model can be trained based on multiple time-series trajectory data of the robot. For example, multiple frames of the robot's state can be obtained from its historical operation data, and trajectory data such as the state of each joint, limb coordinates, and gravity projection in each frame can be directly acquired. Furthermore, each trajectory data of the robot consists of multiple consecutive frames of data. For example, data from 0-10 seconds during a certain operation of the robot can be taken as one trajectory data, data from 0-5 seconds within this trajectory data can be taken as another trajectory data, and data from 3-8 seconds within this trajectory data can also be taken as another trajectory data.

[0028] In some optional implementations, training samples can be multiple trajectory data points randomly extracted from the robot's historical operation data. Alternatively, multiple trajectory data points can be randomly extracted after targeted filtering. This filtering can involve selecting a portion of the historical operation data that includes the moment the robot fell, and a portion that does not. The initial ratio of the two portions of trajectory data can be configured according to actual needs. Furthermore, the difficulty of getting up after falling varies depending on the robot's posture (sitting to standing, push-up, supine to rising), so trajectory data containing different falling postures can be pre-configured.

[0029] Step S102: Input the trajectory data into the strategy model to obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends.

[0030] In some alternative implementations, the strategy model can be a neural network model configured with a basic architecture and initial parameters, which is trained using the robot control model training method of the present invention. Further, the strategy model can be a pre-trained neural network model, which is optimized and updated using the robot control model training method of the present invention.

[0031] After the trajectory data is input into the strategy model, the strategy model can obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends, based on any trajectory data.

[0032] Step S103: Simulate the robot's execution of control commands and obtain the corresponding simulation execution results.

[0033] In some optional implementations, the execution of control commands by the robot can be calculated based on a simulation environment, such as tracking the robot's motion trajectory. The simulation results can include: the robot's ability to get up from its current fallen state, motion trajectory data, etc. The ability to get up from the current fallen state can be represented by a get-up flag; a successful get-up is set to 1, and a failed get-up is set to 0. The motion trajectory data can include the coordinates of the robot's limbs and joint angles.

[0034] Step S104: The discriminator network is used to determine the similarity between the simulation execution result and the action performed by the real person. The action performed by the real person is the action performed by the real person based on the trajectory data at the next moment after the end of the trajectory data.

[0035] In some alternative implementations, the actions performed by the real person are multiple real motion trajectory data of a real person, pre-acquired based on motion capture of the real person. These actions can be stored as a pre-configured database that the discriminator network can access. Alternatively, specified actions by the real person can be stored directly as part of the discriminator network. Furthermore, they can be stored in the cloud and retrieved when the discriminator network compares the simulation results with the real person's actions to determine similarity. In practical applications, appropriate methods can be used to store and update the actions performed by the real person according to actual needs.

[0036] In some alternative implementations, both the discriminator network and the policy model are pre-configured neural network models, with their neural network architecture and initial parameters pre-configured according to requirements. Similarly, the discriminator network can be configured with only the network architecture and initial parameters, or it can be a pre-trained discriminator network.

[0037] In some alternative implementations, a discriminator network can directly output the similarity between the simulation execution results and the actions performed by a real person.

[0038] Step S105: Update the discriminator network and policy model based on the action similarity corresponding to multiple trajectory data.

[0039] In some alternative implementations, the updating of the discriminator network can refer to the following principles: If the discriminator network input is a human performing an action, the discriminator network outputs an action similarity score that compares this input to the human's action. Theoretically, this output similarity score should be close to 1. Therefore, the closer the actual output similarity score is to 1, the higher the reward for the discriminator network. If the discriminator network input is a robot performing an action, the discriminator network outputs an action similarity score that compares this input to the human's action. Theoretically, this output similarity score should be close to 0. Therefore, the closer the actual output similarity score is to 0, the higher the reward for the discriminator network.

[0040] Here, a fixed similarity threshold and a fixed reward value for the discriminator network can be configured. When the similarity between the simulated execution result and the actions performed by a real person is higher than the pre-configured similarity threshold, a fixed reward value is input to the discriminator network. This reward value can be configured to update the gradient or loss function of the discriminator network as needed. When the similarity threshold is lower than the pre-configured similarity threshold, the reward value input to the discriminator network is zero.

[0041] The similarity threshold can also be configured as an interval range of multiple similarity thresholds, and a reward value can be configured for each similarity threshold interval. First, determine the interval range of the similarity threshold to which the action similarity belongs, and obtain the corresponding reward value. Based on this reward value, the discriminator network is updated.

[0042] In some alternative implementations, the policy model can be updated based on the judgment results of the discriminator network. Specifically, the discriminator network uses the original motion characteristics of human-executed actions such as standing up from a sitting position, doing push-ups, and getting up from a supine position as a reference. By comparing the overall style differences between the robot's actions and those of human-executed actions, it outputs an implicit reward signal for updating the policy model.

[0043] Specifically, when the similarity between the simulation result and the action performed by a real person is higher than the pre-configured similarity threshold, the robot's action is judged as "human-like" and the reward value is increased.

[0044] Similar to the discriminator network, this can also be configured with only a fixed similarity threshold and a fixed reward value for the policy network. When the similarity between the simulated execution result and the actions performed by a real person is higher than the pre-configured similarity threshold, a fixed reward value is input into the policy network. This reward value can be configured to update the gradient or loss function of the policy network as needed. When the similarity threshold is lower than the pre-configured similarity threshold, the reward value input into the policy network is zero.

[0045] The similarity threshold can also be configured as an interval range of multiple similarity thresholds, and a reward value can be configured for each similarity threshold interval. First, determine the interval range of the similarity threshold to which the action similarity belongs, and obtain the corresponding reward value. Based on this reward value, the policy network is updated.

[0046] Therefore, there is no need to manually distinguish between different action types or design separate display reward functions for different action types. By analyzing the similarity between the robot's actions and those of a human, the rationality and "human-like" characteristics of the robot's actions can be judged more accurately.

[0047] In some optional implementations, the simulation results are directly used to update the strategy model. Specifically, the simulation results may include the number of steps the robot takes to maintain normal operation after executing control commands, information on whether a robot successfully gets up, etc. If the number of steps maintained is greater than a set number or the number of successful gets-ups is greater than a set number, the model parameters of the current strategy model are maintained. If the number of steps maintained is less than a set number or the number of failed gets-ups is less than a set number, the model parameters of the current strategy model are adjusted to execute the next round of model training. Other strategy model update methods can also be configured.

[0048] This invention discloses a robot control model training method. After obtaining control commands from training samples, it simulates the robot's execution of these commands. A discriminator network directly determines the similarity between the robot's actions and those of a human. Based on this similarity, the discriminator network and policy model are updated. Therefore, during model training, implicit rewards are only needed for each simulation result based on the action similarity obtained from the discriminator, eliminating the need for designing and configuring explicit reward functions for different actions. This achieves high adaptability to the multiple action types included in the training samples while effectively avoiding the design difficulty and model complexity associated with designing explicit reward functions for multiple action types. Furthermore, it avoids the adaptation difficulty and bias caused by multi-action type identification and reward function adaptation. Moreover, updating the discriminator network and policy model through action similarity effectively improves the convergence speed of model training and significantly enhances model training efficiency.

[0049] This embodiment provides a robot control model training method, which can be used for the aforementioned robot. Figure 2 This is a flowchart of a robot control model training method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain training samples for training the robot control model. The training samples include multiple time-series trajectory data of the robot.

[0050] For specific implementation details, please refer to the above. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0051] Step S202: Input the trajectory data into the strategy model to obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends.

[0052] In some optional implementations, multiple trajectory data points are input into the policy model according to a set frequency set to achieve step S202 above. The set frequency set indicates the sample training ratio for each trajectory in the current training round, and the sample training ratio is the ratio of the number of times each trajectory data point is input into the policy model to the total number of times all trajectory data points in the current training round are input into the policy model. For example, during the first training of the model, multiple trajectory data points can be input into the policy model once each. Alternatively, trajectory data points including those at the moment the robot falls and those not at the moment the robot falls can be input into the policy model according to a set ratio. Furthermore, multiple trajectory data points can be configured in the training samples according to actual needs.

[0053] Step S203: Simulate the robot's execution of control commands and obtain the corresponding simulation execution results.

[0054] In some optional implementations, during each round of model training, an auxiliary pulling force value corresponding to the robot's initial state can be configured for trajectory data in different initial states. For each trajectory data, it can first be determined whether the trajectory data represents the robot's initial state (lying supine, prone, or sideways). Then, based on the control commands for that trajectory data, the robot's execution of the control commands is simulated while applying the auxiliary pulling force value corresponding to the robot's initial state, thus achieving step S203 above.

[0055] Here, the auxiliary pull value refers to the value of a vertically upward auxiliary pull applied at the center of the robot's torso, approximately at the chest cavity, in addition to the power required for the robot's normal operation. During the simulation of control commands, the auxiliary pull value is a configurable variable external force applied to the robot, which can be configured through parameters in the simulation software.

[0056] Specifically, the training samples include at least a portion of the robot's trajectory data, which represents the robot in various states such as lying supine, prone, or sideways, requiring external assistance to stand up. Through continuous training and updating of the policy network and discriminator network, the robot's autonomous standing action is achieved by gradually reducing or eliminating the need for assistance.

[0057] In some optional embodiments, the robot control model training method of the present invention further includes steps S204 and S205.

[0058] Step S204: After obtaining the corresponding simulation execution results, calculate the degree of action completion of the robot in the next moment based on the simulation execution results.

[0059] In some alternative implementations, the simulation results may include the degree of completion of the robot's action at the next moment based on any trajectory data.

[0060] The completion rate of an action can be calculated in the following ways: If a trajectory data point is extracted from the original robot trajectory of length M seconds, then the robot's reference data is the trajectory from seconds 0 to M. The trajectory data in the training samples is the trajectory data from seconds 0 to N extracted from the trajectory from seconds 0 to M. Based on the current simulation environment, starting from second N, control commands are issued to the robot for imitation tracking. At second K, the robot fails to track, where N < K < M. Then, the degree of action completion of the robot's action to be performed in the next moment based on this trajectory data is (KN) / (MN). Therefore, based on the simulation execution results, the degree of action completion of the robot's action to be performed in the next moment based on each trajectory data point can be calculated.

[0061] In practical applications, other suitable methods can also be used to calculate the degree of robot completion for the next action based on different trajectory data. For example, the percentage of times the robot successfully gets up based on a certain trajectory data.

[0062] Step S205: Based on the completion rate of the action, adjust the frequency set of inputting multiple trajectory data into the strategy model.

[0063] In some alternative implementations, step S205 may include: Step S2051: Reduce the training ratio of samples corresponding to trajectory data with high action completion rate.

[0064] Step S2052: Increase the training ratio of the samples corresponding to the trajectory data with low action completion rate.

[0065] Specifically, for trajectory data with high action completion rate, it indicates that the robot can easily complete subsequent actions such as getting up based on this trajectory data. The current policy model and discriminator network architecture and parameters are already well adapted to the corresponding trajectory data. Therefore, the training ratio of the samples corresponding to trajectory data with high action completion rate is reduced accordingly. Conversely, for trajectory data with low action completion rate, it indicates that the robot has not yet been able to complete subsequent actions such as getting up based on this trajectory data. The current policy model and discriminator network architecture and parameters are not yet adapted to the corresponding trajectory data. Therefore, the training ratio of the samples corresponding to trajectory data with low action completion rate is increased accordingly. Here, high and low action completion rate are relative concepts. In practical applications, action completion rate can be divided into multiple ranges, and a corresponding training sample ratio can be configured for each range. Alternatively, the training ratio of the samples for a particular trajectory data in the next round of model training can be configured based on the difference in action completion rate between two adjacent rounds of model training. For example, if the difference in action completion rate for a certain trajectory data exceeds a set difference threshold during two adjacent rounds of model training, and the action completion rate in the later round is greater than that in the earlier round, then the sample training ratio for that trajectory data will be reduced accordingly.

[0066] Therefore, this invention employs a dynamic sampling probability distribution that is inversely proportional to the degree of action completion. The sampling probability of a single type of action trajectory data decreases as the corresponding action completion of the trajectory data increases during model training. The sampling probabilities of all action types are normalized and constrained to the 0-1 range. This increases the proportion of training sample resources for the policy model and discriminator network training towards trajectory data with lower action completion, effectively balancing the training efficiency of each action, accelerating the model convergence speed, and effectively improving the model training efficiency and accuracy.

[0067] In some optional embodiments, the robot control model training method of the present invention further includes steps S206 and S207.

[0068] Step S206: After obtaining the corresponding simulation execution results, calculate the degree of action completion of the robot in the next moment based on the simulation execution results and the trajectory data of different states.

[0069] Here, the success rate of executing actions is affected by factors such as the magnitude of the auxiliary pulling force, disturbances such as randomly added thrust during robot execution, and the effectiveness of the strategy model itself. Different states can include prone, supine, and other positions. The state of a particular trajectory can be pre-labeled in the training samples.

[0070] In some alternative implementations, the robot's action completion rate for the next action based on trajectory data from different states focuses only on the following data: the success rate of the robot completing the next action based on trajectory data starting from second 0. Specifically, for trajectory data of a certain state in the current model training round, the ratio of the number of trajectory data points with 100% completion starting from second 0 to the total number of trajectory data points starting from second 0 can be calculated. This ratio is used as the robot's action completion rate for the next action based on trajectory data of this state.

[0071] The training samples can pre-label the robot's state at each moment in the robot trajectory data, such as standing, falling, sitting, etc. Each state can also be further classified and represented in more specific ways; for example, falling can include falling face down, falling sideways, falling supine, etc.

[0072] Therefore, it is possible to calculate the degree of action completion of the robot in the next moment based on trajectory data from different states.

[0073] Step S207: After obtaining the corresponding simulation execution results, adjust the auxiliary pulling force value configured for the trajectory data of the robot in the corresponding state according to the degree of action completion.

[0074] In some optional scenarios, auxiliary pulling forces are provided to the robot, and the magnitude of these forces is continuously adjusted. The purpose is to provide adaptive support for different standing movements of the robot. For example, upward pulling forces can be applied to the robot's head and waist to simulate a low-gravity environment by offsetting some of its own weight, while simultaneously lowering the body's center of gravity and reducing the load on the torso when lifting. For scenarios involving rolling movements, such as standing up from a prone position, additional torque in the same direction can be applied to assist the robot's torso in rotating around a support point, reducing the force required from the joints. When the success rate of a certain type of movement exceeds a preset success rate threshold in a training cycle, the auxiliary pulling force value corresponding to that movement can be gradually reduced in 1% increments, transitioning to a state without auxiliary pulling forces. Here, the movement completion rate can refer to the percentage of autonomous standing movements that the robot executes control commands to complete autonomously.

[0075] In practical applications, other auxiliary tension value adjustment methods can also be configured, but this invention does not impose specific limitations on them.

[0076] Specifically, if the completion rate of a standing-up movement from a given initial state reaches a set completion threshold, the assist pull value is adjusted according to a pre-configured method. For example, after each training round, if the success rate of standing up from a given initial state reaches the set completion threshold, the corresponding assist pull value is reduced by a set amount. If the completion rate of standing up from a given initial state does not reach the set completion threshold, the corresponding assist pull value remains unchanged. Alternatively, if the completion rate of standing up from a given initial state is less than or equal to a minimum completion threshold, the assist pull value for that standing-up movement is increased according to set rules. Furthermore, other assist pull value adjustment methods can be configured according to actual needs.

[0077] Step S208: The discriminator network is used to determine the similarity between the simulation execution result and the action performed by the real person. The action performed by the real person is the action performed by the real person based on the trajectory data at the next moment after the end of the trajectory data.

[0078] Please refer to the above for details. Figure 1 The step in the illustrated embodiment is S104, which will not be described again here.

[0079] Step S209: Update the discriminator network and policy model based on the action similarity corresponding to multiple trajectory data.

[0080] In some optional implementations, the gradient parameters of the policy model can be updated based on the action completion degree and the action similarity corresponding to multiple trajectory data to achieve the above step S209.

[0081] Specifically, referring to the gradient reward method for the policy model used when updating the policy network in step S105 above, the gradient reward value for the policy model based on action completion can be calculated. Similar to the discriminator network, a fixed action completion threshold and a fixed completion reward value for the policy network can be configured here. When the action completion degree is higher than the pre-configured action completion threshold, a fixed completion reward value is input to the policy network. This completion reward value can be configured to update the gradient or loss function of the policy network according to actual needs. Alternatively, the action completion threshold can be configured as a range of multiple action completion thresholds, and a completion reward value can be configured for each range. First, determine the range of action completion thresholds to which the action similarity belongs, and then obtain the corresponding action completion reward value.

[0082] Furthermore, taking into account the reward value determined based on action similarity, the action completion reward value is superimposed with the reward value determined based on action similarity in step S105 to obtain a comprehensive gradient reward value for the policy model. Based on the obtained comprehensive gradient reward value, the policy model is updated.

[0083] For other detailed procedures, please refer to the above. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0084] Therefore, the robot control model training method of the present invention realizes a unified learning framework for multiple action types through implicit style rewards. By combining dynamic sampling and coordinated adjustment of auxiliary pulling force, it avoids training imbalance caused by a single strategy. At the same time, it can improve the robot's adaptability to different standing actions without the need for manually designing complex type differentiation rules, shortening the training cycle while ensuring the human-likeness and physical feasibility of the actions.

[0085] This embodiment provides a robot control model training method, which can be used for the aforementioned robot. Figure 3 This is a flowchart illustrating a specific application example of the robot control model training method according to an embodiment of the present invention, such as... Figure 3 As shown, the training system for the robot control model training method can be divided into a sampling probability scheduler, an auxiliary pull scheduler, a simulation environment, a discriminator network, and a policy model.

[0086] The process includes the following steps: Step S301: The sampling probability scheduler inputs training samples into the simulation environment according to the preset sampling probability.

[0087] For specific implementation details, please refer to the above. Figure 1 Step S101 of the illustrated embodiment and Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0088] Step S302: The strategy model inputs the control instructions that the robot needs to execute at the next moment after the trajectory data ends into the simulation environment.

[0089] For specific implementation details, please refer to the above. Figure 1 Step S102 of the illustrated embodiment and Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0090] Step S303: Simulate the robot's execution of control commands in a simulation environment and obtain the corresponding simulation execution results.

[0091] For specific implementation details, please refer to the above. Figure 1 Step S103 of the illustrated embodiment and Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0092] Step S304: Update the strategy model based on the simulation execution results.

[0093] Step S305: Input the simulation execution results into the discriminator network and update the discriminator network.

[0094] Step S306: Update the policy model based on the output of the discriminator network.

[0095] In some alternative implementations, style rewards and completion rewards can be calculated based on the output of the discriminator network. A combined reward is then applied to the policy module based on these two rewards. The style reward is the reward mentioned above based on the similarity between actions performed by the robot and those performed by a human. Updating the policy module can be done by updating the gradients of the neural network.

[0096] Step S307: Based on the simulation execution results, calculate the degree of action completion of the robot in the next moment based on different trajectory data.

[0097] Please refer to the above for details. Figure 2 Step S204 shown will not be repeated here.

[0098] Step S308: Based on the completion rate of the action, adjust the frequency set of inputting multiple trajectory data into the strategy model.

[0099] Please refer to the above for details. Figure 2 Step S205 shown will not be repeated here.

[0100] Step S309: Based on the simulation execution results, calculate the success rate of the robot completing the action to be performed in the next moment based on the trajectory data of different states.

[0101] Step S310: Adjust the auxiliary pulling force value configured for the trajectory data of the robot in the corresponding state based on the success rate.

[0102] It should be noted that in this embodiment, when simulating the robot's execution of control commands in the simulation environment in step S303, step S203 above can be referred to. In each round of model training, corresponding auxiliary pulling force values ​​are configured for the robot's trajectory data in different states. When the corresponding auxiliary pulling force values ​​are applied to the robot, the robot's execution of control commands is simulated.

[0103] above Figure 3 For further implementation details and beneficial effects of the illustrated embodiments, please refer to [link / reference]. Figure 1 and Figure 2Examples are not described here.

[0104] This embodiment also provides a robot control model training device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0105] This embodiment provides a robot control model training device, such as... Figure 4 As shown, it includes: The acquisition module 401 is used to acquire training samples for training the robot control model. The training samples include multiple time-series trajectory data of the robot. The instruction generation module 402 is used to input trajectory data into the strategy model to obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends. The simulation module 403 is used to simulate the robot's execution of control commands and obtain the corresponding simulation execution results; The discrimination module 404 is used to determine the similarity between the simulation execution result and the action performed by a real person through the discriminator network. The action performed by the real person is the action performed by the real person based on the trajectory data at the next moment after the end of the trajectory data. The model update module 405 is used to update the discriminator network and policy model based on the action similarity corresponding to multiple trajectory data.

[0106] In some alternative implementations, the instruction generation module 402 includes: The first input unit is used to input multiple trajectory data into the policy model according to a set frequency set, wherein the set frequency set is used to indicate the sample training ratio of the number of times each trajectory data is input into the policy model in the current training round to the total number of times all trajectory data is input into the policy model in the current training round.

[0107] In some alternative embodiments, the apparatus further includes: The completion calculation module is used to calculate the completion degree of the robot's action at the next moment based on different trajectory data after obtaining the corresponding simulation execution results. The frequency adjustment module is used to adjust the set of frequencies for inputting multiple trajectory data into the strategy model based on the completion rate of the action.

[0108] In some optional implementations, the frequency adjustment module includes: The reduction unit is used to reduce the training ratio of samples corresponding to trajectory data with high motion completion rate; Add units to increase the training ratio of samples corresponding to trajectory data with low action completion rate; The sample training ratio is the ratio of the number of times each trajectory data is input to the policy model to the total number of times all trajectory data is input to the policy model in the current training round.

[0109] In some alternative implementations, simulation module 403 includes: The auxiliary pull unit is used to configure corresponding auxiliary pull values ​​for the robot's trajectory data in different states during each round of model training. When the corresponding auxiliary pull value is applied to the robot, the robot's execution of control commands is simulated.

[0110] In some alternative embodiments, the apparatus further includes: The success rate module is used to calculate the success rate of the robot completing the next action based on the trajectory data of different states after obtaining the corresponding simulation execution results. The tension adjustment module is used to adjust the auxiliary tension value configured for the robot's trajectory data in the corresponding state based on the success rate.

[0111] The robot control model training device provided in this embodiment of the invention can execute the robot control model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the various modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0112] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0113] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0114] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0115] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the robot control model training method of the embodiments of the present invention.

[0116] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0117] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the robot control model training method shown in the above embodiments is implemented.

[0118] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0119] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended technical solutions.

Claims

1. A method for training a robot control model, characterized in that, The method includes: Acquire training samples for training the robot control model, the training samples including multiple time-series trajectory data of the robot; The trajectory data is input into the strategy model to obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends. The robot's execution of the control commands is simulated to obtain the corresponding simulation execution results; The similarity between the simulation execution result and the action performed by a real person is determined by a discriminator network, wherein the action performed by the real person is the action performed by the real person at the next moment after the end of the trajectory data based on the trajectory data. The discriminator network and the policy model are updated based on the action similarity corresponding to multiple trajectory data.

2. The method according to claim 1, characterized in that, The step of inputting the trajectory data into the strategy model includes: Multiple trajectory data sets are input into the strategy model according to a set frequency. The set of frequencies is used to indicate the sample training ratio of each trajectory in the current training round; The sample training ratio is the ratio of the number of times each trajectory data is input to the policy model to the total number of times all trajectory data is input to the policy model in the current training round.

3. The method according to claim 2, characterized in that, After obtaining the corresponding simulation execution results, the method further includes: Based on the simulation results, the degree of action completion of the robot in the next moment is calculated based on different trajectory data. Based on the completion rate of the action, the frequency set of inputting multiple trajectory data into the strategy model is adjusted.

4. The method according to claim 3, characterized in that, The step of adjusting the set of frequencies for inputting multiple trajectory data into the strategy model based on the action completion degree includes: Reduce the training ratio of samples corresponding to trajectory data with high action completion rate; The corresponding training ratio of trajectory data with low motion completion rate is increased.

5. The method according to claim 3, characterized in that, The updating of the discriminator network and the policy model includes: The gradient parameters of the policy model are updated based on the action completion degree and the action similarity corresponding to multiple trajectory data.

6. The method according to claim 1, characterized in that, The simulation of the robot's execution of the control commands includes: In each round of model training, corresponding auxiliary pulling force values ​​are configured for the robot's trajectory data in different states. When the corresponding auxiliary pulling force values ​​are applied to the robot, the robot's execution of the control commands is simulated.

7. The method according to claim 6, characterized in that, After obtaining the corresponding simulation execution results, the method further includes: Based on the simulation results, calculate the success rate of the robot completing the action to be performed in the next moment based on trajectory data of different states; If the success rate is greater than the set success ratio threshold, the auxiliary pulling force value configured for the trajectory data of the robot in the corresponding state will be reduced according to the set reduction rules.

8. A robot control model training device, characterized in that, The device includes: The acquisition module is used to acquire training samples for training the robot control model, the training samples including multiple time-series trajectory data of the robot; The instruction generation module is used to input the trajectory data into the strategy model to obtain the control instructions that the robot needs to execute at the next moment after the trajectory data ends. The simulation module is used to simulate the robot's execution of the control commands and obtain the corresponding simulation execution results; The discrimination module is used to determine the similarity between the simulation execution result and the action performed by a real person through a discriminator network, wherein the action performed by the real person is the action performed by the real person based on the trajectory data at the next moment after the end of the trajectory data. The model update module is used to update the discriminator network and the policy model based on the action similarity corresponding to multiple trajectory data.

9. An electronic device, characterized in that, include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the robot control model training method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer instructions for causing a computer to execute the robot control model training method according to any one of claims 1 to 7.

Images