Simulation training method, device, medium and program product for robot control strategy

Abstract

This application relates to the field of robot control technology, and discloses a simulation training method, device, medium, and program product for robot control strategies. The method includes: constructing a simulation environment tailored to the characteristics of the robot; generating a paired training dataset consisting of healthy trajectory data and fault trajectory data for N failure modes; pre-training and fine-tuning a D&P network; constructing fault tokens; training a conditional shared policy network with potential spatial smoothness constraints and support for token mixing; and combining post-training domain randomization with multi-stage course learning throughout the entire process to complete the policy verification and deployment from simulation to the real environment, achieving stable control of the robot under multiple fault scenarios. Thus, it provides strong technical support for the robustness and reliability of robot control in complex fault scenarios.

Description

Technical Field

[0001] This application relates to the field of robot control technology, specifically to a simulation training method, equipment, medium, and program product for robot control strategies. Background Technology

[0002] With the widespread application of robotics in complex outdoor environments, higher demands are being placed on the robustness and reliability of robot control strategies. In practical deployments, robots inevitably encounter various physical failure modes, such as joint lock-up or power supply voltage drops. These failure modes can cause a sharp decline in the performance of existing control strategies, or even lead to complete robot failure.

[0003] Currently, existing single-failure-mode handling solutions, such as the "Saving the Limping" approach, only design control strategies for one joint failure mode, failing to address situations where multiple failure modes coexist or alternate in outdoor environments, resulting in extremely poor generalization ability. Existing domain randomization solutions (such as the AcL approach) only perform simple randomization on robot torque parameters, leading to significant differences between the simulation environment and the real outdoor environment, making it difficult to transfer the trained strategy to real-world scenarios. Existing fault detection and motion control separation solutions first use an independent fault detection module to identify the fault type before switching to a preset fault-tolerant controller, resulting in delays in fault detection and discrete switching leading to unsmooth control, making it difficult to handle continuously changing faults. Furthermore, most existing reinforcement learning training is conducted directly in the fault environment, resulting in long training cycles, large sample requirements, and a lack of paired data between healthy and fault states, making it impossible to effectively learn the differences between fault and healthy modes.

[0004] Therefore, in order to improve the robustness and adaptability of mobile robots in complex failure scenarios, there is an urgent need for a robot control strategy simulation training method that can systematically cope with multiple failure modes, reduce the complexity of training samples, and achieve efficient transfer from simulation to reality (Sim2Real). Summary of the Invention

[0005] This application provides a simulation training method, equipment, medium, and program product for robot control strategies, which improves the robustness and reliability of robot control in complex fault scenarios.

[0006] Firstly, this application provides a simulation training method for robot control strategies. The method includes: constructing a simulation environment for the robot, the simulation environment being related to at least one of the following: joint dynamics, motor characteristics, ground contact model, or environmental factors; obtaining a paired training dataset corresponding to a target training task based on the simulation environment, the paired training dataset including a health trajectory dataset and a fault trajectory dataset corresponding to N failure modes, where N is a positive integer; performing network pre-training on the paired training dataset using contrastive learning to obtain a pre-trained fault detection and parameter prediction (D&P) network, the D&P network being used for robot failure mode recognition; fine-tuning the pre-trained D&P network using reinforcement learning to obtain a trained D&P network; and constructing K tokens corresponding to the N failure modes, each token representing a fault scenario, where K... The input is a positive integer. Based on fault conditions, a conditional shared policy network is trained in a simulation environment to obtain the trained shared policy network. During the training process, a latent spatial smoothness constraint is added, and the fault conditions are periodically set as the embedding vector of the tokens or a mixture of embedding vectors of different tokens. The shared policy network is used for robot action decisions. In the later stages of training, at least one of the physical parameters, environmental parameters, and fault parameters in the simulation environment is domain randomized. A multi-stage learning strategy is adopted throughout the training process of the shared policy network to gradually adjust the training difficulty of the simulation environment. Based on the trained D&P network and the trained shared policy network, policy performance is verified from the simulation environment to the real environment, and the successfully verified shared policy network is deployed to the robot.

[0007] Thus, firstly, by using the D&P network for comparative learning pre-training followed by reinforcement learning fine-tuning, the training sample complexity of the shared policy network can be reduced by 5 to 10 times, decreasing the number of training iterations and computational resource consumption, and shortening the overall training cycle. Secondly, by normalizing fault scenarios with K tokens, combined with latent space smoothness constraints and token hybrid training, the shared policy can stably adapt to N failure modes and various parameters such as dynamic fault parameters, improving the accuracy of fault scenario adaptation, enhancing action smoothness, and significantly improving generalization ability, thereby reducing the robot's runaway rate. Furthermore, domain randomization of simulation environment parameters in the later stages of training, combined with multi-stage course learning throughout the process, reduces the deviation between simulation and real environment parameters, lowers the risk of policy transfer failure, shortens the real environment adaptation cycle, and ensures that the shared policy quickly adapts to the real robot. Moreover, the collaborative construction of closed-loop control by the D&P network and the shared policy helps improve the fault identification accuracy of the D&P network and the response time of the optimal action output of the shared policy. Combined with course learning and domain randomization, this improves the robot's task completion rate and operational stability in complex fault scenarios. Therefore, this solution effectively solves the problems of low training efficiency, poor adaptation to multiple faults, weak transfer, and complex deployment in traditional methods, and achieves efficient training, accurate adaptation, stable transfer, and reliable deployment, providing strong technical support for the robustness and reliability of robot control in complex fault scenarios.

[0008] In one possible implementation of the first aspect above, the simulation environment for constructing the robot includes: building a high-fidelity model of the robot using a set physics engine, the high-fidelity model containing X joints, where X is a positive integer; configuring joint parameters for each of the X joints, the joint parameters including at least one of the following: mass, length, or coefficient of friction; setting a control frequency, observation space, and motion space for the high-fidelity model, wherein the observation space includes: inertial data of the torso of the high-fidelity model, the positions and velocities of the X joints; the motion space includes: the positions or torques of the X joints; and generating terrain using a set terrain generator, the terrain including at least one of flat land, slopes, and uneven terrain.

[0009] In one possible implementation of the first aspect above, the joint dynamics are related to at least one of the inertia, damping, and friction models of each joint of the robot; the motor characteristics are related to the motor delay and / or torque response curves of the robot; the surface contact model is related to at least one of the Coulomb friction, collision detection, and contact force feedback between the robot and the ground; and the environmental factors are related to at least one of the terrain profile, ground slope, ground friction coefficient, and wind resistance of the environment in which the robot is located.

[0010] In one possible implementation of the first aspect above, the aforementioned N failure modes include two or more of the following: increased joint friction, motor delay, sensor noise, brake jamming, and power supply voltage drop; and each failure mode has a parameterized parameter range and a dynamic activation mechanism.

[0011] In one possible implementation of the first aspect above, each fault trajectory data has a set of fault labeling information, which includes: the time of fault occurrence, joint identifier, failure mode identifier, and fault severity quantification value.

[0012] In one possible implementation of the first aspect above, the D&P network includes: a feature encoder, a joint prediction head, and a fault classification head, wherein the feature encoder is used to input the robot's observed state and output latent features corresponding to the fault, the observed state including the robot's joint position, joint velocity, and inertial data; the fault classification head is used to input the latent features and output the probability distribution of the joints on N failure modes; and the joint prediction head is used to input the latent features and output the joint position deviation.

[0013] In one possible implementation of the first aspect above, the above-mentioned pre-training of the network using contrastive learning on the paired training dataset to obtain the pre-trained fault detection and parameter prediction (D&P) network includes: constructing positive sample pairs and negative sample pairs on the paired training dataset, wherein the positive sample pair includes a healthy trajectory data and a fault trajectory data under the same task, and the negative sample pair includes two fault trajectory data under different failure modes or two fault trajectory data under different tasks; and training the network based on the positive sample pairs and negative sample pairs to obtain the pre-trained D&P network.

[0014] In one possible implementation of the first aspect above, the first total loss in the pre-training process of the D&P network is a weighted sum of the joint state deviation reconstruction loss, the contrastive learning loss, and the fault classification loss; wherein, the reconstruction loss is used to measure whether the D&P network accurately predicts the joint coupling mode under normal dynamics; the contrastive learning loss is used to distinguish between healthy joint coupling modes and faulty joint coupling modes; and the fault classification loss is used to train the ability to identify failure modes on the hidden layer features.

[0015] In one possible implementation of the first aspect above, the fine-tuning of the pre-trained D&P network based on reinforcement learning to obtain the trained D&P network includes: for each failure mode, training an expert policy network using reinforcement learning in a simulation environment to obtain N expert policy networks corresponding one-to-one with the N failure modes, wherein each expert policy network is optimized on the fault trajectory dataset corresponding to the corresponding failure mode; and jointly fine-tuning the pre-trained D&P network and the student policy network in a simulation environment to obtain the trained D&P network and the trained student policy network, wherein the student policy network learns from the N expert policy networks by using the fault-related latent features output by the pre-trained D&P network as conditional input, and the student policy network is used for robot action decision-making.

[0016] In one possible implementation of the first aspect above, the method further includes: verifying policy performance from a simulation environment to a real environment based on the trained D&P network and student policy network, and deploying the successfully verified student policy network to the robot.

[0017] In one possible implementation of the first aspect above, the above-mentioned training of a conditional shared policy network in a simulation environment based on fault conditions to obtain a trained shared policy network includes: using the trained student policies as the initial weights of the shared policy, and then training the shared policy network in a simulation environment based on fault conditions to obtain a trained shared policy network.

[0018] In one possible implementation of the first aspect above, each of the tokens includes: a failure mode identifier, a failure severity quantification value, and a joint identifier.

[0019] In one possible implementation of the first aspect above, the physical parameters include at least one of the following: joint friction coefficient, motor delay time, mass or rod length; the environmental parameters include at least one of the following: terrain elevation map, slope angle, ground friction coefficient, wind interference; and the failure parameters include: parameters for each failure mode.

[0020] In one possible implementation of the first aspect above, the multi-stage course learning strategy includes the following stages in sequence: a stage where flat ground corresponds to the first fault severity; a stage where terrain with a slope less than the first set slope corresponds to the second fault severity; a stage where terrain undulations and changes in the ground friction coefficient correspond to the third fault severity; and a stage where a slope greater than the second set slope and multiple failure modes combined with wind interference correspond to the fault, wherein the second set slope is greater than the first set slope and the second fault severity is greater than the first fault severity.

[0021] In a second aspect, this application provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0022] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect and any possible implementation thereof.

[0023] Fourthly, this application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the methods described in the first aspect above and any possible implementation thereof.

[0024] The beneficial effects of the second to sixth aspects mentioned above can be referred to the relevant descriptions in the first aspect and its various possible implementations, which will not be repeated here. Attached Figure Description

[0025] Figure 1 The diagram shown is a simulation training scenario of a robot control strategy provided in an embodiment of this application.

[0026] Figure 2 The diagram shown is a schematic representation of a simulation training component provided in an embodiment of this application.

[0027] Figure 3 The diagram shown is a flowchart illustrating a simulation training method for a robot control strategy provided in an embodiment of this application.

[0028] Figure 4 The diagram shown is a flowchart illustrating a simulation training method for a jointly optimized robot control strategy provided in an embodiment of this application.

[0029] Figure 5 The figure shown is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0030] Figure 6 The diagram shown is a schematic representation of the hardware structure of a robot provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] It should also be stated that the steps in the methods and processes in this application are numbered for ease of reference, not to limit the order of steps. If there is an order between the steps, the textual description shall prevail.

[0033] The illustrative embodiments of this application include, but are not limited to, simulation training methods, devices, media, and program products for robot control strategies.

[0034] This application provides a simulation training method for robot control strategies, which can be applied to robot control scenarios with multiple failure models where the robot migrates from a simulation environment to a real environment.

[0035] As mentioned earlier, current robot control strategy simulation training methods cannot guarantee the robustness and adaptability of mobile robots in complex fault scenarios due to their inability to identify multiple failure modes and cope with continuously changing faults.

[0036] To improve the robustness and adaptability of mobile robots in complex fault scenarios, this application provides a simulation training method for robot control strategies. Specifically, this method constructs a simulation environment that fits the characteristics of the robot, generates a paired training dataset consisting of healthy trajectory data and fault trajectory data of N failure modes, pre-trains and fine-tunes a fault detection and parameter prediction (D&P) network, constructs fault tokens, trains a conditional shared policy network with potential spatial smoothness constraints and support for token mixing, and combines domain randomization in the later stage of training with multi-stage course learning throughout the process to complete the policy verification and deployment from simulation to the real environment, achieving stable control of the robot in multi-fault scenarios. It can be understood that, firstly, by using paired datasets to support the comparative learning pre-training of the D&P network, followed by reinforcement learning fine-tuning, the complexity of the training samples for the shared policy network can be reduced by 5 to 10 times, the number of training iterations and computational resource consumption can be reduced, and the overall training cycle can be shortened. Secondly, by normalizing fault scenarios using K tokens and combining latent space smoothness constraints with token hybrid training, the shared strategy can stably adapt to N failure modes and various parameters such as dynamic fault parameters. This improves the accuracy of fault scenario adaptation, enhances action smoothness, and significantly strengthens generalization ability, thereby reducing the robot's runaway rate. Furthermore, domain randomization of simulation environment parameters in the later stages of training, combined with multi-stage course learning throughout the process, reduces the deviation between simulation and real environment parameters, lowers the risk of strategy migration failure, shortens the real environment adaptation cycle, and ensures that the shared strategy quickly adapts to the real robot. Moreover, the D&P network and the shared strategy collaboratively construct closed-loop control, which helps improve the fault identification accuracy of the D&P network and the optimal action output response time of the shared strategy. Combined with course learning and domain randomization, this improves the robot's task completion rate and operational stability in complex fault scenarios. Therefore, this solution effectively solves the problems of low training efficiency, poor multi-fault adaptation, weak transfer, and complex deployment in traditional methods, achieving efficient training, accurate adaptation, stable transfer, and reliable deployment, providing strong technical support for the robustness and reliability of robot control in complex fault scenarios.

[0037] Reference Figure 1 The diagram illustrates a simulation training scenario for a robot control strategy provided in an embodiment of this application. Scenario 00 includes an electronic device 10 and at least one robot 20 in a real-world environment. The electronic device 10 uses a simulation training component 01 to perform simulation training and control strategy decision-making on the robot 20. The robot 20 executes the control strategy and provides action feedback. The strategy is trained and validated in the electronic device 10, and then the mature strategy is deployed to the real robot 20 for execution via a Simulation to Reality (Sim2Real) deployment and migration path.

[0038] Specifically, the electronic device 10 is used to perform simulation training. It internally constructs a high-fidelity simulation environment, including a simulated robot model 02 and simulated terrain. Furthermore, the electronic device 10 trains a control policy network (such as a shared policy network) through a pre-trained D&P network and a multi-stage training process (including reinforcement learning and tokenized latent space training). This training process can also incorporate 3D domain randomization and multi-stage curriculum learning strategies to ensure that the control policy can generalize to the real environment.

[0039] Robot 20 is the execution terminal, equipped with a pre-trained control strategy learning and D&P network. In actual operation, Robot 20's sensors collect and observe its state. From the observed state via D&P network Extracting hidden fault features And through the control policy network based on the observed state And generate action commands from hDP based on fault hidden features This enables the robot 20 to perform tasks robustly even under various failure modes (such as increased joint friction, motor delay, etc.).

[0040] The aforementioned simulation training components include, but are not limited to, simulation training components corresponding to physics engines such as Isaac Gym, Multi-Joint Dynamics with Contact (MuJoCo), Webots, or Genesis. It is understood that the simulation training components used in the embodiments of this application can be selected based on application scenarios such as academic research, industrial manufacturing, and reinforcement learning training, and no specific limitations are imposed.

[0041] The aforementioned electronic device 10 may include general-purpose computing devices such as personal computers (PCs), laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The aforementioned electronic device 10 may also include computing acceleration devices such as graphics processing unit (GPU) servers, tensor processing unit (TPU) servers, or neural network processing unit (NPU) servers. Alternatively, the aforementioned electronic device 10 may also include cloud computing resources such as cloud GPU instances or container clusters, as well as cloud-based distributed training clusters such as multi-node GPU clusters. Figure 1 The following explanation uses a laptop computer as an example of electronic device 10.

[0042] As an example, the robot 20 described above is a mobile robot, including but not limited to legged robots that move using legs and feet, wheeled / tracked robots that move using wheels or tracks, and composite mobile robots that work using a mobile platform and a robotic arm. For example, the legged robots described above may include multi-jointed outdoor mobile robots such as bipedal robots or quadrupedal robots. Figure 1 The following explanation uses an outdoor quadruped robot as an example.

[0043] Reference Figure 2 The diagram shown is a schematic representation of the composition of a simulation training component 01 provided in an embodiment of this application. The simulation training component 01 includes: a high-fidelity simulation environment construction module 11, a fault injection and data generation module 12, a D&P network and policy training module 13, and a Sim2Real adaptation module 14.

[0044] The high-fidelity simulation environment construction module 11 is used to build high-fidelity physical models (i.e., robot models) and virtual simulation worlds with diverse terrains and parameters. As an example, module 11 can use physics engines such as Isaac Gym or MuJoCo to accurately simulate robot joint dynamics, motor characteristics, ground contact models, etc. Module 11 can use a terrain generator to generate terrains such as flat ground, slopes, or uneven terrain, and configure environmental factors such as ground friction coefficient and wind force. Furthermore, module 11 can also build robot models, such as creating a high-fidelity model of a quadruped robot with X joints (e.g., 12), and configuring key parameters such as mass or length.

[0045] The fault injection and data generation module 12 is used to systematically generate training data corresponding to predefined failure models. As an example, this module 12 predefines N failure modes, including increased joint friction, motor delay, sensor noise, brake jamming, and power supply voltage drop. Here, N is a positive integer greater than 1, such as N equals 5. Furthermore, this module 12 can generate paired health trajectory datasets by injecting faults. and fault trajectory dataset The dataset consists of paired datasets, and each fault trajectory is labeled with fault information (such as fault type, joint ID, severity, etc.).

[0046] The D&P network and policy training module 13 is used to train the D&P network and the control policy network. As an example, module 13 can perform pre-training and reinforcement learning on the D&P network, where the input to the D&P network is the observed state. (e.g., 34-dimensional), the output is the hidden features of the fault. (e.g., 128-dimensional), used to provide fault information for control strategies.

[0047] As an example, module 13 can independently train an expert policy network for each failure mode. Furthermore, it learns expert policies through distillation techniques and outputs latent features as part of the D&P network. As conditional inputs, a deployable student policy network is ultimately formed. For example, module 13 can use reinforcement learning to jointly fine-tune the D&P network during the training of the student policy network.

[0048] As an example, module 13 can train a conditional shared policy network under fault conditions. For instance, during the training phase, module 13 can tokenize fault conditions. As input, a conditional shared policy network is trained. And introduce potential spatial smoothness constraints. Training strategy that combines tokens.

[0049] Sim2Real adaptation module 14 is used to predict and adjust the gap between simulation and reality. For example, this module 14 can randomize the physical parameters, environmental parameters, and fault parameters in the simulation environment in the later stages of training the shared policy network, thereby enhancing the generalization ability of the policy. Furthermore, this module 14 can adopt a multi-stage course learning strategy in the simulation environment, gradually increasing the training difficulty from easy to difficult (e.g., from flat ground and gentle slopes, from low faults to extreme conditions).

[0050] During the deployment phase, after the electronic device 10 has undergone simulation verification (e.g., ensuring a task completion rate of >90% in the generalization evaluation), the shared policy network or student policy network trained in the electronic device 10 is deployed to the robot 20.

[0051] In some embodiments, the robot 20 includes a perception and decision-making module 21 and an execution module 22.

[0052] The perception and decision-making module 21 is used to acquire the observation state of the robot 20 in real time through its onboard sensors (such as encoders or inertial measurement units, IMUs). (e.g., 34-dimensional). Furthermore, this module 21 can receive the observation status of the robot 20 through the D&P network with frozen parameters. To output the corresponding latent features Furthermore, module 21 can observe the state by controlling a policy network, such as a shared policy network. and corresponding latent features As input, calculate and output action commands in real time. (e.g., target position or torque of 12 joints).

[0053] Execution module 22 controls the actuators (such as joint motors) of robot 20 to receive motion commands. And execute. As an example, when the control policy is a student policy network, since the student policy network has learned from the expert policy through distillation and can learn from latent features... It can detect faults. Therefore, even in failure modes such as increased joint friction and motor delay, robot 20 can maintain stable and robust movement and complete tasks such as walking and climbing.

[0054] Next, refer to Figure 3 The flowchart illustrates a simulation training method for a robot control strategy provided in this application embodiment. The execution entity of this method can be... Figure 1 The illustrated scenario includes an electronic device 10 and a robot 20. Specifically, the process includes the following steps:

[0055] S301: Electronic device 10 constructs a simulation environment for the robot.

[0056] The simulation environment is related to at least one of the following: joint dynamics, motor characteristics, ground contact model, or environmental factors.

[0057] It is understandable that the electronic device 10 can construct a high-fidelity physical simulation environment through the high-fidelity simulation environment construction module 11, accurately model joint dynamics, motor characteristics, ground contact, and terrain contours, and ensure that the fault modes learned in the simulation can be transferred to the real environment.

[0058] In some embodiments, the electronic device can construct a simulation environment for the robot's target task. For example, the target task could be a quadruped robot walking on rugged terrain, but it is not limited to this.

[0059] In some embodiments, joint dynamics are related to at least one of the inertia, damping, and friction models for each joint of the robot. Thus, in a simulation environment, the motion response of each joint is ensured to be highly consistent with that of a real robot through inertia, damping, and friction models, providing a realistic physical basis for fault injection (such as increased friction).

[0060] In some embodiments, motor characteristics are related to the robot's motor delay and / or torque response curves. Therefore, in a simulation environment, by simulating the motor delay and torque response curves, the actuator behavior is made to approximate the real hardware, avoiding the generation of idealized torque outputs in the simulation.

[0061] In some embodiments, the above-described ground contact model is related to at least one of Coulomb friction, collision detection, and contact force feedback between the robot and the ground. Therefore, in a simulation environment, the ground contact model accurately simulates the complex interaction process between the foot and the ground through joint modeling of Coulomb friction, collision detection, and contact force feedback, ensuring the effectiveness of gait training.

[0062] In some embodiments, the aforementioned environmental factors are related to at least one of the terrain profile, ground slope, ground friction coefficient, and wind resistance of the robot's environment. Therefore, in the simulation environment, by configuring a combination of parameters such as terrain profile, ground slope, friction coefficient variation, and wind resistance, the simulation environment can realistically reproduce the diversity and uncertainty of outdoor scenes.

[0063] Thus, modeling joint dynamics, motor characteristics, ground contact models, or environmental factors in a simulation environment helps to narrow the physical gap between simulation and reality (Sim2Real gap), enabling control strategies trained in the simulation environment to be effectively transferred to real robots. At the same time, it provides a parameterizable physical basis for subsequent systematic fault injection, domain randomization, and course learning.

[0064] In some embodiments, the process of constructing the simulation environment for the robot described above may include the following steps 1 to 4:

[0065] Step 1) Use the set physics engine to build a high-fidelity model of the robot. The high-fidelity model contains X joints, where X is a positive integer.

[0066] The physics engine mentioned above can be Isaac Gym or MuJoCo. Using a professional physics engine like Isaac Gym or MuJoCo to build a high-fidelity robot model with multiple joints ensures the physical accuracy of rigid body dynamics and collision detection. Furthermore, Isaac Gym's GPU parallelism allows it to run thousands of environments simultaneously, significantly accelerating reinforcement learning training.

[0067] As an example, for a quadruped robot walking on rugged terrain, the electronic device 10 can use the IsaacGym physics engine to build a quadruped robot model containing 12 joints (3 degrees of freedom × 4 legs). In this case, X equals 12.

[0068] Step 2) Configure joint parameters for each of the X joints. The joint parameters include at least one of the following: mass, length, or coefficient of friction. In this way, by configuring mass, length, and coefficient of friction for each joint, the joint inertial load, kinematic range, and motion resistance are affected respectively, providing a quantifiable physical parameter basis for subsequent fault injection (such as increased joint friction).

[0069] As an example, for a quadruped robot to walk on rugged terrain, the electronic device 10 can configure the joint parameters of 12 joints, including mass, length, coefficient of friction, etc., and these joint parameters are referenced to the specifications of a real robot (such as Unitree Go2).

[0070] Step 3) Set the control frequency, observation space, and motion space for the high-fidelity model. The observation space includes: the inertial data of the model's torso, and the positions and velocities of X joints. The motion space includes: the positions or torques of the X joints.

[0071] As an example, for a quadruped robot walking on rugged terrain, the electronic device 10 defines the control frequency as 100Hz, sets the observation space to include 12-dimensional position, 12-dimensional velocity and IMU information, and sets the motion space to include 12-dimensional target position or torque.

[0072] For example, the aforementioned IMU information may include 10 dimensions of information, such as 3D angular velocity, 3D acceleration, and 4D attitude quadruples. Therefore, by setting a control frequency of 100Hz, both real-time performance and computational efficiency can be balanced. A 34-dimensional observation space is defined, containing 10-dimensional trunk inertial data, 12 joint positions, and 12 joint velocities, providing the policy network with complete state-aware information. Simultaneously, a 12-dimensional motion space is defined, supporting both position control and torque control modes to flexibly adapt to different control requirements.

[0073] Step 4) Use the set terrain generator to generate terrain, which includes at least one of flat land, slope, and uneven terrain.

[0074] The shape generator supports the random generation of flat, sloping, and uneven terrain.

[0075] As an example, for a quadruped robot walking on rugged terrain, the electronic device 10 can generate uneven terrain by setting a terrain generator to construct a high-fidelity physical simulation environment suitable for walking tasks on rugged terrain. Therefore, using a terrain generator to generate flat ground, slopes, and uneven terrain covers typical outdoor scenes, providing a difficulty gradient from easy to difficult for course learning and enhancing the environmental adaptability of the strategy.

[0076] In this way, the above four steps jointly construct a physically realistic, parameter-adjustable, and terrain-diverse simulation environment, laying the foundation for subsequent fault injection, policy training, and Sim2Real migration.

[0077] S302: Electronic device 10 obtains the paired training dataset corresponding to the target training task based on the simulation environment. The paired training dataset includes a health trajectory dataset and a fault trajectory dataset corresponding to N failure modes, where N is a positive integer.

[0078] In some embodiments, the above N failure modes include two or more of the following: increased joint friction, motor delay, sensor noise, brake jamming, and power supply voltage drop; and each failure mode has a parameterized parameter range and a dynamic activation mechanism.

[0079] Referring to Table 1, examples of parameterized descriptions and key parameter ranges for each mode are provided.

[0080] Table 1:

[0081]

[0082] The parameterized description of failure mode M1 shown in Table 1 is as follows: .in, This represents the joint friction force under fault conditions. This represents the joint friction force in a healthy state. This represents the proportionality coefficient for the increase in friction. Furthermore, the parameter range corresponding to failure mode M1 is... The parameter range is [0, 2.0].

[0083] The parameterized description of failure mode M2 ​​shown in Table 1 is as follows: .in, This indicates the actual output torque of the motor. Indicates a delayed control command. This indicates the motor delay time, with a parameter range of [0, 200ms]. At this time, the control command passes through... The motor executes the signal after a time delay, which may be due to analog signal transmission delay, motor aging, or controller response lag.

[0084] The parameterized descriptions of failure mode M3 shown in Table 1 are as follows: ,in, Indicates the observation angle with noise. Indicates the actual joint angle. This indicates that the mean is 0 and the variance is 0. Gaussian noise, This represents the noise standard deviation, with a parameter range of [0.001, 0.05] rad. It can be understood that the joint angle observations are contaminated by Gaussian noise, and the inaccurate measurements are caused by electromagnetic interference from the analog encoder or sensor aging.

[0085] The parameterized description of failure mode M4 shown in Table 1 is as follows: =0 indicates the actual output torque of the joint when a fault is activated. The parameter range for the fault activation state is a Boolean value {True, False}. It can be understood that when a fault is activated, the joint output torque is forcibly set to zero, simulating brake lock-up or motor drive circuit failure, which is the most severe failure mode.

[0086] The parameterized description of failure mode M4 shown in Table 1 is as follows: ,in This indicates the actual output power of the power supply. This indicates the nominal power of the power supply. This represents the voltage drop ratio, with a parameter range of [0, 0.3]. It can be understood that a drop in power supply voltage causes a proportional decrease in the actuator's available power, simulating insufficient power due to battery discharge, power supply line impedance, or poor contact.

[0087] Thus, this application defines five typical robot failure modes, covering common failure types in key subsystems such as actuators, sensors, and power supplies. After parameterizing these five modes, a progressive fault injection from slight degradation to complete failure can be achieved by continuously adjusting the severity parameter. This provides a quantifiable fault space for paired dataset generation, domain randomization, and curriculum learning, enabling the trained control strategy to effectively cope with various physical faults that real robots may encounter.

[0088] In some embodiments, each fault trajectory data has a set of fault annotation information, including: fault occurrence time, joint identifier, failure mode identifier, and fault severity quantification value. These fault annotation information collectively constitute complete prior fault knowledge, providing refined supervisory signals for the multi-task learning of the D&P network, enabling the network to accurately extract multi-dimensional information such as fault type, location, time, and severity from the observed state.

[0089] As an example, the electronic device systematically generates paired datasets for each training task (such as walking or climbing). Specifically, for each task, a health trajectory dataset is collected. The system records the complete trajectory of the robot completing the task under healthy conditions. Simultaneously, under the same initial conditions and task parameters, five failure modes (increased joint friction, motor delay, sensor noise, brake jamming, and power supply voltage drop) are injected, and corresponding fault trajectory datasets are collected. Each fault trajectory is marked as a quadruple (...). ),in Indicates the time when the fault occurred. Indicates the affected joints. Indicates the failure mode identifier (1 to 5). Indicates the severity of the fault (a continuous value from 0 to 1).

[0090] S303: Electronic device 10 uses contrastive learning to pre-train the network on the paired training dataset to obtain the pre-trained D&P network.

[0091] Among them, the D&P network is used for failure mode identification of robots.

[0092] In some embodiments, the architecture of the D&P network described above includes: a feature encoder, a joint prediction head, and a fault classification head.

[0093] The feature encoder is used to input the robot's observed state and output the hidden features corresponding to the fault. The observed state includes the robot's joint position, joint velocity, and inertial data.

[0094] For example, the feature encoder described above includes two hidden layers (e.g., 256 units), and the input is the robot's observation state. (i.e., a vector), outputting latent features (e.g., 128-dimensional implicit representation). For example, the feature encoder is a 2-layer, 256-unit multilayer perceptron (MLP). It can be understood that during simulation training, the input to the feature encoder is the robot's observed state in the simulation environment. At this point, the feature encoder can extract core information related to the fault from the high-dimensional sensor data (i.e., the observed state) and remove redundancy, while outputting a continuous differentiable vector (i.e., latent features) to provide fault condition input for the subsequent policy network.

[0095] The fault classification head takes latent features as input and outputs the probability distribution of joints across N failure modes. For example, the fault classification head can be a classifier used to identify which of the N failure modes the robot's current failure mode is. Thus, the fault classification head enables automatic identification of fault types (such as distinguishing between increased joint friction, motor delay, and sensor noise), and its classification loss provides supervision signals for the pre-training of the D&P network, accelerating the convergence of the feature encoder and enhancing the discriminative power of the features.

[0096] A joint prediction head is used to input latent features and output joint position deviations. For example, the joint prediction head mentioned above can be a small network used to predict the joint position deviation for the next step. In this way, the joint prediction head learns the normal coupling relationship between joints in a healthy state, providing reconstruction loss for the subsequent contrastive learning pre-training of the D&P network; in a fault state, the difference between the predicted deviation and the actual deviation will increase significantly, thereby improving the sensitivity of fault detection.

[0097] Thus, based on the aforementioned feature encoder, joint prediction head, and fault classification head, the D&P network can extract fault information end-to-end from the raw observations without the need for manual design of fault features. At the same time, it can fully mine data information through multi-task pre-training, providing high-quality fault feature input for subsequent reinforcement learning and reducing sample complexity by 5-10 times.

[0098] In some embodiments, the electronic device 10 can construct positive and negative sample pairs on a paired training dataset, wherein the positive sample pair includes a healthy trajectory data and a fault trajectory data under the same task, i.e. Negative sample pairs include two fault trajectory data under different failure modes or two fault trajectory data under different tasks; based on positive and negative sample pairs, the network is trained to obtain a pre-trained D&P network.

[0099] As an example, electronic device 10 can use the NT-Xent loss function for contrastive learning training and set temperature parameters. The training process iterates for 100 epochs with a learning rate of 0.001. After training, the pre-training performance is evaluated on the validation set to determine if it meets the first condition. For example, the first condition includes a fault classification accuracy higher than 95% and a reconstruction loss lower than 0.1. If the first condition is met, the weights of the pre-trained D&P network are saved to provide network initialization for subsequent reinforcement learning stages, thereby reducing the sample complexity of subsequent policy training.

[0100] In some embodiments, the electronic device 10 may pre-train a D&P network on a paired training dataset based on a first total loss and using a contrastive learning objective.

[0101] For example, the total loss during the pre-training process of the D&P network mentioned above is a weighted sum of the joint state deviation reconstruction loss, the contrastive learning loss, and the fault classification loss.

[0102] As an example, the aforementioned first total loss The objective function can be expressed by the following formula (1).

[0103] (1).

[0104] in, and To balance the hyperparameters of each loss term.

[0105] Specifically, reconstruction loss The D&P network is used to measure whether it accurately predicts joint coupling patterns under normal dynamics. This is achieved by calculating the mean square error between the predicted joint position deviation and the actual deviation. This loss term forces the network to learn the inherent cooperative relationships between joints in a healthy state.

[0106] Comparative learning loss The NT-Xent loss function can be used, with the temperature parameter set to 0.1. Its function is to distinguish between the healthy joint coupling mode and the fault mode in the latent feature space, so that the features of the same sample pair (healthy trajectory and fault trajectory under the same task) are close to each other, while the features of different sample pairs (different fault modes or different tasks) are far apart.

[0107] Fault Classification Loss Cross-entropy loss function can be used to train the classifier on hidden layer features to identify five failure modes, enabling the network to accurately determine whether the current fault is due to increased joint friction, motor delay, sensor noise, brake jamming, or power supply voltage drop.

[0108] Thus, through the joint optimization of the above three losses, the D&P network can simultaneously learn health dynamics modeling, fault mode discrimination, and fault type classification capabilities during the pre-training stage, providing high-quality network initialization weights for joint fine-tuning in the subsequent reinforcement learning stage.

[0109] S304: Electronic device 10 adjusts the pre-trained D&P network based on reinforcement learning to obtain the trained D&P network.

[0110] In some embodiments, the electronic device 10 can use Deep Reinforcement Learning (RL) in a simulation environment to train the robot's control policy network and generate a reinforced control policy network. As an example, during the reinforcement learning training loop of the control policy network, the electronic device 10 can further fine-tune the pre-trained D&P to obtain the trained D&P network.

[0111] As an example, in an electronic device, the pre-trained D&P network and the control policy network are jointly optimized end-to-end within a reinforcement learning framework. The latent features output by the D&P network... As a conditional input to the control policy network, the loss gradient of the control policy network is obtained through latent features. Backpropagation to the D&P network allows it to move beyond simply maximizing fault classification accuracy and instead learn the most useful fault feature representations for policy decisions. Furthermore, the joint fine-tuning process is performed online within the reinforcement learning training loop, continuously optimizing the D&P network using the dynamic data distribution generated during policy exploration. This adapts the network to the current state distribution encountered by the policy, and the fault representation capability of the D&P network increases in tandem with policy performance. Using the pre-trained D&P network weights as initialization preserves the health dynamics modeling and fault discrimination capabilities gained from contrastive learning, avoiding learning fault representations from scratch. Simultaneously, guided by reinforcement learning tasks, the D&P network learns to compress redundant information unnecessary for the policy, retaining only the core fault features relevant to action decisions. This two-stage training paradigm of pre-training and joint fine-tuning enables the D&P network to possess both general fault perception capabilities and task-oriented optimization capabilities, reducing the sample complexity of subsequent reinforcement learning by 5 to 10 times compared to direct training from scratch.

[0112] S305: Electronic device 10 constructs K tokens corresponding to N failure modes, where each token represents a failure scenario and K is a positive integer.

[0113] In some embodiments, each token includes: a failure mode identifier, a failure severity quantification value, and a joint identifier.

[0114] As an example, electronic device 10 discretizes each failure mode and its parameter range into tokens. Each token contains three dimensions of information: failure mode identifier. Quantitative value of fault severity and joint markings ,Right now The failure mode identifier distinguishes between various fault types, such as increased joint friction, motor delay, sensor noise, brake jamming, and power supply voltage drop. The fault severity quantification discretizes continuous parameters (e.g., friction increase ratio 0-2.0) into low, medium, and high levels. The joint identifier locates the specific joint where the fault occurred (e.g., one of the 12 joints). Thus, the three-dimensional structure of the token preserves fine-grained information on fault type, severity, and location, providing precise fault condition inputs for the strategy, enabling it to learn differentiated control behaviors for different fault scenarios.

[0115] S306: Electronic device 10 trains a conditional shared policy network in a simulation environment based on fault conditions.

[0116] In this process, a potential spatial smoothness constraint is added during the training of the shared policy network, and the fault condition is periodically set to the embedding vector of the token or a vector resulting from the mixture of embedding vectors of different tokens. The shared policy network is used for the robot's action decision.

[0117] In some embodiments, the electronic device 10 can map each of the K tokens to a d-dimensional latent vector through an embedding layer. This transforms discrete fault conditions into continuous vector representations, which serve as input to the shared policy network. This enables fault conditions to be input into the shared policy network in a differentiable manner, thus supporting end-to-end conditional policy training.

[0118] In some embodiments, the mathematical form of the above-mentioned shared policy network can be represented by the following formula (2).

[0119] (2).

[0120] In formula (2), ⊕ represents a fusion operation (such as splicing or adding). To observe the state, Let be the embedding vector of the token (i.e., the fault token). Then, the shared policy network... latent vector of input tokens The fault conditions and the robot's observed state in the simulation environment are represented. Output the corresponding action command .

[0121] During training, the electronic device adds a latent spatial smoothness constraint to the training objective. The latent space of the embedding vector of the forced token is smooth and interpolable.

[0122] As an example, the latent spatial smoothness constraint is determined based on the semantic distance between the different tokens. For example, the latent spatial smoothness constraint described above can be expressed by the following formula (3).

[0123] (3).

[0124] in, For tokens and Semantic distance between them To prevent division by zero of small constants, this regularization term... This forces tokens with similar semantics to have similar embedding vectors in the latent space, while tokens with different semantics are pushed apart to a certain extent.

[0125] Furthermore, in some embodiments, the electronic device 10 periodically performs token mixing and interpolation training during the training process of the shared policy network. As an example, the token mixing process can be represented by the following formula (4).

[0126] (4).

[0127] Two tokens are randomly selected. With tokens The embedding vector, with The mixing coefficients generate the mixed embedding vector. The mixed vector is then used as the fault condition input to the shared policy network. In this way, the shared policy network learns to adapt to any intermediate state in the fault parameter space, enhancing the policy's robustness and interpolation ability at any location in the continuous fault parameter space.

[0128] In some embodiments, the architecture of the shared policy network may include an input layer, an encoder, and a decoder. As an example, the electronic device 10 may define a token set of K=150 tokens, covering a combination of 5 failure modes, 3 severity levels, and 10 joints. Furthermore, each token is mapped to a 64-dimensional latent vector through an embedding layer. Furthermore, the input layer of the shared policy network receives the observed state. (Including 12-dimensional joint positions and 12-dimensional joint velocities) and the embedding vector of the token This observation status First, features are extracted using an encoder. The encoded features are then concatenated with the token embedding to form a 288-dimensional fused vector. Subsequently, the decoder outputs a 12-dimensional action command, i.e., the output... Action instructions in .

[0129] As an example, during the training of the shared policy network, electronic device 10 collects the trajectory for each token cyclically (under the fault condition specified by the corresponding token) and performs policy optimization using the Proximal Policy Optimization (PPO) algorithm. Furthermore, a latent spatial smoothness constraint regularization term is added to the training objective. Weighting coefficient This forces the latent space of the token embedding vectors to be smooth and interpolable. Furthermore, token mixing training is performed every 300 training steps: two tokens are randomly selected. and ,according to Generate hybrid embedding vectors as fault conditions for training, allowing the policy to learn to adapt to intermediate fault states. The electronic device 10 continues training until the average reward across all token combinations exceeds 95% of the baseline performance, resulting in the trained shared policy network.

[0130] S307: In the later stage of training of the shared policy network, the electronic device 10 performs domain randomization on at least one of the physical parameters, environmental parameters and fault parameters in the simulation environment.

[0131] It is understandable that, for outdoor multi-joint mobile robots, this application may employ a three-dimensional synthesis domain randomization strategy to enhance the generalization ability of the strategy.

[0132] In some embodiments, the physical parameters mentioned above include at least one of the following: joint friction coefficient, motor delay time, mass, or rod length.

[0133] For example, joint friction coefficient The randomization range can be ∈[ ×0.5, [×2.0] That is, the joint friction coefficient is randomly sampled within the range of 0.5 to 2.0 times the nominal value to simulate changes in joint wear or lubrication conditions.

[0134] For example, motor delay time The randomization range can be ∈[0ms, 200ms]. That is, the motor delay time is randomly sampled within the range of 0 to 200 milliseconds to simulate signal transmission delay or motor aging.

[0135] For example, quality The randomization range can be That is, the mass of the connecting rod is randomly sampled within the range of 0.9 to 1.2 times the nominal value to simulate the uncertainty of load changes or mass distribution.

[0136] For example, rod length The randomization range can be That is, the rod length is randomly sampled within the range of 0.95 to 1.05 times the nominal value to simulate manufacturing tolerances or assembly errors.

[0137] In some embodiments, environmental parameters include at least one of the following: topographic elevation map, slope angle, ground friction coefficient, and wind disturbance.

[0138] For example, terrain height maps can be generated using Berlin noise or a simple undulation pattern to create random terrain.

[0139] For example, slope angle The randomization range can be .

[0140] For example, the coefficient of ground friction The randomization range is They simulate different ground materials such as soil, grass, and rocks.

[0141] For example, wind interference The randomization range of wind force is Furthermore, the direction is random to simulate natural wind disturbances in an outdoor environment.

[0142] In addition, the fault parameter randomization combines the parameter space of the aforementioned N failure modes to randomly sample the severity parameters of each failure mode.

[0143] It is understandable that the above-mentioned three-dimensional domain randomization strategy is activated in the later stage of training, so that the strategy undergoes a wide variety of simulation condition changes during the training process, forcing the strategy to learn robust control behavior to changes in physical parameters, environmental changes and fault changes.

[0144] As an example, electronic device 10 can enable comprehensive domain randomization, such as randomization of physical parameters, environmental parameters, and fault parameters, in the last third of the training steps of the shared control strategy, thereby forcing the strategy to adapt to a wide range of simulation condition variations.

[0145] S308: Electronic device 10 adopts a multi-stage course learning strategy throughout the training process of the shared policy network, gradually adjusting the training difficulty of the simulation environment.

[0146] In some embodiments, the above-mentioned multi-stage course learning strategy includes the following four stages in sequence.

[0147] The first stage corresponds to a flat surface and the first level of fault severity (low fault severity). In this stage, the robot walks on a flat surface, and the fault severity is controlled within a low range (e.g., the increase in joint friction does not exceed 0.5, and the motor delay does not exceed 50ms), allowing the strategy to first learn basic motion control capabilities.

[0148] The second stage corresponds to terrain with a slope less than the first set slope (e.g., 15°) and a second fault severity level (medium fault severity). In this stage, the robot begins to walk on gentle slopes, and the fault severity increases to a medium range (e.g., friction increases by 0.5~1.0, motor delay by 50~100ms). The strategy learns its adaptability under slope and medium fault conditions.

[0149] The third stage corresponds to the terrain undulations and changes in the ground friction coefficient, and the third level of fault severity (high fault severity). In this stage, the robot faces complex undulating terrain (such as height maps generated using Berlin noise) and varying ground friction coefficients (simulating different materials such as soil, grass, and rock). The fault severity is further increased (such as friction increasing by 1.0~2.0, and motor delay by 100~200ms). The strategy learns robust control capabilities under complex terrain and high fault conditions.

[0150] The fourth stage involves slopes greater than a second predetermined slope (e.g., 30°) and multiple failure modes occurring simultaneously, along with wind interference. Specifically, the second predetermined slope is greater than the first predetermined slope, the severity of the second failure is greater than the severity of the first failure, and the severity of the third failure is greater than the severity of the second failure. In this stage, the robot faces extreme conditions, including steep slopes, the simultaneous occurrence of multiple failure modes (e.g., a combined failure of increased joint friction and motor delay), and wind interference from random directions (0~200N). The strategy learns the robot's comprehensive coping ability under these extreme conditions.

[0151] Through the progressive training in the above four stages, the strategy gradually transitions from simple scenarios to complex scenarios, and stably learns robust control strategies that can adapt to various terrains, fault severity levels, and environmental disturbances.

[0152] S309: Electronic device 10 verifies policy performance from a simulation environment to a real environment based on the trained D&P network and the trained shared policy network, and deploys the successfully verified shared policy network to the robot.

[0153] In some embodiments, the above verification process first involves a comprehensive evaluation in a simulation environment. The shared policy network is tested under completely random combinations of terrain and faults, with the parameter values ​​used in the test being outside the range of the training set parameters, to verify the shared policy network's generalization ability to unknown scenarios. Then, a generalization evaluation of the shared policy network is performed, for example, sampling within a 5×5 grid parameter space to test the policy performance under all combinations of failure modes, covering cross-combinations of various fault types and their severity. Finally, when the task completion rate in the simulation scenario exceeds 90%, the shared policy network can be transferred to a real robot.

[0154] During the real-world deployment phase of the control strategy, the complexity of the test scenarios can be gradually increased. For example, in a real-world environment, validation can begin on flat ground to ensure the robot's basic walking ability. Then, it can transition to slopes to test its slope adaptability. Next, it can expand to rugged terrain to verify robustness in complex ground environments. Finally, comprehensive validation should be conducted under extreme conditions (such as steep slopes, multi-mode combined failures, and wind interference) to ensure the reliability and robustness of the strategy in real-world outdoor environments.

[0155] Thus, the robot control strategy simulation training method provided in this application, through a complete pipeline including simulation environment construction, fault injection and paired data generation, D&P network pre-training and joint fine-tuning, tokenized conditional policy training, course learning, domain randomization, and Sim2Real verification and deployment, can achieve simulation training of robot control strategies for multiple failure modes. In this way, this application can address multiple failure modes with a single strategy, smoothly adapt to continuous fault parameters, and effectively transfer from simulation to reality, improving overall sample efficiency by 5-10 times, and achieving a task completion rate of over 90% in simulation verification.

[0156] Reference Figure 4 The flowchart illustrates a simulation training method for a robot control strategy provided in this application embodiment. The execution entity of this method can be... Figure 1 Electronic device 10 in the scene shown. Among them, Figure 4 The process shown is the same as Figure 3 The only difference is that Figure 4 In the joint optimization process of the D&P network and the control policy network, a teacher-student learning mechanism is adopted. Specifically, the process includes the following steps:

[0157] S401 to S403. As an example, S401 to S403 are... Figure 3 S301 to S303 shown are the same, and will not be described again here.

[0158] S404A: For each failure mode, the electronic device 10 trains an expert policy network in a simulation environment using reinforcement learning, thereby obtaining N expert policy networks that correspond one-to-one with the N failure modes.

[0159] In some embodiments, the electronic device 10 can train independent expert policies for each of the multiple failure modes. Each expert policy network is optimized on the fault trajectory dataset corresponding to its respective failure mode. The objective of the expert policy network is to maximize the cumulative expected reward. ,in This is the fault trajectory dataset for failure mode m. For example, electronic device 10 can use algorithms such as PPO algorithm and SoftActor-Critic (SAC) to train each expert policy network independently.

[0160] As an example, each expert policy network is trained using the PPO algorithm, with a network structure of 2 hidden layers, each with 200 units, and a learning rate of [value missing]. For example, the reward function during the training process of an expert policy network. It can be expressed by the following formula (5).

[0161] (5).

[0162] The weights are respectively , =0.1、 =0.01, which respectively encourages forward speed, penalizes sudden changes in movement, and penalizes energy consumption. Furthermore, It can also be expressed as This is a penalty item used to punish sudden changes in movement and encourage smoother movements. This is a penalty term used to punish large movements and encourage energy conservation and efficiency. Furthermore, during training, trajectories are collected in the simulation environment of the corresponding failure modes for PPO updates. The strategy is evaluated every 500,000 steps (on 100 test trajectories, under no-failure conditions), and the average reward is recorded. The optimal expert model is saved after training until the reward converges or reaches 2 million steps.

[0163] S404B: Electronic device 10 performs joint fine-tuning of the pre-trained D&P network and the student policy network in a simulation environment to obtain the trained D&P network and the trained student policy network.

[0164] The student policy network takes the fault-related latent features output by the pre-trained D&P network as conditional input and learns from N expert policy networks through distillation. The student policy network is used for the robot's action decision-making.

[0165] As an example, student policy networks A two-layer MLP structure with 200 units per layer is used. The input is the observed state `s`, and the output is the action instruction `a`. Furthermore, the student policy network... Fault-related latent features output by the pre-trained D&P network Given the conditional input, learn from N expert policy networks through distillation.

[0166] As an example, the distillation objective is to minimize the difference between the student's policy and the expert's policy output, and the loss function can be expressed by formula (6).

[0167] (6).

[0168] It is understandable that the expert selection mechanism works as follows: for each trajectory segment, latent features are calculated. The expert with the highest similarity to the embedding vectors of each expert is dynamically selected as the teacher. The distillation learning rate is set to 0.0001, and training is iterated until the generalization performance of the student policy under multiple failure mode combinations exceeds 95%. After training, the distilled student policy is saved as the final policy for the robot's action decision-making.

[0169] It is understood that in this embodiment, independent expert policy networks are trained for multiple failure modes, and each expert policy network is optimized within a specific failure mode range, making each expert a controller for the corresponding failure mode. Furthermore, the student policy network interacts with multiple expert policy networks but does not directly receive failure mode labels. The distillation process is implemented through the latent features of the D&P network; that is, the observed state is processed by the encoder of the D&P network to output a latent representation. As a conditional input to the student policy network, this enables the network to learn to select the optimal action by observing implicit fault indicators (rather than explicit labels). This enhances the generalization ability of the student policy, allowing it to perform well even with fault parameter combinations outside the training range.

[0170] S405: Electronic device 10 constructs K tokens corresponding to N failure modes, where each token represents a failure scenario and K is a positive integer.

[0171] As an example, the above S405 and Figure 3 The S305 shown is the same, so it will not be described again here.

[0172] S406: Electronic device 10 trains a conditional shared policy network in a simulation environment based on fault conditions.

[0173] As an example, the above S406 and Figure 3 The similarities between S306 and S406 will not be repeated here. The only difference is that the shared policy network in S406 can be trained on the basis of the student policy network.

[0174] In some embodiments, the electronic device 10 uses the trained student policy as the initial weights of the shared policy, and then trains the shared policy network in a simulated environment based on fault conditions to obtain the trained shared policy network. Thus, by using the trained student policy as the initial weights for conditional training of the shared policy, the basic motor skills already mastered by the student policy are transferred to the shared policy through transfer learning. This allows the shared policy to focus on learning action adjustments under fault conditions without having to learn walking ability from scratch. Consequently, training time is significantly shortened, sample efficiency is improved, and policy performance is typically superior to independent training, making it a highly efficient optimization implementation method.

[0175] S407: In the later stages of training the shared policy network, the electronic device 10 performs domain randomization on at least one of the physical parameters, environmental parameters, and fault parameters in the simulation environment.

[0176] S408: Electronic device 10 adopts a multi-stage course learning strategy throughout the training process of the shared policy network, gradually adjusting the training difficulty of the simulation environment.

[0177] As an example, S408 and S409 are respectively with Figure 3 S308 and S309 are shown to be the same, and will not be described in detail here.

[0178] S409: Electronic device 10 verifies policy performance from a simulation environment to a real environment based on a trained D&P network, and a trained student policy network or shared policy network, and deploys the successfully verified policy network to the robot.

[0179] Thus, the above-mentioned verification and deployment scheme takes the student policy network as the deployment object, utilizes its blind adaptation capability (not dependent on explicit fault tokens) and the real-time fault feature extraction capability of the D&P network, and achieves safe, reliable, and zero-label migration deployment from simulation to the real environment by using the quantitative verification standard of 90% task completion rate in simulation and progressive testing in the real environment on flat ground, slope, rugged terrain and extreme conditions, under the premise of fully verifying the policy generalization capability.

[0180] Next, the hardware structure of the electronic device and robot in the embodiments of this application will be described.

[0181] Figure 5 A schematic diagram of the hardware structure of an electronic device is shown according to an embodiment of this application.

[0182] like Figure 5As shown, the electronic device 10 includes a processor 1001 and a memory 1002, used to store instructions executed by the processor 1001, or input data required by the processor 1001 to execute instructions, or data generated after the processor 1001 executes instructions. Furthermore, the electronic device 10 also includes an interface circuit 1003. The processor 1001 and the interface circuit 1003 are coupled to each other. It can be understood that the interface circuit 1003 can be a transceiver, an input / output interface, or a communication module.

[0183] The aforementioned electronic device 10 can be used to perform Figure 3 or Figure 4 The method is illustrated. For example, processor 1001 is used to execute processes such as simulation environment construction, fault injection, paired training data generation, D&P network pre-training, policy reinforcement learning, conditional policy training, domain randomization, and policy simulation and deployment.

[0184] Figure 6 A schematic diagram of the hardware structure of a robot is shown according to an embodiment of this application.

[0185] like Figure 6 As shown, the robot 20 includes modules such as processor 201, sensor unit 202, actuator 203, driver 204 and transmission device 205.

[0186] The processor 201 is the central processing unit of the robot 20, responsible for data fusion, algorithm execution, and instruction issuance. For example, the processor 201 may include a microcontroller (MCU), a high-performance computing platform, etc.

[0187] Sensor unit 202 is used to acquire information about its own state and the external environment. For example, sensor unit 202 includes, but is not limited to, an inertial measurement unit (IMU) 2021, a vision sensor 2022, a LiDAR (Light Detection and Ranging) system 2023, a tactile sensor 2024, and an ultrasonic sensor 2025. The inertial measurement unit 2021 measures acceleration and angular velocity for attitude estimation and balance control. The vision sensor 2022 can be an RGB-D depth camera (RealSense) or a camera. The LiDAR 2023 is used for Simultaneous Localization and Mapping (SLAM) and navigation. The tactile sensor 2024 may include a joint torque sensor and a six-dimensional force sensor, which are key to achieving compliant control. The ultrasonic sensor 2025 is used for obstacle avoidance.

[0188] Actuator 203 and driver 204 are used to convert motion commands into physical actions. Specifically, driver 204 is used to convert control signals into high-voltage electrical signals, such as servo drivers or robot joint drive circuits. Actuator 203 can be a servo motor or a DC motor / servo motor. Servo motors are used for high-precision position / speed control and are the core of humanoid robots and robotic arm joints. DC motors / servo motors are used for wheeled movement or simple joint drive.

[0189] The transmission device 205 can be a harmonic reducer to achieve high torque output at the joint.

[0190] In addition, robot 20 may also include a mobility structure, such as a wheeled / tracked chassis, a robotic arm, or multi-degree-of-freedom bionic legs like those of the Unitree G1. Furthermore, robot 20 may also include a power system such as a lithium battery, mechanical structures such as body links (often made of aluminum alloy, engineering plastics, or carbon fiber), wiring harnesses, and a cooling system (including a fan and a VC heat sink).

[0191] This application also provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by the one or more processors, cause the electronic device to execute the robot control strategy simulation training method provided in the above embodiments.

[0192] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the robot control strategy simulation training method provided in the above embodiments.

[0193] This application also provides a computer program product for implementing the robot control strategy simulation training method provided in the above embodiments.

[0194] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer program modules or module code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0195] Computer program modules or module code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0196] Module code can be implemented using a high-level modular language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used to implement module code when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0197] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer-readable storage media. Therefore, machine-readable storage media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable storage media include any type of machine-readable storage media suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0198] In this specification, the reference to "an embodiment" or "an embodiment" means that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least one exemplary implementation or technology disclosed according to an embodiment of this application. The appearance of the phrase "in an embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0199] The disclosure of embodiments of this application also relates to means for performing operations in text. This means may be specifically constructed for the claimed purpose or may include a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, application-specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, and each may be coupled to a computer system bus. Furthermore, the computer mentioned in the specification may include a single processor or may be an architecture employing multiple processors for increased computing power.

[0200] Furthermore, the language used in this specification has been primarily chosen for readability and instructional purposes and may not have been chosen to depict or limit the disclosed subject matter. Therefore, the embodiments disclosed herein are intended to illustrate, and not limit, the scope of the concepts discussed herein.

Claims

1. A simulation training method of a robot control strategy, characterized by, The method includes: Construct a simulation environment for the robot, wherein the simulation environment is related to at least one of the following: joint dynamics, motor characteristics, ground contact model, or environmental factors; Based on the simulation environment, a paired training dataset corresponding to the target training task is obtained. The paired training dataset includes a health trajectory dataset and a fault trajectory dataset corresponding to N failure modes, where N is a positive integer. On the paired training dataset, contrastive learning is used to pre-train the network to obtain a pre-trained fault detection and parameter prediction (D&P) network, which is used for failure mode identification of the robot. The pre-trained D&P network is fine-tuned using reinforcement learning to obtain the trained D&P network. Construct K tokens corresponding to the N failure modes, where each token represents a failure scenario and K is a positive integer; Based on fault conditions, a conditional shared policy network is trained in the simulation environment to obtain the trained shared policy network. In the training process of the shared policy network, a latent spatial smoothness constraint is added, and the fault conditions are periodically set as the embedding vector of the token or a vector composed of the embedding vectors of different tokens. The shared policy network is used for the robot's action decision. In the later stages of training the shared policy network, at least one of the physical parameters, environmental parameters, and fault parameters in the simulation environment is domain randomized. A multi-stage course learning strategy is adopted throughout the training process of the shared policy network to gradually adjust the training difficulty of the simulation environment. Based on the trained D&P network and the trained shared policy network, the policy performance is verified from the simulation environment to the real environment, and the successfully verified shared policy network is deployed to the robot.

2. The method according to claim 1, characterized in that, The simulation environment for building the robot includes: A high-fidelity model of the robot is built using a set physics engine. The high-fidelity model contains X joints, where X is a positive integer. Configure joint parameters for each of the X joints, the joint parameters including at least one of the following: mass, length, or coefficient of friction; A control frequency, observation space, and motion space are set for the high-fidelity model, wherein the observation space includes: the inertial data of the torso of the high-fidelity model, and the positions and velocities of the X joints; the motion space includes: the positions or torques of the X joints; Use a terrain generator to generate terrain, which includes at least one of flat land, slope, and uneven terrain.

3. The method according to claim 1 or 2, characterized in that, The joint dynamics are related to at least one of the inertia, damping, and friction models of each joint of the robot; The motor characteristics are related to the motor delay and / or torque response curve of the robot; The ground contact model is related to at least one of Coulomb friction, collision detection, and contact force feedback between the robot and the ground; The environmental factors are related to at least one of the following: terrain profile, ground slope, ground friction coefficient, and wind resistance of the environment in which the robot is located.

4. The method according to claim 1 or 2, characterized in that, The N failure modes include two or more of the following: increased joint friction, motor delay, sensor noise, brake jamming, and power supply voltage drop; and each failure mode has a parameterized parameter range and a dynamic activation mechanism.

5. The method according to claim 1, characterized in that, Each fault trajectory data has a set of fault labeling information, which includes: the time of fault occurrence, joint identifier, failure mode identifier, and fault severity quantification value.

6. The method according to claim 1, characterized in that, The D&P network comprises: a feature encoder, a joint prediction head, and a fault classification head, wherein... The feature encoder is used to input the observation state of the robot and output the hidden features corresponding to the fault. The observation state includes the joint position, joint velocity and inertial data of the robot. The fault classification head is used to input the hidden features and output the probability distribution of the joint on the N failure modes; The joint prediction head is used to input the latent features and output the joint position deviation.

7. The method according to claim 1 or 6, characterized in that, The process involves pre-training the network using contrastive learning on the paired training dataset to obtain a pre-trained fault detection and parameter prediction (D&P) network, including: On the paired training dataset, positive sample pairs and negative sample pairs are constructed, wherein the positive sample pairs include a healthy trajectory data and a fault trajectory data under the same task, and the negative sample pairs include two fault trajectory data under different failure modes or two fault trajectory data under different tasks. Based on the positive sample pairs and the negative sample pairs, the network is trained to obtain the pre-trained D&P network.

8. The method according to claim 7, characterized in that, The first total loss in the pre-training process of the D&P network is the weighted sum of the joint state deviation reconstruction loss, the contrastive learning loss, and the fault classification loss. The reconstruction loss is used to measure whether the D&P network accurately predicts the joint coupling mode under normal dynamics. The contrastive learning loss is used to distinguish between healthy joint coupling patterns and faulty joint coupling patterns. The fault classification loss is used to train the ability to identify failure modes on hidden layer features.

9. The method according to claim 1 or 6, characterized in that, The step of fine-tuning the pre-trained D&P network based on reinforcement learning to obtain the trained D&P network includes: For each of the aforementioned failure modes, an expert policy network is trained using reinforcement learning in the simulation environment to obtain N expert policy networks that correspond one-to-one with the N failure modes, wherein each expert policy network is optimized on the fault trajectory dataset corresponding to the corresponding failure mode. In the simulation environment, the pre-trained D&P network and the student policy network are jointly fine-tuned to obtain the trained D&P network and the trained student policy network. The student policy network learns from the N expert policy networks by using the fault-related latent features output by the pre-trained D&P network as conditional input and through distillation. The student policy network is used for the robot's action decision.

10. The method according to claim 9, characterized in that, The method further includes: Based on the trained D&P network and the student policy network, the policy performance is verified from the simulation environment to the real environment, and the successfully verified student policy network is deployed to the robot.

11. The method according to claim 9, characterized in that, The step of training a conditional shared policy network under the simulation environment based on fault conditions to obtain the trained shared policy network includes: The trained student policy is used as the initial weight of the shared policy, and the shared policy network is trained in the simulation environment based on the fault conditions to obtain the trained shared policy network.

12. The method according to claim 1, characterized in that, Each token contains: a failure mode identifier, a failure severity quantification value, and a joint identifier.

13. The method according to claim 1, characterized in that, The physical parameters include at least one of the following: joint friction coefficient, motor delay time, mass or rod length; The environmental parameters include at least one of the following: topographic elevation map, slope angle, ground friction coefficient, and wind interference; The fault parameters include parameters for each of the failure modes.

14. The method according to claim 1, characterized in that, The multi-stage course learning strategy includes the following stages in sequence: The stage corresponding to the severity of the first fault is a flat surface; The stage corresponding to the second fault severity is a terrain with a slope less than the first set slope. The terrain undulations and changes in the ground friction coefficient correspond to the stage of the severity of the third fault; The stage where the slope is greater than the second set slope and multiple failure modes are combined with wind interference, wherein the second set slope is greater than the first set slope and the severity of the second failure is greater than the severity of the first failure.

15. An electronic device, characterized in that, include: One or more processors; One or more memories; the one or more memories storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 14.

17. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1 to 14.

Images