Robot control method and system based on blind privilege information implicit alignment algorithm, equipment and medium
By employing an implicit alignment algorithm in the robot blind-walking algorithm and combining real-time and historical observation parameters with the Actor_Critic model, the problems of training complexity and information loss are solved, resulting in more stable and accurate terrain information estimation and improved policy robustness.
Patent Information
- Application Number
- CN202511570913.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-30
- Publication Date
- 2026-01-13
AI Technical Summary
Existing robot blind navigation algorithms suffer from complex training and information loss in environments with limited perception. In particular, the two-stage training process of the teacher-student framework and the DreamWaQ algorithm lack explicit privileged information supervision, leading to training instability and information loss.
An implicit alignment algorithm based on privileged information for blind walking is adopted. By acquiring real-time and historical observation parameters of the robot and combining them with the Actor_Critic model, features are extracted using the encoding and decoding module. Privileged information distillation is performed during end-to-end training to avoid the complex two-stage training process.
It achieves a more stable training process, faster convergence, more accurate terrain information estimation, and more robust and stable policies, avoiding information loss and inconsistency in the teacher-student framework.
Smart Images
Figure CN121325880A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot algorithm control technology, and more specifically, to a robot control method, system, device, and medium based on a blind privileged information implicit alignment algorithm. Background Technology
[0002] Blind navigation technology for robots is primarily applied in unstructured environments where external perception is limited or completely unknown. Its applications include autonomous movement across complex indoor and outdoor terrains, disaster relief site exploration, industrial facility inspection, and military reconnaissance.
[0003] Traditional quadruped robots rely on external sensors such as vision or lidar for environmental perception and navigation, but these methods have significant limitations in the aforementioned applications. In recent years, reinforcement learning-based motion control methods have made significant progress, enabling robots to adapt to complex terrains solely through proprioception by combining simulation training with real-world deployment. This technological advancement provides a new technical path for robust movement in perception-constrained environments and has also spurred innovation in related algorithms and processing devices.
[0004] like Figure 4 As shown, the currently commonly used algorithm is the teacher-student framework, which utilizes a two-stage learning process. First, a privileged-policy is trained using privileged information (terrain scan points, aircraft speed, etc.). Since privileged information is often unavailable in the real environment, the teacher-student framework will then have a "distillation" stage to distill the knowledge in the privileged-policy into a deploy-policy with only "internal perception information" as input. However, the teacher-student model has the problem of a complex two-stage model training process, and a lot of useful information is lost during the distillation process from the privileged-policy to the deploy-policy (only successful experiences are distilled).
[0005] like Figure 5 As shown, the DreamWaQ algorithm can also be used. The DreamWaQ algorithm uses local perception historical observations to estimate privileged information such as terrain information and aircraft speed. It also uses an asymmetric PPO algorithm (the inputs of policy and critic are different, only the input of critic contains privileged information). However, it still uses VAE to estimate terrain information, the KL term has no upper or lower bounds, it is easy to crash during training, and there is no explicit privileged information supervision signal. Instead, it uses historical observations to predict the next observation to implicitly infer privileged information. Summary of the Invention
[0006] The purpose of this invention is to provide a robot control method, system, device, and medium based on a blind privileged information implicit alignment algorithm to solve the above-mentioned problems in the prior art.
[0007] This invention is achieved through the following technical solution: A robot control method based on a blind privileged information implicit alignment algorithm includes: Obtain real-time and historical observation parameters of the robot's real-time operation; Acquire privileged observation parameters, which include a second parameter acquired by the robot during runtime; Input the real-time observation parameters, historical observation parameters, and privileged observation parameters into the Actor_Critic model to obtain the joint action and the corresponding action evaluation value. Pass the action into the environment and receive the reward; Calculate the loss function and update the model until the network converges.
[0008] Preferred options also include: Feature extraction is performed on historical observation parameters and privileged observation parameters respectively to obtain historical observation parameter features Z_p and privileged observation parameter features Z_c; The historical observation parameter features Z_p and privileged observation parameter features Z_c are input into the Actor_Critic model.
[0009] Preferably, the feature extraction of historical observation parameters and privileged observation parameters respectively includes: Construct an encoding / decoding module and train it using historical observation parameters and privileged observation parameters to obtain a trained encoding / decoding module; The encoder in the trained encoder-decoder module extracts features from historical observation parameters and privileged observation parameters, respectively.
[0010] Preferably, the encoding / decoding module includes a GRU_Encoder encoder, an MLP_Encoder encoder, and an MLP_Decoder decoder; Historical observation parameters are used to obtain the first pre-privileged observation parameters through the GRU_Encoder encoder and MLP_Decoder decoder; The privileged observation parameters are used to obtain the second pre-privileged observation parameters through the MLP_Encoder encoder and MLP_Decoder decoder; The losses of historical observation parameter features and privileged observation parameter features, first pre-privileged observation parameters and privileged observation parameters, and first pre-privileged observation parameters and second pre-privileged observation parameters are calculated respectively. Based on the comprehensive analysis of the above losses, the encoding and decoding modules are updated.
[0011] Preferably, the historical observation parameters are used to obtain the first pre-privileged observation parameters through the GRU_Encoder and MLP_Decoder, including: Historical observation parameters are obtained by the GRU_Encoder to obtain historical observation parameter features, and the first pre-privileged observation parameters are obtained by the MLP_Decoder based on the historical observation parameter features.
[0012] Preferably, the privileged observation parameters are used to obtain the second pre-privileged observation parameters through the MLP_Encoder and MLP_Decoder. Privileged observation parameters are encoded by the MLP_Encoder to obtain privileged observation parameter features, and based on the privileged observation parameter features, second pre-privileged observation parameters are obtained by the MLP_Decoder.
[0013] Preferably, calculating the loss includes:
[0014] In the formula, For loss function, The total number of samples, For the sample true value, These are the predicted values for the sample.
[0015] Secondly, the present invention also provides a robot control system based on a blind privileged information implicit alignment algorithm, for executing the above-mentioned robot control method based on the blind privileged information implicit alignment algorithm, comprising: The data processing module is configured to acquire real-time and historical observation parameters of the robot during its operation; acquire privileged observation parameters, including second parameters acquired by the robot during operation; input the real-time, historical, and privileged observation parameters into the Actor_Critic model to obtain joint actions and corresponding action evaluation values; pass the actions to the environment to obtain rewards; calculate the loss function and update the model until the network converges. The execution module outputs the obtained joint motion actions to the robot control module.
[0016] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described robot control method based on a blind privileged information implicit alignment algorithm.
[0017] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described robot control method based on a blind privileged information implicit alignment algorithm.
[0018] The technical solution of the present invention has at least the following advantages and beneficial effects: The method provided by this invention mainly includes acquiring real-time observation parameters and historical observation parameters of the robot during real-time operation; acquiring privileged observation parameters, which include second parameters acquired by the robot during operation; and inputting the real-time observation parameters, historical observation parameters, and privileged observation parameters into the Actor_Critic model to obtain joint actions and corresponding action evaluation values. Through this method, without using VAE, the training process is more stable and converges faster; the terrain information estimated by policy_ae is more accurate and interpretable (with Z_c of critic_ae as supervision); end-to-end training avoids the complex two-stage training process, and learns while distilling (distilling privileged information latent variables, unlike policy distillation in the teacher-student framework), which avoids the teacher-student framework only distilling "successful" experiences, making the policy more robust and stable. The policy in simulation training is consistent with the deployed policy, avoiding the gap between Teacher and Student in the teacher-student training framework. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the network model of the present invention; Figure 2 This is a schematic diagram of the network model incorporating Z_c and Z_p according to the present invention; Figure 3 This is the encoding / decoding module of the present invention; Figure 4 This is a schematic diagram of the training model for the teacher-student framework in existing technologies; Figure 5 This is a schematic diagram of the DreamWaQ algorithm in the prior art. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0022] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of modules in this application is a logical division; in practical applications, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the connection, coupling, or communication in this application can be a direct connection, coupling, or communication between related objects, or an indirect connection, coupling, or communication through other devices. Moreover, the connection, coupling, or communication between objects can be electrical or other similar forms, which are not limited in this application. The independently described modules or sub-modules may or may not be physically separated; they may be implemented in software or hardware, and some modules or sub-modules may be implemented in software, with the processor calling the software to implement the function of these modules or sub-modules, while other modules or sub-modules may be implemented in hardware, such as through hardware circuits. Furthermore, some or all of the modules can be selected to achieve the purpose of this application's solution according to actual needs.
[0023] Please refer to Figure 1 This invention provides a robot control method based on an implicit alignment algorithm using privileged information for blind walking, which is a robot blind walking algorithm trained on an Actor_Critic model. Using privileged information to assist training, the robot can complete stable walking tasks with fewer observations during actual operation. Specifically, it includes: S101: Obtain real-time and historical observation parameters of the robot's real-time operation; Among them, real-time observation parameters and historical observation parameters can include robot angular velocity, gravity projection, key positions, etc. S102: Obtain privileged observation parameters, the privileged observation parameters including the second parameters obtained by the robot during operation; The second parameter can be a parameter that is difficult for sensors to obtain directly during actual robot operation but is important for robot control, such as robot linear velocity and terrain scanning points.
[0024] S103: Input the real-time observation parameters, historical observation parameters and privileged observation parameters into the Actor_Critic model to obtain the joint action and the corresponding action evaluation value. The Actor_Critic model consists of two parts: the Actor, responsible for executing actions, determines the appropriate action based on the current state. Its role is the policy itself; and the Critic, responsible for evaluating actions, assesses the quality of the Actor's actions in a given state. Its role is the value function.
[0025] The Actor part takes the current policy, the input state, and outputs an action. The Critic part (usually the state value function V) is responsible for evaluation. The Critic calculates the state value of the current state and the state value of the next state.
[0026] S104: Pass the action into the environment and receive the reward; The `step(action)` method of the environment is invoked. This process is the standard interface for reinforcement learning simulators (environments), allowing the agent's algorithm to operate independently of the specific environment problem. Whether the environment is a game, robot simulation, or financial trading simulation, the agent interacts with it through the same `step` interface.
[0027] S105: Calculate the loss function and update the model until the network converges.
[0028] The goal of Critic is to make its estimates of state value more accurate. Therefore, its loss function is to make the predicted value closer to the target value. Critic Loss (Value Loss): Mean squared error is typically used.
[0029] The Actor's goal is to adjust its strategy to maximize expected reward. The Critic's calculations tell the Actor whether the current action is good or bad. Its loss function is:
[0030] In the formula, Let S be the loss function for the Actor, where A is the action and S is the state. The result is calculated for Critic.
[0031] During actual operation, the robot only needs to input the actual observation values into the Actor network to obtain the action. Each joint of the robot can move according to the corresponding parameters in the action to complete the smooth movement on complex terrain.
[0032] The method provided by this invention mainly includes acquiring real-time observation parameters and historical observation parameters of the robot during real-time operation; acquiring privileged observation parameters, which include second parameters acquired by the robot during operation; and inputting the real-time observation parameters, historical observation parameters, and privileged observation parameters into the Actor_Critic model to obtain joint actions and corresponding action evaluation values. Through this method, without using VAE, the training process is more stable and converges faster; the terrain information estimated by policy_ae is more accurate and interpretable (with Z_c of critic_ae as supervision); end-to-end training avoids the complex two-stage training process, and learns while distilling (distilling privileged information latent variables, unlike policy distillation in the teacher-student framework), which avoids the teacher-student framework only distilling "successful" experiences, making the policy more robust and stable. The policy in simulation training is consistent with the deployed policy, avoiding the gap between Teacher and Student in the teacher-student training framework.
[0033] like Figure 2-3 As shown, this is an exemplary embodiment of the present invention. However, historical observation information and privileged information are often much larger in dimensionality than real-time observation information. Therefore, an encoder should be used to extract features from them first. The Z_c and Z_p obtained from the feature extraction are latent variables containing feature information, which are used as part of the input to the Actor_Critic model. It also includes: Feature extraction is performed on historical observation parameters and privileged observation parameters respectively to obtain historical observation parameter features Z_p and privileged observation parameter features Z_c; The historical observation parameter features Z_p and privileged observation parameter features Z_c are input into the Actor_Critic model.
[0034] Specifically, the feature extraction of historical observation parameters and privileged observation parameters includes: An encoding / decoding module is constructed and trained using historical observation parameters and privileged observation parameters to obtain a trained encoding / decoding module. The encoder in the trained encoding / decoding module is then used to extract features from the historical observation parameters and privileged observation parameters, respectively.
[0035] In one exemplary embodiment of the present invention, the encoding and decoding module includes a GRU_Encoder encoder, an MLP_Encoder encoder, and an MLP_Decoder decoder; Historical observation parameters are used to obtain the first pre-privileged observation parameters through the GRU_Encoder encoder and MLP_Decoder decoder; The privileged observation parameters are used to obtain the second pre-privileged observation parameters through the MLP_Encoder encoder and MLP_Decoder decoder; The losses of historical observation parameter features and privileged observation parameter features, first pre-privileged observation parameters and privileged observation parameters, and first pre-privileged observation parameters and second pre-privileged observation parameters are calculated respectively. Based on the comprehensive analysis of the above losses, the encoding and decoding modules are updated.
[0036] In one exemplary embodiment of the present invention, obtaining the first pre-privileged observation parameter from the historical observation parameters through the GRU_Encoder and MLP_Decoder includes: obtaining historical observation parameter features from the historical observation parameters through the GRU_Encoder, and obtaining the first pre-privileged observation parameter based on the historical observation parameter features through the MLP_Decoder; obtaining the second pre-privileged observation parameter from the privileged observation parameters through the MLP_Encoder and MLP_Decoder includes: obtaining privileged observation parameter features from the privileged observation parameters through the MLP_Encoder, and obtaining the second pre-privileged observation parameter based on the privileged observation parameter features through the MLP_Decoder.
[0037] Specifically, calculating the loss includes:
[0038] In the formula, For loss function, The total number of samples, For the sample true value, These are the predicted values for the sample.
[0039] It's important to note that on the critic side, the `critic_ae` network learns an implicit representation of the privileged information `Privileged_Obs`. Since the critic network is only used in the simulator, it can obtain a very accurate implicit representation of the privileged information `Z_c`. Because the features we want to extract from historical observations and the privileged information are the same, `Z_c` and `Z_p` should contain the same information; therefore, `Z_c` will also serve as a supervisory signal for `Z_p`. Furthermore, the decoder `MLP_Decoder` is shared between `policy_ae` and `critic_ae`.
[0040] Secondly, the present invention also provides a robot control system based on a blind privileged information implicit alignment algorithm, for executing the above-mentioned robot control method based on the blind privileged information implicit alignment algorithm, comprising: The data processing module is configured to acquire real-time and historical observation parameters of the robot during its operation; acquire privileged observation parameters, including second parameters acquired by the robot during operation; input the real-time, historical, and privileged observation parameters into the Actor_Critic model to obtain joint actions and corresponding action evaluation values; pass the actions to the environment to obtain rewards; calculate the loss function and update the model until the network converges. The execution module outputs the obtained joint motion actions to the robot control module.
[0041] Wherein, History Obs are historical observation parameters, Obs are real-time observation parameters, Privileged Obs are privileged observation parameters, Policy is the policy, Critic is the value function, Pre_Privileged Obs1 is the first pre-privileged observation parameter, and re_Privileged Obs2 is the second pre-privileged observation parameter.
[0042] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0043] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer software product, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0044] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A robot control method based on a blind privileged information implicit alignment algorithm, characterized in that, include: Obtain real-time and historical observation parameters of the robot's real-time operation; Acquire privileged observation parameters, which include a second parameter acquired by the robot during runtime; Input the real-time observation parameters, historical observation parameters, and privileged observation parameters into the Actor_Critic model to obtain the joint action and the corresponding action evaluation value. Pass the action into the environment and receive the reward; Calculate the loss function and update the model until the network converges.
2. The robot control method based on the implicit alignment algorithm of blind privileged information according to claim 1, characterized in that, Also includes: Feature extraction is performed on historical observation parameters and privileged observation parameters respectively to obtain historical observation parameter features Z_p and privileged observation parameter features Z_c; The historical observation parameter features Z_p and privileged observation parameter features Z_c are input into the Actor_Critic model.
3. The robot control method based on the implicit alignment algorithm of blind privileged information according to claim 2, characterized in that, The feature extraction of historical observation parameters and privileged observation parameters respectively includes: Construct an encoding / decoding module and train it using historical observation parameters and privileged observation parameters to obtain a trained encoding / decoding module; The encoder in the trained encoder-decoder module extracts features from historical observation parameters and privileged observation parameters, respectively.
4. The robot control method based on the implicit alignment algorithm of blind privileged information according to claim 3, characterized in that, The encoding / decoding module includes a GRU_Encoder encoder, an MLP_Encoder encoder, and an MLP_Decoder decoder; Historical observation parameters are used to obtain the first pre-privileged observation parameters through the GRU_Encoder encoder and MLP_Decoder decoder; The privileged observation parameters are used to obtain the second pre-privileged observation parameters through the MLP_Encoder encoder and MLP_Decoder decoder; The losses of historical observation parameter features and privileged observation parameter features, first pre-privileged observation parameters and privileged observation parameters, and first pre-privileged observation parameters and second pre-privileged observation parameters are calculated respectively. Based on the comprehensive analysis of the above losses, the encoding and decoding modules are updated.
5. A robot control method based on a blind privileged information implicit alignment algorithm according to claim 3, characterized in that, The historical observation parameters, obtained through the GRU_Encoder and MLP_Decoder, include the following first pre-privileged observation parameters: Historical observation parameters are obtained by the GRU_Encoder to obtain historical observation parameter features, and the first pre-privileged observation parameters are obtained by the MLP_Decoder based on the historical observation parameter features.
6. A robot control method based on a blind privileged information implicit alignment algorithm according to claim 5, characterized in that, The privileged observation parameters are used to obtain the second pre-privileged observation parameters through the MLP_Encoder and MLP_Decoder. Privileged observation parameters are encoded by the MLP_Encoder to obtain privileged observation parameter features, and based on the privileged observation parameter features, second pre-privileged observation parameters are obtained by the MLP_Decoder.
7. A robot control method based on a blind privileged information implicit alignment algorithm according to claim 5, characterized in that, The calculation of the loss includes: In the formula, For loss function, The total number of samples, For the sample true value, These are the predicted values for the sample.
8. A robot control system based on an implicit alignment algorithm using blind privileged information, characterized in that, A robot control method for executing the implicit alignment algorithm based on blind privileged information as described in any one of claims 1-7, comprising: The data processing module is configured to acquire real-time and historical observation parameters of the robot during its operation; acquire privileged observation parameters, including second parameters acquired by the robot during operation; input the real-time, historical, and privileged observation parameters into the Actor_Critic model to obtain joint actions and corresponding action evaluation values; pass the actions to the environment to obtain rewards; calculate the loss function and update the model until the network converges. The execution module outputs the obtained joint motion actions to the robot control module.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a robot control method based on the implicit alignment algorithm of blind privileged information as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements a robot control method based on a blind privileged information implicit alignment algorithm as described in any one of claims 1-7.