Perception and control model training method, robot control method and device
By employing a multi-stage training method for perception and control models and establishing a unified latent representation interface using depth and elevation image codecs, the problems of sensitivity to data distribution and inconsistent deployment during the model training stage in existing technologies are solved, enabling precise control and stable movement of robots on complex terrains.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU XIAOPENG MOTORS TECH CO LTD
- Filing Date
- 2026-01-26
- Publication Date
- 2026-04-17
AI Technical Summary
In existing robotics technologies, the training phase of perception and control models is sensitive to data distribution, leading to unstable performance after policy transfer. The perception interfaces in the training and deployment phases are inconsistent, resulting in high engineering costs, long cycles, and insufficient real-time performance, making it difficult to maintain a sufficient frame rate and low jitter on lightweight platforms.
By employing a multi-stage training method that integrates perception and control, and utilizing depth and elevation image codecs to perform reinforcement learning on the action policy neural network, a unified latent representation interface is established, avoiding the simulation rendering link and improving sampling efficiency and training stability.
It improves model training efficiency, helps robots accurately perceive the environment, achieve precise action control, and make reliable foot lifting and stepping decisions in complex terrain such as stairs, reducing cross-domain migration and engineering integration costs.
Smart Images
Figure CN121879147A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotics, and in particular to a method for training a perception and control model, a robot control method, and a device. Background Technology
[0002] With the development of robotics technology, enabling robots to move autonomously in human living and working environments has become a key requirement.
[0003] Among related technologies, image networks that directly regress elevation maps from depth maps, learning methods that use point clouds as intermediate representations, end-to-end learning strategies driven by camera images, and schemes that rely on graphics rendering for joint training of visual control in simulations all have the following problems: such as the original image or depth directly driving the policy network, which is extremely sensitive to data distribution during the training phase, leading to unstable performance after policy transfer; and the inconsistency between the perception interfaces in the training and deployment phases, making it difficult to seamlessly switch policies. Summary of the Invention
[0004] To address or partially address the problems existing in related technologies, this application provides a training method for a perception and control model, a robot control method, and an apparatus. Through the interaction of perception and control, it is beneficial to improve the model training efficiency and can help the robot accurately perceive the working environment and achieve precise action control.
[0005] The first aspect of this application provides a training method for a perception and control model, comprising: acquiring multiple sample image sets; wherein each sample image set includes a depth image and an elevation image representing the same terrain; performing multi-stage training on an initial model based on the multiple sample image sets to obtain a perception and control model for controlling robot actions; wherein the multi-stage training includes: training an elevation image codec using elevation images to obtain a trained elevation image codec; training a depth image codec using depth images and the trained elevation image codec to obtain a trained depth image codec; and performing reinforcement learning training on an action policy neural network using the trained depth image codec; the initial model includes an elevation image codec, a depth image codec, and an action policy neural network.
[0006] In some embodiments, training a depth image codec using a depth image and a trained elevation image codec to obtain a trained depth image codec includes: inputting the depth image into a first encoder in the depth image codec to obtain a first multidimensional vector output by the first encoder; inputting the elevation image corresponding to the depth image into a second encoder in the trained elevation image codec to obtain a second multidimensional vector output by the second encoder; inputting the first multidimensional vector into a first decoder in the depth image codec to obtain a first predicted elevation image output by the first decoder; determining a first loss function value based on the first multidimensional vector, the second multidimensional vector, the first predicted elevation image, and the elevation image; and adjusting the parameters of the depth image codec using the first loss function value to obtain a trained depth image codec.
[0007] In some embodiments, training an elevation image codec using an elevation image to obtain a trained elevation image codec includes: inputting an elevation image into the elevation image codec to obtain a second predicted elevation image output by the elevation image codec; determining a second loss function value based on the second predicted elevation image and the elevation image; and adjusting the parameters of the elevation image codec using the second loss function value to obtain a trained elevation image codec.
[0008] In some embodiments, reinforcement learning training of the action policy neural network is performed using a trained deep image codec, including: obtaining a reference multidimensional vector output by the trained deep image codec; outputting the reference multidimensional vector and the true value of the joint's real position information to the action policy neural network to obtain the predicted position information of the joint output by the action policy neural network; and performing reinforcement learning training on the action policy neural network based on the difference between the predicted position information and the real position information.
[0009] A second aspect of this application provides a robot control method, comprising: acquiring a target image collected by the robot; and using a perception and control model to perform a robot control action task based on the target image; wherein the perception and control model is trained according to the above method.
[0010] In some embodiments, using a perception and control model to perform a robot action task based on a target image includes: using the perception and control model to obtain position information of robot joints corresponding to the target image; generating motion commands based on the position information; and performing the robot action task based on the motion commands.
[0011] A third aspect of this application provides a training apparatus for a perception and control model, the apparatus comprising: a first acquisition module for acquiring multiple sample image sets; wherein each sample image set includes a depth image and an elevation image representing the same terrain; a training module for performing multi-stage training on an initial model based on the multiple sample image sets to obtain a perception and control model for controlling robot actions; wherein the multi-stage training includes: training an elevation image codec using elevation images to obtain a trained elevation image codec; training a depth image codec using depth images and the trained elevation image codec to obtain a trained depth image codec; and performing reinforcement learning training on an action policy neural network using the trained depth image codec; the initial model includes an elevation image codec, a depth image codec, and an action policy neural network.
[0012] A fourth aspect of this application provides a robot control device, comprising: a second acquisition module for acquiring a target image collected by the robot; and a control module for executing a robot action task based on the target image using a perception and control model; wherein the perception and control model is trained according to the above method.
[0013] The fifth aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.
[0014] A sixth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.
[0015] The technical solution provided in this application may include the following beneficial effects: The technical solution provided in this application, through the interaction of perception and control, is conducive to improving model training efficiency and can help robots accurately perceive the working environment and achieve precise action control.
[0016] The technical solution of this application can also enable the robot to make more reliable decisions on lifting its feet and taking steps when approaching complex terrain such as stairs.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0018] The above and other objects, features and advantages of this application will become more apparent from the following description of exemplary embodiments of this application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of this application.
[0019] Figure 1 This is a flowchart illustrating the training method of the perception and control model shown in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the training of an elevation image codec according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the training of a depth image codec according to an embodiment of this application; Figure 4 This is a flowchart illustrating the action strategy neural network in an embodiment of this application; Figure 5 This is a schematic diagram illustrating a robot control method according to an embodiment of this application; Figure 6 This is a flowchart illustrating a robot control method according to another embodiment of this application; Figure 7 This is a schematic diagram of the structure of the training device for the perception and control model shown in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the robot control device shown in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation
[0020] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.
[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0022] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0023] Among related technologies, image networks that directly regress elevation maps from depth maps, learning methods that use point clouds as intermediate representations, end-to-end learning strategies driven by camera images, and schemes that rely on graphics rendering for joint training of visual control in simulations all suffer from the following problems: Strong coupling between perception and control, amplified domain gap: The policy network is directly driven by the original image or depth at the end, and the training phase is extremely sensitive to the data distribution; the simulation and reality differ significantly in terms of noise, exposure, reflection, occlusion, etc., which leads to unstable performance after policy transfer, requiring a large amount of domain randomization and retraining, resulting in high engineering costs and long cycles.
[0024] Simulation training relies on rendering, which limits efficiency and latency: To provide image / depth input to the RL policy network, rendering and encoding are performed in real time during simulation, and then coupled iteratively with dynamic simulation. The rendering pipeline significantly slows down the sampling rate, limiting the data throughput of reinforcement learning; at the same time, even small changes in rendering parameters can alter the input distribution, burdening training stability and experimental reproducibility.
[0025] Inconsistent representations and difficulties in cross-modal alignment: Actual deployments can only obtain depth sensor data, while simulations can directly obtain "true" elevation values. The lack of an aligned and replaceable unified latent representation between the "simulation-obtainable elevation representation" and the "real-world-obtainable depth representation" leads to inconsistencies in the perception interfaces between the training and deployment phases, making it difficult to seamlessly switch strategies.
[0026] Complex engineering integration and insufficient real-time performance: 3D reconstruction / dense fusion methods have high computational and storage overhead, making it difficult to run stably within the strict latency budget of gait control; end-to-end vision strategies have high requirements for inference devices, making it difficult to maintain sufficient frame rate and low jitter on lightweight platforms.
[0027] To address the aforementioned issues, this application provides a training method for a perception and control model. Through the interaction between perception and control, the model training efficiency can be improved, enabling robots to accurately perceive their working environment and achieve precise action control.
[0028] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.
[0029] Figure 1 This is a flowchart illustrating the training method of the perception and control model shown in the embodiments of this application.
[0030] See Figure 1 A training method for a perception and control model, the method comprising: Step 101: Obtain multiple sample image sets; wherein each sample image set includes depth images and elevation images representing the same terrain.
[0031] For example, a simulation scenario is constructed, creating various training terrains: flat ground, randomly distributed obstacles, steps of different heights, and gentle slopes, etc. Then, images are acquired to obtain multiple sample image sets.
[0032] For example, the robot's location and terrain are randomly generated, and for each location, the following data triples are collected simultaneously: depth map and elevation map.
[0033] A depth map simulates the image seen by a depth camera mounted on the head of a robot. This simulates real-world sensor data, which includes noise, occlusion, and viewpoint distortion. The elevation map is generated by directly obtaining the actual ground elevation information of the area in front of the robot's feet from the simulation engine, creating a grid map. The value of each grid point represents the elevation of that point. This is the "God's-eye view" truth value in the simulation world. Multiple sample image sets, such as a dataset containing tens of thousands of sets (depth maps, elevation maps).
[0034] Step 102: The initial model is trained in multiple stages based on multiple sample image sets to obtain a perception and control model for controlling the robot's actions.
[0035] For example, multi-stage training may include: training an elevation image codec using elevation images to obtain a trained elevation image codec; training a depth image codec using depth images and the trained elevation image codec to obtain a trained depth image codec; and using the trained depth image codec to perform reinforcement learning training on an action policy neural network; the initial model includes an elevation image codec, a depth image codec, and an action policy neural network.
[0036] It is understandable that by training an elevation image codec using elevation images, a trained elevation image codec is obtained. Then, by training a depth image codec using depth images and the trained elevation image codec, a trained depth image codec is obtained. This allows the two codecs to be aligned, thereby learning the mapping from depth images to elevation images and generating a unified latent representation.
[0037] It is understandable that by using a trained deep image codec to perform reinforcement learning training on the action policy neural network, that is, by using the latent representation generated by the elevation map for reinforcement learning training, the robot learns a walking strategy to control the robot's actions.
[0038] The training method for the perception and control model in this application, through the interaction of perception and control, helps to improve the training efficiency of the model, and enables the robot to accurately perceive the working environment and achieve precise action control.
[0039] In some embodiments, training an elevation image codec using an elevation image to obtain a trained elevation image codec includes: inputting an elevation image into the elevation image codec to obtain a second predicted elevation image output by the elevation image codec; determining a second loss function value based on the second predicted elevation image and the elevation image; and adjusting the parameters of the elevation image codec using the second loss function value to obtain a trained elevation image codec.
[0040] Figure 2 This is a schematic diagram illustrating the training of an elevation image codec according to an embodiment of this application. See also... Figure 2 The elevation image codec takes an elevation image as input (as ground truth) and outputs a reconstructed elevation image, learning a compact, task-relevant latent representation about terrain elevation. This latent representation serves as the standard for subsequent alignment.
[0041] For example, an elevation image codec can be trained to perfectly compress and reconstruct elevation maps. This can be done using a large dataset of simulated elevation map data (input elevation map, label elevation map). The input and label maps are the same elevation map. The loss function measures the difference between the reconstructed elevation map and the original input elevation map. By adjusting the parameters of the elevation image codec using a second loss function value, a well-trained elevation image codec can be obtained, capable of encoding any elevation map into a high-quality latent representation.
[0042] In some embodiments, training a depth image codec using a depth image and a trained elevation image codec to obtain a trained depth image codec includes: inputting the depth image into a first encoder in the depth image codec to obtain a first multidimensional vector output by the first encoder; inputting the elevation image corresponding to the depth image into a second encoder in the trained elevation image codec to obtain a second multidimensional vector output by the second encoder; inputting the first multidimensional vector into a first decoder in the depth image codec to obtain a first predicted elevation image output by the first decoder; determining a first loss function value based on the first multidimensional vector, the second multidimensional vector, the first predicted elevation image, and the elevation image; and adjusting the parameters of the depth image codec using the first loss function value to obtain a trained depth image codec.
[0043] Figure 3 This is a schematic diagram illustrating the training of a depth image codec according to an embodiment of this application. See also... Figure 3 A depth image codec takes a depth image, such as one obtained from a depth camera, as input and learns how to extract a latent representation from real sensor data with noise and distortion that is equivalent to a trained elevation image codec.
[0044] For example, training and alignment, i.e., ensuring that the latent representation generated by the depth image codec is consistent with the latent representation of the trained elevation image codec.
[0045] With fixed parameters for the trained elevation image codec, prepare triplet data (depth map, corresponding elevation map, and the first multidimensional vector of the latent representation generated by the trained elevation image codec). Then, input the depth map into the encoder of the depth image codec to obtain the latent representation Z_depth; input the same elevation map into the encoder of the trained elevation image codec to obtain the latent representation Z_height (as a supervision label); input Z_depth into the decoder of the depth image codec to output the predicted elevation map. The loss function consists of two parts: a weighted summation and a latent representation consistency loss, which calculates the difference between Z_depth and Z_height. This forces the encoder of the depth image codec to learn to generate a latent representation equivalent to the ideal elevation map. The reconstruction loss calculates the difference between the predicted elevation map decoded by the depth image codec and the true elevation map label. This ensures that the encoder of the depth image codec is itself an effective depth-to-elevation map converter. By adjusting the parameters of the depth image codec using the first loss function value, a trained depth image codec is obtained, which can convert the depth map into a latent representation Z_depth that is consistent with the latent representation Z_height of the elevation map.
[0046] It is understandable that depth image codecs and elevation image codecs can form a model, such as a map prediction model, that can generate a latent representation that is semantically equivalent to an elevation map based on a depth image.
[0047] In some embodiments, reinforcement learning training of the action policy neural network is performed using a trained deep image codec, including: obtaining a reference multidimensional vector output by the trained deep image codec; outputting the reference multidimensional vector and the true value of the joint's real position information to the action policy neural network to obtain the predicted position information of the joint output by the action policy neural network; and performing reinforcement learning training on the action policy neural network based on the difference between the predicted position information and the real position information.
[0048] Figure 4 This is a flowchart illustrating the action policy neural network in an embodiment of this application. See also... Figure 4 Action policy neural networks can serve as walking policy models, and their output is used to control robot actions. Action policy neural networks avoid the time-consuming image rendering process.
[0049] For example, an elevation map of the area beneath the robot's feet is obtained. This elevation map is then input into the encoder of a pre-trained elevation image codec to obtain a latent representation Z_height. This latent representation Z_height, along with the ground truth information of the robot's joints, is used as an observation and input into a reinforcement learning policy network (i.e., an action policy neural network) to obtain the predicted joint position information output by the action policy neural network.
[0050] For example, reward functions can be designed to encourage robots to walk and climb stairs stably and efficiently. This is understandable because the input is a lightweight latent representation rather than the original image or point cloud, and rendering is skipped, resulting in extremely fast sampling and significantly improving the training efficiency of reinforcement learning.
[0051] Figure 5 This is a flowchart illustrating the robot control method in an embodiment of this application.
[0052] See Figure 5 A robot control method, the method comprising: Step 501: Obtain the target image captured by the robot.
[0053] Step 502: Use the perception and control model to execute the task of controlling the robot's actions based on the target image.
[0054] For example, the perception and control model is trained using the method described above.
[0055] This is understandable, such as deploying a trained perception and control model onto a real-world robot. The robot acquires a depth image of its surroundings using an onboard depth camera, thus obtaining the target image. This depth image is then input into the perception and control model, which in turn executes the robot's actions based on the model's output.
[0056] In some embodiments, using a perception and control model to perform a robot action task based on a target image includes: using the perception and control model to obtain position information of robot joints corresponding to the target image; generating motion commands based on the position information; and performing the robot action task based on the motion commands.
[0057] Figure 6 This is a flowchart illustrating a robot control method according to another embodiment of this application. See also... Figure 6The depth image, i.e., the target image, is input into the decoder of a pre-trained depth image codec to obtain the latent representation Z_depth. This latent representation Z_depth is then used as the observation input to a pre-trained action policy neural network. The action policy neural network outputs the position information of the robot joints corresponding to the target image, generating action commands for the robot to execute and control its actions. For example, based on the control commands, it can perform actions such as lifting a foot or taking a step to complete tasks like climbing stairs.
[0058] It is understandable that during the training of the perception and control model, the loss function forces the alignment of Z_depth and Z_height. Therefore, the Z_depth received by the action policy neural network during deployment is semantically highly consistent with the Z_height received during training. This enables a smooth transfer from simulation to reality, thereby allowing for correct decision-making.
[0059] To better understand this application, the following embodiments further illustrate the content of this application, but this application is not limited to the following embodiments.
[0060] This embodiment addresses the following key issues by dividing the training into two networks and aligning them at the latent representation level: Establish a unified and replaceable latent representation interface: By aligning the latent representations of the "elevation map autoencoder" and the "depth map autoencoder", a stable and task-relevant latent representation can be obtained using ground truth elevation during the simulation phase. In the deployment phase, an equivalent latent representation can be directly generated using the depth map, ensuring that the input semantics of the policy network are consistent and the distribution is stable during training and deployment.
[0061] Overcoming simulation rendering bottlenecks and improving reinforcement learning (RL) sampling efficiency: The reinforcement learning stage directly uses the elevation map obtained from simulation, and obtains the latent representation through the elevation encoder. This completely removes the image / depth rendering link, significantly improving sampling throughput and training speed, while reducing the uncertainty caused by rendering changes.
[0062] Improve the reliability of occlusion completion and noise resistance: By implementing phased training and loss design of the perceptual network (emphasizing both reconstruction consistency and latent representation consistency), the robust completion capability against noise, bad pixels, slight calibration errors and blind spots is enhanced without relying on heavy 3D reconstruction, providing a more stable input for height assessment before landing.
[0063] Reduce the cost of cross-domain migration and model replacement: Since the strategy deals with a stable, task-related latent representation that is decoupled from sensor details, in actual deployment, only the autoencoder on the depth side needs to be calibrated to adapt to different cameras, different mounting poses and different bodies, which greatly shortens the migration and parameter tuning cycle.
[0064] Balancing real-time performance with engineering feasibility: The latent representation dimension is controllable, the inference chain is short, and the computational burden is small, which can meet the latency budget of gait control; at the same time, it is easy to integrate into existing robot systems.
[0065] The embodiments of the present invention enable robots to make more reliable foot-lifting and stepping decisions when approaching complex terrains such as steps by achieving uniformity of the perception-control interface, consistency of training-deployment, sampling efficiency and real-time performance, and robustness to occlusion and noise.
[0066] For example, training can be done using two separate networks.
[0067] Map prediction component: Supervised learning is employed, with offline data collection used to train the network. Network input: depth image; output: elevation map.
[0068] It can consist of two parts: an elevation map codec and a depth map codec.
[0069] The Height AE (Height Assortment) takes a rectangular grid map as input, compresses it into a latent representation using an encoder, and then reconstructs the elevation map using a decoder. The label represents the input height from the dataset. The Height AE can employ a hybrid architecture combining MLP (Medium Level Portfolio) and CNN (Neural Network Array).
[0070] The depth map encoder / decoder (Depth AE) takes the depth map as input, compresses it into a latent representation using the encoder, and then restores it to an elevation map using the decoder. The label is the height in the dataset, i.e., the input to the height AE. The depth AE can use a CNN structure.
[0071] Training is divided into Phase I and Phase II.
[0072] Phase I: First, train the height ae, including the encoder and decoder, to ensure that the prediction is as accurate as possible with the input. The loss function is the input-output loss.
[0073] Phase II: Only the encoder is trained, and the height encoder (AE) is introduced into the pipeline with fixed parameters. The loss function is loss = latent loss + reconstruction loss. The latent loss means that the latent value of the depth encoder should be as consistent as possible with the latent value of the height encoder. The reconstruction loss means that the elevation map reconstructed using the latent value compressed by this depth encoder should be consistent with the label. The total loss function is obtained by weighting the two loss functions.
[0074] Training the walking component: A pre-trained image prediction model is obtained from the above, and the height encoder is introduced into the RL pipeline. In the simulator, height maps represented by yellow dots are collected and input into the height encoder to obtain the latency. This latency is then used as observation input to the RL network. This avoids introducing the camera rendering function from the simulator, as rendering would severely slow down the operation. Thus, the robot can learn actions such as stepping up and down stairs based on the height map.
[0075] Actual Deployment: Since a direct heightmap is unavailable, a depth encoder is needed. The input is the depth image obtained from the depth camera, and the output is the latency. This latency is then used as observation input to the RL network using the same method. Because the training of the map prediction network (AE) already ensures consistency between the depth and heightmap latency for the same terrain through latency loss, it can be used safely here. The RL network will determine the appropriate time to step up or down stairs based on the latency.
[0076] For example, a humanoid robot could use an onboard depth camera to autonomously and stably lift its foot and step onto a step about 20 centimeters high without falling or missing a step. The training is divided into two main parts. The first part trains the "terrain understanding brain" (perceptual model): learning to map depth and elevation maps to the same latent representation space. The second part trains the "walking control cerebellum" (policy model): in simulation, it uses the latent representation of the elevation map to learn walking and stair-climbing skills. During deployment, the "terrain understanding brain" and the "walking control cerebellum" are assembled to obtain a perception and control model, which drives the robot using real depth maps.
[0077] Step 1: Data Preparation and Simulation Environment Setup. Create the dataset and environment for training the perception model and policy model.
[0078] Choose a simulation platform: Use NVIDIA Isaac Sim or PyBullet as the physics simulator. They can simulate robot dynamics and sensors with high precision. Constructing simulation scenarios: Create various training terrains in the simulator, including flat ground, randomly scattered small obstacles, steps of different heights (15cm, 20cm, 25cm), and gentle slopes. Automated data acquisition: In the simulation, robot positions and terrain are generated randomly and programmatically. For each position, the following data triples are acquired simultaneously: depth map, elevation map, and robot state.
[0079] A depth map simulates the image seen by a depth camera mounted on the head of a robot. This simulates real-world sensor data, which includes noise, occlusion, and viewpoint distortion. The elevation map is generated by directly obtaining the actual ground elevation information of the area in front of the robot's feet from the simulation engine, creating a grid map. The value of each grid point represents the elevation of that point. This is the "God's-eye view" truth value in the simulation world. Robot status, including joint angles, angular velocity, and trunk posture.
[0080] The final result is a dataset containing tens of thousands of sets (depth maps, elevation maps, robot states). Step 2: Train the perceptual model; train two autoencoders and align their intermediate representations (latent representations). Phase I: Training the elevation map autoencoder.
[0081] Network structure: Convolutional neural network with encoder-decoder structure. Encoder: Input a 128×128 elevation map, pass through 4 convolutional layers (with pooling), and finally compress it into a 32-dimensional vector, i.e., the latent representation Z_height. Decoder: The 32-dimensional Z_height is passed through 4 transposed convolutional layers and upsampled back to a 128×128 elevation map. Training data: The elevation map collected in the first step is used as input and also as labels (self-supervised). Loss function: pixel-level mean squared error loss. The goal is to make the elevation map output by the decoder as identical as possible to the input elevation map. Training process: Use the Adam optimizer with a batch size of 64 and train until the loss converges. Completion indicator: When given a simulated elevation map, the network can reconstruct it almost perfectly. This means the encoder has learned how to efficiently compress terrain information into the 32-dimensional vector Z_height. Phase II: Training and aligning the depth map autoencoder.
[0082] Network structure: The depth map encoder uses a similar CNN structure to the elevation map encoder, and the output is also a 32-dimensional latent representation Z_depth. The decoder structure is also the same. Training data: The depth map collected in the first step is used as input, and the corresponding elevation map is used as the reconstruction label for the decoder. Loss function: The total loss consists of two weighted components.
[0083] Reconstruction loss: Calculates the mean squared error (MSE) between the predicted elevation map output by the depth map decoder and the actual elevation map label. Latent representation alignment loss: Input the depth map into the depth encoder to obtain Z_depth. Input the corresponding elevation map into the elevation encoder with frozen parameters to obtain Z_height. Calculate the MSE between Z_depth and Z_height. Total loss + reconstruction loss + λ×alignment loss (λ is a hyperparameter, for example, set to 1.0). Training process: Only the parameters of the depth map autoencoder are trained, while the parameters of the elevation map autoencoder are kept frozen. Completion indicator: For the same terrain, given a depth map, the Z_depth generated by the network is very close to the Z_height generated from the elevation map. This means that even without having seen the real depth map, the latent representation generated by the elevation encoder becomes the "teacher" for the depth encoder. Step 3: Train the "walking control cerebellum" (reinforcement learning strategy) to teach the robot to make walking decisions based on terrain latent representations. Build an RL environment and load a humanoid robot model (such as NASA-Valkyrie) into PyBullet. The simulation environment provides the following observations to the RL agent: The terrain latent representation Z_height (32-dimensional vector) is calculated in real time by directly using the elevation map provided by the simulator and inputting it into a frozen, trained elevation encoder. This completely skips the time-consuming and unstable camera rendering and depth map generation processes. Robot proprioception (e.g., 50-dimensional vector): including joint position, velocity, torso orientation, etc. The output of the RL agent is the target joint position of the robot (such as a 12-dimensional vector). Reward function design: forward reward, encouraging the robot to move forward; survival reward, rewarding the robot as long as it does not fall; energy penalty, penalizing large joint torques to encourage efficient walking; stability penalty, penalizing violent torso swaying; successful step-climbing reward, giving a large reward when the robot's feet successfully step onto the step surface. RL algorithms can use reinforcement learning algorithms such as PPO or SAC. Training: The robot is trained on various terrains including steps, performing millions of simulated steps. Because the input is a lightweight latent representation rather than an image, sampling is extremely fast, resulting in high training efficiency. Completion sign: In the simulation, the robot can steadily walk towards the steps and successfully climb them. Step 4: Deployment. Deploy the trained model (perception and control model) onto the real robot. Prepare a realistic humanoid robot and install a depth camera on its torso. Load the trained depth map encoder and RL policy network onto the robot's onboard computer. The depth camera captures a depth map frame. Input this depth map frame into the depth map encoder to obtain a 32-dimensional latent representation Z_depth. Concatenate the Z_depth with the proprioceptive information read by the robot's body sensors to form a complete observation vector, which is then input into the RL policy network. The RL policy network outputs the target joint position. The underlying controller (such as a PD controller) drives the joint motors to the target position, thus generating a stepping motion. When the robot approaches a real-world step, the Z_depth extracted by the depth map encoder from the noisy depth map is semantically highly consistent with the Z_height it saw during simulation training. The RL policy network "thinks" it has seen a familiar step again, and thus naturally issues the command to lift its foot and step, allowing the robot to successfully and stably step onto the step.
[0084] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a map generation device, electronic device, and corresponding embodiments.
[0085] Figure 7 This is a schematic diagram of the structure of the training device for the perception and control model shown in the embodiments of this application.
[0086] See Figure 7 The training device 700 for the perception and control model in this embodiment includes a first acquisition module 710 and a training module 720.
[0087] The first acquisition module 710 is used to acquire multiple sample image sets; wherein each sample image set includes depth images and elevation images representing the same terrain.
[0088] The training module 720 is used to train the initial model in multiple stages based on multiple sample image sets to obtain a perception and control model for controlling the robot's actions.
[0089] The multi-stage training includes: training an elevation image codec using elevation images to obtain a trained elevation image codec; training a depth image codec using depth images and the trained elevation image codec to obtain a trained depth image codec; and using the trained depth image codec to perform reinforcement learning training on the action policy neural network; the initial model includes an elevation image codec, a depth image codec, and an action policy neural network.
[0090] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.
[0091] According to embodiments of this application, any plurality of modules in the first acquisition module 710 and training module 720 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first acquisition module 710 and training module 720 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the first acquisition module 710 and training module 720 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.
[0092] Figure 8 This is a schematic diagram of the structure of the robot control device shown in the embodiments of this application.
[0093] See Figure 8 The robot control device 800 in this embodiment includes a second acquisition module 810 and a control module 820.
[0094] The second acquisition module 810 is used to acquire the target image collected by the robot.
[0095] The control module 820 is used to execute the robot's action task based on the target image using a perception and control model; wherein, the perception and control model is trained according to the above method.
[0096] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.
[0097] According to embodiments of this application, any plurality of modules in the second acquisition module 810 and the control module 820 can be combined into one module, or any one of the modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the second acquisition module 810 and the control module 820 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the second acquisition module 810 and the control module 820 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.
[0098] Figure 9 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.
[0099] See Figure 9 The electronic device 900 includes a memory 910 and a processor 920.
[0100] The processor 920 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0101] Memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 920 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.
[0102] The memory 910 stores executable code, which, when processed by the processor 920, can cause the processor 920 to execute part or all of the methods described above.
[0103] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.
[0104] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.
[0105] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A training method for a perception and control model, characterized in that, include: Acquire multiple sample image sets; where each sample image set includes depth and elevation images representing the same terrain. The initial model is trained in multiple stages based on multiple sets of sample images to obtain a perception and control model for controlling the robot's actions. The multi-stage training includes: training an elevation image codec using elevation images to obtain a trained elevation image codec; training a depth image codec using depth images and the trained elevation image codec to obtain a trained depth image codec; and using the trained depth image codec to perform reinforcement learning training on an action policy neural network; the initial model includes the elevation image codec, the depth image codec, and the action policy neural network.
2. The method according to claim 1, characterized in that, The step of training a depth image codec using a depth image and the trained elevation image codec to obtain a trained depth image codec includes: The depth image is input into the first encoder in the depth image codec to obtain the first multidimensional vector output by the first encoder. The elevation image corresponding to the depth image is input into the second encoder in the trained elevation image codec to obtain the second multidimensional vector output by the second encoder. The first multidimensional vector is input into the first decoder in the depth image codec to obtain the first predicted elevation image output by the first decoder. The first loss function value is determined based on the first multidimensional vector, the second multidimensional vector, the first predicted elevation image, and the elevation image; The parameters of the depth image codec are adjusted using the first loss function value to obtain the trained depth image codec.
3. The method according to claim 1, characterized in that, The process of training an elevation image codec using elevation images to obtain a trained elevation image codec includes: The elevation image is input into the elevation image codec to obtain the second predicted elevation image output by the elevation image codec; The second loss function value is determined based on the second predicted elevation image and the elevation image; The parameters of the elevation image codec are adjusted using the second loss function value to obtain the trained elevation image codec.
4. The method according to claim 1, characterized in that, The step of using the trained deep image codec to perform reinforcement learning training on the action policy neural network includes: Using the trained depth image codec, a reference multidimensional vector output by the depth image codec is obtained; The reference multidimensional vector and the true value of the joint's actual position information are output to the action policy neural network to obtain the predicted position information of the joint output by the action policy neural network. The action policy neural network is trained using reinforcement learning based on the difference between the predicted location information and the actual location information.
5. A robot control method, characterized in that, include: Acquire the target image captured by the robot; The robot performs a control task based on the target image using a perception and control model; wherein the perception and control model is trained by the method according to any one of claims 1-4.
6. The method according to claim 5, characterized in that, The task of controlling the robot's actions based on the target image using a perception and control model includes: Using a perception and control model, the position information of the robot joints corresponding to the target image is obtained. Generate action instructions based on the location information; The robot performs its actions according to the action instructions.
7. A training device for a perception and control model, characterized in that, include: The first acquisition module is used to acquire multiple sample image sets; wherein each sample image set includes depth images and elevation images representing the same terrain. The training module is used to perform multi-stage training on the initial model based on multiple sets of sample images to obtain a perception and control model for controlling the robot's actions. The multi-stage training includes: training an elevation image codec using elevation images to obtain a trained elevation image codec; training a depth image codec using depth images and the trained elevation image codec to obtain a trained depth image codec; and using the trained depth image codec to perform reinforcement learning training on an action policy neural network; the initial model includes the elevation image codec, the depth image codec, and the action policy neural network.
8. A robot control device, characterized in that, include: The second acquisition module is used to acquire the target image collected by the robot; A control module is used to perform a task of controlling the robot's actions based on the target image using a perception and control model; wherein the perception and control model is trained by the method according to any one of claims 1-5.
9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-6.
10. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-6.