Three-dimensional perception model training and application method and device, equipment and robot

Abstract

The application provides a method and device for training and applying a three-dimensional perception model, equipment and a robot. The method comprises: obtaining an environment model corresponding to a real environment, a running state of a robot when the robot runs in the real environment, and a real image collected by the robot; controlling a robot model corresponding to the robot in a simulation environment to run in the environment model based on the running state and collect a simulation image; determining real semantic labels of a plurality of first voxels corresponding to the simulation image; processing training data based on the three-dimensional perception model to obtain predicted semantic labels of a plurality of second voxels corresponding to the training data; the training data comprises the real image and / or the simulation image; training the three-dimensional perception model based on the predicted semantic labels of the plurality of second voxels and the real semantic labels of the plurality of first voxels to obtain a trained three-dimensional perception model. The application can improve the efficiency of obtaining training data for training the three-dimensional perception model and reduce the cost of obtaining the training data.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a training method for a three-dimensional perception model, an application method for a three-dimensional perception model, a training device for a three-dimensional perception model, an application device for a three-dimensional perception model, a computing device, a robot, and a computer-readable storage medium. Background Technology

[0002] 3D environmental perception is a core technology for mobile robots, aiming to construct 3D geometric and semantic representations of the environment through sensors such as LiDAR and cameras, including point cloud maps, voxel maps, or bird's-eye view (BEV) feature maps. Currently, the common technical approach mainly originates from the field of autonomous driving, widely adopting "Multi-View Stereo (MVS)" or "Learning-based Occupancy Networks (OCC Networks)" model architectures to project 2D image features into 3D space to achieve 3D environmental perception.

[0003] However, training a 3D environment perception model usually requires collecting a large amount of real-world data from the entire scene as training data, which results in extremely high cost and low efficiency in acquiring training data. Summary of the Invention

[0004] This application provides a method for training a 3D perception model, a method for applying a 3D perception model, a training device for a 3D perception model, an application device for a 3D perception model, a computing device, a robot, and a computer-readable storage medium, which can improve the efficiency of acquiring training data for training a 3D perception model and reduce the acquisition cost. The technical solution of this application is implemented as follows: In a first aspect, embodiments of this application provide a training method for a three-dimensional perception model. The method includes: acquiring an environment model corresponding to a real environment and real machine data of a robot running in the real environment; the real machine data includes the robot's operating state and real images collected by the robot; in a simulation environment, controlling a robot model corresponding to the robot to run in the environment model based on its operating state and collecting simulation images; determining the true semantic labels of multiple first voxels corresponding to the simulation images; processing the training data based on the three-dimensional perception model to obtain predicted semantic labels of multiple second voxels corresponding to the training data; the training data includes real images and / or simulation images; training the three-dimensional perception model based on the predicted semantic labels of the multiple second voxels and the true semantic labels of the multiple first voxels to obtain the trained three-dimensional perception model.

[0005] In this embodiment, a training data acquisition method of "scanning-simulation-hybridization" can be realized. That is, the real physical environment is reconstructed by high-precision static scanning, and the robot's running state is reproduced in the simulation environment by combining the robot's real motion trajectory. In this way, a large amount of high-precision simulation data can be efficiently synthesized in the simulation environment to achieve the generation of large-scale training data with low cost and high precision. This can reduce the cost of acquiring training data and improve the acquisition efficiency.

[0006] Secondly, embodiments of this application provide a method for applying a three-dimensional perception model to a robot. The robot is equipped with a trained three-dimensional perception model. The method includes: acquiring real-time joint coding data of the robot while it is running in a real environment, and acquiring two-dimensional images through an image acquisition device in the robot; determining the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the robot's base coordinate system based on the robot's preset parameters and real-time joint coding data; projecting the image features of the two-dimensional images into a three-dimensional space based on the dynamic extrinsic parameters through the trained three-dimensional perception model, and determining multiple voxels in the three-dimensional space; and processing the multiple voxels based on the trained three-dimensional perception model to obtain target semantic labels for the multiple voxels.

[0007] In this embodiment, dynamic extrinsic parameters are calculated based on the robot's real-time joint state and injected into the network projection layer to correct posture jitter. Thus, regardless of how the robot moves, the 3D map output based on the 3D perception model is always gravity-aligned and stable, ensuring the continuity of planning and avoiding perception jitter of non-rigid robots.

[0008] Thirdly, embodiments of this application provide a training device for a three-dimensional perception model. The device includes: a first acquisition module configured to acquire an environment model corresponding to a real environment and real machine data of a robot running in the real environment; the real machine data includes the robot's operating state and real images collected by the robot; an acquisition module configured to control a robot model corresponding to the robot to run in the environment model based on its operating state in a simulation environment, and to collect simulation images; a first determination module configured to determine the true semantic labels of multiple first voxels corresponding to the simulation images; a first processing module configured to process the training data based on the three-dimensional perception model to obtain predicted semantic labels of multiple second voxels corresponding to the training data; the training data includes real images and / or simulation images; and a training module configured to train the three-dimensional perception model based on the predicted semantic labels of the multiple second voxels and the true semantic labels of the multiple first voxels to obtain a trained three-dimensional perception model.

[0009] Fourthly, embodiments of this application provide an application device for a three-dimensional perception model, configured in a robot. The robot is equipped with a trained three-dimensional perception model. The device includes: a second acquisition module configured to acquire real-time joint coding data of the robot when the robot is running in a real environment, and to acquire two-dimensional images through an image acquisition device in the robot; a second determination module configured to determine the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the base coordinate system of the robot based on preset parameters of the robot and the real-time joint coding data; a third determination module configured to project the image features of the two-dimensional image into a three-dimensional space based on the dynamic extrinsic parameters through the trained three-dimensional perception model, and to determine multiple voxels in the three-dimensional space; and a second processing module configured to process the multiple voxels based on the trained three-dimensional perception model to obtain target semantic labels for the multiple voxels.

[0010] Fifthly, embodiments of this application provide a computing device, which includes a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement the steps in a method for training a three-dimensional perception model.

[0011] Sixthly, embodiments of this application provide a robot equipped with a trained three-dimensional perception model, which is used to implement the steps in the application method of the three-dimensional perception model.

[0012] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described training method or application method of the three-dimensional perception model.

[0013] Eighthly, embodiments of this application provide a program product comprising a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described training method or application method of the three-dimensional perception model.

[0014] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0015] Figure 1 A schematic diagram illustrating the implementation process of a training method for a three-dimensional perception model provided in an embodiment of this application; Figure 2 A schematic diagram illustrating another method for training a three-dimensional perception model provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the determination of the true semantic label of a target voxel, provided as an embodiment of this application; Figure 4A A schematic diagram of a voxel labeling result provided in an embodiment of this application; Figure 4B A schematic diagram illustrating another voxel labeling result provided in an embodiment of this application; Figure 5 A flowchart for loading an object model into an environment model is provided as an embodiment of this application; Figure 6A A schematic diagram of the original image of an operational model provided in an embodiment of this application; Figure 6B A schematic diagram of a 2D segmentation extraction object mask of an object model provided in an embodiment of this application; Figure 6C A depth estimation map of an operational object model provided in an embodiment of this application; Figure 6D A schematic diagram illustrating the target 6D pose detection result of an object model provided in an embodiment of this application; Figure 6E A schematic diagram illustrating pose registration in virtual space for an object model, provided as an embodiment of this application; Figure 6F This application provides a schematic diagram illustrating the generation of the three-dimensional occupancy truth value of an operand model in an embodiment of the present application. Figure 7 A schematic diagram of a dynamic feature projection provided in an embodiment of this application; Figure 8 A schematic diagram of the architecture of a three-dimensional perception model provided in an embodiment of this application; Figure 9 A schematic diagram illustrating the implementation process of an application method for a three-dimensional perception model provided in this application embodiment; Figure 10 A schematic diagram illustrating an application method of a three-dimensional perception model provided in an embodiment of this application; Figure 11 A schematic diagram illustrating the structural composition of a training device for a three-dimensional perception model provided in an embodiment of this application; Figure 12 This is a schematic diagram of the composition structure of an application device for a three-dimensional perception model provided in an embodiment of this application.

[0016] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0020] The following explains the relevant terms used in this application: Simultaneous Localization and Mapping (SLAM): Using SLAM technology, robots can estimate the environmental map and their pose relative to the environmental map while operating in an unknown environment, using their onboard sensors (such as LiDAR and cameras).

[0021] Simulation to Reality / Virtual-Real Transfer: This refers to a technological paradigm that uses simulated synthetic data in a virtual simulation environment to train a model, and then transfers and deploys the trained model to a real physical robot system. This allows robots to learn skills at the lowest cost and fastest speed, and then safely apply them to the real world.

[0022] Voxel: A volumetric element. It is the smallest unit in three-dimensional space, similar to a pixel in a two-dimensional bitmap. In computer graphics and robot perception, voxels are often used to discretize continuous three-dimensional space. Each voxel contains geometric properties (such as whether it is occupied) or appearance properties (such as color and density) of that spatial location.

[0023] Occupation network is a deep learning-based 3D perception algorithm. It discretizes a continuous 3D space into regular grid cells (voxels) and infers the category of each grid cell and the probability of it being occupied by an object based on sensor input (such as an image or point cloud). Compared to traditional object detection, it can represent the geometric details of unstructured environments with greater precision.

[0024] Ray tracing is a computer graphics and geometric computation technique. It determines the visibility of voxels by simulating the propagation path of light rays (such as rays) in three-dimensional space.

[0025] Rigid body: refers to an object whose shape and size do not change during motion or after being subjected to forces, and whose distance between any two points inside remains constant.

[0026] Non-rigid bodies: These are objects whose geometry, size, or relative positions of internal points change during motion or under stress. They include flexible bodies and articulated objects. Flexible bodies include materials, liquids, and soft robots; articulated objects are the core of humanoid robots, formed by connecting multiple rigid bodies through joints. Although the limbs and legs of a humanoid robot are themselves rigid bodies, the overall shape of the robot is constantly changing because its joints can move.

[0027] In related technologies, 3D environmental perception is a core technology for mobile robots and autonomous driving systems. It aims to construct 3D geometric and semantic representations of the environment using sensors such as LiDAR and cameras, including point cloud maps, voxel maps, or BEV feature maps. Currently, the common technical approach originates from the field of autonomous driving, widely employing multi-view stereo vision (MVS) or learning-based occupancy network (OVN) model architectures to project 2D image features into 3D space to achieve 3D environmental perception.

[0028] In the field of autonomous driving, vehicles are usually regarded as rigid bodies. Their camera parameters can be pre-calibrated, and the extrinsic parameters (height and angle) between the camera and the vehicle body often do not change. The vehicle body area in the imaging results is relatively fixed, and the area corresponding to the effective field of view is all environmental targets. It is only necessary to accurately perceive the type and pose of the targets in the environment and then directly avoid obstacles.

[0029] In 3D ground truth annotation, the LiDAR on autonomous driving data acquisition vehicles is typically mounted at the same height as the camera, with a similar field of view. Generally, image-level depth or semantic ground truth (derived from image segmentation annotation results) is generated simply by directly projecting the LiDAR point cloud. Therefore, in autonomous driving, the ground truth for 3D perception can be conveniently determined based on the vehicle-mounted LiDAR point cloud reconstruction data. In autonomous driving occupancy network perception, a point cloud projection method is generally used to project sparse point clouds onto a 2D image to obtain the corresponding semantic category, and then the semantic category corresponding to the 2D image is back-projected onto 3D space. Furthermore, occluded areas are usually ignored or simply filled in, because autonomous driving systems do not need to concern themselves with the categories of occluded objects or other obstacles outside the road.

[0030] When transferring autonomous driving perception technology to the field of embodied intelligence (such as humanoid robots), the following technical problems arise because humanoid robots (hereinafter referred to as robots) are non-rigid bodies, and the LiDAR and camera perspectives within robots differ significantly, and they cannot easily process occluded areas: (1) Perception target confusion and operation logic conflict: The relevant technologies lack the ability to semantically segment the robot body and the objects operated by the robot (i.e. the manipulated objects). When the robot is working, its arms, legs and objects in its hands (such as goods) often appear in the robot's own field of vision, but traditional three-dimensional environment perception algorithms usually misidentify them as external obstacles, which can lead to the robot deadlock (i.e., self-obstacle avoidance) or be unable to operate the objects normally.

[0031] (2) Poor adaptability to posture changes leading to projection misalignment: View projection in related technologies relies on fixed extrinsic parameters. Since the robot is a non-rigid body, it experiences drastic posture changes such as squatting, walking, and bending over. This causes the robot's camera extrinsic parameters to fluctuate drastically in real time. As a result, using fixed parameters will cause serious misalignment of two-dimensional image features in three-dimensional space. The constructed three-dimensional map will shake with the robot's body, making the map unusable.

[0032] (3) Difficulty in obtaining voxel ground truth and parallax problem: Due to the limited size of the robot, it is impossible to install a LiDAR parallel to the camera's line of sight like a vehicle, which will lead to a serious parallax occlusion problem. Therefore, it is difficult to generate pixel-level aligned voxel ground truth based on the traditional point cloud projection method. In addition, the cost of collecting a large amount of real machine data in the whole scene as training data is extremely high and the efficiency is low.

[0033] (4) Occlusion ambiguity and efficiency bottleneck in voxel ground truth generation: When generating voxel ground truth using point cloud projection, it is impossible to correctly determine the occlusion area behind the object, and unknown areas are often incorrectly marked as air (Free), causing the model to produce illusions during training. At the same time, the geometric calculation efficiency based on the central processing unit (CPU) is low and cannot meet the production needs of massive data (such as tens of millions of frames).

[0034] Based on the above-mentioned technical problems, the embodiments of this application provide a training method for a three-dimensional perception model, an application method for a three-dimensional perception model, a training device for a three-dimensional perception model, an application device for a three-dimensional perception model, a computing device, a robot, and a computer-readable storage medium, which can improve the efficiency of acquiring training data for training a three-dimensional perception model and reduce the acquisition cost.

[0035] This application provides a method for training a three-dimensional perception model, which can be executed by a computing device. The computing device can be a server, laptop, tablet, desktop computer, smart TV, mobile device, or other device with data processing capabilities. Figure 1 This is a schematic diagram illustrating the implementation process of a training method for a three-dimensional perception model provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes steps S101 to S105: S101, acquire the environmental model corresponding to the real environment and the real machine data of the robot running in the real environment.

[0036] refer to Figure 2 A schematic diagram of another training method for a 3D perception model is shown below. Figure 2 As shown, when acquiring training data for training a 3D environment perception model, data from the physical world (i.e., the real environment) can be acquired first. For example, a high-precision handheld scanner (such as a LiDAR scanner) can be used to statically scan the real environment (or the real working environment) to reconstruct an environmental model as the ground truth. This environmental model can be a high-fidelity 3D mesh map or a 3D Gaussian Splatting (3DGS) model. For example, since the observed real environment differs at different heights, when reconstructing the environmental model, the real environment can be scanned at different heights to obtain environmental models at different heights. The ground truth values ​​from these different heights are then merged to obtain the final environmental model.

[0037] Furthermore, the robot can operate in this real-world environment, and its trajectory can be recorded to obtain real-world data. This real-world data can include the robot's operating state and real images captured by the robot. The robot's operating state can include its joint motion states and its position (pose) in the real-world environment. For example, the robot can be equipped with image acquisition devices (such as cameras) and LiDAR. The image acquisition device is used to acquire two-dimensional images during the robot's operation, and the LiDAR is used to acquire LiDAR data (i.e., point cloud data) during the robot's operation. During operation, the robot can acquire real images at various times to obtain a real image stream (or camera image stream) through the image acquisition device, and acquire LiDAR data at various times through the LiDAR; it can also acquire the robot's joint motion states at various times (such as joint angles, joint displacements, etc.). The LiDAR data at each time point can be used to determine the robot's position in the real-world environment at that moment. It should be noted that the real images, LiDAR data, and joint motion states at each time point are all timestamp-aligned data.

[0038] S102, In the simulation environment, the robot model corresponding to the control robot runs in the environment model based on the running state and acquires simulation images.

[0039] Next, digital twin data (i.e., simulation data) can be synthesized in the simulation environment based on the data obtained in the real environment. For example, a simulation environment can be built on a computing device, and the environment model corresponding to the real environment and the robot's operating state at each moment can be loaded into the simulation environment. Furthermore, the Unified Robot Description Format (URDF) digital model of the robot (hereinafter referred to as the robot model) can be loaded into the simulation environment to generate the robot's true value; a three-dimensional model of the manipulated object (i.e., the object manipulated by the robot) can also be loaded to generate the object's true value. The URDF digital model is used to describe the robot's structure, joints, dynamic properties, etc. By writing URDF files, the robot's various components, the connection relationships between components, and kinematic constraints can be defined.

[0040] After the model is loaded, the robot model can be controlled to run in the simulation environment based on its running state at each moment, simulating the robot's running state in a real environment. Furthermore, virtual image acquisition devices (such as virtual cameras) can be configured in the simulation environment to simulate the robot's real camera viewpoint (i.e., the virtual image acquisition device has the same real viewpoint as the robot). Thus, during the robot model's operation, two-dimensional synthetic images corresponding to each moment can be acquired through the virtual image acquisition device, such as rendering two-dimensional camera images corresponding to each moment using the virtual camera.

[0041] S103, determine the true semantic labels of multiple first voxels corresponding to the simulation image.

[0042] In some implementations, during the operation of the robot model, the ground truth (i.e., ground semantic ID) of the voxel (hereinafter referred to as the first voxel) corresponding to each pixel in the simulation image in three-dimensional space can be determined as the ground truth (GT) during the training process of the three-dimensional perception model. This ground truth is the supervision signal (label). For example, the real robot camera view can be simulated in the simulation environment, and the depth map and two-dimensional semantic segmentation map corresponding to the simulation image at each time moment can be rendered as the ground truth during the training process.

[0043] For example, the true semantic label of the first voxel may include an occupied layer, a free layer, and an occluded layer. The occupied layer indicates that the voxel is a voxel within an object, that is, the voxel is occupied by an object; the free layer indicates that the voxel is a voxel that is not an object (such as air); and the occluded layer indicates that the voxel is occluded by other objects and currently belongs to an unknown type.

[0044] S104, based on the 3D perception model, the training data is processed to obtain the predicted semantic labels of multiple second voxels corresponding to the training data.

[0045] In some implementations, the real images and simulated images corresponding to each time step can be used as training data input into the 3D perception model to be trained. That is, the real images at each time step can be used as real-device data (e.g., training group A), and the simulated images at each time step can be used as simulated data (e.g., training group B), allowing for hybrid training of the 3D perception model based on both real-device and simulated data. In other words, during the training of the 3D perception model, its input can be real camera images (capable of preserving realistic lighting and texture) or simulated synthetic images. After acquiring the training data corresponding to each time step, the 3D perception model performs 3D perception processing on the training data to determine the predicted semantic labels (i.e., predicted semantic IDs) of multiple voxels (hereinafter referred to as second voxels) corresponding to multiple pixels in the 2D training data in 3D space, thereby determining the category of each voxel in the environment.

[0046] S105, the 3D perception model is trained based on the predicted semantic labels of multiple second voxels and the real semantic labels of multiple first voxels to obtain the trained 3D perception model.

[0047] In some implementations, the true values ​​determined above can be used as real semantic labels to supervise the training of the predicted semantic labels determined by the 3D perception model. That is, the 3D perception model can be trained based on the predicted semantic labels of multiple second voxels and the real semantic labels of multiple first voxels to obtain the trained 3D perception model.

[0048] In this embodiment, a training data acquisition method of "scanning-simulation-hybridization" can be realized. That is, the real physical environment is reconstructed by high-precision static scanning, and the robot's running state is reproduced in the simulation environment by combining the robot's real motion trajectory. In this way, a large amount of high-precision simulation data can be efficiently synthesized in the simulation environment to achieve the generation of large-scale training data with low cost and high precision. This can reduce the cost of acquiring training data and improve the acquisition efficiency.

[0049] In some implementations, S103 includes: generating a scene mesh model of the robot model running in the environment model based on the running state; emitting simulated rays to each first voxel in the scene mesh model through a parallel processing unit with the optical center of the virtual image acquisition device as the origin; and determining the real semantic label of each first voxel based on the intersection state between each first voxel and the simulated rays corresponding to each first voxel.

[0050] In some implementations, when controlling the robot model to run within the environment model based on its operational state at each moment, the scene model at each moment can be divided into multiple first voxels to generate scene mesh models for each moment. The scene model at each moment is the scene model observed from the perspective of the virtual image acquisition device. The scene model may include an environment model, a model of the robot's body (such as the robot's arm), or a model of the object the robot manipulates. In other words, the scene model at each moment is a 3D model of the simulation image at that moment. The voxel type of the first voxels in the scene model can include non-object voxels and object voxels. Object voxels may include robot body voxels, voxels of objects manipulated by the robot, or voxels of environmental objects; non-object voxels are voxels not occupied by objects, such as air voxels.

[0051] Next, the true semantic labels of multiple first voxels in the scene mesh model corresponding to each time step can be determined. For example, in a simulation environment, a processor can emit simulated rays (such as beams) from the optical center of a virtual image acquisition device (such as a virtual camera) to the center of each first voxel within its view frustum (i.e., each first voxel in the scene mesh model). For example, the processor emitting the simulated rays can be a parallel processing unit, such as a graphics processing unit (GPU), thereby improving the rendering efficiency of the simulated rays.

[0052] Next, the intersection state between each first voxel and its corresponding simulated ray can be determined, and the true semantic label of each first voxel can be determined based on the intersection state. Object voxels can intersect with simulated rays; non-object voxels do not intersect with simulated rays, meaning the simulated ray passes through the non-object voxel. The intersection state between each first voxel and its corresponding simulated ray can include: the simulated ray corresponding to the first voxel passes through the first voxel (i.e., no intersection with the first voxel), the simulated ray corresponding to the first voxel intersects with the first voxel, and the simulated ray corresponding to the first voxel does not reach the first voxel (i.e., intersects with other voxels before reaching the first voxel). Accordingly, the true semantic label of each first voxel can include: idle layer, occupied layer, and occluded layer.

[0053] In this embodiment, by generating simulated light through a parallel processing unit, the generation speed of voxel ground truth can be increased from the second or minute level of traditional CPU methods to the millisecond level (millions of frames per day), thereby achieving low-cost construction of massive amounts of embodied intelligence training data.

[0054] refer to Figure 3 The diagram shown illustrates how to determine the true semantic label of a target voxel. Figure 3As shown, the method includes steps S301 to S314: S301 emits rays targeting the voxel.

[0055] For example, the following describes how to determine the true semantic label of a target voxel by emitting a simulated ray (hereinafter referred to as the target simulated ray) at a target voxel among a plurality of first voxels. The target simulated ray can be a ray beam.

[0056] S302, GPU raycasting and physics intersection calculations.

[0057] For example, the target simulated ray can be rendered using a graphics processing unit (GPU). After the target simulated ray is emitted towards the target voxel, the intersection state between the target simulated ray and the target voxel can be determined. The intersection state between the target voxel and the target simulated ray can include: the target simulated ray passes through the target voxel (i.e., there is no intersection with the target voxel), the target simulated ray intersects with the target voxel, and the target simulated ray does not reach the target voxel (i.e., it intersects with other voxels before reaching the target voxel).

[0058] S303, determine whether the simulated ray intersects with multiple first voxels.

[0059] For example, when determining the intersection state between the target simulated ray and the target voxel, it can be first determined whether the target simulated ray intersects with at least one of the multiple first voxels. If the target simulated ray does not intersect with any of the multiple first voxels, then S304 is executed; if the target simulated ray intersects with at least one of the multiple first voxels, then S305 is executed.

[0060] S304, if the simulated ray has no intersection with multiple first voxels, then the true semantic label of the target voxel is the idle layer.

[0061] If the simulated ray tracing the target voxel has no intersection with any of the first voxels, it means that the simulated ray traverses the target voxel and all other first voxels in the scene mesh model. In this case, the true semantic label of the target voxel can be marked as Free. That is, at this time, the simulated ray tracing the target voxel has no intersection with the target voxel, and the target voxel is a non-object voxel.

[0062] S305, if the simulated ray intersects with multiple first voxels, obtain the depth of the nearest intersection point.

[0063] If the target simulated ray corresponding to the target voxel intersects with at least one of the multiple first voxels, then the intersection point when the target simulated ray first intersects with the multiple first voxels in the scene mesh model can be determined, that is, the depth of the nearest intersection point can be determined. .

[0064] S306, determine the magnitude relationship between the intersection depth and the voxel depth of the target voxel.

[0065] Next, the depth of the intersection can be determined. voxel depth relative to the target voxel The size relationship between them is determined, and the intersection state between the target simulated ray and the target voxel is determined based on the size relationship, thereby determining the true semantic label of the target voxel.

[0066] S307, if the intersection depth is less than the voxel depth of the target voxel, then mark the true semantic label of the target voxel as the occluded layer.

[0067] For example, if the intersection depth Voxel depth smaller than the target voxel This indicates that the simulated ray intersects with other first voxels and generates an intersection point before reaching the target voxel (i.e., the simulated ray does not reach the target voxel). Therefore, the true semantic label of the target voxel can be determined as an occluded layer. In other words, the simulated ray has not reached the target voxel at this point; the target voxel could be an object voxel or a non-object voxel. Since the voxel type of the target voxel is unknown at this time, it can also be labeled as an unknown voxel.

[0068] S308. If the intersection depth is greater than the voxel depth of the target voxel, then mark the true semantic label of the target voxel as an idle layer.

[0069] For example, if the intersection depth Voxel depth greater than the target voxel If the simulated ray of light passes through the target voxel and intersects with other first voxels to generate an intersection point (i.e., the simulated ray of light passes through the target voxel), then the true semantic label of the target voxel can be determined to be an empty layer. In other words, if the simulated ray of light has no intersection with the target voxel, then the target voxel is a non-object voxel.

[0070] S309, if the intersection depth is within the voxel depth range of the target voxel, then mark the true semantic label of the target voxel as the occupied layer.

[0071] For example, if the intersection depth Voxel depth located at the target voxel Within a given range, if the simulated ray intersects with the target voxel inside the target voxel, then the true semantic label of the target voxel can be determined as the occupancy layer. That is, at this point, the simulated ray intersects with the target voxel, and the target voxel is considered an object voxel.

[0072] S310, if the true semantic label of the target voxel is the occupying layer, then query the semantic identifier of the target voxel.

[0073] In some implementations, if the true semantic label (i.e., candidate semantic label) of the target voxel is an occupied layer, that is, when the target voxel is determined to be "Occupied", the voxel type (i.e., semantic identifier) ​​of the hit object mesh corresponding to the target voxel can be further determined, and the occupancy state of the target voxel can be further subdivided to determine the true semantic label of the target voxel. The voxel type of the target voxel may include the robot's body voxel, the robot's operational object voxel, or the environment object voxel; the environment object voxel may include voxels of static or dynamic environmental objects.

[0074] S311, if the semantic identifier of the target voxel is a static environment object voxel, then the true semantic label of the target voxel is determined to be the static environment layer.

[0075] For example, if the semantic identifier of the target voxel is a voxel of a static environment (e.g., ID is Environment), such as voxels of objects like walls, floors, and furniture, then the true semantic label of the target voxel can be determined to be the static environment layer (e.g., labeled as Static Env).

[0076] S312, if the semantic identifier of the target voxel is a dynamic environment voxel, then the true semantic label of the target voxel is determined to be the dynamic environment layer.

[0077] For example, if the semantic identifier of the target voxel is a voxel of a dynamic environment (e.g., ID: DynamicActor), such as a voxel of a moving pedestrian, vehicle, or other object, then the true semantic label of the target voxel can be determined to be the dynamic environment layer (e.g., labeled as Dynamic Env).

[0078] S313, if the semantic identifier of the target voxel is the robot's ontology voxel, then the true semantic label of the target voxel is determined to be the ontology layer.

[0079] For example, if the semantic identifier of the target voxel is a robot's ontological voxel (e.g., ID RobotURDF), such as voxels of the robot model's arms, legs, torso, etc., then the true semantic label of the target voxel can be determined to be the ontological layer (e.g., labeled as ontological Robot Self).

[0080] S314. If the semantic identifier of the target voxel is the object element operated by the robot, then the true semantic label of the target voxel is determined to be the object layer.

[0081] For example, if the semantic identifier of the target voxel is the object element operated by the robot (e.g., ID is TargetObject), that is, the voxel of the item currently grasped or moved by the robot, then the true semantic label of the target voxel can be determined to be the operation object layer (e.g., marked as the operation item Manipulation Object).

[0082] refer to Figure 4A A schematic diagram of a voxel labeling result is shown. Figure 4B A schematic diagram of another voxel labeling result is shown. (See diagram below.) Figure 4A and Figure 4B As shown, a scanner can be used to perform a static scan of the real environment, such as scanning the front left, back left, front right, and back right of a specific location in the real environment, to obtain a high-fidelity 3D mesh map or a 3D Gaussian splash model as the environment model. After determining the true semantic labels corresponding to all the first voxels, first voxels with different true semantic labels can be identified by different colors. For example, Figure 4A and Figure 4B In this model, blue and gray grids represent the first voxels with the true semantic label "Occupied," such as obstacles and the ground; red grids represent the first voxels with the true semantic label "Occluded"; and transparent grids represent the first voxels with the true semantic label "Free." This allows the true semantic labels corresponding to each first voxel to be used as ground truth values ​​for training the 3D perception model. Furthermore, since voxel occupancy is very dense, processing with serial processors (such as CPUs) would be extremely slow. Therefore, parallel computing with parallel processing units (such as GPUs) can achieve large-scale voxelization processing, improving the speed of voxel ground truth value generation. This enables the generation of unambiguous voxel ground truth values ​​that conform to physical occlusion relationships by simulating optical paths in a digital twin environment, achieving efficient and large-scale data production.

[0083] In this embodiment, by introducing semantic tags for occluded states, the false negative noise caused by line-of-sight occlusion in traditional methods can be eliminated, thereby improving the upper limit of the 3D perception model's understanding of complex environments. Furthermore, when a voxel is determined to be an occupied layer, by further accurately distinguishing the voxel type (environment, body, or manipulated object), the robot can navigate safely without unintentionally triggering obstacle avoidance, thus avoiding the problem of the robot confusing the target.

[0084] In some implementations, reference Figure 5This is a flowchart illustrating the loading of an object model into an environment model. (Example:) Figure 5 As shown, the method includes S501 to S507: S501, Input the image of the object to be operated on.

[0085] refer to Figure 6A A schematic diagram of the original image of a manipulating object model is shown, as follows. Figure 6A As shown, you can first input a two-dimensional original image of the object being manipulated in the simulation environment. This two-dimensional image can be an RGB image.

[0086] S502, 2D visual perception extraction body mask.

[0087] refer to Figure 6B The diagram shown illustrates a 2D segmentation of an object model to extract an object mask. Figure 6B As shown, the two-dimensional image can then be processed by two-dimensional (2D) visual perception to segment the objects in the two-dimensional image and extract the object mask corresponding to the objects.

[0088] S503, depth estimation extracts 3D information of the scene.

[0089] refer to Figure 6C The image shown is a depth estimation map of an operational object model, such as... Figure 6C As shown, depth estimation can be performed on the two-dimensional image to obtain a depth estimation map, thereby determining the distance of each pixel in the image relative to the shooting source and extracting the three-dimensional information corresponding to the two-dimensional image.

[0090] S504, 6D Pose Calculation: Calculates the pose of the manipulated object under the camera.

[0091] refer to Figure 6D The diagram shown illustrates the target 6D pose detection results of an object model. Figure 6D As shown, next, based on the object mask, 3D information, and a preset model library (such as an object CAD model library) corresponding to the manipulated object, the pose of the manipulated object in the 2D image can be calculated (such as 6D pose calculation), thereby determining the precise position of the manipulated object under the camera. The 6D pose of the manipulated object can include its 3D coordinates (X, Y, Z) and 3D orientation (rotation angles around the X, Y, and Z axes), i.e., pose information with 6 degrees of freedom.

[0092] For example, a preset model library can store preset 3D models of the manipulated object. These preset 3D models include high-precision 3D geometric information (such as shape, size, and topology) of the manipulated object, serving as a reference template for 6D pose calculation. After extracting the 3D information of the scene, this information can be matched with the preset 3D models of the manipulated object (such as CAD models) in the preset model library to calculate the precise position (i.e., 3D coordinates) and orientation (i.e., rotation angle) of the manipulated object in the camera coordinate system through geometric comparison. For instance, when the object grasped by the robot is a cargo box, the CAD model of the cargo box can be called and matched with the extracted 3D information of the cargo box to determine the spatial coordinates of the grasping point. The object mask is used to reduce the interference of background noise on pose calculation, ensuring that the pose calculation for 6D pose is based solely on the features of the manipulated object itself, avoiding deviations in the calculation results due to mismatches of background objects.

[0093] S505, Virtual Scene Reconstruction: Place the CAD model of the object to the appropriate position in virtual space.

[0094] refer to Figure 6E The diagram shown illustrates pose registration in the virtual space of an object model. Figure 6E As shown, after determining the pose information of the manipulated object, it can be loaded into the environment model based on the pose information. That is, pose registration is performed in the environment model (i.e., virtual space) loaded in the simulation environment so that the preset 3D model of the manipulated object can be placed according to the pose information of the manipulated object.

[0095] S506, Ray Tracing and Voxelization: Computing which meshes the model of the operand occupies.

[0096] refer to Figure 6F The diagram shown illustrates the generation of the 3D occupancy truth value of a manipulator model. Figure 6F As shown, after placing the preset 3D model of the object to be manipulated in the environment model, the robot model corresponding to the robot can be controlled to run in the environment model based on its running state to perform tasks on the object, thereby obtaining a scene mesh model. Then, ray tracing can be used to determine which meshes in the scene mesh model are occupied by the preset 3D model of the object, that is, to determine which voxels in the scene mesh model are the voxels corresponding to the preset 3D model of the object.

[0097] S507, Output: True value of 3D occupancy of the operand.

[0098] For example, the voxels of the preset 3D model of the object can be output as 3D occupancy ground truth to train the 3D perception model.

[0099] In this embodiment, the preset model library can provide digital twins of objects for virtual scene reconstruction and provide high-precision object localization when the robot manipulates the objects. Furthermore, the standardized models stored in the preset model library can support generalized recognition of similar objects, eliminating the need to remodel each object of the same category. Additionally, by extracting the object's mask to segment it from the background, interference from background noise, lighting changes, or occlusion on 3D information extraction can be avoided, ensuring the accuracy of depth estimation and pose calculation.

[0100] In some implementations, S104 includes: acquiring dynamic extrinsic parameters; the dynamic extrinsic parameters are the transformation matrix of the camera coordinate system relative to the robot's base coordinate system, determined by the robot based on preset parameters of the robot and real-time joint coding data of the robot running in a real environment; the camera coordinate system is the coordinate system corresponding to the image acquisition device in the robot, which is used to acquire real images; projecting the image features of the training data into a three-dimensional space based on the dynamic extrinsic parameters through a three-dimensional perception model to determine multiple second voxels in the three-dimensional space; and processing the multiple second voxels based on the three-dimensional perception model to obtain predicted semantic labels for the multiple second voxels.

[0101] refer to Figure 7 The diagram shown is a schematic of a dynamic feature projection, such as... Figure 7 As shown, because robots are non-rigid bodies, they undergo drastic posture changes such as squatting, walking, and bending over. This causes significant fluctuations in the extrinsic parameters of the image acquisition devices within the robot (such as camera extrinsic parameters) when acquiring real-time 2D images at various moments. Therefore, to avoid the problem of severe misalignment of 2D image features in 3D space due to the use of fixed parameters, dynamic feature projection can be used to project the features of the 2D image into 3D space.

[0102] For example, when a robot is running in a real-world environment, it can generate real-time joint encoding data at each moment using real-time joint encoders configured at each joint. This real-time joint encoding data can include data such as the joint's rotation angle and rotation speed. Based on the robot's preset parameters and the real-time joint encoding data at each moment, the robot can perform real-time extrinsic calculations, that is, calculate in real-time the pose transformation matrix of the camera coordinate system relative to the robot's base (or foot) coordinate system (Base_link) or the world coordinate system at the current moment. The transformation matrix This refers to real-time extrinsic parameters (or dynamic extrinsic parameters). The robot's preset parameters can be determined based on the robot's URDF file, and these preset parameters may include parameters related to the robot's structure, joints, and dynamic properties. The z-axis of the robot's base coordinate system and world coordinate system is perpendicular to the direction of gravity.

[0103] For example, dynamic extrinsic parameters may include a three-dimensional translation vector and a three-dimensional rotation matrix. The three-dimensional translation vector represents the spatial position offset of the camera coordinate system relative to the robot base or world coordinate system at the current moment. As the robot's joints move, the three-dimensional translation vector is dynamically updated with changes in joint angles to reflect real-time changes in camera height and position. The three-dimensional rotation matrix represents the orientation change of the camera coordinate system relative to the robot base or world coordinate system at the current moment, used to ensure that the projection direction is aligned with the direction of gravity.

[0104] For example, a dynamic view transformation (Dynamic View Transformer) can be performed on a 2D image using dynamic extrinsic parameters, based on an LSS (Lift, Splat, Shoot) model or the cross-attention mechanism in a Transformer, to project 2D image features into a 3D space or BEV space. For instance, in the Lift stage of the LSS model, dynamic extrinsic parameters can replace pre-calibrated fixed extrinsic parameters, projecting pixels in the 2D image along the depth direction onto a voxel mesh in 3D space based on the current dynamic extrinsic parameters. Thus, as the robot moves, the projected depth position and direction change with the camera height, ensuring that the position of all pixels in 3D space remains consistent with the robot's base or the ground. In the cross-attention mechanism of the Transformer model, dynamic extrinsic parameters can be used to replace the pre-calibrated fixed extrinsic parameters, thereby injecting the dynamic extrinsic parameters into the spatial position encoding. When the robot body shakes, the displacement can be dynamically compensated based on the spatial position encoding, so that the three-dimensional perception result output by the model is a stable scene under the robot base or world coordinate system, rather than a shaking camera perspective.

[0105] In this embodiment of the application, by calculating real-time dynamic extrinsic parameters, the three-dimensional spatial structure generated by the three-dimensional perception network can always remain stable relative to the robot base or world coordinate system, regardless of whether the robot is in a crouching operation or standing and walking motion state, so as to avoid spatial mapping jitter caused by the robot body shaking.

[0106] refer to Figure 8 The diagram shown is an architectural schematic of a three-dimensional perception model, as follows: Figure 8As shown, the 3D perception model can include an image feature extraction network, an occupancy network 3D spatial encoder, and a viewpoint transformation encoder. For example, real images captured by the robot (such as images from a surround-view camera) can be used as input data for the image feature extraction network to extract image features from the real images, obtaining the depth and image features of the real images. Next, the transformation relationship from the camera space to the occupancy network's 3D base coordinate system can be determined based on multi-camera parameters (such as the camera's intrinsic and extrinsic parameters) and the robot's joint coding states (i.e., joint coding data). Then, the depth and image features of the real images, as well as the transformation relationship, are processed based on the viewpoint transformation encoder to obtain the occupancy network 3D spatial features corresponding to the real images. The transformation relationship from camera space to 3D base space can be determined based on multi-camera parameters and joint coding states. If a gravity sensor (such as an inertial measurement unit, IMU) is configured in the robot, a more accurate transformation matrix from camera space to 3D base space can be determined based on the robot's pose information relative to the global environment and the total acceleration of the robot joints in the three axes measured by the gravity sensor (which can be used to determine the tilt angle of the robot joints relative to the direction of gravity).

[0107] Then, the occupancy network's 3D spatial features can be processed based on the occupancy network 3D spatial encoder and prediction head to obtain the occupancy network OCC prediction results of the 3D perception model for real images. The ground truth OCC can be obtained by reconstructing the real environment using a scanner in a simulation environment to obtain a global dense point cloud and real-time joint pose information of the robot during its operation in the real environment, thus reproducing the robot's running state and obtaining local dense point cloud data from the robot's perspective. Based on this, the OCC results obtained by ground truth processing of the local dense point cloud data can be used for supervised training of the prediction results output by the occupancy network OCC model (i.e., the 3D perception model). Supervised training can be achieved based on loss function calculation and gradient backpropagation, thereby driving model training and optimization based on massive amounts of data.

[0108] Based on the foregoing embodiments, this application provides a method for applying a three-dimensional perception model, which can be executed by a robot equipped with a three-dimensional perception model trained based on the above embodiments. Figure 9 This is a schematic diagram illustrating the implementation process of an application method for a three-dimensional perception model provided in an embodiment of this application, as shown below. Figure 9 As shown, the method includes S901 to S904: S901: When the robot is running in a real environment, it acquires the robot's real-time joint coding data and captures two-dimensional images through the image acquisition device in the robot.

[0109] refer to Figure 10 The diagram shown illustrates an application method of a three-dimensional perception model, as follows: Figure 10 As shown, after obtaining the trained 3D perception model based on the above embodiments, the trained 3D perception model can be configured in the robot. Thus, when the robot runs in a real environment, it can perceive its surroundings based on the trained 3D perception model to determine the semantic labels (hereinafter referred to as target semantic labels) corresponding to each voxel in 3D space. For example, when the robot runs in a real environment, it can first obtain real-time joint encoding data at each moment through the joint encoders configured in each joint, i.e., robot joint state information (Proprioception); and then acquire corresponding 2D images at each moment through the image acquisition device in the robot, such as surround-view camera images (Visual Data).

[0110] S902 determines the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the robot's base coordinate system, based on the robot's preset parameters and real-time joint coding data.

[0111] In some implementations, after acquiring the real-time joint coding data at each moment, the robot can determine the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the robot's base coordinate system or world coordinate system based on the robot's preset parameters and the real-time joint coding data at each moment. It is understood that the method for determining the dynamic extrinsic parameters can be referred to the description in S104, and will not be repeated here.

[0112] S903, through a trained 3D perception model, projects the image features of a 2D image into a 3D space based on dynamic extrinsic parameters, and determines multiple voxels in the 3D space.

[0113] In some implementations, after determining the dynamic extrinsic parameters at each time step, the image features of the corresponding two-dimensional image at each time step can be projected into the three-dimensional space based on the dynamic extrinsic parameters at each time step using the trained three-dimensional perception model, and multiple voxels in the three-dimensional space can be determined based on the features projected into the three-dimensional space.

[0114] S904 processes multiple voxels based on the trained 3D perception model to obtain target semantic labels for multiple voxels.

[0115] In some implementations, the trained 3D perception model is a multi-semantic layer occupancy network, possessing fine-grained semantic discrimination capabilities. The robot can classify multiple voxels based on the trained 3D perception model to determine the target semantic label corresponding to each voxel and output a voxelized 3D semantic grid. The target semantic label can include an idle layer, an occupied layer, or an occluded layer (i.e., the current blind spot); the occupied layer can include a body layer (such as the robot's arms, legs, torso, etc.), an object layer corresponding to the object being manipulated by the robot (such as an item being carried or grasped), or an environment layer corresponding to environmental object elements; the environment layer can include a dynamic environment layer corresponding to dynamic environment object elements (such as moving obstacles like pedestrians or vehicles) or a static environment layer corresponding to static environment voxels (such as stationary obstacles like walls, floors, and furniture).

[0116] In this embodiment, dynamic extrinsic parameters are calculated based on the robot's real-time joint state and injected into the network projection layer to correct posture jitter. Thus, regardless of how the robot moves, the 3D map output based on the 3D perception model is always gravity-aligned and stable, ensuring the continuity of planning and avoiding perception jitter of non-rigid robots.

[0117] In some embodiments, the above method further includes: if the target semantic label of the voxel to be processed among the multiple voxels is the ontology layer, then the voxel to be processed is filtered; if the target semantic label of the voxel to be processed is the operation object layer, then the voxel to be processed is dynamic planning; if the target semantic label of the voxel to be processed is the environment layer, then the voxel to be processed is obstacle avoidance; if the target semantic label of the voxel to be processed is the occluded layer, then the voxel to be processed is pending.

[0118] For example, after determining the 3D semantic grid, the robot can process each voxel based on its target semantic label in the 3D semantic grid using a planning algorithm. Taking any voxel among multiple voxels (hereinafter referred to as the voxel to be processed) as an example, if the robot determines that the target semantic label of the voxel to be processed is the self-body layer based on the trained 3D perception model, the voxel to be processed can be filtered to prevent the robot from deadlocking during obstacle avoidance.

[0119] If the target semantic label of the voxel to be processed is the operation object layer, the operation object can be regarded as an extension of the body, and dynamic follow-up planning can be performed on the robot end effector and the voxel to be processed as a whole, instead of treating the operation object as an obstacle to be simply bypassed. This allows the robot to correctly operate the operation object. For example, when the robot runs on a slope and carries a box, the view of the camera in the robot will tilt and change in height due to the terrain undulation. In this process, by calculating the dynamic extrinsic parameters in real time, it can be ensured that the spatial position and orientation of the robot body, the operation object and the surrounding environment voxels in the perceived 3D environment remain stable relative to the robot base or world coordinate system when the robot runs to different heights. The robot end effector (such as a robotic arm) can be dynamically adjusted according to the voxels of the operation object, so that the robot can keep the box stable and avoid tipping over when carrying the box.

[0120] If the target semantic label of the voxel to be processed is an environment layer (such as a static environment layer or a dynamic environment layer), then the voxel to be processed can be subjected to conventional static or dynamic obstacle avoidance processing to avoid collisions between the robot and environmental objects. If the target semantic label of the voxel to be processed is an occluded layer, then the voxel to be processed can be marked as unknown for pending processing, thereby avoiding illusions when the model processes occluded areas.

[0121] In this embodiment, by dividing voxels in three-dimensional space into a static environment layer, a dynamic environment layer, a body layer, a manipulated object layer, and an occluded layer, and processing different types of voxels accordingly based on a planning algorithm, the body layer can be filtered to prevent self-obstacle avoidance deadlock, and the manipulated object layer can be subjected to specific dynamic planning instead of simple detours. Furthermore, unknown processing can be performed on occluded areas, eliminating false idle noise caused by line-of-sight occlusion in traditional methods. This improves the model's understanding of complex environments and prevents the model from misinterpreting and incorrectly processing voxels in occluded areas. Moreover, it enables the robot to accurately distinguish between the environment, the body, and the manipulated object, allowing the robot to navigate safely without unintentionally triggering obstacle avoidance, thus avoiding target confusion.

[0122] Based on the above embodiments, this application provides a 3D perception and training system that can adapt to changes in robot posture, accurately distinguish between the body and the environment, and use GPUs or other parallel processing units to accelerate ray tracing to generate unambiguous truth values.

[0123] 1. Multi-semantic layer network construction like Figure 8 and Figure 10As shown, a surround perception model (i.e., a 3D perception model) with fine-grained semantic discrimination capabilities is established. Unlike traditional models that only predict the 3D occupancy of the environment, this application constructs a multi-semantic layer occupancy network. Its input receives images from a surround-view camera and the robot's current joint state information, and its output outputs a voxelized 3D semantic grid (including voxel point coordinates and semantic labels). For example, voxels in 3D space can be divided into the following independent semantic categories: static environment layer (e.g., immovable objects such as walls, floors, and furniture), dynamic environment layer (e.g., moving obstacles such as pedestrians and other vehicles), ontology layer (e.g., the robot's arms, legs, and torso), manipulation object layer (e.g., the object the robot is currently grasping or carrying), and occlusion layer. The unknown processing of voxels in occluded areas prevents the model from experiencing illusions in these areas. This allows the planning algorithm to filter out the ontology layer to prevent self-obstacle avoidance deadlock, perform specific dynamic planning on the manipulation object layer instead of simple detours, and handle the unknown processing of occluded areas to avoid obvious errors caused by the model forcibly filling in the gaps.

[0124] 2. Dynamic viewpoint adaptive projection based on real-time kinematics calculation like Figure 7 As shown, to address the changes in camera extrinsic parameters caused by the robot's varied postures, this application proposes a dynamic feature projection mechanism: Real-time extrinsic parameter calculation: Based on the robot's URDF file and real-time joint encoder data, the transformation matrix of the camera coordinate system relative to the robot's base coordinate system or world coordinate system at the current moment is calculated in real time.

[0125] Dynamic View Transformer: When projecting 2D image features into 3D space (or BEV space), instead of using pre-calibrated fixed extrinsic parameters, the aforementioned dynamically calculated extrinsic parameters are injected into the projection layer (such as depth projection in the LSS layer or cross-attention position encoding in the Transformer). Therefore, regardless of whether the robot is crouching or standing, the 3D spatial structure generated by the perception network remains stable relative to the robot's feet or the world coordinate system, eliminating spatial mapping jitter caused by body swaying.

[0126] 3. Scalable data production pipeline based on scan reconstruction and hybrid simulation like Figure 2 As shown, in order to address the difficulties in obtaining the truth and the differences in perspectives, this application presents an embodiment of a data closed-loop system of "scanning-simulation-hybrid".

[0127] Step 1: Offline Scene Reconstruction. A high-precision handheld scanner can be used to perform a static scan of the real working environment and reconstruct a high-fidelity 3D mesh map or a 3D Gaussian splash model as the ground truth of the environment.

[0128] Step 2: Real-world trajectory recording. Control the robot to run in the real environment, record the time-stamped camera image stream and its own joint motion state, as well as LiDAR data, and determine the robot's position (Pose) in the environment based on the LiDAR data.

[0129] Step 3: Digital twin data synthesis.

[0130] Load the "environment model" reconstructed in the first step into the simulation engine; load the recorded robot pose; load the robot's URDF digital model (for generating ontology ground truth); load the 3D model of the object being transported (for generating object ground truth). The loading method can be found in [reference needed]. Figure 5 As shown in Figure 6, by simulating the real camera viewpoint in the simulation, 2D camera images can be quickly synthesized.

[0131] Step 4: A truth generation engine based on GPU parallel processing ray tracing.

[0132] Simulate the robot's real camera view in a simulation environment to render a depth map and a semantic segmentation map, which can be used as the ground truth (GT) for training.

[0133] Parallel ray emission mechanism: such as Figure 3 As shown, using GPU parallel processing, tens of millions of simulated rays are emitted toward the center of each voxel within the view frustum, with the optical center of the virtual camera configured in the simulation environment as the origin.

[0134] Physical intersection calculation: Calculates the nearest intersection point between the ray and all Mesh (including environment, body, and objects) in the scene.

[0135] Unambiguous labeling of voxel three states: Occupied: When a ray intersects with a Mesh inside a voxel, it is marked with the corresponding semantic ID. Please refer to the semantic subdivision determination section below.

[0136] Free: A ray intersects the Mesh (or has no intersection) after passing through a voxel -> marked as Free.

[0137] Occluded: The ray was blocked by another Mesh before reaching this voxel -> marked as Unknown.

[0138] Semantic Refinement: When a voxel is determined to be "Occupied", the semantic ID of the hit mesh is further retrieved, which can subdivide the occupancy state into four categories: Static Occupancy: Rays hit the background map Mesh.

[0139] Dynamic environment: Dynamic Mesh such as ray hitting pedestrians and non-interactive vehicles.

[0140] Main body: The ray hits the robot's own limb, Mesh.

[0141] Manipulate objects: The ray hits the Mesh that is being grabbed or interacted with.

[0142] Step 5: Sim-Real Hybrid Training. During training, the input can be real camera images (preserving realistic lighting and texture) or simulated synthetic images. The supervision signal (Label) is an accurate 3D ground truth. This enables low-cost, high-precision, large-scale generation of training data.

[0143] Voxels calculated using the X-ray emission mechanism can be used as a reference. Figure 4A and Figure 4B As shown, blue and gray represent Occupied, which represent obstacles and the ground respectively, red represents Occluded, and transparent invisible voxels represent Free. It can be seen that the voxel occupancy ground value is very dense, so it would be very slow to process using a CPU. However, in this embodiment, large-scale voxelization processing can be achieved through GPU parallel computing.

[0144] In this embodiment, a multi-semantic layer occupancy network is constructed to divide voxels in 3D space into a static environment layer, a dynamic environment layer, a ontology layer, an operation object layer, and an occluded layer. This allows the planning algorithm to filter out the ontology layer to prevent self-obstacle avoidance deadlock, perform specific dynamic planning for the operation object layer instead of simple detours, and handle the unknown in occluded areas to avoid obvious errors caused by the model forcibly filling in the blanks. Furthermore, through dynamic viewpoint adaptive projection based on real-time kinematics calculation, the 3D spatial structure generated by the 3D perception network remains stable relative to the robot's feet or the world coordinate system, regardless of whether the robot is crouching or standing, thus eliminating spatial mapping jitter caused by robot body swaying. Moreover, through a large-scale data production pipeline based on scan reconstruction and hybrid simulation, parallel ray tracing using GPUs can be used to simulate physical collisions of light paths, thereby accurately calculating which voxels are Occluded, Occupied, and Free, enabling large-scale and rapid production of high-precision 3D ground truth. In addition, by synthesizing digital twin simulation data and Sim-Real hybrid training, it is possible to optimize the occupancy network model based on extremely high data production efficiency and accurate ground truth.

[0145] In this embodiment, during Sim-Real hybrid training, all simulated images can be used through Domain Randomization. However, extreme textures, lighting, and noise can be applied to the simulated images to force the 3D perception network to learn geometric features, thereby adapting it to the real world. Alternatively, Style Transfer / CycleGAN can be used to transform simulated images into real-style images for training, or real images can be transformed into simulated styles for inference. Alternatively, self-supervised learning can be used, leveraging the photometric consistency of real-world video sequences for self-supervised training, without requiring explicit 3D ground truth.

[0146] The application method and training method of the 3D perception model provided in this application embodiment can be applied to various scenarios, such as: 1. Online operation scenario (actual robot operation stage) (1) Accurate perception of the surroundings of complex humanoid / non-rigid robots Scenario description: When a humanoid robot squats, twists its waist, turns to the side, or walks on a bumpy road, its body posture and the camera's perspective will change drastically.

[0147] Application Value: Based on the scheme of real-time dynamic extrinsic parameter calculation and injection into the network projection layer in the embodiments of this application, it is possible to ensure that even when the robot twists its body violently, the 3D map output by the 3D perception model remains gravity-aligned and stable. This avoids the map jitter or ghosting problems caused by large camera movements in traditional perception algorithms, ensuring the continuity of subsequent planning modules.

[0148] (2) Embodied intelligence close contact object handling and manipulation Scenario description: The robot needs to carry a box or hand tools with both hands to perform tasks, or move when its body is very close to an obstacle.

[0149] Application value: Based on the scheme of distinguishing between the body, environment and manipulated object in the embodiments of this application, the robot can clearly know that the box it is holding is not an obstacle, thereby avoiding the common problem of lifting a stone to drop it on its own foot in traditional algorithms (that is, the robot mistakenly thinks there is an obstacle in front of it because it sees something in its hand and brakes suddenly), thus realizing smooth object-holding navigation and close-range operation.

[0150] (3) Navigation in unstructured environments with many blind spots or dynamic changes Scenario description: In a home service, a chaotic warehouse, or post-disaster ruins, there are numerous obstructions to the view (such as under a table or behind a door).

[0151] Application value: By introducing the occluded state, the robot can clearly distinguish between areas that are not visible due to occlusion and areas that are considered free. This avoids false negative noise caused by line-of-sight obstruction, prevents the robot from rashly entering unexplored dangerous blind spots, and improves safety in complex environments.

[0152] 2. Offline Development and Training Scenarios (Algorithm Development Phase) (1) Low-cost, large-scale data production with embodied intelligent perception (data factory) Scenario description: Training a high-precision 3D occupancy network typically requires millions of frames of labeled data, and manually annotating 3D voxels is both costly and inefficient.

[0153] Application Value: Based on the "scan reconstruction-simulation playback" digital twin pipeline and GPU-accelerated ray tracing scheme in the embodiments of this application, the optical path of the physical world can be reproduced in a simulation environment. This allows computing devices to automatically generate unambiguous 3D truth values ​​containing the Occupied / Free / Occluded states at millisecond-level speeds (millions of frames per day). This significantly reduces the time and cost of building massive amounts of training data.

[0154] (2) Sim-Real Hybrid Supervised Training Scenario Description: To avoid the problem of 3D perception models trained solely on simulation data failing to adapt to the real world (Sim-to-Real Gap).

[0155] Application Value: Based on the hybrid training mode in this application embodiment, the input to the 3D perception model can be real-world images (preserving real lighting, sensor noise, and texture details) and simulated images, while the supervision signal (Label) is an accurate 3D ground truth generated through a digital twin system. This allows for both the preservation of visual realism and the possession of accurate simulation ground truth, making it suitable for training perception models that are sensitive to lighting but require precise spatial geometric understanding.

[0156] In summary, the embodiments of this application use GPU parallel computing to replace manual annotation, quickly generating massive amounts of high-quality training data, which can improve the training speed of 3D perception models; and enable the robot to maintain stable 3D perception even when it is shaking, holding heavy objects, or when there is occlusion in the surrounding area.

[0157] Based on the foregoing embodiments, this application provides a training device for a three-dimensional perception model. This device includes various units and modules within each unit, which can be implemented using a processor in a computing device; alternatively, they can be implemented using specific logic circuits. During implementation, the processor can be a Central Processing Unit (CPU), a Microprocessor Unit (MPU), a Digital Signal Processor (DSP), or a Field Programmable Gate Array (FPGA), etc. When the processor or logic circuit in the computing device executes the program, it implements the aforementioned training method for the three-dimensional perception model.

[0158] Figure 11 This is a schematic diagram of the composition structure of a training device for a three-dimensional perception model provided in an embodiment of this application, as shown below. Figure 11 As shown, the training device 1100 for the 3D perception model can be configured on the computing device 11. The training device 1100 for the 3D perception model includes: a first acquisition module 1110, a collection module 1120, a first determination module 1130, a first processing module 1140, and a training module 1150, wherein: The first acquisition module 1110 is configured to acquire an environment model corresponding to the real environment and real machine data of the robot running in the real environment. The real machine data includes the robot's running state and real images captured by the robot. The acquisition module 1120 is configured to control the robot model corresponding to the robot to run in the environment model based on the running state in the simulation environment and to capture simulation images. The first determination module 1130 is configured to determine the true semantic labels of multiple first voxels corresponding to the simulation images. The first processing module 1140 is configured to process the training data based on the 3D perception model to obtain predicted semantic labels of multiple second voxels corresponding to the training data. The training data includes real images and / or simulation images. The training module 1150 is configured to train the 3D perception model based on the predicted semantic labels of multiple second voxels and the true semantic labels of multiple first voxels to obtain the trained 3D perception model.

[0159] In some implementations, the first determining module 1130 is specifically configured to: generate a scene mesh model of the robot model running in the environment model based on the running state; the virtual image acquisition device has the same viewpoint as the robot; emit simulated light rays to each first voxel in the scene mesh model with the optical center of the virtual image acquisition device as the origin through a parallel processing unit; and determine the real semantic label of each first voxel based on the intersection state between each first voxel and the simulated light rays corresponding to each first voxel.

[0160] In some implementations, the plurality of first voxels includes non-object voxels and object voxels; the first determining module 1130 is configured to at least: determine the intersection point position when the target simulated ray intersects with the plurality of first voxels for the first time; the target simulated ray is the simulated ray corresponding to the target voxel among the plurality of first voxels; the first determining module 1130 is further configured to at least one of the following: if the intersection point position is located at the target voxel, then determine the true semantic label of the target voxel as an occupied layer; the target voxel is an object voxel; if the target simulated ray generates an intersection point after passing through the target voxel, then determine the true semantic label of the target voxel as an idle layer; the target voxel is a non-object voxel; if the target simulated ray generates an intersection point before reaching the target voxel, then determine the true semantic label of the target voxel as an occluded layer.

[0161] In some implementations, the first determining module 1130 is specifically configured to: if the intersection point is located at the target voxel, determine the candidate semantic label of the target voxel as an occupying layer; determine the voxel type of the target voxel; the voxel type includes the robot's body voxel, the robot's operating object voxel, or the environment voxel; determine the true semantic label of the target voxel based on the candidate semantic label and the voxel type; the true semantic label of the target voxel includes one of the body layer, the operating object layer, and the environment layer.

[0162] In some implementations, the first processing module 1140 is specifically configured to: acquire dynamic extrinsic parameters; the dynamic extrinsic parameters are the transformation matrix of the camera coordinate system relative to the robot's base coordinate system, determined by the robot based on the robot's preset parameters and real-time joint coding data during the robot's operation in a real environment; the camera coordinate system is the coordinate system corresponding to the image acquisition device in the robot, which is used to acquire real images; project the image features of the training data into a three-dimensional space based on the dynamic extrinsic parameters through a three-dimensional perception model to determine multiple second voxels corresponding to the training data; and process the multiple second voxels based on the three-dimensional perception model to obtain predicted semantic labels for the multiple second voxels.

[0163] In some implementations, the first determining module 1130 is specifically configured to: acquire a two-dimensional image corresponding to the object operated by the robot; extract the object mask corresponding to the object and perform depth estimation on the two-dimensional image to extract the three-dimensional information corresponding to the two-dimensional image; perform pose detection on the object based on the object mask, the three-dimensional information and the preset three-dimensional model corresponding to the object to obtain the pose information of the object; load the preset three-dimensional model corresponding to the object into the environment model based on the pose information of the object; and control the robot model corresponding to the robot to run in the environment model based on the running state to perform operations on the preset three-dimensional model corresponding to the object to obtain a scene mesh model.

[0164] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0165] Based on the foregoing embodiments, this application provides an application device for a three-dimensional perception model. This device includes various units and modules within each unit, which can be implemented using a processor in a robot; alternatively, it can be implemented using specific logic circuits. During implementation, the processor can be a Central Processing Unit (CPU), a Microprocessor Unit (MPU), a Digital Signal Processor (DSP), or a Field Programmable Gate Array (FPGA), etc. When the processor or logic circuit in the robot executes the program, it implements the aforementioned application method for the three-dimensional perception model.

[0166] Figure 12This is a schematic diagram of the composition structure of an application device for a three-dimensional perception model provided in an embodiment of this application, as shown below. Figure 12 As shown, the 3D perception model application device 1200 can be configured on a robot 12, which is equipped with a trained 3D perception model. The 3D perception model application device 1200 includes: a second acquisition module 1210, a second determination module 1220, a third determination module 1230, and a second processing module 1240. Specifically: the second acquisition module 1210 is configured to acquire real-time joint coding data of the robot while it is running in a real environment, and acquire 2D images through an image acquisition device in the robot. The second determination module 1220 is configured to determine the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the robot's base coordinate system based on the robot's preset parameters and real-time joint coding data. The third determination module 1230 is configured to project the image features of the 2D image onto a 3D space based on the dynamic extrinsic parameters using the trained 3D perception model, and determine multiple voxels in the 3D space. The second processing module 1240 is configured to process the multiple voxels based on the trained 3D perception model to obtain target semantic labels for the multiple voxels.

[0167] like Figure 12 As shown, the application device 1200 for the three-dimensional perception model also includes a third processing module 1250.

[0168] In some implementations, the target semantic label includes one of the following: an occlusion layer, a body layer corresponding to the robot's body voxel, an operation object layer corresponding to the operation object voxel operated by the robot, and an environment layer corresponding to the environment voxel; the third processing module 1250 is configured to perform at least one of the following: if the target semantic label of the voxel to be processed among the multiple voxels is a body layer, then the voxel to be processed is filtered; if the target semantic label of the voxel to be processed is an operation object layer, then the voxel to be processed is dynamically planned; if the target semantic label of the voxel to be processed is an environment layer, then the voxel to be processed is obstacle avoidance; if the target semantic label of the voxel to be processed is an occlusion layer, then the voxel to be processed is marked as an unknown voxel.

[0169] It should be noted that, in the embodiments of this application, if the above-mentioned training method or application method of the 3D perception model is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0170] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor in a computing device, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0171] This application provides a computer program including computer-readable code, wherein when the computer-readable code is run in a computing device, a processor executes some or all of the steps in the above-described method.

[0172] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computing device, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0173] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0174] This application provides a computer storage medium that stores one or more programs that can be executed by a processor to implement the steps of the training method or application method of the three-dimensional perception model in the above embodiments.

[0175] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0176] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0177] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for a three-dimensional perception model, characterized in that, The method includes: Obtain an environmental model corresponding to the real environment and real machine data of the robot running in the real environment; the real machine data includes the robot's operating status and real images captured by the robot; In the simulation environment, the robot model corresponding to the robot is controlled to run in the environment model based on the running state, and simulation images are acquired; Determine the true semantic labels of multiple first voxels corresponding to the simulated image; The training data is processed based on a 3D perception model to obtain predicted semantic labels for multiple second voxels corresponding to the training data; the training data includes the real image and / or the simulated image. The 3D perception model is trained based on the predicted semantic labels of multiple second voxels and the real semantic labels of multiple first voxels to obtain the trained 3D perception model.

2. The method according to claim 1, characterized in that, Determining the true semantic labels of multiple first voxels corresponding to the simulated image includes: Generate a scene mesh model of the robot model based on the operating state while it is running in the environment model; Simulated light rays are emitted from the optical center of the virtual image acquisition device to each of the first voxels in the scene mesh model via a parallel processing unit; the virtual image acquisition device has the same viewpoint as the robot. Based on the intersection state between each first voxel and the simulated light corresponding to each first voxel, the true semantic label of each first voxel is determined.

3. The method according to claim 2, characterized in that, The plurality of first voxels include non-object voxels and object voxels; the determination of the true semantic label of each first voxel based on the intersection state between each first voxel and the simulated light rays corresponding to each first voxel includes at least: Determine the intersection point when the target simulated ray first intersects with multiple first voxels; the target simulated ray is the simulated ray corresponding to the target voxel among the multiple first voxels; It also includes at least one of the following: If the intersection point is located within the target voxel, then the true semantic label of the target voxel is determined to be the occupancy layer; the target voxel is an object voxel. If the target simulated ray generates the intersection point after passing through the target voxel, then the true semantic label of the target voxel is determined to be an idle layer; the target voxel is a non-object voxel. If the target simulated ray generates the intersection point before reaching the target voxel, then the true semantic label of the target voxel is determined to be the occluded layer.

4. The method according to claim 3, characterized in that, If the intersection point is located within the target voxel, then the true semantic label of the target voxel is determined to be the occupancy layer, including: If the intersection point is located within the target voxel, then the candidate semantic label of the target voxel is determined to be the occupancy layer; Determine the voxel type of the target voxel; the voxel type includes the robot's body voxel, the robot's operating object voxel, or the environment object voxel; The true semantic label of the target voxel is determined based on the candidate semantic label and the voxel type; the true semantic label of the target voxel includes one of the ontology layer, the operation object layer, and the environment layer.

5. The method according to any one of claims 1 to 4, characterized in that, The process of processing the training data based on the 3D perception model to obtain predicted semantic labels for multiple second voxels corresponding to the training data includes: The dynamic extrinsic parameters are obtained; the dynamic extrinsic parameters are the transformation matrix of the camera coordinate system relative to the robot's base coordinate system, which is determined by the robot based on the robot's preset parameters and the robot's real-time joint coding data during operation in the real environment; the camera coordinate system is the coordinate system corresponding to the image acquisition device in the robot, which is used to acquire the real image; Using the three-dimensional perception model, the image features of the training data are projected onto three-dimensional space based on the dynamic extrinsic parameters to determine multiple second voxels corresponding to the training data; Based on the three-dimensional perception model, multiple second voxels are processed to obtain predicted semantic labels for multiple second voxels.

6. The method according to any one of claims 2 to 4, characterized in that, The generation of the scene mesh model based on the robot model running in the environment model in the operating state includes: Obtain a two-dimensional image of the object operated by the robot; Extract the object mask corresponding to the manipulated object, and perform depth estimation on the two-dimensional image to extract the three-dimensional information corresponding to the two-dimensional image; Based on the object mask, the three-dimensional information and the preset three-dimensional model corresponding to the manipulated object, the pose detection of the manipulated object is performed to obtain the pose information of the manipulated object; Based on the pose information of the manipulated object, the preset three-dimensional model corresponding to the manipulated object is loaded into the environment model; The robot model corresponding to the robot is controlled to run in the environment model based on the running state to perform operations on the preset three-dimensional model corresponding to the manipulated object, so as to obtain the scene mesh model.

7. A method for applying a three-dimensional perception model, characterized in that, Applied to a robot equipped with a trained 3D perception model, the method includes: When the robot is running in a real environment, the robot acquires real-time joint coding data and captures two-dimensional images through the image acquisition device in the robot. Based on the robot's preset parameters and the real-time joint coding data, the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the robot's base coordinate system are determined; Using the trained 3D perception model, the image features of the 2D image are projected into 3D space based on the dynamic extrinsic parameters, and multiple voxels in the 3D space are determined. Based on the trained 3D perception model, multiple voxels are processed to obtain target semantic labels for multiple voxels.

8. The method according to claim 7, characterized in that, The target semantic tag includes one of the following: an occlusion layer, a ontology layer corresponding to the robot's ontology voxels, an operation object layer corresponding to the operation object voxels operated by the robot, and an environment layer corresponding to the environment voxels; the method further includes at least one of the following: If the target semantic tag of the voxel to be processed among the multiple voxels is the ontology layer, then the voxel to be processed is filtered. If the target semantic label of the voxel to be processed is the operation object layer, then the voxel to be processed is subjected to dynamic planning processing. If the target semantic label of the voxel to be processed is the environment layer, then obstacle avoidance processing is performed on the voxel to be processed. If the target semantic label of the voxel to be processed is the occluded layer, then the voxel to be processed is marked as an unknown voxel.

9. A training device for a three-dimensional perception model, characterized in that, include: The first acquisition module is configured to acquire the environmental model corresponding to the real environment and the real machine data of the robot running in the real environment; The real machine data includes the robot's operating status and the real images captured by the robot; The acquisition module is configured to control the robot model corresponding to the robot to run in the environment model based on the running state in the simulation environment, and to acquire simulation images; The first determining module is configured to determine the true semantic labels of a plurality of first voxels corresponding to the simulated image; The first processing module is configured to process the training data based on a 3D perception model to obtain predicted semantic labels for multiple second voxels corresponding to the training data; the training data includes the real image and / or the simulated image. The training module is configured to train the 3D perception model based on the predicted semantic labels of multiple second voxels and the real semantic labels of multiple first voxels, so as to obtain the trained 3D perception model.

10. An application device for a three-dimensional perception model, characterized in that, Configured in a robot, the robot having a trained 3D perception model, the device includes: The second acquisition module is configured to acquire the robot's real-time joint coding data and acquire two-dimensional images through the image acquisition device in the robot when the robot is running in a real environment. The second determining module is configured to determine the dynamic extrinsic parameters of the camera coordinate system corresponding to the image acquisition device relative to the base coordinate system of the robot based on the preset parameters of the robot and the real-time joint coding data. The third determining module is configured to project the image features of the two-dimensional image into a three-dimensional space based on the dynamic extrinsic parameters using the trained three-dimensional perception model, and determine multiple voxels in the three-dimensional space; The second processing module is configured to process multiple voxels based on the trained 3D perception model to obtain target semantic labels for the multiple voxels.

11. A computing device, characterized in that, The computing device includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the program to implement the steps of the method according to any one of claims 1 to 6.

12. A robot, characterized in that, The robot is used to perform the steps in the method of claim 7 or 8.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the control component, it implements the training method of the three-dimensional perception model according to any one of claims 1 to 6, or the application method of the three-dimensional perception model according to claim 7 or 8.

Images