A multi-modal three-dimensional semantic occupancy prediction method for a wild scene and a system thereof

Abstract

The application discloses a kind of multi-modal three-dimensional semantic occupancy prediction methods for field scene, method includes: based on multiple frames of point cloud data with semantic information, generate three-dimensional semantic occupancy label;Acquire multi-modal sensor data under field scene, carry out feature extraction and historical frame superposition of sensor data, utilize knowledge distillation function to transfer the geometric knowledge in the extracted point cloud feature to image branch network, the features of extracted image and point cloud are dynamically fused, and output fusion feature;The occupancy prediction head constructed using three-dimensional convolution transforms fusion feature, generates three-dimensional semantic occupancy prediction value, realizes the automatic reconstruction of field scene.The application realizes the generation of three-dimensional semantic occupancy label, significantly reduces the error of reconstructed scene, and fills the vacancy of three-dimensional semantic occupancy prediction method under field scene.

Description

Technical Field

[0001] This application relates to the field of environmental perception for autonomous driving, and in particular to a multimodal three-dimensional semantic occupancy prediction method and system for outdoor scenes. Background Technology

[0002] The goal of 3D semantic occupancy prediction is to predict the semantic label of each occupancy voxel. In recent years, deep learning-based 3D semantic occupancy prediction has been widely applied in the field of autonomous driving. The 3D semantic occupancy prediction task divides the space within the perception range into multiple voxels, and for each voxel, predicts its occupancy state and semantic information. Deep learning-based 3D semantic occupancy prediction methods can perform fast inference on GPUs (Graphics Processing Units), exhibiting a certain degree of real-time performance. However, existing occupancy prediction methods mainly focus on road environments, and there is currently no research on 3D semantic occupancy perception for off-road environments. Because the ground in off-road scenarios is uneven and contains a large number of obstacles with height information (trees, shrubs, etc.), 3D representation can provide more comprehensive information compared to 2D representation, which loses height information.

[0003] The initial goal of knowledge distillation is to compress the model by transferring knowledge from the teacher model to the student model. Due to its effectiveness, knowledge distillation has been applied to the 3D semantic occupancy prediction task. However, existing methods all perform knowledge distillation within the same modality and cannot achieve cross-modal knowledge distillation.

[0004] Furthermore, current off-road perception mainly focuses on traversable area segmentation tasks, with relatively little research on reconstructing the off-road environment. Since scene reconstruction can provide more comprehensive perception information, introducing 3D semantic occupancy prediction tasks into off-road environments is particularly important.

[0005] However, most current 3D semantic occupancy prediction labels are for road environments, and there are currently no corresponding labels for field environments. Research has found that current methods for generating 3D semantic occupancy prediction labels for road environments produce significant errors when applied to field scenes.

[0006] Therefore, in order to make full use of the rich geometric information in lidar data, it is urgent to propose a cross-modal knowledge distillation function to transfer the geometric information in the lidar branch to the camera branch more fully, so as to improve the performance of the camera branch when working alone; at the same time, it is urgent to have a three-dimensional semantic occupancy prediction method with small error that can be applied to field scenes. Summary of the Invention

[0007] To address the significant errors that arise in existing field scenarios, a multimodal 3D semantic occupancy prediction method and system for field scenarios are proposed.

[0008] In a first aspect, embodiments of this application provide a multimodal 3D semantic occupancy prediction method for outdoor scenes, applied to camera branches, radar branches, and multimodal fusion branches. The method includes:

[0009] The steps for generating 3D semantic occupancy labels are as follows: Based on multi-frame point cloud data with semantic information, coordinate projection and semantic coordinate classification methods are used to generate 3D semantic occupancy labels.

[0010] Feature extraction steps: Collect multimodal sensing data in the field scene, extract features from the sensing data and overlay historical frames. The multimodal sensing data includes multiple frames of images from the camera and multiple frames of point cloud data from the lidar.

[0011] Knowledge transfer steps: The camera branch and radar branch are trained using the three-dimensional semantic occupancy labels. The geometric knowledge in the extracted point cloud features is transferred to the image branch network using the knowledge distillation function. The extracted image and point cloud features are dynamically fused to output the fused features.

[0012] Prediction result generation steps: The occupancy prediction head constructed using 3D convolution transforms the fused features to generate a predicted value of 3D semantic occupancy, thereby achieving automatic reconstruction of the field scene.

[0013] In a specific embodiment of the present invention, the above feature extraction step includes:

[0014] In the camera branch, a deep convolutional neural network is used to extract multi-scale two-dimensional features from the image, and the LSS method is used to project the two-dimensional features into a three-dimensional voxel space to obtain image voxel features.

[0015] In the radar branch, voxel networks are used to extract features from point cloud data accumulated from multiple frames, and three-dimensional voxel space is used for representation to obtain point cloud voxel features.

[0016] In a specific embodiment of the present invention, the above-mentioned knowledge transfer steps include:

[0017] Adaptive fusion steps: When training the camera branch, a soft-supervised distillation function is used to efficiently transfer the rich geometric knowledge in the LiDAR branch to the camera branch;

[0018] Multimodal fusion steps: The point cloud voxel features and image voxel features are dynamically fused using an adaptive fusion function to output the fused features.

[0019] In a specific embodiment of the present invention, the distillation function L of the above-mentioned soft supervision distill as follows:

[0020]

[0021] Among them, fI (x,y,z) is the image voxel feature F I Features at coordinates (x, y, z), f L (x,y,z) is a point voxel feature F L The feature at coordinates (x, y, z) is defined such that if the voxel at (x, y, z) in the label is already occupied, then parameter M... x,y,z =1, otherwise M x,y,z =0, H represents the length of the voxel feature space, W represents the width of the voxel feature space, and D represents the height of the voxel feature space.

[0022] In a specific embodiment of the present invention, the above-mentioned adaptive fusion function F M for:

[0023] F M =σ(W)×F I +(1-σ(W))×F L ,

[0024] Among them, F I For image voxel features, F L σ represents the voxel features of the point cloud, W is the trainable parameter output by the 3D convolution, σ is the sigmoid function, and × represents the element-wise product.

[0025] In a specific embodiment of the present invention, the above-mentioned three-dimensional semantic occupancy tag generation step includes:

[0026] Point cloud stitching steps: In the field scene to be detected, each frame of point cloud with semantic information is projected onto the world coordinate system using absolute pose information. The projected multi-frame point clouds are stitched together to generate a dense point cloud of the current field scene with semantic information.

[0027] Voxel generation steps: Based on the semantic category labels attached to the point cloud data, all point clouds are divided into ground class and non-ground class. Point clouds belonging to the ground class are reconstructed using coarse-grained Poisson surface, and point clouds belonging to the non-ground class are reconstructed using fine-grained Poisson surface. The reconstructed mesh structure is then voxelized to generate voxels.

[0028] Label generation steps: Use the K-nearest neighbor algorithm to classify the center point of each voxel, and assign the center point label to the corresponding voxel according to the semantic category of the nearest point to generate a three-dimensional semantic occupancy label.

[0029] Secondly, embodiments of this application provide a method for constructing three-dimensional semantic occupancy labels for outdoor scenes, applied to the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes described above. The label construction method includes:

[0030] Point cloud stitching steps: In the field scene to be detected, each frame of point cloud with semantic information is projected onto the world coordinate system using absolute pose information. The projected multi-frame point clouds are stitched together to generate a dense point cloud of the current field scene with semantic information.

[0031] Voxel generation steps: Based on the semantic category labels attached to the point cloud data, all point clouds are divided into ground class and non-ground class. Point clouds belonging to the ground class are reconstructed using coarse-grained Poisson surface, and point clouds belonging to the non-ground class are reconstructed using fine-grained Poisson surface. The reconstructed mesh structure is then voxelized to generate voxels.

[0032] Label generation steps: Use the K-nearest neighbor algorithm to classify the center point of each voxel, and assign the center point label to the corresponding voxel according to the semantic category of the nearest point to generate a three-dimensional semantic occupancy label.

[0033] Thirdly, embodiments of this application provide a multimodal 3D semantic occupancy prediction system for outdoor scenes, employing the multimodal 3D semantic occupancy prediction method for outdoor scenes as described above. The system is applied to camera branches, radar branches, and multimodal fusion branches, and includes:

[0034] 3D semantic occupancy label generation module: Based on multi-frame point cloud data with semantic information, it generates 3D semantic occupancy labels using coordinate projection and semantic coordinate classification methods;

[0035] Feature extraction module: Collects multimodal sensing data in the field scene, performs feature extraction of the sensing data and overlays it with historical frames. The multimodal sensing data includes multiple frames of images from the camera and multiple frames of point cloud data from the lidar.

[0036] Knowledge transfer module: The camera branch is trained using 3D semantic occupancy labels. The knowledge distillation function is used to transfer the geometric knowledge in the extracted point cloud features to the image branch network. The extracted image and point cloud features are dynamically fused to output the fused features.

[0037] Prediction Result Generation Module: The occupancy prediction head, constructed using 3D convolution, transforms the fused features to generate a predicted value of 3D semantic occupancy, enabling automatic reconstruction of the field scene.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes.

[0039] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described above.

[0040] Compared with existing technologies, it has the following outstanding advantages:

[0041] 1) The method of the present invention consists of a camera branch, a lidar branch and a multimodal fusion branch. By extracting features from camera images and lidar point clouds, it can reconstruct the surrounding environment and is suitable for field scene understanding tasks in autonomous driving systems.

[0042] 2) The method of this invention proposes a cross-modal knowledge distillation function, which can more fully transfer the geometric information in the lidar branch to the camera branch, so as to improve the performance of the camera branch when it works alone;

[0043] 3) The method of this invention proposes a coarse-to-fine tag generation framework to optimize the generation of 3D semantic occupancy tags, which significantly reduces the error of scene reconstruction. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1 This is a schematic diagram of the multimodal three-dimensional semantic occupancy prediction method of the present invention;

[0046] Figure 2 This is a schematic diagram of a multimodal three-dimensional semantic occupancy prediction method according to a specific embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the multimodal 3D semantic tag generation method of the present invention;

[0048] Figure 4 This is a schematic diagram of the multimodal 3D semantic tag generation method according to a specific embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of a multimodal three-dimensional semantic occupancy prediction system according to an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the computer hardware of the present invention. Detailed Implementation

[0051] It should be noted that the processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0052] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0053] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0054] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0055] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0057] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0058] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0059] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0061] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0062] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0065] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0066] This invention aims to propose a method and apparatus for multimodal 3D semantic occupancy prediction in outdoor scenes using deep learning. The method consists of a camera branch, a LiDAR branch, and a multimodal fusion branch. By extracting features from camera images and LiDAR point clouds, it reconstructs the surrounding environment and is suitable for outdoor scene understanding tasks in autonomous driving systems.

[0067] This invention proposes a method and apparatus for predicting 3D semantic occupancy in off-road scenarios, as well as a method for constructing 3D semantic occupancy labels for outdoor scenes, aiming to fill the gap in 3D semantic occupancy prediction methods for outdoor scenes. This framework improves the accuracy of 3D semantic occupancy prediction by fusing multi-frame images and point cloud information into voxel space and utilizing a cross-modal distillation function to transfer geometric knowledge from the LiDAR branch to the camera branch. Simultaneously, a coarse-to-fine label generation framework is proposed to optimize the generation of 3D semantic occupancy labels, significantly reducing errors in scene reconstruction.

[0068] This invention provides a deep learning-based multimodal 3D semantic occupancy prediction method for outdoor scenes. This method combines camera and LiDAR data, achieving high-precision 3D semantic occupancy prediction of outdoor scenes through multimodal fusion and cross-modal distillation. To address unsafe outdoor environments, both the camera and LiDAR branches can operate independently. The method includes the following steps:

[0069] (1) Input data is acquired through multimodal sensors, including multi-frame images from the camera and multi-frame point cloud data from the lidar, and feature extraction and historical frame overlay are performed respectively.

[0070] (2) The image obtained in step (1) is fused with the point cloud features. When training the camera branch separately, the geometric knowledge in the point cloud features is transferred to the image branch network using the knowledge distillation function.

[0071] (3) The features obtained in step (2) are converted into predicted values ​​of three-dimensional semantic occupancy using an occupancy prediction head constructed by three-dimensional convolution.

[0072] In step (1), the deep convolutional neural network is a ResNet-101 pre-trained network model on ImageNet. In step (2), the multimodal fusion uses an adaptive fusion module to dynamically fuse image voxel features and point voxel features. In step (2), the distillation function is the weakly supervised knowledge distillation function proposed in this invention.

[0073] The following detailed description is provided in conjunction with specific embodiments:

[0074] Example 1

[0075] like Figure 1 As shown, Figure 1 This is a schematic diagram of the multimodal 3D semantic occupancy method of the present invention. This application provides a multimodal 3D semantic occupancy prediction method for outdoor scenes, applied in camera branch, radar branch, and multimodal fusion branch. The method includes:

[0076] Step 101 of generating 3D semantic occupancy labels: Based on multi-frame point cloud data with semantic information, coordinate projection and semantic coordinate classification methods are used to generate 3D semantic occupancy labels;

[0077] Feature extraction step 102: Collect multimodal sensing data in the field scene, extract features from the sensing data and overlay historical frames. The multimodal sensing data includes multiple frames of images from the camera and multiple frames of point cloud data from the lidar.

[0078] Knowledge transfer step 103: The camera branch and radar branch are trained using the three-dimensional semantic occupancy labels. The geometric knowledge in the extracted point cloud features is transferred to the image branch network using the knowledge distillation function. The extracted image and point cloud features are dynamically fused to output the fused features.

[0079] Prediction result generation step 104: The occupancy prediction head constructed using 3D convolution transforms the fused features to generate a predicted value of 3D semantic occupancy, thereby realizing the automatic reconstruction of the field scene.

[0080] In a specific embodiment of the present invention, the above-mentioned three-dimensional semantic occupancy tag generation step 101 includes:

[0081] Point cloud stitching steps: In the field scene to be detected, each frame of point cloud with semantic information is projected onto the world coordinate system using absolute pose information. The projected multi-frame point clouds are stitched together to generate a dense point cloud of the current field scene with semantic information.

[0082] Voxel generation steps: Based on the semantic category labels attached to the point cloud data, all point clouds are divided into ground class and non-ground class. Point clouds belonging to the ground class are reconstructed using coarse-grained Poisson surface, and point clouds belonging to the non-ground class are reconstructed using fine-grained Poisson surface. The reconstructed mesh structure is then voxelized to generate voxels.

[0083] Label generation steps: Use the K-nearest neighbor algorithm to classify the center point of each voxel, and assign the center point label to the corresponding voxel according to the semantic category of the nearest point to generate a three-dimensional semantic occupancy label.

[0084] In a specific embodiment of the present invention, the feature extraction step 102 includes:

[0085] In the camera branch, a deep convolutional neural network is used to extract multi-scale two-dimensional features from the image, and the LSS method is used to project the two-dimensional features into a three-dimensional voxel space to obtain image voxel features.

[0086] In the radar branch, voxel networks are used to extract features from point cloud data accumulated from multiple frames, and three-dimensional voxel space is used for representation to obtain point cloud voxel features.

[0087] like Figure 2 As shown, in the camera branch, a deep convolutional neural network (ResNet, FPN) is used to extract multi-scale two-dimensional features from the image, and the LSS method is used to project the two-dimensional features into a three-dimensional voxel space. In the LiDAR branch, a voxel network (VoxelNet) is used to extract features from the accumulated point cloud data of multiple frames, and the features are represented using a three-dimensional voxel space.

[0088] In a specific embodiment of the present invention, the knowledge transfer step 103 includes:

[0089] Adaptive fusion steps: When training the camera branch, a soft-supervised distillation function is used to efficiently transfer the rich geometric knowledge in the LiDAR branch to the camera branch;

[0090] Multimodal fusion steps: The point cloud voxel features and image voxel features are dynamically fused using an adaptive fusion function to output the fused features.

[0091] In a specific embodiment of the present invention, when training the camera branch, the knowledge distillation function designed in this invention is used to transfer richer geometric knowledge from the point cloud data to the camera branch model. During multimodal fusion, different modalities are dynamically fused based on the input.

[0092] After obtaining the features in the three-dimensional voxel space, the final model prediction results are output using the designed three-dimensional occupancy prediction head.

[0093] like Figure 2 As shown, Figure 2 This is a flowchart of a specific embodiment of the 3D occupancy prediction method of the present invention. The method consists of a camera branch, a LiDAR branch, and multimodal fusion. For the LiDAR branch, since the point cloud of a single frame is too sparse, the point cloud of the current moment and the previous 9 consecutive frames are used as input. A voxel encoder is used to voxelize the input point cloud and extract features. The voxel features of multiple historical frames are then spatially and temporally fused: given the input... in The voxel features of the point cloud at time ti are represented by the absolute pose at the corresponding time. t-i The formula for aligning voxel features across multiple frames is as follows:

[0094]

[0095] The above formula uses pose information to align features from historical frames with features in the current vehicle coordinate system. For features... The coordinates of each voxel in the data are obtained using... The coordinates of the nearest voxel are used as the new voxel coordinates. A voxel encoder consisting of 3D convolutional layers, batch normalization (BatchNorm) layers, and ReLU layers is then used to extract the final voxel features. For the camera branch, the overall structure is similar to the LiDAR branch, except that a combination of ResNet and FPN is used to extract image features, and the LSS algorithm is used to project these features onto a 3D voxel space. Subsequently, a space-time fusion and feature extractor with the same structure are used to obtain the final voxel features. The goal of multimodal fusion is to fully utilize information from all modalities; therefore, this invention uses an adaptive fusion module to dynamically fuse the image voxel features F. I and point cloud voxel features F L :

[0096] F M =σ(W)×F I +(1-σ(W))×F L ,

[0097] In the above formula, W is a trainable parameter output by 3D convolution, σ is the sigmoid function, and × represents element-wise multiplication. When training the camera branch, this invention proposes a soft-supervised distillation function to efficiently transfer the rich geometric knowledge from the LiDAR branch to the camera branch. The cross-modal distillation function is as follows:

[0098]

[0099] Among them, f I (x,y,z) is the image voxel feature F IFeatures at coordinates (x, y, z), f L (x,y,z) is a point voxel feature F L The feature at coordinates (x, y, z) is defined such that if the voxel at (x, y, z) in the label is already occupied, then parameter M... x,y,z =1, otherwise M x,y,z =0, H represents the length of the voxel feature space, W represents the width of the voxel feature space, and D represents the height of the voxel feature space.

[0100] Example 2

[0101] like Figure 3 As shown, Figure 3 This is a schematic diagram of the three-dimensional semantic occupancy label construction method of the present invention. This application provides a method for constructing three-dimensional semantic occupancy labels for outdoor scenes, applied to the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described above. The label construction method includes:

[0102] Point cloud stitching step 201: In the field scene to be detected, the point cloud with semantic information of each frame is projected onto the world coordinate system using absolute pose information, and the projected point cloud is stitched together to generate a dense point cloud of the current field scene with semantic information.

[0103] Voxel generation step 202: Based on the semantic category labels attached to the point cloud data, all point clouds are divided into ground class and non-ground class. Point clouds belonging to the ground class are reconstructed using coarse-grained Poisson surface, and point clouds belonging to the non-ground class are reconstructed using fine-grained Poisson surface. The reconstructed mesh structure is then voxelized to generate voxels.

[0104] Label generation step 203: Use the K-nearest neighbor algorithm to classify the center point of each voxel, and assign the center point label to the corresponding voxel according to the semantic category of the nearest point to generate a three-dimensional semantic occupancy label.

[0105] like Figure 4 As shown, Figure 4 This is a schematic diagram of the three-dimensional semantic occupancy tag generation framework of the present invention. The embodiments of the present invention provide a tag generation framework from coarse to fine, including the following steps:

[0106] (1) Input multi-frame point cloud data with semantic information into the system, first project the point cloud of each frame into the world coordinate system, and superimpose the point clouds of multiple frames to generate the point cloud representation of the current scene.

[0107] (2) Classify the point cloud obtained in step (1), and then use different grain sizes to reconstruct the surface to obtain the reconstructed mesh output. Then, perform voxelization on the reconstructed mesh.

[0108] (3) Classify each voxel obtained in step (2).

[0109] Furthermore, in step (1), in the scene to be detected, the point cloud of each frame with semantic information is projected onto the world coordinate system using absolute pose information. The projected point clouds are then stitched together to generate a dense point cloud of the current scene with semantic information.

[0110] The classification method in step (2) is as follows: Based on the semantic category labels attached to the point cloud data, all point clouds are divided into ground-type (grass, cement, etc.) and non-ground-type (trees, shrubs, etc.). Point clouds belonging to the ground-type are reconstructed using a coarse-grained Poisson surface, while point clouds belonging to the non-ground-type are reconstructed using a fine-grained Poisson surface. The reconstructed mesh output is obtained, and the reconstructed mesh is then voxelized.

[0111] In step (3), the classification method involves using the K-nearest neighbor algorithm to classify the center point of each voxel, and assigning a label to the corresponding voxel based on the semantic category of the nearest point. This ensures that each voxel accurately reflects its corresponding semantic category, thereby generating dense three-dimensional semantic occupancy labels.

[0112] Example 3

[0113] like Figure 5 As shown in the figure, this application provides a multimodal 3D semantic occupancy prediction system for outdoor scenes. It employs the multimodal 3D semantic occupancy prediction method for outdoor scenes as described above. The system is applied to camera branches, radar branches, and multimodal fusion branches. The system includes:

[0114] 3D semantic occupancy label generation module 301: Based on multi-frame point cloud data with semantic information, it generates 3D semantic occupancy labels using coordinate projection and semantic coordinate classification methods;

[0115] Feature extraction module 302: Collects multimodal sensing data in the field scene, performs feature extraction of the sensing data and overlays historical frames, wherein the multimodal sensing data includes multi-frame images from the camera and multi-frame point cloud data from the lidar;

[0116] Knowledge transfer module 303: The camera branch is trained using 3D semantic occupancy labels. The geometric knowledge in the extracted point cloud features is transferred to the image branch network using the knowledge distillation function. The extracted image and point cloud features are dynamically fused to output the fused features.

[0117] Prediction Result Generation Module 304: The occupancy prediction head constructed using 3D convolution transforms the fused features to generate a predicted value of 3D semantic occupancy, thereby realizing the automatic reconstruction of the field scene.

[0118] Example 4

[0119] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes.

[0120] Example 5

[0121] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described above.

[0122] In addition, combined Figure 1 The multimodal 3D semantic occupancy prediction method for outdoor scenes described in this application embodiment can be implemented by electronic devices, such as computer devices. Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application.

[0123] In some embodiments, the computer device may further include a communication interface 83 and a bus 80. For example, Figure 6 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0124] Specifically, the processor 81 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0125] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0126] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any of the multimodal three-dimensional semantic occupancy prediction methods for field scenes in the above embodiments.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for multi-modal three-dimensional semantic occupancy prediction for outdoor scenes, the method comprising: The method is applied to a camera branch, a radar branch and a multi-modal fusion branch, and the method comprises: a three-dimensional semantic occupancy label generation step: based on multi-frame point cloud data with semantic information, a coordinate projection and semantic coordinate classification method is used to generate a three-dimensional semantic occupancy label; a feature extraction step: multi-modal sensor data in an outdoor scene is collected, and feature extraction and historical frame superposition of the sensor data are performed, wherein the multi-modal sensor data comprises multi-frame images from a camera and multi-frame point cloud data from a laser radar; a knowledge transfer step: the three-dimensional semantic occupancy label is used to train the camera branch and the radar branch, a knowledge distillation function is used to transfer geometric knowledge in the extracted point cloud features to an image branch network, the features of the extracted images and the point cloud are dynamically fused, and fused features are output; a prediction result generation step: the fused features are converted using an occupancy prediction head constructed by a three-dimensional convolution to generate a prediction value of three-dimensional semantic occupancy, and automatic reconstruction of an outdoor scene is realized; the three-dimensional semantic occupancy label generation step comprises: a point cloud splicing step: in a to-be-detected outdoor scene, each frame of point cloud with semantic information is projected into a world coordinate system using absolute pose information, and the projected multi-frame point cloud is spliced to generate a dense point cloud of the current outdoor scene with semantic information; a voxel generation step: according to the semantic category label attached to the point cloud data, all point clouds are divided into ground and non-ground, a coarse-grained Poisson surface reconstruction is used for the point cloud belonging to the ground, a fine-grained Poisson surface reconstruction is used for the point cloud belonging to the non-ground, and voxelization processing is performed on the reconstructed grid structure to generate voxels; a label generation step: a K-nearest neighbor algorithm is used to classify the center points of each voxel, the center point label is assigned to the corresponding voxel according to the semantic category of the nearest point, and the three-dimensional semantic occupancy label is generated.

2. The method of claim 1, wherein, the feature extraction step comprises: in the camera branch, a deep convolutional neural network is used to extract multi-scale two-dimensional features from the image, and an LSS method is used to project the two-dimensional features into a three-dimensional voxel space to obtain image voxel features; in the radar branch, a voxel network is used to extract features from the multi-frame accumulated point cloud data, and a three-dimensional voxel space is used for representation to obtain point cloud voxel features.

3. The method of claim 1, wherein, the knowledge transfer step comprises: an adaptive fusion step: when training the camera branch, a soft-supervised distillation function is used to efficiently transfer the rich geometric knowledge in the laser radar branch to the camera branch; a multi-modal fusion step: the point cloud voxel features and the image voxel features are dynamically fused using an adaptive fusion function to output fused features.

4. The method of claim 3, wherein, The soft supervised distillation function As follows: ， wherein is an image voxel feature coordinate at the feature, is a point voxel feature coordinate at the feature, if the voxel at is occupied in the label, the parameter , otherwise H represents the length of the voxel feature space, W represents the width of the voxel feature space, and D represents the height of the voxel feature space.

5. The method of claim 3, wherein, The adaptive fusion function Is: ， wherein, is an image voxel feature, is a point cloud voxel feature is a trainable parameter output by a 3D convolution, is a sigmoid function, represents an element-wise product.

6. A method for constructing a three-dimensional semantic occupancy label for a wild scene, applied to the multi-modal three-dimensional semantic occupancy prediction method for a wild scene in any one of claims 1-5, characterized in that, the label construction method comprises: a point cloud splicing step: in a to-be-detected outdoor scene, each frame of point cloud with semantic information is projected into a world coordinate system using absolute pose information, and the projected multi-frame point cloud is spliced to generate a dense point cloud of the current outdoor scene with semantic information; The voxel generation step: according to the semantic category label attached to the point cloud data, all point clouds are divided into ground and non-ground categories, the point clouds belonging to the ground category are reconstructed using a coarse-grained Poisson surface, the point clouds belonging to the non-ground category are reconstructed using a fine-grained Poisson surface, and the reconstructed grid structure is voxelized to generate voxels; The label generation step: using the K-nearest neighbor algorithm to classify the center point of each voxel, according to the semantic category of the nearest point, the center point label is assigned to the corresponding voxel to generate the three-dimensional semantic occupancy label.

7. A multi-modal three-dimensional semantic occupancy prediction system for outdoor scenes, employing the multi-modal three-dimensional semantic occupancy prediction method for outdoor scenes according to any one of claims 1-5, characterized in that, The system is applied to a camera branch, a radar branch and a multi-modal fusion branch, and the system comprises: A three-dimensional semantic occupancy label generation module: based on multi-frame point cloud data with semantic information, a coordinate projection and semantic coordinate classification method is used to generate a three-dimensional semantic occupancy label; A feature extraction module: collects multi-modal sensing data in an outdoor scene, and performs feature extraction and historical frame superposition on the sensing data, wherein the multi-modal sensing data comprises multi-frame images from a camera and multi-frame point cloud data from a laser radar; A knowledge transfer module: the camera branch is trained using the three-dimensional semantic occupancy label, the geometric knowledge in the extracted point cloud features is transferred to the image branch network using a knowledge distillation function, the features of the extracted images and the point clouds are dynamically fused, and a fusion feature is output; A prediction result generation module: a three-dimensional convolution is used to construct an occupancy prediction head to convert the fusion feature to generate a three-dimensional semantic occupancy prediction value, and realize automatic reconstruction of an outdoor scene; The three-dimensional semantic occupancy label generation module comprises: A point cloud stitching module: in a to-be-detected outdoor scene, each frame of point cloud with semantic information is projected into a world coordinate system using absolute pose information, and the projected multi-frame point clouds are stitched to generate a current outdoor scene dense point cloud with semantic information; A voxel generation module: according to the semantic category label attached to the point cloud data, all point clouds are divided into ground and non-ground categories, the point clouds belonging to the ground category are reconstructed using a coarse-grained Poisson surface, the point clouds belonging to the non-ground category are reconstructed using a fine-grained Poisson surface, and the reconstructed grid structure is voxelized to generate voxels; A label generation module: using the K-nearest neighbor algorithm to classify the center point of each voxel, according to the semantic category of the nearest point, the center point label is assigned to the corresponding voxel to generate the three-dimensional semantic occupancy label.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the multi-modal three-dimensional semantic occupancy prediction method for an outdoor scene in any one of claims 1-5.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the multi-modal three-dimensional semantic occupancy prediction method for an outdoor scene in any one of claims 1-5.

Images