Multi-modal three-dimensional semantic occupancy prediction method and system for field scene

Abstract

The invention discloses a multi-modal three-dimensional semantic occupancy prediction method for a field scene, and the method comprises the steps: generating a three-dimensional semantic occupancy tag based on multi-frame point cloud data with semantic information; the method comprises the following steps: acquiring multi-modal sensing data in a field scene, performing feature extraction and historical frame superposition on the sensing data, transmitting geometric knowledge in extracted point cloud features to an image branch network by using a knowledge distillation function, performing dynamic fusion on the extracted image and point cloud features, and outputting fusion features; and converting the fusion features by using an occupation prediction head constructed by three-dimensional convolution, generating a predicted value of three-dimensional semantic occupation, and realizing automatic reconstruction of a field scene. According to the method, the generation of the three-dimensional semantic occupancy label is realized, the error of scene reconstruction is remarkably reduced, and the vacancy of a three-dimensional semantic occupancy prediction method in a field scene is filled.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving environment perception, and in particular to a multimodal three-dimensional semantic occupancy prediction method and system for outdoor scenes. Background Art

[0002] The goal of the 3D semantic occupancy prediction task is to predict the semantic label of each occupied voxel. In recent years, 3D semantic occupancy prediction based on deep learning has been widely used in the field of autonomous driving. The 3D semantic occupancy prediction task divides the space within the perception range into multiple voxels, and for each voxel, predicts its occupancy state and semantic information. The 3D semantic occupancy prediction method based on deep learning can be quickly inferred in the GPU (graphics processing unit) and has a certain real-time performance. However, the existing occupancy prediction methods mainly focus on the road environment, and there is currently no research on 3D semantic occupancy perception for off-road environments. Since the ground in the off-road scene is uneven and contains a large number of obstacles with height information (trees, bushes, etc.), the 3D representation can provide more comprehensive information compared to the 2D representation that loses the height information.

[0003] The initial goal of knowledge distillation is to compress the model by transferring the knowledge of the teacher model to the student model. Due to its effectiveness, knowledge distillation has been applied to the task of 3D semantic occupancy prediction. However, all existing methods perform knowledge distillation in the same modality and cannot achieve cross-modal knowledge distillation.

[0004] In addition, current off-road perception mainly focuses on the task of traversable area segmentation, and relatively little research is done on reconstructing off-road environments. Since scene reconstruction can provide more comprehensive perception information, it is particularly important to introduce the 3D semantic occupancy prediction task into off-road environments.

[0005] However, most of the current 3D semantic occupancy prediction labels belong to road environments, and there are no corresponding labels for wild environments. After research, it was found that the current method used to generate 3D semantic occupancy prediction labels in road environments will produce large errors when applied to wild scenes.

[0006] Therefore, in order to make full use of the rich geometric information in the lidar data, it is urgent to propose a cross-modal knowledge distillation function to transfer the geometric information in the lidar branch to the camera branch more fully, so as to improve the performance of the camera branch when working alone; at the same time, there is an urgent need for a three-dimensional semantic occupancy prediction method with small error that can be applied to field scenes. Summary of the invention

[0007] In order to solve the technical problem that large errors may occur in the above-mentioned existing wild scenes, a multimodal three-dimensional semantic occupancy prediction method and system for wild scenes are proposed.

[0008] In a first aspect, an embodiment of the present application provides a multimodal three-dimensional semantic occupancy prediction method for outdoor scenes, which is applied to a camera branch, a radar branch, and a multimodal fusion branch. The method includes:

[0009] Three-dimensional semantic occupancy label generation step: based on multi-frame point cloud data with semantic information, three-dimensional semantic occupancy labels are generated by using coordinate projection and semantic coordinate classification methods;

[0010] Feature extraction step: Collect multi-modal sensor data in the field scene, perform feature extraction and historical frame superposition of the sensor data, where the multi-modal sensor data includes multi-frame images from the camera and multi-frame point cloud data from the lidar;

[0011] Knowledge transfer step: using the three-dimensional semantic occupancy label to train the camera branch and the radar branch, using the knowledge distillation function to transfer the geometric knowledge in the extracted point cloud features to the image branch network, dynamically fusing the extracted image and point cloud features, and outputting the fused features;

[0012] Prediction result generation steps: Use the occupancy prediction head constructed using three-dimensional convolution to transform the fused features, generate three-dimensional semantic occupancy prediction values, and realize automatic reconstruction of field scenes.

[0013] In a specific embodiment of the present invention, the feature extraction step includes:

[0014] In the camera branch, a deep convolutional neural network is used to extract multi-scale two-dimensional features from the image, and the LSS method is used to project the two-dimensional features into the three-dimensional voxel space to obtain the image voxel features;

[0015] In the radar branch, a voxel network is used to extract features from point cloud data accumulated from multiple frames, and three-dimensional voxel space is used to represent them to obtain point cloud voxel features.

[0016] In a specific embodiment of the present invention, the above-mentioned knowledge transfer step includes:

[0017] Adaptive fusion step: When training the camera branch, a soft-supervised distillation function is used to efficiently transfer the rich geometric knowledge in the lidar branch to the camera branch;

[0018] Multimodal fusion steps: dynamically fuse the point cloud voxel features and image voxel features using an adaptive fusion function and output the fused features.

[0019] In a specific embodiment of the present invention, the above-mentioned soft-supervised distillation function L distill as follows:

[0020]

[0021] Among them, fI (x,y,z) is the image voxel feature F I The feature at coordinate (x,y,z), f L (x,y,z) is the point voxel feature F L The feature at the coordinate (x, y, z). If the voxel at (x, y, z) in the label is occupied, the parameter M x,y,z =1, otherwise M x,y,z =0, H represents the length of the voxel feature space, W represents the width of the voxel feature space, and D represents the height of the voxel feature space.

[0022] In a specific embodiment of the present invention, the above-mentioned adaptive fusion function F M for:

[0023] F M =σ(W)×F I +(1-σ(W))×F L ,

[0024] Among them, F I is the image voxel feature, F L is the point cloud voxel feature, W is the trainable parameter output by the 3D convolution, σ is the sigmoid function, and × represents the element-by-element product.

[0025] In a specific embodiment of the present invention, the above three-dimensional semantic occupancy label generation step includes:

[0026] Point cloud stitching step: In the field scene to be detected, use the absolute pose information to project each frame of point cloud with semantic information into the world coordinate system, stitch the projected multi-frame point clouds to generate a dense point cloud of the current field scene with semantic information;

[0027] Voxel generation step: According to the semantic category labels attached to the point cloud data, all point clouds are divided into ground and non-ground categories. The point clouds belonging to the ground category are reconstructed using coarse-grained Poisson surface reconstruction, and the point clouds belonging to the non-ground category are reconstructed using fine-grained Poisson surface reconstruction. The reconstructed grid structure is voxelized to generate voxels.

[0028] Label generation step: Use the K-nearest neighbor algorithm to classify the center point of each voxel, assign the center point label to the corresponding voxel according to the semantic category of the nearest point, and generate a three-dimensional semantic occupancy label.

[0029] In a second aspect, an embodiment of the present application provides a method for constructing a three-dimensional semantic occupancy label for a wild scene, which is applied to the multimodal three-dimensional semantic occupancy prediction method for a wild scene as described above, and the label construction method includes:

[0030] Point cloud stitching step: In the field scene to be detected, use the absolute pose information to project each frame of point cloud with semantic information into the world coordinate system, stitch the projected multi-frame point clouds to generate a dense point cloud of the current field scene with semantic information;

[0031] Voxel generation step: According to the semantic category labels attached to the point cloud data, all point clouds are divided into ground and non-ground categories. The point clouds belonging to the ground category are reconstructed using coarse-grained Poisson surface reconstruction, and the point clouds belonging to the non-ground category are reconstructed using fine-grained Poisson surface reconstruction. The reconstructed grid structure is voxelized to generate voxels.

[0032] Label generation step: Use the K-nearest neighbor algorithm to classify the center point of each voxel, assign the center point label to the corresponding voxel according to the semantic category of the nearest point, and generate a three-dimensional semantic occupancy label.

[0033] In a third aspect, an embodiment of the present application provides a multimodal three-dimensional semantic occupancy prediction system for outdoor scenes, which adopts the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described above, and the system is applied to a camera branch, a radar branch, and a multimodal fusion branch, and the system includes:

[0034] 3D semantic occupancy label generation module: Based on multi-frame point cloud data with semantic information, coordinate projection and semantic coordinate classification methods are used to generate 3D semantic occupancy labels;

[0035] Feature extraction module: collects multi-modal sensor data in field scenes, extracts features of sensor data and superimposes historical frames. Multi-modal sensor data includes multi-frame images from cameras and multi-frame point cloud data from lidar.

[0036] Knowledge transfer module: The camera branch is trained with 3D semantic occupancy labels, and the geometric knowledge in the extracted point cloud features is transferred to the image branch network using the knowledge distillation function. The extracted image and point cloud features are dynamically fused and the fused features are output.

[0037] Prediction result generation module: The occupancy prediction head constructed using three-dimensional convolution transforms the fused features to generate three-dimensional semantic occupancy prediction values ​​to achieve automatic reconstruction of field scenes.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes.

[0039] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described are implemented.

[0040] Compared with the related prior art, it has the following outstanding beneficial effects:

[0041] 1) The method of the present invention consists of a camera branch, a lidar branch and a multimodal fusion branch. It realizes the reconstruction of the surrounding environment by extracting features from camera images and laser point clouds, and is suitable for field scene understanding tasks in autonomous driving systems;

[0042] 2) The method of the present invention proposes a cross-modal knowledge distillation function to fully transfer the geometric information in the lidar branch to the camera branch to improve the performance of the camera branch when working alone;

[0043] 3) The method of the present invention proposes a coarse-to-fine label generation framework to optimize the generation of three-dimensional semantic occupancy labels, which significantly reduces the error of reconstructed scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 It is a schematic diagram of the multimodal three-dimensional semantic occupancy prediction method of the present invention;

[0046] Figure 2 A schematic diagram of a method for multimodal three-dimensional semantic occupancy prediction according to a specific embodiment of the present invention;

[0047] Figure 3 This is a flow chart of a method for generating multimodal three-dimensional semantic labels according to the present invention;

[0048] Figure 4 This is a flowchart of a method for generating multimodal three-dimensional semantic labels according to a specific embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a multimodal three-dimensional semantic occupancy prediction system according to an embodiment of the present invention;

[0050] Figure 6 Schematic diagram of computer hardware of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that the processor described in the present invention is the control center of the electronic device, which may be a processor or a general term for multiple processing elements. For example, it may be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0052] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0053] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0054] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0055] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0056] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0057] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0058] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0059] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0061] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0062] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0063] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0064] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0065] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0066] The present invention aims to propose a method and device for realizing multimodal three-dimensional semantic occupancy prediction in outdoor scenes using deep learning methods. The method consists of a camera branch, a lidar branch, and a multimodal fusion branch. By extracting features from camera images and laser point clouds, the surrounding environment can be reconstructed, which is suitable for outdoor scene understanding tasks in autonomous driving systems.

[0067] The present invention proposes a 3D semantic occupancy prediction method and device for off-road scenes, as well as a 3D semantic occupancy label construction method for outdoor scenes, aiming to fill the gap in 3D semantic occupancy prediction methods in outdoor scenes. The framework fuses multi-frame images and point cloud information into voxel space, and uses cross-modal distillation functions to transfer the geometric knowledge of the lidar branch to the camera branch, thereby improving the accuracy of 3D semantic occupancy prediction. At the same time, a coarse-to-fine label generation framework is proposed to optimize the generation of 3D semantic occupancy labels, significantly reducing the error of reconstructed scenes.

[0068] The embodiment of the present invention provides a method for predicting multimodal 3D semantic occupancy of outdoor scenes based on deep learning. The method combines camera and LiDAR data, and realizes high-precision 3D semantic occupancy prediction of outdoor scenes through multimodal fusion and cross-modal distillation. To cope with unsafe outdoor environments, the camera branch and the LiDAR branch can work independently, including the following steps:

[0069] (1) Obtain input data through multimodal sensors, including multiple frames of images from the camera and multiple frames of point cloud data from the lidar, and perform feature extraction and historical frame superposition respectively;

[0070] (2) Fusing the image obtained in step (1) with the point cloud features, and using the knowledge distillation function to transfer the geometric knowledge in the point cloud features to the image branch network when training the camera branch alone;

[0071] (3) The features obtained in step (2) are converted into three-dimensional semantic occupancy prediction values ​​using an occupancy prediction head constructed using three-dimensional convolution.

[0072] In step (1), the deep convolutional neural network is a pre-trained network model of ResNet-101 on ImageNet. In step (2), the multimodal fusion uses an adaptive fusion module to dynamically fuse image voxel features and point voxel features. In step (2), the distillation function is the weakly supervised knowledge distillation function proposed in the present invention.

[0073] The following is a detailed description with reference to specific embodiments:

[0074] Embodiment 1

[0075] like Figure 1 As shown, Figure 1 The flowchart of the multimodal three-dimensional semantic occupancy method of the present invention is as follows. The embodiment of the present application provides a multimodal three-dimensional semantic occupancy prediction method for outdoor scenes, which is applied to the camera branch, the radar branch and the multimodal fusion branch. The method includes:

[0076] 3D semantic occupancy label generation step 101: based on multi-frame point cloud data with semantic information, a 3D semantic occupancy label is generated by using a coordinate projection and semantic coordinate classification method;

[0077] Feature extraction step 102: collecting multi-modal sensor data in a field scene, performing feature extraction and historical frame superposition of the sensor data, wherein the multi-modal sensor data includes multiple frames of images from a camera and multiple frames of point cloud data from a laser radar;

[0078] Knowledge transfer step 103: using the three-dimensional semantic occupancy label to train the camera branch and the radar branch, using the knowledge distillation function to transfer the geometric knowledge in the extracted point cloud features to the image branch network, dynamically fusing the extracted image and point cloud features, and outputting the fused features;

[0079] Prediction result generation step 104: using the occupancy prediction head constructed by three-dimensional convolution to transform the fused features, generate a prediction value of three-dimensional semantic occupancy, and realize automatic reconstruction of the field scene.

[0080] In a specific embodiment of the present invention, the above-mentioned three-dimensional semantic occupancy label generation step 101 includes:

[0081] Point cloud stitching step: In the field scene to be detected, use the absolute pose information to project each frame of point cloud with semantic information into the world coordinate system, stitch the projected multi-frame point clouds to generate a dense point cloud of the current field scene with semantic information;

[0082] Voxel generation step: According to the semantic category labels attached to the point cloud data, all point clouds are divided into ground and non-ground categories. The point clouds belonging to the ground category are reconstructed using coarse-grained Poisson surface reconstruction, and the point clouds belonging to the non-ground category are reconstructed using fine-grained Poisson surface reconstruction. The reconstructed grid structure is voxelized to generate voxels.

[0083] Label generation step: Use the K-nearest neighbor algorithm to classify the center point of each voxel, assign the center point label to the corresponding voxel according to the semantic category of the nearest point, and generate a three-dimensional semantic occupancy label.

[0084] In a specific embodiment of the present invention, the feature extraction step 102 includes:

[0085] In the camera branch, a deep convolutional neural network is used to extract multi-scale two-dimensional features from the image, and the LSS method is used to project the two-dimensional features into the three-dimensional voxel space to obtain the image voxel features;

[0086] In the radar branch, a voxel network is used to extract features from point cloud data accumulated from multiple frames, and three-dimensional voxel space is used to represent them to obtain point cloud voxel features.

[0087] like Figure 2 As shown in the figure, in the camera branch, a deep convolutional neural network (ResNet, FPN) is used to extract multi-scale two-dimensional features from the image, and the LSS method is used to project the two-dimensional features into the three-dimensional voxel space. In the lidar branch, a voxel network (VoxelNet) is used to extract features from the point cloud data accumulated from multiple frames, and the three-dimensional voxel space is used to represent them;

[0088] In a specific embodiment of the present invention, the knowledge transfer step 103 includes:

[0089] Adaptive fusion step: When training the camera branch, a soft-supervised distillation function is used to efficiently transfer the rich geometric knowledge in the lidar branch to the camera branch;

[0090] Multimodal fusion steps: dynamically fuse the point cloud voxel features and image voxel features using an adaptive fusion function and output the fused features.

[0091] In a specific embodiment of the present invention, when training the camera branch, the knowledge distillation function designed in the present invention is used to transfer the richer geometric knowledge in the point cloud data to the camera branch model. When performing multimodal fusion, the fusion between different modalities is dynamically performed according to the input;

[0092] After obtaining the features in the 3D voxel space, the designed 3D occupancy prediction head is used to output the prediction results of the final model.

[0093] like Figure 2 As shown, Figure 2 This is a flowchart of a three-dimensional occupancy prediction method according to a specific embodiment of the present invention. The method consists of a camera branch, a lidar branch, and multimodal fusion. For the lidar branch, since a single-frame point cloud is too sparse, the current moment and the previous nine consecutive frames of point cloud are used as input, and a voxel encoder is used to voxelize the point cloud input and extract features. The voxel features of multiple historical frames are spatially-temporally fused: Given an input in Represents the point cloud voxel features at time ti, using the absolute pose Pose at the corresponding time t-i For the alignment of multi-frame voxel features, the formula is as follows:

[0094]

[0095] The above formula uses the pose information to align the features in the historical frame with the features in the current vehicle coordinate system. The coordinates of each voxel in , using The voxel coordinates with the closest distance in the image are taken as the new voxel coordinates. Then, a voxel encoder composed of a three-dimensional convolutional layer, a batch normalization (BatchNorm) layer, and a ReLU layer is used to extract the final voxel features. For the camera branch, the overall structure is similar to that of the lidar branch, except that a combination of ResNet and FPN is used to extract image features, and the LSS algorithm is used to project the image features into a three-dimensional voxel space. Subsequently, the spatial-temporal fusion and feature extractor with the same structure are used to obtain the final voxel features. The goal of multimodal fusion is to make full use of the information in all modalities. Therefore, the present invention uses an adaptive fusion module to dynamically fuse the image voxel features F I and point cloud voxel features F L :

[0096] F M =σ(W)×F I +(1-σ(W))×F L ,

[0097] In the above formula, W is a trainable parameter output by 3D convolution, σ is a sigmoid function, and × represents the element-by-element product. When training the camera branch, the present invention proposes a soft-supervised distillation function to efficiently transfer the rich geometric knowledge in the lidar branch to the camera branch. The cross-modal distillation function is as follows:

[0098]

[0099] Among them, f I (x,y,z) is the image voxel feature F IThe feature at coordinate (x,y,z), f L (x,y,z) is the point voxel feature F L The feature at the coordinate (x, y, z). If the voxel at (x, y, z) in the label is occupied, the parameter M x,y,z =1, otherwise M x,y,z =0, H represents the length of the voxel feature space, W represents the width of the voxel feature space, and D represents the height of the voxel feature space.

[0100] Embodiment 2

[0101] like Figure 3 As shown, Figure 3 The flowchart of the method for constructing a three-dimensional semantic occupancy label of the present invention is as follows. The embodiment of the present application provides a method for constructing a three-dimensional semantic occupancy label for a wild scene, which is applied to the multimodal three-dimensional semantic occupancy prediction method for a wild scene as described above. The label construction method includes:

[0102] Point cloud stitching step 201: In the field scene to be detected, each frame of point cloud with semantic information is projected into the world coordinate system using the absolute pose information, and multiple frames of point cloud after the projection are stitched to generate a dense point cloud of the current field scene with semantic information;

[0103] Voxel generation step 202: according to the semantic category labels attached to the point cloud data, all point clouds are divided into ground and non-ground categories, the point clouds belonging to the ground category are reconstructed using a coarse-grained Poisson surface, and the point clouds belonging to the non-ground category are reconstructed using a fine-grained Poisson surface, and the reconstructed grid structure is voxelized to generate voxels;

[0104] Label generation step 203: Use the K-nearest neighbor algorithm to classify the center point of each voxel, assign the center point label to the corresponding voxel according to the semantic category of the nearest point, and generate a three-dimensional semantic occupancy label.

[0105] like Figure 4 As shown, Figure 4 Schematic diagram of a three-dimensional semantic occupancy label generation framework of the present invention. The embodiment of the present invention provides a coarse-to-fine label generation framework, including the following steps:

[0106] (1) Input multiple frames of point cloud data with semantic information into the system, first project the point cloud of each frame into the world coordinate system, and then superimpose the point clouds of multiple frames to generate a point cloud representation of the current scene;

[0107] (2) classifying the point cloud obtained in step (1), reconstructing the surface using different granularities, obtaining a reconstructed mesh output, and voxelizing the reconstructed mesh;

[0108] (3) Classify each voxel obtained in step (2).

[0109] Furthermore, in step (1), in the scene to be detected, each frame of the point cloud with semantic information is projected into the world coordinate system using the absolute pose information, and the projected multiple frames of point clouds are spliced ​​to generate a dense point cloud of the current scene with semantic information.

[0110] Classification method in step (2): According to the semantic category labels attached to the point cloud data, all point clouds are divided into ground classes (grass, cement, etc.) and non-ground classes (trees, shrubs, etc.). For point clouds belonging to the ground class, a coarse-grained Poisson surface reconstruction is used, and for point clouds belonging to the non-ground class, a fine-grained Poisson surface reconstruction is used. The reconstructed mesh output is obtained and the reconstructed mesh is voxelized;

[0111] In step (3), the classification method is: using the K-nearest neighbor algorithm to classify the center point of each voxel, and assigning its label to the corresponding voxel according to the semantic category of the nearest point, ensuring that each voxel can accurately reflect its corresponding semantic category, thereby generating a dense three-dimensional semantic occupancy label.

[0112] Embodiment 3

[0113] like Figure 5 As shown, the embodiment of the present application provides a multimodal three-dimensional semantic occupancy prediction system for outdoor scenes, adopting the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described, the system is applied to the camera branch, the radar branch and the multimodal fusion branch, and the system includes:

[0114] 3D semantic occupancy label generation module 301: based on multi-frame point cloud data with semantic information, using coordinate projection and semantic coordinate classification method, generates 3D semantic occupancy labels;

[0115] Feature extraction module 302: collects multi-modal sensor data in outdoor scenes, performs feature extraction and historical frame superposition of the sensor data, wherein the multi-modal sensor data includes multi-frame images from a camera and multi-frame point cloud data from a laser radar;

[0116] Knowledge transfer module 303: using 3D semantic occupancy labels to train the camera branch, using the knowledge distillation function to transfer the geometric knowledge in the extracted point cloud features to the image branch network, dynamically fusing the extracted image and point cloud features, and outputting the fused features;

[0117] Prediction result generation module 304: uses the occupancy prediction head constructed by three-dimensional convolution to transform the fused features, generate the prediction value of three-dimensional semantic occupancy, and realize the automatic reconstruction of the field scene.

[0118] Embodiment 4

[0119] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes.

[0120] Embodiment 5

[0121] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes are implemented.

[0122] In addition, combined Figure 1 The multimodal three-dimensional semantic occupancy prediction method for outdoor scenes described in the embodiment of the present application can be implemented by an electronic device, such as a computer device. Figure 6 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0123] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 6 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0124] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0125] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0126] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any one of the multimodal three-dimensional semantic occupancy prediction methods for outdoor scenes in the above embodiments.

[0127] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multimodal three-dimensional semantic occupancy prediction method for outdoor scenes, characterized in that: Applied to the camera branch, radar branch and multi-modal fusion branch, the method includes: Three-dimensional semantic occupancy label generation step: based on multi-frame point cloud data with semantic information, three-dimensional semantic occupancy labels are generated by using coordinate projection and semantic coordinate classification methods; Feature extraction step: collecting multi-modal sensor data in a field scene, performing feature extraction and historical frame superposition of the sensor data, wherein the multi-modal sensor data includes multi-frame images from a camera and multi-frame point cloud data from a laser radar; Knowledge transfer step: using the three-dimensional semantic occupancy label to train the camera branch and the radar branch, using the knowledge distillation function to transfer the geometric knowledge in the extracted point cloud features to the image branch network, dynamically fusing the extracted features of the image and the point cloud, and outputting the fused features; Prediction result generation step: using the occupancy prediction head constructed by three-dimensional convolution to transform the fused features, generate a prediction value of three-dimensional semantic occupancy, and realize automatic reconstruction of the field scene.

2. The multimodal three-dimensional semantic occupancy prediction method for outdoor scenes according to claim 1, characterized in that: The feature extraction step comprises: In the camera branch, a deep convolutional neural network is used to extract multi-scale two-dimensional features from the image, and the two-dimensional features are projected into a three-dimensional voxel space using an LSS method to obtain image voxel features; In the radar branch, a voxel network is used to extract features from point cloud data accumulated from multiple frames, and three-dimensional voxel space is used to represent the features to obtain point cloud voxel features.

3. The multimodal three-dimensional semantic occupancy prediction method for outdoor scenes according to claim 1, characterized in that: The knowledge transfer step includes: Adaptive fusion step: When training the camera branch, a soft-supervised distillation function is used to efficiently transfer the rich geometric knowledge in the lidar branch to the camera branch; Multimodal fusion step: dynamically fuse the point cloud voxel features and the image voxel features using an adaptive fusion function to output fusion features.

4. The multimodal three-dimensional semantic occupancy prediction method for outdoor scenes according to claim 3, characterized in that: The soft-supervised distillation function L distill as follows: Among them, f I (x,y,z) is the image voxel feature F I The feature at coordinate (x,y,z), f L (x,y,z) is the point voxel feature F L The feature at the coordinate (x, y, z). If the voxel at (x, y, z) in the label is occupied, the parameter M x,y,z =1, otherwise M x,y,z =0, H represents the length of the voxel feature space, W represents the width of the voxel feature space, and D represents the height of the voxel feature space.

5. The multimodal three-dimensional semantic occupancy prediction method for outdoor scenes according to claim 3, characterized in that: The adaptive fusion function F M for: F M =σ(W)×F I +(1-σ(W))×F L , Among them, F I is the image voxel feature, F L is the point cloud voxel feature, W is the trainable parameter output by the 3D convolution, σ is the sigmoid function, and × represents the element-by-element product.

6. The multimodal three-dimensional semantic occupancy prediction method for outdoor scenes according to claim 1, characterized in that: The step of generating a three-dimensional semantic occupancy label comprises: Point cloud stitching step: In the field scene to be detected, use the absolute pose information to project each frame of point cloud with semantic information into the world coordinate system, stitch the projected multi-frame point clouds to generate a dense point cloud of the current field scene with semantic information; Voxel generation step: according to the semantic category labels attached to the point cloud data, all point clouds are divided into ground class and non-ground class, the point clouds belonging to the ground class are reconstructed using coarse-grained Poisson surface, the point clouds belonging to the non-ground class are reconstructed using fine-grained Poisson surface, and the reconstructed grid structure is voxelized to generate voxels; Label generation step: Use the K-nearest neighbor algorithm to classify the center point of each voxel, assign the center point label to the corresponding voxel according to the semantic category of the nearest point, and generate the three-dimensional semantic occupancy label.

7. A method for constructing three-dimensional semantic occupancy labels for outdoor scenes, applied to the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as claimed in any one of claims 1 to 6, characterized in that: The label construction method comprises: Point cloud stitching step: In the field scene to be detected, use the absolute pose information to project each frame of point cloud with semantic information into the world coordinate system, stitch the projected multi-frame point clouds to generate a dense point cloud of the current field scene with semantic information; Voxel generation step: according to the semantic category labels attached to the point cloud data, all point clouds are divided into ground class and non-ground class, the point clouds belonging to the ground class are reconstructed using coarse-grained Poisson surface, the point clouds belonging to the non-ground class are reconstructed using fine-grained Poisson surface, and the reconstructed grid structure is voxelized to generate voxels; Label generation step: Use the K-nearest neighbor algorithm to classify the center point of each voxel, assign the center point label to the corresponding voxel according to the semantic category of the nearest point, and generate the three-dimensional semantic occupancy label.

8. A multimodal three-dimensional semantic occupancy prediction system for outdoor scenes, using the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as claimed in any one of claims 1 to 6, characterized in that: The system is applied to the camera branch, the radar branch and the multi-modal fusion branch, and the system includes: 3D semantic occupancy label generation module: Based on multi-frame point cloud data with semantic information, coordinate projection and semantic coordinate classification methods are used to generate 3D semantic occupancy labels; Feature extraction module: collects multi-modal sensor data in field scenes, performs feature extraction and historical frame superposition of the sensor data, wherein the multi-modal sensor data includes multi-frame images from a camera and multi-frame point cloud data from a lidar; Knowledge transfer module: using the three-dimensional semantic occupancy label to train the camera branch, using the knowledge distillation function to transfer the geometric knowledge in the extracted point cloud features to the image branch network, dynamically fusing the extracted image and point cloud features, and outputting fused features; Prediction result generation module: The occupancy prediction head constructed using three-dimensional convolution is used to transform the fused features, generate a prediction value of three-dimensional semantic occupancy, and realize automatic reconstruction of the field scene.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes described in any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the multimodal three-dimensional semantic occupancy prediction method for outdoor scenes as described in any one of claims 1 to 6 are implemented.

Images