3D semantic scene completion method

Abstract

The invention relates to a 3D semantic scene completion method. The method comprises the following steps: step 1, acquiring an RGB image of a 3D scene by using a binocular camera; 2, constructing a voxel occupancy prediction network, and inputting the collected image into the voxel occupancy prediction network to obtain a voxel grid; 3, constructing an attention enhancement feature fusion module, and performing fine adjustment on the voxel grid by using the attention enhancement feature fusion module to obtain a three-dimensional voxel query set; 4, extracting two-dimensional features from the collected images, associating the three-dimensional voxel query set with the two-dimensional features through a variable cross attention mechanism, and updating voxel data; 5, introducing mask marks to supplement voxel data, and extracting voxel features from all voxel data by using a self-attention mechanism; and 6, inputting the voxel features into an AUHead semantic segmentation head for semantic segmentation, and projecting a result to an output space to obtain a 3D scene graph. According to the invention, the space occupation integrity and semantic segmentation accuracy of 3D semantic scene completion are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision processing technology, and in particular to a 3D semantic scene completion method. Background Art

[0002] For the perception system of autonomous vehicles, it is crucial to fully and accurately understand the 3D scene, as this directly affects core tasks such as path planning and map building. However, due to the limitations of sensor resolution, limited field of view, and incomplete observation caused by object occlusion, obtaining accurate and comprehensive 3D information of the real world faces major challenges.

[0003] Estimating 3D information from 2D images has always been a research challenge in the field of computer vision. Although humans can easily understand scenes and process geometric and semantic information from a single image, years of research have shown that this process is technically quite complex. Therefore, in order to reduce the technical difficulty of 3D estimation, many algorithms tend to use specialized depth sensors such as LiDAR or depth cameras. However, these specialized sensors are usually more expensive, larger, and more intrusive than ordinary cameras, which are widely used in devices such as smartphones, drones, and cars. Therefore, if we can accurately estimate 3D scenes from images, it will provide broad possibilities for the development of new applications.

[0004] To overcome the challenges faced by existing technologies, the task of 3D semantic scene completion (SSC) aims to reconstruct geometric and semantic structures based on observation data and fill in occluded regions using prior knowledge. Progress in this task benefits from the fact that humans can naturally infer the geometry and semantic content of a scene from partial observations. However, in driving scenarios, there is still a significant performance gap between current state-of-the-art SSC methods and human perception capabilities.

[0005] Currently, most 3D SSC techniques rely on 3D signal inputs with depth information, such as voxels, point clouds, depth maps, and truncated signed distance fields (TSDFs), which require expensive sensors to acquire. Compared with lidar, cameras are cheaper and easier to deploy, which has stimulated research interest in camera-based SSC solutions. For example, dense feature projection technology is used to convert 2D image inputs into 3D data. However, this projection method inevitably assigns 2D features of the visible area to blank or occluded voxels. For example, blank voxels occluded by a car still receive visual features related to the car. Therefore, the generated 3D features contain a lot of fuzzy information, which is not conducive to subsequent geometric completion and semantic segmentation, resulting in unsatisfactory performance. Summary of the invention

[0006] The present invention provides a 3D semantic scene completion method to solve the above technical problems.

[0007] In order to solve the above technical problems, the present invention provides a 3D semantic scene completion method, comprising the following steps:

[0008] Step 1: Use a binocular camera to collect RGB images of the 3D scene;

[0009] Step 2: construct a voxel occupancy prediction network, input the image collected in step 1 into the voxel occupancy prediction network, and obtain a voxel grid;

[0010] Step 3: construct an attention enhancement feature fusion module, and use the attention enhancement feature fusion module to perform depth correction on the voxel grid to obtain a three-dimensional voxel query set;

[0011] Step 4: extracting two-dimensional features from the image collected in step 1, associating the three-dimensional voxel query set with the two-dimensional features through a variable cross-attention mechanism, and updating the voxel data;

[0012] Step 5: Introduce mask marks to supplement voxel data, and use the self-attention mechanism to extract voxel features from all voxel data;

[0013] Step 6: Input the voxel features into the AUHead semantic segmentation head for semantic segmentation, and project the results into the output space to obtain a 3D scene graph.

[0014] Preferably, the image captured by the binocular camera at least includes image information of a current timestamp.

[0015] Preferably, the RGB image captured by the binocular camera also includes image information of the first four historical timestamps.

[0016] Preferably, the attention enhancement feature fusion module is a multi-stage codec architecture, and the calculation process includes:

[0017] Perform a 2D convolution operation on the X and Y dimensions while converting the Z dimension to the feature dimension;

[0018] After two rounds of downsampling, the boundary feature fusion module is used to fuse deep features and shallow features;

[0019] Use transposed convolution to upsample the fused features, and use the boundary feature fusion module to fuse them with the features of the corresponding layer for the second time;

[0020] Atrous spatial pyramid pooling is used for multi-scale feature extraction, and 3D convolution is used to complete occupancy segmentation.

[0021] Preferably, the calculation process of the boundary feature fusion module includes:

[0022] Define shallow convolution feature maps and deep convolution feature maps;

[0023] Extract vectors from the pixel positions corresponding to the above two feature maps, and perform convolution and batch normalization to obtain feature maps;

[0024] Upsampling is performed to generate a feature map that incorporates boundary information.

[0025] Preferably, the attention-enhanced feature fusion module also includes a deformable convolution attention module, which adjusts the sampling grid by an integer offset.

[0026] Preferably, the computational process of the deformable convolutional attention module includes: extracting deformation information from feature maps using additional convolutional layers and constructing offset fields; the design of the convolutional layer responsible for calculating the offset follows the convolution kernel size and dilation coefficient of the corresponding convolutional layer.

[0027] Preferably, bilinear interpolation is used to calculate the offset image outside the feature grid.

[0028] Preferably, the AUHead semantic segmentation head adopts an encoder-decoder architecture. In the encoder stage, multi-scale feature representation is extracted through a downsampling operation; in the decoder stage, the extracted features are mapped back to the original image size through an upsampling operation.

[0029] Preferably, at the decoder stage, a 3D convolutional attention mechanism is introduced in the upsampling operation.

[0030] Compared with the prior art, the 3D semantic scene completion method provided by the present invention has the following advantages:

[0031] 1. The present invention uses 2D images as input and utilizes voxel features and voxel boundary information of different scales to improve the semantic segmentation accuracy of 3D semantic scene completion;

[0032] 2. The present invention proposes an attention-enhanced feature fusion module (AFFM), which performs weighted fusion of voxel features of different scales in the voxel occupancy prediction stage and focuses on voxel boundary information, so as to more effectively restore three-dimensional spatial information;

[0033] 3. The present invention designs a semantic segmentation head driven by attention UNet (AUHead), which realizes the deep fusion of feature information of different scales through 3D UNet, improves the accuracy of spatial occupancy determination and the performance of semantic segmentation; introduces a 3D convolutional attention mechanism in the jump connection part to finely fuse features, aiming to enrich voxel feature information while highlighting core information, further improving the accuracy of spatial occupancy and the performance of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a process framework of a 3D semantic scene completion method in a specific implementation of the present invention;

[0035] Figure 2 This is a network structure diagram of an attention enhancement feature fusion module in a specific embodiment of the present invention;

[0036] Figure 3 This is a network structure diagram of a boundary feature fusion module in a specific implementation manner of the present invention;

[0037] Figure 4 A network structure diagram of a deformable convolutional attention module in a specific embodiment of the present invention;

[0038] Figure 5 This is a network structure diagram of the AUHead semantic segmentation head in a specific implementation of the present invention;

[0039] Figure 6 A comparison chart of qualitative results of various 3D semantic scene completion methods. DETAILED DESCRIPTION

[0040] In order to describe the technical solution of the above invention in more detail, specific embodiments are listed below to demonstrate the technical effects; it should be emphasized that these embodiments are used to illustrate the present invention but not to limit the scope of the present invention.

[0041] The 3D semantic scene completion method provided by the present invention is as follows: Figure 1 As shown, the following steps are included:

[0042] Step 1: Use a binocular camera to collect RGB images of the 3D scene;

[0043] Step 2: construct a voxel occupancy prediction network, input the image collected in step 1 into the voxel occupancy prediction network, and obtain a voxel grid;

[0044] Step 3: construct an attention enhancement feature fusion module, and use the attention enhancement feature fusion module to perform depth correction on the voxel grid to obtain a three-dimensional voxel query set;

[0045] Step 4: extracting two-dimensional features from the image collected in step 1, associating the three-dimensional voxel query set with the two-dimensional features through a variable cross-attention mechanism, and updating the voxel data;

[0046] Step 5: Introduce mask marks to supplement voxel data, and use the self-attention mechanism to extract voxel features from all voxel data;

[0047] Step 6: Input the voxel features into the AUHead semantic segmentation head for semantic segmentation, and project the results into the output space to obtain a 3D scene graph.

[0048] The present invention takes 2D images as input and utilizes voxel features and voxel boundary information of different scales to improve the semantic segmentation accuracy of 3D semantic scene completion.

[0049] like Figure 1 As shown in the figure, 3D semantic scene completion (SSC) aims to predict labels C = {c0,c1,c2,…,c M} to jointly infer the geometric shape and semantic information of the 3D scene, where M represents the total number of categories of interest and c0 represents "empty". In previous SSC research, 2.5D or 3D data (such as point clouds, depth information, etc.) are usually used as input. The difference is that this application only uses RGB images as input to infer the dense semantic scene within a certain spatial range around the vehicle. Specifically, this application uses the current image I t As input data, the 3D scene information within a certain distance around the vehicle is inferred, and a voxel grid map Y∈{c0,c1,c2,…,c M} W×H×Z In this map, each voxel may be marked as "empty" (c0) or assigned {c1,c2,…,c M}, where H, W, and Z represent the length, width, and height of the voxel grid, respectively.

[0050] In summary, the overall goal of this application is to train a neural network To generate a semantic voxel grid map Y t , and strive to make it as close to the ground truth as possible

[0051] like Figure 1 As shown, this application constructs a two-stage framework to achieve voxel-based semantic segmentation for scene representation.

[0052] Phase I (OccPred):

[0053] In the first stage, a lightweight voxel occupancy prediction network is used. The network integrates depth prediction and occupancy prediction functions to efficiently extract a set of sparse voxel queries from the image. In order to further improve the accuracy of spatial geometric occupancy prediction, this application designs an attention-enhanced feature fusion module (AFFM). This network architecture specifically performs depth correction on the predicted voxel depth to ensure the accuracy and reliability of the voxel query results. By introducing the attention enhancement mechanism and utilizing multi-scale feature fusion technology, AFFM can effectively improve the quality of voxel prediction.

[0054] Second stage (3D-ReSeg):

[0055] In the second stage, this application draws on the sparse to dense processing idea of ​​MAE and adopts a transformer-based architecture. A phased approach is used to enhance voxel features and achieve accurate semantic segmentation, thereby completing voxel-based semantic scene representation.

[0056] First, voxel features are enhanced through a variable cross-attention mechanism between the image and the voxel. Subsequently, for unproposed voxels, they are associated with a learnable mask token. Next, the entire voxel set is processed with self-attention to extract voxel features. Finally, accurate semantic segmentation is achieved through a semantic segmentation head driven by an attention UNet (AUHead), thus completing a voxel-based semantic scene representation.

[0057] Specifically, in the first stage (OccPred), category-independent query proposals are generated through a deep occupancy network. After entering the second stage (3D-ReSeg), two-dimensional features are first extracted from the RGB image. Then, the sparse three-dimensional voxel query set generated in the first stage is used and associated with the two-dimensional features extracted from the image through a variable cross-attention mechanism, thereby effectively enhancing the voxel features. Next, the voxel data is supplemented by introducing mask labels, and features are extracted from all voxels using a self-attention mechanism. Finally, all extracted voxel features are input into the AUHead semantic segmentation head for semantic segmentation, and the results are projected into the output space to obtain a 3D scene graph.

[0058] In some embodiments, the image captured by the binocular camera includes at least the image information of the current timestamp, which is recorded as the AEFF-SSC version. This version only uses the binocular camera image of the current timestamp as input data. In other embodiments, the RGB image captured by the binocular camera also includes the image information of the first four historical timestamps, which is recorded as the AEFF-SSC+ version. That is, in addition to processing the current timestamp, this version also integrates the information of the first four historical timestamps, enriching the input data by splicing the features of all frames along the channel, thereby improving the spatial occupancy accuracy and semantic segmentation accuracy of the system.

[0059] The first stage is a sparse geometric reconstruction network based on image depth information. Its core is a sparse geometric reconstruction network based on image depth information. The main goal of this stage is to focus on occupied voxels while using depth information for sparse reconstruction and separate empty voxels from the processing group. The specific process is as follows:

[0060] First, a pair of stereo 2D RGB images (denoted as ( and ) as input. Subsequently, through the depth estimation network, a 2.5D representation of the scene can be obtained. However, the point cloud data directly obtained by depth estimation projection often contains significant noise. Therefore, this application further adopts a 3D grid-based occupancy correction network, namely the Attention-Enhanced Feature Fusion Module (AFFM) to refine voxel information. This process aims to improve the accuracy and reliability of reconstruction and lay a solid foundation for subsequent 3D scene understanding and analysis.

[0061] For depth estimation, an off-the-shelf depth estimation model (such as MobileStereoNet) can be used to directly predict the depth of each image pixel. Subsequently, the depth map d is back-projected into a 3D point cloud.

[0062] In a scene where the origin of the camera coordinate system and the world coordinate system coincides, the depth of description of the same object in the two coordinate systems is the same, that is, d w =d c Based on this premise, the following transformation can be used to map pixel coordinates in a 2D image to coordinates in 3D space:

[0063]

[0064] In the above equation, (u0,v0) represents the optical center, f u and f v Represent the horizontal and vertical focal lengths, respectively. However, when projecting depth information into 3D space, depth errors are often amplified due to the presence of noise and artifacts. In addition, limited by the depth estimation algorithm, only some pixels in distant areas may obtain valid depth values. This situation may cause the point cloud in some areas to be very sparse after projection, thus affecting the overall quality of the point cloud.

[0065] Directly voxelizing point clouds often results in lower accuracy due to the influence of depth estimation errors and inaccurate point cloud projection. In addition, not all voxels are of interest to us. In order to enhance robustness, it is necessary to eliminate invisible and empty voxels and appropriately reduce the resolution. Doing so can focus more on voxels containing valid information, thereby improving the accuracy and efficiency of the overall processing.

[0066] In order to generate high-quality voxel queries, this paper designs a lightweight 3D space occupancy prediction network named AFFM (Attention Enhanced Feature Fusion Module). The network is able to generate low-resolution space occupancy maps, which helps to effectively correct image depth. First, the point cloud data is converted into a binary voxel grid map Q INITIn this process, if a voxel is occupied by at least one point, it is marked as “1” to ensure the accuracy of spatial occupancy. This method can more accurately identify and locate occupied voxels, providing strong support for subsequent depth correction and voxel query. Next, the designed AFFM is applied to refine the occupancy map, which can be expressed as M by the following equation: out =AFFM(Q INIT ) to obtain the corrected occupancy map.

[0067] Here, M out ∈{1,0} h×w×z represents the low-resolution output after refinement, and Q INIT ∈{1,0} H×W×Z represents the original high-resolution input. It is worth noting that the output M out The resolution is lower than the input Q INIT The AFFM mentioned here is a multi-level encoder-decoder architecture that enhances feature fusion based on the attention mechanism. The detailed structure of AFFM is as follows Figure 2 shown.

[0068] like Figure 2 As shown, Q INIT As input, a 2D convolution operation is performed on the X and Y dimensions, while the Z dimension is converted to the feature dimension to keep the architecture lightweight. After two rounds of downsampling, the deep features and shallow features are fused with the help of the boundary feature fusion module (BFFM). Subsequently, the fused features are upsampled using transposed convolution and fused again with the features of the corresponding layer using the BFFM module. Finally, ASPP (atrous spatial pyramid pooling) is used to achieve multi-scale feature extraction, and 3D convolution is used to complete occupancy segmentation.

[0069] During the propagation of the encoder-decoder network, as the network depth increases, the receptive field of the feature map gradually increases. However, this increase is often accompanied by the loss of detailed information. For tasks that require high-precision segmentation, the rich details in advanced convolutions are essential. Although the context branch is rich in diverse feature information, when performing segmentation tasks, more attention needs to be paid to the spatial and geometric details of the boundary area. Inspired by the attention mechanism, this application specially designs a boundary feature fusion module, namely Figure 2 The module intelligently utilizes boundary features and promotes deep fusion of detailed information and context information, thereby improving segmentation accuracy.

[0070] like Figure 2As shown in the figure, the boundary feature fusion module (BFFM) plays a key role in the 3D space occupancy prediction network. This module fuses and concatenates the low-level convolutional features obtained by downsampling in the encoding process with the high-level convolutional features obtained by upsampling in the decoding process. Through this fusion method, feature maps with clearer boundary information can be successfully generated, which significantly improves the performance of the network in the 3D space occupancy prediction task. Specifically, AFFM (Attention Enhanced Feature Fusion Module) consists of two boundary feature fusion modules (BFFM) and two deformable convolutional attention modules. The processing process is as follows:

[0071]

[0072] in, and Represents deep features, represents shallow features, Up represents upsampling, DAtten represents the deformable convolution attention module described in the subsequent section, Tran represents transposed convolution, and Seg represents segmentation head, as shown in Figure 2 As shown in the lower right corner. Through this design, the present application can better capture and utilize feature information, thereby improving the performance of the network.

[0073] Figure 3 The computational flow of the boundary feature fusion module (BFFM) is shown. This module defines two feature maps: the shallow convolution feature map F s And the deep convolutional feature map F d . Extract vectors from the corresponding pixel locations of these two feature maps and perform convolution and batch normalization to obtain the feature map F sn and F dn In addition, for F dn Upsample to get the same value as F s Feature maps F with the same spatial dimensions dn ′. In order to calculate the similarity between these two vectors, a channel-independent similarity graph is introduced, denoted as A. The similarity graph is obtained by sn and F dn The elements of ′ are multiplied element by element, summed along the channel dimension, and the sigmoid activation function is applied to obtain. Finally, the deep convolutional feature map F d Upsample to get the same value as F s Feature maps F with the same spatial dimensions d ′.

[0074] Calculate the similarity matrix, then subtract matrix A from the all-1 matrix J element by element to get matrix B, and compare it with F respectively d ′ and F sPerform element-by-element multiplication, and then concatenate the two weighted feature maps along the channel dimension to generate a feature map that incorporates boundary information. The specific process is as follows:

[0075]

[0076] B=JA (5)

[0077] Out = Concat(F s ⊙B,F d ′⊙A) (6)

[0078] Among them, Sigmoid represents the activation function, J represents the all-1 matrix, ⊙ represents the Hadamard product (element-by-element multiplication), and Concat represents channel concatenation.

[0079] like Figure 2 As shown in Figure 1, this application designs an attention module with deformable convolution and applies it to the encoder-decoder structure. Inspired by the attention mechanism, this application cleverly embeds the attention mechanism based on deformable convolution in the encoder-decoder architecture. Figure 4 As shown, deformable convolution demonstrates its unique ability to freely adjust the sampling grid via integer offsets to achieve flexible deformation.

[0080] Additional convolutional layers extract deformation information from the feature map and subsequently construct the offset field. The design of the convolutional layer responsible for calculating the offset follows the kernel size and dilation coefficient of the corresponding convolutional layer. For the offset image outside the feature grid, this application uses bilinear interpolation for calculation. This method, which relies on feature self-learning deformation, leads to the formation of adaptive convolution kernels.

[0081] Compared with the traditional encoder-decoder structure, the AFFM proposed in this application not only inherits the classic encoder-decoder structure, but also significantly enhances the deep feature extraction capability. Figure 4 As shown, the deformable convolutional attention can be expressed as:

[0082] F′=GELU(Conv(F)) (7)

[0083] Atten=Conv(DConv(GELU(Conv(F)))) (8)

[0084]

[0085] Among them, Conv stands for conventional convolution, DConv stands for deformable convolution, GELU is Gaussian Error Linear Units, and Atten stands for attention mechanism. represents the multiplication operation, represents the addition operation, and F represents the voxel features fused by BFFM (boundary feature fusion module).

[0086] After the first stage, query proposals are used to focus on image features, aiming to learn a rich visual representation of the 3D scene. To improve efficiency, this application uses both deformable cross-attention and deformable self-attention mechanisms. Deformable attention interacts with local regions of interest and only samples around reference points to calculate the attention results.

[0087] In the deformable criss-cross attention mechanism, the query grid coordinates are mapped to different image frames to generate reference points for the deformable attention. Subsequently, sampling is performed around these reference points to extract the corresponding image features as input. After several layers of deformable criss-cross attention, these features are further refined using deformable self-attention, resulting in more refined voxel features.

[0088] Inspired by 3D voxel segmentation technology and attention mechanism, this application developed a three-layer UNet semantic segmentation module. At present, many semantic segmentation methods usually adopt the method of directly upsampling features, but this method has significant limitations in accurately restoring the feature information of the original image. In contrast, the AUHead network designed in this application adopts a unique encoder-decoder architecture. In the encoder stage, the network effectively extracts multi-scale feature representations through downsampling operations, thereby significantly expanding the receptive field of the features. In the decoder stage, these carefully extracted features are accurately mapped back to the original image size through upsampling. This architecture gives the AUHead network stronger capabilities to more comprehensively preserve and restore feature details, thereby achieving more accurate semantic segmentation. For the detailed structure of the AUHead network, please refer to Figure 5 .

[0089] In the initial stage, a downsampling module is used to gradually reduce the size of the input feature voxels from the original size H×W×D to the original size H / 2×W / 2×D / 2 and H / 4×W / 4×D / 4. The specific operation details are as follows:

[0090] First, the input feature voxel F t 3D Perform two downsampling operations to obtain voxel features and The purpose of this step is to increase the receptive field of the convolution operation. Next, the deconvolution function (i.e., upsampling module) is used to reconstruct the features obtained after two downsampling operations. and Upsampling is performed to generate new voxel features and The main purpose of designing this deconvolution module is to capture high-level semantic information from the image and restore the spatial details lost in the downsampling process as much as possible. By combining downsampling and upsampling operations, it is possible to effectively extract and restore feature information while maintaining computational efficiency, providing strong support for subsequent 3D dense reconstruction and semantic segmentation tasks.

[0091] The jump connection in the UNet architecture is a core component responsible for connecting feature maps with the same resolution in the encoder to feature maps with the corresponding resolution in the decoder. The core role of this connection is to directly transmit low-level features to the decoder, thereby helping the decoder to more accurately restore the details of the original image and significantly improve the accuracy of the segmentation results.

[0092] Different from the traditional element-by-element addition method, this application innovatively introduces the 3D convolution attention mechanism into the jump connection. This mechanism enables the up-sampled features and the encoding stage features to be fused with higher accuracy. The 3D attention mechanism is similar to Figure 4 The deformable convolution attention mechanism in , simply upgrades all 2D convolutions to 3D convolutions and replaces deformable convolutions with ordinary 3D convolutions. The calculation formula is as follows:

[0093] V′=GELU(Conv 3D (V)) (10)

[0094] Atten 3D =Conv 3D (Conv 3D (GELU(Conv 3D (V)))) (11)

[0095] Out = Conv(Atten 3D ⊙V)+V (12)

[0096] Among them, GELU stands for Gaussian Error Linear Units, Atten 3D Represents the 3D attention mechanism, Conv 3D represents 3D convolution, ⊙ represents element-wise multiplication, and V represents voxel features.

[0097] Through this mechanism, the model can dynamically adjust the fusion weights of different features according to the needs of specific tasks, thereby achieving more efficient feature fusion. After that, the refined voxel features are obtained, and they are semantically classified, and the results are projected into the output space to obtain the final result. The mathematical expression of this process can be described as follows:

[0098]

[0099] Among them, Downsample represents the downsampling function, and Upsample represents the upsampling function. and is the feature obtained after downsampling, and and It is the feature obtained by upsampling the downsampled feature after upsampling. 3D represents the attention mechanism, Seg 3D represents 3D semantic segmentation, Y t Represents the final result after semantic segmentation.

[0100] For the loss function:

[0101] In the first stage, this application uses binary cross-entropy loss (BCELoss) to perform occupancy prediction at low spatial resolution.

[0102] In the second stage, this application chooses the weighted cross-entropy loss (Weighted Cross-Entropy Loss) L ce To guide the training process. Following the MonoScene approach, scene class affinity losses are also added. and is incorporated into the training. Therefore, training is performed by optimizing the following loss function:

[0103]

[0104] Next, experiments were conducted on the method provided in this application.

[0105] like Figure 1 As shown, in the first stage, this application obtains the predicted depth map from MobileStereoNet and performs depth projection on it. To ensure compatibility with the data size of the SemanticKITTI dataset, the depth map is converted into a sparse voxel query Q of size 256×256×32. INIT To improve the accuracy of voxel occupancy and optimize computational resources, an AFFM network is then employed to deeply correct and reduce the resolution of the sparse voxel queries. Figure 2 As shown, the voxel Q INIT Input into the AFFM network. After deep correction, a Q matrix of size 128×128×16×20 is obtained. p , where 20 represents the feature dimension, which is consistent with the dimension of image features.

[0106] In the second stage, when some selected voxel features are aligned to the image, the unselected voxels are replaced by mask labels to construct a complete 3D voxel feature. Then, through the cross-attention and self-attention mechanisms, the voxel features are obtained. Subsequently, the feature is semantically segmented through the AUHead network to generate a map of size 256×256×32×20. Here, the voxel size of 256×256×32 matches the size of the true label in the dataset, while the number 20 represents the number of semantic categories of interest. It is worth noting that each voxel corresponds to 0.2 cubic meters in the real world, and these voxel queries are defined in the self-vehicle coordinate system.

[0107] In this application, the proposed AEFF-SSC method is systematically validated using the SemanticKITTI dataset. The SemanticKITTI dataset contains not only LiDAR data but also stereo image data corresponding to each LiDAR scan. SemanticKITTI provides detailed semantic annotations for each LiDAR scan, which are derived from an odometry benchmark test set. The benchmark test set covers a wide range of 22 different outdoor driving scenarios, ensuring a comprehensive and diverse validation. In addition, the dataset provides a 256×256×32 voxel grid labeled with 20 different classes. These voxels are generated by post-processing the LiDAR scans, and the size of each voxel is 0.2 cubic meters. The voxel grid is labeled with 20 classes (19 semantic classes and 1 empty class). Regarding the target output, SemanticKITTI provides ground truth semantic voxel grid data by aggregating and voxelizing semantic point clouds that are continuously annotated in the spatial range. The SSC model is able to accept a single voxelized LiDAR scan or an RGB stereo pair as input. In this application, we focus on image-based SSC, utilizing stereo image data as input and performing a comparative analysis between the final output and annotated LiDAR data.

[0108] This application uses voxel-level intersection over union (IoU) as an occupancy prediction indicator, and uses the mean intersection over union (mIoU) on 19 semantic categories to evaluate the performance of semantic segmentation.

[0109] Occupancy prediction without considering semantics is considered as category-agnostic perception. It only focuses on understanding spatial geometry and determining whether each voxel in 3D space is occupied or empty. For this perception, a common evaluation metric is the voxel-level Intersection over Union (IoU), expressed as:

[0110]

[0111] Among them, TP, FP and FN represent the number of true positives, false positives and false negatives respectively. A true positive indicates that a voxel that is actually occupied is correctly predicted to be occupied.

[0112] Simultaneously inferring the occupancy state of voxels and occupancy predictions for semantic categories can be considered semantic geometry awareness. In this case, mean Intersection over Union (mIoU) is often used as an evaluation metric. The mIoU metric calculates the IoU for each semantic category separately and then averages these IoU values ​​over all categories (excluding the "empty" category):

[0113]

[0114] Among them, TP i , FP i , and FN i They represent the number of true positives, false positives, and false negatives for a specific semantic category i, respectively. C represents the total number of semantic categories.

[0115] There is a strong interaction between IoU and mIoU. For example, simply sacrificing IoU can improve mIoU. We adopt the mainstream evaluation method of the current SSC task and evaluate IoU and mIoU within 51.2m×51.2m in front of the vehicle. In the field of autonomous driving, people pay more attention to the environmental information near the vehicle because autonomous vehicles have less reaction time. In order to show the advantage in close range, this application evaluates IoU and mIoU in the range of 25.6m×25.6m and 12.8m×12.8m.

[0116] This paper compares AEFF-SSC with state-of-the-art open source SSC methods, which can be divided into RGB-based inference methods MonoScene, VoxFormer-S, and 2.5D / 3D-based methods JS3CNet, LMSCNet and SSCNet, as well as LMSCNet* that uses pseudo-lidar point clouds generated from stereo depth as input.

[0117] Table 1 Quantitative comparison with state-of-the-art camera-based SSC methods

[0118]

[0119] Table 1 comprehensively shows the performance of the proposed method on the SemanticKITTI dataset. Experimental results show that the AEFF-SSC and AEFF-SSC+ proposed in this application outperform other comparison methods in various performance indicators. Here, we mainly focus on exploring a lighter solution, namely the AEFF-SSC method with a single frame as input.

[0120] As shown in Table 1, based on the overall data, AEFF-SSC achieves significant improvements in IoU compared to the 2D monocular camera baseline method MonoScene within the labeled range of the dataset (51.2m×51.2m). At a closer distance range of 25.6m×25.6m, the improvements in IoU and mIoU are more significant. In the range of 25.6m×25.6m, IoU increases from 38.55 to 58.34, an improvement of 51.34%; in the range of 12.8m×12.8m, IoU increases from 38.42 to 65.92, an improvement of 71.58%.

[0121] In addition, compared with VoxFormer-S, which also uses a binocular camera, the method proposed in this application shows significant advantages. Without sacrificing IoU, the mIoU of this application is improved by 9.6%, 8.9% and 10.47% in the range of 51.2m, 25.6m and 12.8m, respectively. These results fully demonstrate the advancement and effectiveness of this method. As shown in Table 1, when adding historical frames, that is, using AEFF-SSC+, the method of this application achieves further improvements in both IoU and mIoU. However, due to the limited improvement compared to single-frame input and the need to consume a lot of additional computing resources, it will not be further elaborated.

[0122] Table 2 Detailed quantitative comparison with current state-of-the-art camera-based SSC methods

[0123]

[0124]

[0125] As shown in Table 2, the AEFF-SSC method also achieves significant improvements in the accuracy of various semantic classifications. Compared with the current best camera-based segmentation method VoxFormer-S, our method shows significant improvement in road information segmentation. Specifically, our method achieves significant improvements in multiple key categories, including road category from 54.41 to 58.97, sidewalk category from 25.55 to 29.06, and terrain category from 29.53 to 35.13.

[0126] As shown in Table 2, compared with LMSCNet* which directly uses pseudo point cloud, the improvement of this application is more significant. For example, in the road category, this application improves from 54.90 to 58.97, in the parking category from 9.89 to 15.18, and in the terrain category from 32.30 to 35.13. These significant improvements are mainly attributed to the fact that this application fully utilizes the boundary information of voxel features. By carefully processing the boundaries of voxel features, this method can more accurately identify and segment various elements in road scenes, thereby achieving significant performance improvements on multiple categories.

[0127] In addition, compared with VoxFormer-S, which also uses a binocular camera, this method shows significant advantages. Without sacrificing IoU, the mIoU of this application is improved by 9.6%, 8.9% and 10.47% in the range of 51.2m, 25.6m and 12.8m, respectively. These results fully demonstrate the advancement and effectiveness of this method. As shown in Table 1, in the AEFF-SSC+ version, the method of this application achieves further improvements in both IoU and mIoU. However, due to the limited improvement compared to single-frame input and the need to consume a lot of additional computing resources, it will not be elaborated in depth.

[0128] Table 3 Quantitative comparison with state-of-the-art lidar-based SSC methods

[0129]

[0130] As shown in Table 3, as the distance gradually decreases, the performance gap between the method of the present application and the current state-of-the-art LiDAR method gradually narrows. It is worth noting that when the historical frame information (i.e., the AEFF-SSC+ version) is introduced, the gap is further reduced, and even exceeds some LiDAR-based methods. It should be emphasized that LiDAR data is far superior to camera-based extrapolation in terms of information density and accuracy of depth information. However, the cost of cameras is much lower than that of LiDAR. For example, compared with LMSCNet within the 51.2m distance range, there is a certain performance gap between the AEFF-SSC method and LMSNet due to the advantage of LiDAR in obtaining depth information. Specifically, in terms of IoU, it lags behind LMS by 9.9 points; in terms of mIoU, the gap is 3.19 points. In addition, if the range is narrowed to the 12.8m area, the IoU gap is further reduced to 8.81 points, and mIoU even exceeds LMSCNet. It is worth mentioning that the IoU of AEFF-SSC+ in the 12.8m area exceeds that of the lidar-based SSCNet and JS3CNet, and its mIoU performance is also better than that of the lidar-based SSCNet and LMSCNet.

[0131] This discovery undoubtedly brings positive breakthroughs to autonomous driving technology in acquiring close-range map information. Since AEFF-SSC and AEFF-SSC+ only rely on low-cost cameras during reasoning, they are able to provide more accurate close-range map information, which is of great significance to the practicality and economy of autonomous driving systems.

[0132] This application visually compares the output of AEFF-SSC with the latest mainstream camera-based and lidar-based SSC methods. In order to highlight the advantages of our method in visualization, Figure 6 It also contains the input image (in the leftmost column) and its corresponding true label (GT).

[0133] In outdoor scenes (from the SemanticKITTI dataset), such as Figure 6 As shown in Figure 2, AEFF-SSC performs well in all visual baseline visualization maps compared to the baseline methods, especially in scene completeness. In addition, this application has an absolute advantage in the semantic segmentation task, and its performance is particularly impressive on objects with obvious boundary information, such as cars, roads, sidewalks, and terrain. Figure 6 As shown, the proposed method performs best in the boundary segmentation between the road and the sidewalk. In addition, the proposed method shows a clear advantage in segmenting a row of cars parked on the roadside, and can accurately distinguish each car.

[0134] Compared with the lidar-based LMSCNet, this application has certain advantages in scene completeness in some cases. In addition, in some scenes, it can even reconstruct and segment objects that the lidar method cannot recognize. Figure 6 As shown in the third scene in , the application successfully and clearly reconstructed the traffic sign hidden in the bushes. Similarly, in the sixth scene, the clarity of the traffic sign reconstructed by the application stood out among all the projects, and its reconstruction effect was unparalleled. In addition, in the fifth scene, the application uniquely and accurately segmented the truck from a row of cars. It is important to emphasize that lidar data has significant advantages in point cloud density and depth information accuracy, far exceeding visual depth prediction technology. However, in some specific scene reconstruction tasks, the AEFF-SSC method even outperforms lidar-based technology.

[0135] Table 4 Ablation studies

[0136] Each module IoU QUR This application 44.72 13.20 AUHead in this application 44.30 12.70 AFFM in this application 44.65 12.49 BFFM in this application 44.63 12.20 VoxFormer 44.27 12.04

[0137] As shown in Table 4, this application conducted ablation experiments on each module in the method, and used the BFFM, AFFM network module and AUHead module in this application to verify the effectiveness of the method proposed in this invention. It can be seen from the experimental results in Table 4 that each module in the architecture contributes to achieving the best results. The experimental results of VoxFormer provide accuracy based on the visual baseline, providing a comparison benchmark for this application. For the BFFM module in this application, this enhanced attention module performs weighted processing on the boundary feature information, significantly improving the recognition ability of spatial structure information. When only BFFM is introduced into the system without using the AFFM module and the AUHead module, the spatial structure information is significantly improved compared with the common splicing technology, and the accuracy of semantic segmentation is also slightly improved, with IoU increased by 0.36 and mIoU increased by 0.16.

[0138] In contrast, when the variable convolutional attention mechanism is added and AFFM is used, both the spatial occupancy accuracy and the semantic segmentation accuracy are significantly improved, with IoU improving by 0.38 and mIoU improving by 0.45.

[0139] It is worth noting that the deformable convolutional attention mechanism in this application shows excellent adaptability. It can dynamically assign weights according to the features at different positions in the feature map, greatly improving the network's ability to identify and focus on key information. In addition, by dynamically adjusting the sampling points of the convolution kernel, the mechanism substantially expands the effective receptive field of the model. Therefore, by integrating deformable convolutional attention, the network in this application achieves a significant improvement in voxel segmentation performance.

[0140] Finally, this application compares the performance of direct upsampling and semantic segmentation using only the AUHead module (without the BFFM and AFFM modules), and finds that the spatial occupancy accuracy IoU is slightly improved (0.03), while the semantic segmentation accuracy mIoU is significantly improved (0.66). This shows that by integrating the attention mechanism, the contextual information in the scene can be more effectively captured and utilized, thereby improving the overall performance. In summary, this application successfully improves the accuracy of semantic segmentation while maintaining spatial occupancy accuracy by combining multiple innovative modules.

[0141] Table 5 Ablation experiment of AFFM module

[0142] Fusion Method IoU QUR Splicing 44.27 12.04 BFFM only 44.63 12.20 AFFM 44.65 12.49

[0143] In the SemanticKITTI validation set, in order to further verify the advantages of AFFM (Attention Feature Fusion Module), this application specifically excluded the AUHead module for ablation research. The experimental results are shown in Table 5. In order to evaluate the performance of different skip connection methods, a comparison was performed within a custom deep rectification network.

[0144] During the experiment, we used the "splicing" fusion method as a baseline. This method achieves feature fusion by splicing low-level and high-level features in the network in the channel dimension. However, the experimental results show that the BFFM (boundary feature fusion module) designed in this application surpasses the "splicing" method in performance. Specifically, BFFM achieved an improvement of +0.36 and +0.16 in spatial occupancy accuracy (IoU) and semantic segmentation accuracy (mIoU), respectively. This significant improvement shows that because the BFFM module weights the boundary feature information, it can more effectively integrate and utilize feature information at different levels, thereby improving the accuracy of semantic segmentation.

[0145] Through ablation studies, the important role of the BFFM module in improving semantic segmentation performance is further confirmed, providing solid experimental support for this method. In addition, when the deformable convolutional attention mechanism (i.e., AFFM module) is integrated into the deep rectification network, a significant improvement of 0.38 on IoU and 0.45 on semantic accuracy (mIoU) is achieved. This substantial enhancement highlights the effectiveness of AFFM in improving spatial occupancy accuracy and semantic segmentation accuracy by introducing an enhanced attention feature fusion strategy.

[0146] Table 6 Ablation experiment of AUHead module

[0147] Segmentation method IoU QUR Direct SegHead 44.27 12.04 Two-layer 3D UNet+SegHead 43.77 12.55 Three-layer 3D UNet+SegHead 44.15 12.58 AUHead 44.30 12.70

[0148] As shown in Table 6, this application also conducts an ablation study on the AUHead module to verify the design decisions of this application, while excluding the AFFM and BFFM modules to isolate their effects. The AUHead module is designed based on a three-layer 3D UNet-like network architecture. In the experiment, multiple voxel feature processing methods were explored and their performance differences were compared.

[0149] As a baseline experiment, we directly use voxel features Semantic segmentation was performed without any additional network processing. Subsequently, the voxel features were processed through a two-layer UNet network, and a significant improvement in semantic segmentation accuracy was observed, with mIoU increasing by 0.51. This result confirms the effectiveness of the UNet network in voxel feature processing.

[0150] When the number of UNet layers is further increased to three, both spatial occupancy accuracy and semantic segmentation accuracy are improved compared to the two-layer UNet. This suggests that deeper UNet networks are better able to capture and utilize spatial and contextual information in voxel features. However, due to GPU memory limitations, the impact of four or more layers of UNet networks was not further explored.

[0151] It is worth noting that when this application refers to “3D attention enhanced 3DUNet semantic segmentation head”

[0152] When the UNet network is introduced with AUHead, the segmentation method achieves the highest accuracy. Compared with the above experiments, the use of AUHead significantly improves the spatial occupancy accuracy and semantic segmentation accuracy, with an increase of 0.03 in IoU and 0.66 in mIoU compared to the baseline. This significant improvement fully demonstrates the effectiveness of the AUHead module.

[0153] In summary, the 3D semantic scene completion method provided by the present invention proposes an innovative camera-based 3D semantic scene completion (SSC) method, in which the AFFM network can more effectively integrate and utilize multi-level feature information by utilizing the weighted fusion strategy of attention enhancement and boundary feature information, thereby improving the accuracy of semantic segmentation; the AUHead semantic segmentation head can effectively aggregate multi-layer 3D voxel feature information and accurately extract key feature information, thereby achieving a double improvement in spatial occupancy accuracy and semantic segmentation accuracy; and the present invention can achieve similar results to methods using expensive and complex lidar equipment by relying only on camera data, which is of great significance in practical applications. Overall, the present invention not only improves the accuracy of 3D semantic scene completion, but also provides new possibilities and directions for future camera-based 3D perception systems, enabling this method to play a key role in fields such as autonomous driving, robot navigation, and augmented reality.

[0154] Obviously, those skilled in the art can make various changes and modifications to the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A 3D semantic scene completion method, characterized in that: The steps include: Step 1: Use a binocular camera to collect RGB images of the 3D scene; Step 2: construct a voxel occupancy prediction network, input the image collected in step 1 into the voxel occupancy prediction network, and obtain a voxel grid; Step 3: construct an attention enhancement feature fusion module, and use the attention enhancement feature fusion module to perform depth correction on the voxel grid to obtain a three-dimensional voxel query set; Step 4: extracting two-dimensional features from the image collected in step 1, associating the three-dimensional voxel query set with the two-dimensional features through a variable cross-attention mechanism, and updating the voxel data; Step 5: Introduce mask marks to supplement voxel data, and use the self-attention mechanism to extract voxel features from all voxel data; Step 6: Input the voxel features into the AUHead semantic segmentation head for semantic segmentation, and project the results into the output space to obtain a 3D scene graph.

2. The 3D semantic scene completion method according to claim 1, characterized in that: The image captured by the binocular camera includes at least image information of a current timestamp.

3. The 3D semantic scene completion method according to claim 2, characterized in that: The RGB image captured by the binocular camera also includes image information of the first four historical timestamps.

4. The 3D semantic scene completion method according to claim 1, characterized in that: The attention enhancement feature fusion module is a multi-level codec architecture, and the calculation process includes: Perform a 2D convolution operation on the X and Y dimensions while converting the Z dimension to the feature dimension; After two rounds of downsampling, the boundary feature fusion module is used to fuse deep features and shallow features; Use transposed convolution to upsample the fused features, and use the boundary feature fusion module to fuse them with the features of the corresponding layer for the second time; Atrous spatial pyramid pooling is used for multi-scale feature extraction, and 3D convolution is used to complete occupancy segmentation.

5. The 3D semantic scene completion method according to claim 4, characterized in that: The calculation process of the boundary feature fusion module includes: Define shallow convolution feature maps and deep convolution feature maps; Extract vectors from the pixel positions corresponding to the above two feature maps, and perform convolution and batch normalization to obtain feature maps; Upsampling is performed to generate a feature map that incorporates boundary information.

6. The 3D semantic scene completion method according to claim 4, characterized in that: The attention-enhanced feature fusion module also includes a deformable convolutional attention module, which adjusts the sampling grid by an integer offset.

7. The 3D semantic scene completion method according to claim 6, characterized in that: The computational process of the deformable convolutional attention module includes: using an additional convolutional layer to extract deformation information from the feature map and construct an offset field; the design of the convolutional layer responsible for calculating the offset follows the convolution kernel size and dilation coefficient of the corresponding convolutional layer.

8. The 3D semantic scene completion method according to claim 7, characterized in that: For the offset image outside the feature grid, bilinear interpolation is used for calculation.

9. The 3D semantic scene completion method according to claim 1, characterized in that: The AUHead semantic segmentation head adopts an encoder-decoder architecture. In the encoder stage, multi-scale feature representations are extracted through downsampling operations; in the decoder stage, the extracted features are mapped back to the original image size through upsampling operations.

10. The 3D semantic scene completion method according to claim 9, characterized in that: In the decoder stage, a 3D convolutional attention mechanism is introduced in the upsampling operation.

Images