An implicit scene completion method based on dense voxel representation
By constructing a bird's-eye view-assisted enhancement module and a feature adaptive completion module, the problem of insufficient learning of empty voxel features was solved, and information exchange between empty and non-empty voxels was realized, improving the accuracy and completeness of scene completion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2024-03-06
- Publication Date
- 2026-07-24
AI Technical Summary
Existing implicit scene completion methods lack in-depth exploration of empty voxel features, and there is a lack of information exchange between empty and non-empty voxels, resulting in poor scene completion performance.
A bird's-eye view enhancement module and a feature adaptive completion module are constructed. Dense bird's-eye view features are obtained through two-dimensional convolution, and deformable cross attention and self attention modules are used to realize information exchange between empty voxels and non-empty voxels, thereby improving the feature learning effect.
By exchanging information and enhancing features, the accuracy and completeness of scene completion are improved, generating point cloud representations that are closer to real-world scenes.
Smart Images
Figure CN118135519B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous driving, and in particular relates to an implicit scene completion method based on dense voxel representation. Background Technology
[0002] Scene completion aims to provide dense scene representations for incomplete 3D point clouds, which is of great significance in fields such as autonomous driving. Scene completion algorithms can fill in these missing areas, providing autonomous driving systems with more complete and accurate environmental perception data, thereby enabling safer and more reliable driving decisions. In recent years, implicit scene completion methods have received some attention due to the powerful modeling capabilities and memory efficiency of neural implicit functions. However, due to the sparsity and incompleteness of LiDAR point clouds, the implicit scene completion task remains challenging.
[0003] Current implicit scene completion methods typically use the coordinates of sampling points or related features as input, generating corresponding signed distance function values through a multilayer perceptron. Some implicit neural representation methods, such as SIREN, BACON, and DiGS, can be considered Eikonal implicit scene completion methods. They transform the constraints corresponding to sampling points in different regions into loss functions, which are then used to directly fit the signed distance function corresponding to the 3D point cloud. However, these constraints are applied to discrete coordinates, leading to inaccurate boundary value constraints. Subsequently, LODE, the method closest to this invention, introduces an intermediate embedding space to obtain dense boundary value constraints, thus alleviating this problem. Specifically, it first converts the input point cloud into voxel form and uses sparse convolution for feature extraction. Then, it uses trilinear interpolation to obtain point-level features from the extracted features, concatenates them with the corresponding positional encoding, and inputs them into a multilayer perceptron to generate the corresponding signed distance function values. The above methods lack in-depth exploration of empty voxel feature learning, which is beneficial for scene completion tasks. In addition, current methods lack information exchange between empty and non-empty voxels, failing to capture the contextual information of the scene for completion. Summary of the Invention
[0004] The purpose of this invention is to provide an implicit scene completion method based on dense voxel representation to solve the problems existing in the existing methods.
[0005] The technical solution to achieve the purpose of this invention is: an implicit scene completion method based on dense voxel representation, comprising two processes: a training phase and a testing phase.
[0006] The training phase includes the following steps:
[0007] Step 1: Construct a feature extraction module to extract non-empty voxel features from the input sparse point cloud;
[0008] Step 2: Construct a bird's-eye view enhancement module to enhance the non-empty voxel features extracted in Step 1 using dense bird's-eye view features.
[0009] Step 3: Construct a feature adaptive completion module composed of deformable self-attention modules to adaptively complete all elemental features;
[0010] Step 4: Construct a symbolic distance function value generation module to generate symbolic distance function values corresponding to the sampling points, calculate the corresponding loss function values, and train the implicit scene completion model;
[0011] During the testing phase, the point cloud to be completed is input into the trained implicit scene completion model to obtain the implicit representation of the completed scene point cloud. The testing phase includes the following steps:
[0012] Step 1: Process the point cloud of the scene to be completed using the trained implicit scene completion model, and extract the voxel features after completion. ;
[0013] Step 2: Voxelize the point cloud of the scene to be completed. Treat each voxel in the voxelized scene representation as a point, extract the corresponding point-level features, and generate the corresponding symbolic distance function value using the symbolic distance function value generation module.
[0014] Compared with the prior art, the significant advantages of this invention are:
[0015] (1) This invention constructs a bird's-eye view auxiliary enhancement module, which obtains dense bird's-eye view features through two-dimensional convolution, including information related to empty voxels, and uses a deformable cross-attention module to transfer the information to non-empty voxel features. Compared with previous methods, this invention realizes the exploration of empty voxel features through this structure, which is beneficial to improving the scene completion effect.
[0016] (2) This invention constructs a feature adaptive completion module, which realizes the further transfer of empty voxel information contained in the bird's-eye view features from non-empty voxel features to empty voxel features. Compared with the previous method, this invention realizes the information exchange between non-empty voxel features and empty voxel features, and utilizes scene context information to obtain more expressive voxel features. Attached Figure Description
[0017] Figure 1 This is an overall framework diagram of the method of the present invention.
[0018] Figure 2 This is a framework diagram of the bird's-eye view enhancement module in the method of the present invention.
[0019] Figure 3 This is a framework diagram of the feature adaptive completion module in the method of this invention.
[0020] Figure 4 (a) in the image is a bird's-eye view visualization of the input sparse point cloud in the same scene. Figure 4 (b) in the image is a bird's-eye view visualization of the voxel features extracted from the same scene through sparse convolution. Figure 4 (c) in the figure is a visualization of the bird's-eye view features extracted using the method of the present invention for the same scene. Figure 4 (d) in the image is a bird's-eye view visualization of the ground truth point cloud for the same scene.
[0021] Figure 5 These are scene completion effect diagrams using different methods. Detailed Implementation
[0022] This invention provides an implicit scene completion method based on dense voxel representation, comprising two phases: training and testing. The method uses 2D convolution to extract dense multi-scale features from a bird's-eye view, obtains information related to empty voxels, and learns empty voxel features. A deformable cross-attention module is then used to enhance the features extracted by sparse convolution. Subsequently, a deformable self-attention module is used to achieve information exchange between empty and non-empty voxels.
[0023] The specific steps of the training phase process are as follows:
[0024] Step 1_1: Convert the original input sparse point cloud into voxel form, and extract sparse voxel features using the feature extraction module. The feature extraction module consists of sparse convolutions. express The eigenvalues in the are real numbers. Indicates the resolution of the extracted features. It is the feature dimension.
[0025] Step 1_2: Construct a bird's-eye view auxiliary enhancement module, which includes two parts: bird's-eye view feature extraction and feature enhancement.
[0026] (1) Bird's-eye view feature extraction: This part takes sparse point cloud as input and outputs... Bird's-eye view features at various scales The specific implementation process is as follows: The input point cloud is converted into voxel form, and then max pooling is performed along the z-axis to obtain the initial bird's-eye view features, denoted as... Finally Input a 2D UNet structure for processing, output .
[0027] (2) Feature enhancement: First, for the features obtained in step 1_1 ,filter( ) out of it Non-empty voxel features are used as queries in subsequent attention modules and their shapes are reshaped. ), obtained from One query Compositional characteristics , express The median is a real number. It is the feature dimension:
[0028]
[0029] For query Its coordinates are denoted as The location of the multi-scale bird's-eye view features is The feature point at that location is the corresponding reference point, with coordinates as follows: Sampling from around the reference point Several points were then used, followed by the use of a deformable cross-attention module ( Update the query based on the characteristics of these points to obtain... :
[0030]
[0031] in, This indicates a deformable attention operation. These are learnable attention weights, based on get, These are learnable weights. This refers to the coordinate offset of the sampling points around the reference point relative to the reference. These correspond to the features of these sampling points, obtained by using bilinear interpolation on the bird's-eye view features. (The preceding text appears to be incomplete and requires further context.) In One query Use the corresponding Replacement yields enhanced voxel features. , express The eigenvalues in the are real numbers. express The feature resolution, It is the feature dimension.
[0032] Step 1_3: Construct a feature adaptive completion module, which adaptively completes voxel features using a deformable self-attention module. The specific implementation process is as follows: for the enhanced voxel features obtained in Step 1_2... Filter out the hollow voxel features and use learnable vectors. Assignment, express The value in the middle is a real number, and a new voxel feature is obtained, denoted as . , express The eigenvalues in the are real numbers. express The feature resolution, It is the feature dimension. Each voxel feature within it... These are all queries that need to be updated in the subsequent deformable self-attention module. The queries that need to be updated are in... The position coordinates in the middle are as follows That is, the coordinates of the reference point, sampled around the reference point. 1 point, then use a deformable self-attention module ( Update based on the characteristics of these points To obtain updated features , Completed voxel features:
[0033]
[0034] in, This indicates a deformable attention operation. These are learnable attention weights, based on get, These are learnable weights. This refers to the coordinate offset of the sampling points around the reference point relative to the reference. These correspond to the features of these sampling points.
[0035] Step 1_4: Input the features corresponding to the sampling points into the symbolic distance function value generation module, which is composed of a multilayer perceptron structure with sinusoidal activation. Generate the corresponding symbolic distance function values and train the model using the appropriate loss function. Specifically, the symbolic distance function values for different regions in the scene have different constraints. Therefore, sampling points are generated from different regions of the scene. Features are composed of point-level features and position encoding splicing ( ) and formed, Voxel characteristics Points obtained using trilinear interpolation The coordinates are Corresponding position code It can be represented as:
[0036]
[0037] in, It is a position function, with For parameters, This is a hyperparameter, explained in detail below:
[0038]
[0039] The sampled point features are input into the symbolic distance function value generation module to generate the corresponding symbolic distance function values. This module is a multilayer perceptron structure, including... Fully connected layers and A sinusoidal activation function. The symbolic distance function of the scene is defined as... The process of generating the symbolic distance function value can be represented as follows:
[0040]
[0041] in, Indicates the first Fully connected layer plus the first Layer sinusoidal activation function, Indicates the first The input of a fully connected layer, and They represent the first The weights and biases of fully connected layers and They represent the first The weights and biases of the fully connected layers. The symbolic distance function values of the sampled points are used to calculate the loss function corresponding to the symbolic distance function. The constraints on the symbolic distance function values for different regions are as follows:
[0042]
[0043]
[0044]
[0045]
[0046] in, Indicates the parameter Find the partial derivative. The symbolic distance function representing the scene. coordinates Mapping to high-dimensional shape embedding , Indicating the scene The normal vector at the point, To represent a function, in Use it as input to make its value far away from 0. Represents the entire spatial domain. The surface point representing the truth value, Using these constraints, the corresponding loss function is calculated using the following formula:
[0047]
[0048] This loss function, as part of the total loss function, is used to constrain the symbolic distance function of the sampling points. Represents the regularization parameter. This indicates the calculation of the magnitude of a vector. Calculate the cosine similarity between two vectors.
[0049] The total loss function is defined as:
[0050]
[0051] in, Represents the regularization parameter. The loss is the aforementioned symbolic distance function. It is used in calculating sparse convolution. The loss function value between the voxel occupancy probability predicted by the layer decoder and the occupancy label is used to remove redundant parts. In the stage used for bird's-eye view feature extraction, the loss function value between the normalized output of the single-channel multi-scale bird's-eye view prediction and the corresponding single-channel binarized ground truth bird's-eye view is calculated. and Both use binary cross-entropy ( The loss function, specifically the formula, is as follows:
[0052]
[0053] It is the predicted voxel occupancy probability. This is the corresponding occupied tag.
[0054]
[0055] It is the normalized output of single-channel, multi-scale bird's-eye view prediction. It corresponds to the ground truth value of the single-channel binarized bird's-eye view. This indicates the number of feature scales in a bird's-eye view.
[0056] Calculate the loss function value, and use gradient descent to update the network parameters in the implicit scene completion model until the model converges, thus obtaining the trained implicit scene completion model.
[0057] The specific steps of the testing phase process are as follows:
[0058] Step 2_1: Voxelize the point cloud of the scene to be completed, and process it sequentially using the feature extraction module, bird's-eye view-assisted enhancement module, and feature adaptive completion module in the trained implicit scene completion model to obtain the completed voxel features. .
[0059] Step 2_2: Sampling points according to the initial voxelized point cloud resolution, for Point-level features are obtained using trilinear interpolation, concatenated with the corresponding positional codes, and then input into the symbolic distance function value generation module to generate the corresponding symbolic distance function value, thus obtaining the complete implicit representation of the entire scene.
[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments. The described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0061] The overall framework of the method is as follows Figure 1 As shown, an implicit scene completion method based on dense voxel representation includes two processes: a training phase and a testing phase. The specific steps of the training phase are as follows:
[0062] Step 1_1: Convert the original input sparse point cloud into voxel form, and extract sparse voxel features using the feature extraction module. The feature extraction module consists of sparse convolutions. express The eigenvalues in the are real numbers. This represents the extracted feature resolution, which is set to [value] in our method. , This is the feature dimension, set to 256.
[0063] Step 1_2: Construct the bird's-eye view auxiliary enhancement module. The module structure diagram is as follows: Figure 2 As shown, it includes two parts: bird's-eye view feature extraction and feature enhancement.
[0064] (1) Bird's-eye view feature extraction: This part takes sparse point cloud as input and outputs... Bird's-eye view features at various scales In our method, Set to 3. The specific implementation process is as follows: convert the input point cloud into voxel form, then use max pooling along the z-axis to obtain the initial bird's-eye view features, denoted as... Finally Input a 2D UNet structure for processing, output .like Figure 4As shown, the four sub-figures (a), (b), (c), and (d) are respectively a bird's-eye view of the input sparse point cloud, a bird's-eye view of the voxel features extracted by sparse convolution, a bird's-eye view feature extracted by the method of this invention, and a bird's-eye view of the ground truth point cloud. It can be seen that the bird's-eye view feature extracted by the method of this invention is closest to the bird's-eye view of the ground truth point cloud, and contains relevant information about the location of empty voxels. However, the voxel features extracted by sparse convolution are still relatively sparse. Its bird's-eye view is closer to the bird's-eye view of the input sparse point cloud, but it lacks exploration of the location features of empty voxels.
[0065] (2) Feature enhancement: First, for the features obtained in step 1_1 ,filter( ) out of it Non-empty voxel features are used as queries in subsequent attention modules and their shapes are reshaped. ), obtained from One query Compositional characteristics , express The median is a real number. It is the feature dimension:
[0066]
[0067] For query Its coordinates are denoted as The location of the multi-scale bird's-eye view features is The feature point at that location is the corresponding reference point, with coordinates as follows: Sampling from around the reference point Several points were then used, followed by the use of a deformable cross-attention module ( Update the query based on the characteristics of these points to obtain... :
[0068]
[0069] in, This indicates a deformable attention operation. These are learnable attention weights, based on get, These are learnable weights. This refers to the coordinate offset of the sampling points around the reference point relative to the reference. These correspond to the features of these sampling points, obtained by using bilinear interpolation on the bird's-eye view features. (The preceding text appears to be incomplete and requires further context.) In One query Use the corresponding Replacement yields enhanced voxel features. , express The eigenvalues in the are real numbers. express The feature resolution, It is the feature dimension.
[0070] Step 1_3: Construct the feature adaptive completion module. The module structure diagram is as follows: Figure 3 As shown, voxel features are adaptively supplemented using a deformable self-attention module. The specific implementation process is as follows: for the enhanced voxel features obtained in step 1_2... Filter out the hollow voxel features and use learnable vectors. Assignment, express The value in the middle is a real number, and a new voxel feature is obtained, denoted as . , express The eigenvalues in the are real numbers. express The feature resolution, It is the feature dimension. Each voxel feature within it... These are all queries that need to be updated in the subsequent deformable self-attention module. The queries that need to be updated are in... The position coordinates in the middle are as follows That is, the coordinates of the reference point, sampled around the reference point. 1 point, then use a deformable self-attention module ( Update based on the characteristics of these points To obtain updated features , Completed voxel features:
[0071]
[0072] in, This indicates a deformable attention operation. These are learnable attention weights, based on get, These are learnable weights. This refers to the coordinate offset of the sampling points around the reference point relative to the reference. These correspond to the features of these sampling points.
[0073] Step 1_4: Input the features corresponding to the sampling points into the symbolic distance function value generation module, which is composed of a multilayer perceptron structure with sinusoidal activation. Generate the corresponding symbolic distance function values and train the model using the appropriate loss function. Specifically, the symbolic distance function values for different regions in the scene have different constraints. Therefore, sampling points are generated from different regions of the scene. Features are composed of point-level features and position encoding splicing ( ) and formed, Voxel characteristics Points obtained using trilinear interpolation The coordinates are Corresponding position code It can be represented as:
[0074]
[0075] in, It is a position function, with For parameters, The setting is 10, explained in detail below:
[0076]
[0077] The sampled point features are input into the symbolic distance function value generation module to generate the corresponding symbolic distance function values. This module is a multilayer perceptron structure, including... Fully connected layers and In this method, there are sinusoidal activation functions. Set it to 4. Define the symbolic distance function of the scene as... The process of generating the symbolic distance function value can be represented as follows:
[0078]
[0079] in, Indicates the first Fully connected layer plus the first Layer sinusoidal activation function, Indicates the first The input of a fully connected layer, and They represent the first The weights and biases of fully connected layers and They represent the first The weights and biases of the fully connected layers. The symbolic distance function values of the sampled points are used to calculate the loss function corresponding to the symbolic distance function. The constraints on the symbolic distance function values for different regions are as follows:
[0080]
[0081]
[0082]
[0083]
[0084] in, Indicates the parameter Find the partial derivative. The symbolic distance function representing the scene. coordinates Mapping to high-dimensional shape embedding , Indicating the scene The normal vector at the point, To represent a function, in Use it as input to make its value far away from 0. Represents the entire spatial domain. The surface point representing the truth value, Using these constraints, the corresponding loss function is calculated using the following formula:
[0085]
[0086] This loss function, as part of the total loss function, is used to constrain the symbolic distance function of the sampling points. The regularization parameters are set to 3000, 100, 100, and 50 in this method. This indicates the calculation of the magnitude of a vector. Calculate the cosine similarity between two vectors.
[0087] The total loss function is defined as:
[0088]
[0089] in, These represent the regularization parameters, which are set to 100 and 50 in this method. The loss is the symbolic distance function mentioned above. It is used in computing sparse convolutions The loss function value between the voxel occupancy probability predicted by the layer decoder and the occupancy label is used to remove redundant parts. Set it to 5. In the stage used for bird's-eye view feature extraction, the loss function value between the normalized output of the single-channel multi-scale bird's-eye view prediction and the corresponding single-channel binarized ground truth bird's-eye view is calculated. and Both use binary cross-entropy ( The loss function, specifically the formula, is as follows:
[0090]
[0091] It is the predicted voxel occupancy probability. This is the corresponding occupied tag.
[0092]
[0093] It is the normalized output of single-channel, multi-scale bird's-eye view prediction. It corresponds to the ground truth value of the single-channel binarized bird's-eye view. This indicates the number of feature scales in the bird's-eye view, which is set to 3 in this method.
[0094] Calculate the loss function value, and use gradient descent to update the network parameters in the implicit scene completion model until the model converges, thus obtaining the trained implicit scene completion model.
[0095] The specific steps of the testing phase process are as follows:
[0096] Step 2_1: Voxelize the point cloud of the scene to be completed, and process it sequentially using the feature extraction module, bird's-eye view-assisted enhancement module, and feature adaptive completion module in the trained implicit scene completion model to obtain the completed voxel features. .
[0097] Step 2_2: Sampling points according to the initial voxelized point cloud resolution, for Point-level features are obtained using trilinear interpolation, concatenated with the corresponding positional codes, and then input into the symbolic distance function value generation module to generate the corresponding symbolic distance function value, thus obtaining the complete implicit representation of the entire scene.
[0098] To further verify the feasibility and effectiveness of the method of this invention, the technical effects of this invention are further described in conjunction with experiments. The hardware platform used in the experiments of this invention is: Intel(R) Xeon(R) Gold 6230 CPU @2.10GHz×16, 64G of memory, and two GPUs with 24G of video memory each, model NVIDIA TITAN RTX. The software platform used in the experiments of this invention is: Ubuntu 18.04.6 LTS operating system and Python 3.8.17 and PyTorch 1.8.0 deep learning framework. The validation sets of the SemanticKITTI dataset and the SemanticPOSS dataset are used to analyze the scene completion effect obtained by this method. Here, the mean of the Interactions over Union (IoU) is used as the evaluation index to evaluate the performance of implicit scene completion.
[0099] The method of this invention is used to complete the point cloud of each incomplete scene in the validation sets of the SemanticKITTI dataset and the SemanticPOSS dataset, and the completion result corresponding to each scene is obtained. To test the performance of the method of this invention, the proposed implicit scene completion method based on dense voxel representation is compared with several existing methods. The comparison methods include: the implicit neural representation method SIREN proposed by Vincent Sitzmann et al. in their paper "Implicit Neural Representations with Periodic Activation Functions, NeurIPS, pp. 7462–7473, 2020"; the implicit neural representation method Fourier Features proposed by Matthew Tancik et al. in their paper "Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS, pp. 7537–7547, 2020"; the implicit neural representation method BACON proposed by David B Lindell et al. in their paper "BACON: Band-limited Coordinate Networks for Multiscale Scene Representation, CVPR, 2022, pp. 16252–16262"; and Yizhak et al. Ben-Shabat et al. proposed the implicit neural representation method DiGS in their paper "DiGS: Divergence guided shape implicit neural representation for unoriented point clouds, CVPR, 2022, pp. 19323–19332." Pengfei Li et al. proposed the implicit scene completion method LODE in their paper "LODE: Locally Conditioned Eikonal Implicit Scene Completion from Sparse LiDAR, ICRA, pp. 8269–8276, 2023." Quantitative results are shown in Table 1. The intersection-union ratio (IUU) was used as the evaluation criterion, with values expressed as percentages; a higher value is better.
[0100] Table 1. Quantitative test results on the validation set using the method of the present invention and existing technologies.
[0101] method SemanticKITTI SemanticPOSS SIREN 26.3 36.0 Fourier Features 28.6 30.9 BACON 30.5 40.5 DiGS 31.7 37.4 LODE 51.2 48.7 Method of the present invention 51.9 50.2
[0102] As can be seen from the results in Table 1, the test results obtained by the method of the present invention on both datasets are higher than those of existing methods, indicating that the present invention can obtain a completed point cloud that is more similar to the ground truth point cloud. Qualitative test results are as follows: Figure 5 As shown, the completion results of the method of the present invention are more accurate than those of existing methods.
[0103] The above embodiments should be considered as exemplary and non-limiting, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.
Claims
1. An implicit scene completion method based on dense voxel representation, comprising two processes: a training phase and a testing phase, characterized in that: The training phase includes the following steps: Step 1: Construct a feature extraction module to extract non-empty voxel features from the input sparse point cloud; Step 2: Construct a bird's-eye view enhancement module to enhance the non-empty voxel features extracted in Step 1 using dense bird's-eye view features. Step 3: Construct a feature adaptive completion module composed of deformable self-attention modules to adaptively complete all elemental features; Step 4: Construct a symbolic distance function value generation module to generate symbolic distance function values corresponding to the sampling points, calculate the corresponding loss function values, and train the implicit scene completion model; During the testing phase, the point cloud to be completed is input into the trained implicit scene completion model to obtain the implicit representation of the completed scene point cloud. The testing phase includes the following steps: Step 1: Process the point cloud of the scene to be completed using the trained implicit scene completion model, and extract the voxel features after completion. ; Step 2: Voxelize the point cloud of the scene to be completed. Treat each voxel in the voxelized scene representation as a point, extract the corresponding point-level features, and generate the corresponding symbolic distance function value using the symbolic distance function value generation module.
2. The implicit scene completion method based on dense voxel representation as described in claim 1, characterized in that, In step one of the training phase, the original input sparse point cloud is converted into voxel form, and the sparse voxel features are extracted using the feature extraction module. The feature extraction module consists of sparse convolutions. express The eigenvalues in the are real numbers. Indicates the resolution of the extracted features. It is the feature dimension.
3. The implicit scene completion method based on dense voxel representation as described in claim 1, characterized in that, In step two of the training phase, a bird's-eye view augmentation module is constructed, which includes two parts: bird's-eye view feature extraction and feature augmentation. (1) Bird's-eye view feature extraction: This part takes sparse point cloud as input and outputs... Bird's-eye view features at various scales The specific implementation process is as follows: The input point cloud is converted into voxel form, and then max pooling is performed along the z-axis to obtain the initial bird's-eye view features, denoted as... Finally Input a 2D UNet structure for processing, output ; (2) Feature enhancement: First, the features obtained in step one of the training phase are... Filter out the ones The non-empty voxel features are used as queries in the subsequent attention module, and their shapes are reshaped to obtain the result. One query Compositional characteristics , express The median is a real number. It is the feature dimension: This indicates a filtering operation. This indicates a shape reshaping operation, for queries. Its coordinates are denoted as The location of the multi-scale bird's-eye view features is The feature point at that location is the corresponding reference point, with coordinates as follows: Sampling from around the reference point Then, a deformable cross-attention module is used to update the query based on the features of these points, resulting in... : in, This represents a deformable cross-attention module. This indicates a deformable attention operation. These are learnable attention weights, based on get, These are learnable weights. This refers to the coordinate offset of the sampling points around the reference point relative to the reference. These correspond to the features of these sampling points, obtained by using bilinear interpolation on the bird's-eye view features, and then... In One query Use the corresponding Replacement yields enhanced voxel features. , express The eigenvalues in the are real numbers. express The feature resolution, It is the feature dimension.
4. The implicit scene completion method based on dense voxel representation as described in claim 1, characterized in that, In step three of the training phase, a feature adaptive completion module is constructed, which adaptively completes all geographic features through a deformable self-attention module; the specific implementation process is as follows: For the enhanced voxel features obtained in step two of the training phase Filter out the hollow voxel features and use learnable vectors. Assignment, express The value in the middle is a real number, and a new voxel feature is obtained, denoted as . , express The eigenvalues in the are real numbers. express The feature resolution, It is the feature dimension, where each voxel has features. These are all queries that need to be updated in the subsequent deformable self-attention module. The queries that need to be updated are... The position coordinates in the middle are as follows That is, the coordinates of the reference point, sampled around the reference point. There are 10 points, and then a deformable self-attention module is used to update the value based on the features of these points. To obtain updated features , Completed voxel features: in, This represents a deformable self-attention module. This indicates a deformable attention operation. These are learnable attention weights, based on get, These are learnable weights. This refers to the coordinate offset of the sampling points around the reference point relative to the reference. These correspond to the features of these sampling points.
5. The implicit scene completion method based on dense voxel representation as described in claim 1, characterized in that, In step four of the training phase, the features corresponding to the sampling points are input into the symbolic distance function value generation module, which is composed of a multilayer perceptron structure with sinusoidal activation, to generate the corresponding symbolic distance function value, and the model is trained using the corresponding loss function. Sampling points from different areas of the scene. Features are composed of point-level features and position encoding It is pieced together. Voxel characteristics Points obtained using trilinear interpolation The coordinates are Corresponding position code Represented as: in, It is a position function, with For parameters, For hyperparameters; The sampled point features are input into the symbolic distance function value generation module to generate the corresponding symbolic distance function values. This module is a multilayer perceptron structure, including... Fully connected layers and A sinusoidal activation function is used to define the symbolic distance function of the scene as follows: The process of generating the symbolic distance function value can be represented as follows: in, This indicates a splicing operation. Indicates the first Fully connected layer plus the first Layer sinusoidal activation function, Indicates the first The input of a fully connected layer, and They represent the first The weights and biases of fully connected layers and They represent the first The weights and biases of the fully connected layers are calculated; the symbolic distance function values of the sampled points are used to calculate the loss function corresponding to the symbolic distance function. The constraints on the symbolic distance function values for different regions are as follows: in, Indicates the parameter Find the partial derivative. The symbolic distance function representing the scene. coordinates Mapping to high-dimensional shape embedding , Indicating the scene The normal vector at the point, To represent a function, in For the input, make its value far away from 0. Represents the entire spatial domain. The surface point representing the truth value, Using these constraints, the corresponding loss function is calculated using the following formula: This loss function, as part of the total loss function, is used to constrain the symbolic distance function of the sampling points. Represents the regularization parameter. This indicates the calculation of the magnitude of a vector. Calculate the cosine similarity between two vectors; The total loss function is defined as: in, Represents the regularization parameter. The loss is the aforementioned symbolic distance function. It is used in calculating sparse convolution. The loss function value between the voxel occupancy probability predicted by the layer decoder and the occupancy label is used to remove redundant parts; In the stage of bird's-eye view feature extraction, the loss function value between the normalized output of the single-channel multi-scale bird's-eye view prediction and the corresponding single-channel binarized bird's-eye view ground truth is calculated. and Both use binary cross-entropy ( The loss function, specifically the formula, is as follows: It is the predicted voxel occupancy probability. For the corresponding occupied label; It is the normalized output of single-channel, multi-scale bird's-eye view prediction. It corresponds to the ground truth value of the single-channel binarized bird's-eye view. Indicates the number of feature scales in a bird's-eye view; Calculate the loss function value, and use gradient descent to update the network parameters in the implicit scene completion model until the model converges, thus obtaining the trained implicit scene completion model.
6. The implicit scene completion method based on dense voxel representation as described in claim 1, characterized in that: In step one of the testing phase, the voxelized point cloud of the scene to be completed is processed sequentially using the feature extraction module, the bird's-eye view-assisted enhancement module, and the feature adaptive completion module from the trained implicit scene completion model to obtain the completed voxel features. .
7. The implicit scene completion method based on dense voxel representation as described in claim 1, characterized in that: In step two of the testing phase, sampling points are taken according to the initial voxelized point cloud resolution. Point-level features are obtained using trilinear interpolation, concatenated with the corresponding positional codes, and then input into the symbolic distance function value generation module to generate the corresponding symbolic distance function value, thus obtaining the complete implicit representation of the entire scene.