A visual three-dimensional semantic occupancy prediction method based on foreground feature quantization

Abstract

The application discloses a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization. A vector quantization variational auto-encoding network is constructed based on foreground semantic occupancy labeling, a BEV vector quantization constraint based on Sinkhorn optimal transport is applied on multi-scale foreground hidden features, multi-scale foreground feature codebooks and corresponding BEV foreground quantization feature maps are learned, and codebook utilization is improved. A three-dimensional semantic occupancy prediction network taking multi-view pure visual two-dimensional images as input is constructed, a BEV initial feature is obtained by using a backbone network and feature view conversion, multi-scale foreground feature extraction is performed in the foreground feature quantization branch, and vector quantization is performed on the foreground codebook obtained by combining the variational auto-encoding network, global continuous features are extracted in the global continuous feature branch, and the three-dimensional semantic occupancy prediction result of the scene is output through feature aggregation and three-dimensional voxel decoding, so that the three-dimensional occupancy representation accuracy of the foreground target in a complex road environment is improved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving perception and computer vision technology, and in particular to a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization. Background Technology

[0002] 3D semantic occupancy prediction, used to characterize the occupancy state and semantic category of an environment within a unified voxel space, is a key issue in autonomous driving and environmental perception. Existing methods are mostly based on multi-view camera images, extracting image features through a 2D visual backbone network, and then using feature transformation methods such as LSS to map the multi-view features onto the BEV bird's-eye view space. 3D semantic occupancy prediction is then trained on the BEV bird's-eye view space, achieving some success.

[0003] However, in real-world road scenes, background areas such as road surfaces, buildings, and vegetation occupy the majority of the image and 3D voxels, while foreground targets such as vehicles, pedestrians, non-motorized vehicles, traffic cones, and crash barriers are small and sparsely distributed. Existing methods mostly model and supervise learning the entire scene on a unified feature space, and the loss function is also mainly based on all voxels. This causes the network to tend to prioritize fitting background class features that appear more frequently, and the response of foreground targets is easily overwhelmed by large areas of background features during BEV aggregation and subsequent feature extraction. Even if methods such as class reweighting are introduced to alleviate the class imbalance problem, because the foreground and background still share the same continuous feature representation space, the semantic features of the foreground are often not prominent enough. The prediction accuracy of the 3D semantic occupancy of small targets, distant targets, and occluded targets is still limited, and it is easy to have blurred foreground target boundaries and missed detections. Summary of the Invention

[0004] The purpose of this invention is to address the problem in existing visual 3D semantic occupancy prediction technology that foreground targets are easily submerged by the background, resulting in insufficient prediction accuracy for small targets and distant targets. This invention provides a visual 3D semantic occupancy prediction method based on foreground feature quantization, which highlights the semantic representation of the foreground and improves the 3D occupancy prediction performance of targets.

[0005] The objective of this invention is achieved through the following technical solution: a visual 3D semantic occupancy prediction method based on foreground feature quantization, comprising: Based on the foreground dataset, a vector quantization variational autoencoder network is constructed and trained. The vector quantization variational autoencoder network reconstructs the foreground semantic occupancy result and performs vector quantization on the foreground latent features at each scale to obtain the foreground feature codebook and the BEV foreground quantization feature map. A deep learning network is constructed that takes multi-view pure visual 2D images as input and outputs 3D semantic occupancy results. The foreground quantization feature map of the BEV bird's-eye view is added to the supervision signal to train the network. The network includes: extracting 2D image features and discrete depth estimation from each viewpoint, mapping and aggregating the 2D image features from each viewpoint to a unified BEV bird's-eye view space, using the feature pyramid module in the network to extract continuous foreground features and global continuous features from the initial features of the BEV bird's-eye view, quantizing the continuous foreground features based on the foreground feature codebook, and outputting semantic occupancy prediction values ​​through a 3D voxel decoding network and a fully connected layer. Using a trained deep learning network, predictions are made on multi-view pure visual 2D images.

[0006] Furthermore, the process of constructing the foreground dataset includes: performing label transformation on the training dataset with three-dimensional semantic occupancy annotations of all types, keeping the foreground categories unchanged, merging the background category and the empty class into an empty class, and obtaining foreground semantic occupancy annotations that only include the foreground categories and the empty class.

[0007] Furthermore, the vector quantization variational autoencoder network includes a three-dimensional encoder, a two-dimensional encoder, a vector quantization module based on Sinkhorn optimal transmission, an upsampling module, and a two-dimensional decoder.

[0008] Foreground semantic occupancy annotations are used as network inputs and are encoded sequentially by a 3D encoder and a 2D encoder. The BEV continuous feature maps of different scales output by the two encoders are input into a vector quantization module based on Sinkhorn optimal transmission to obtain foreground quantized feature maps and foreground feature codebooks of different scales. Then, the foreground quantized feature maps of each scale are spliced ​​to the same spatial resolution by an upsampling module, spliced ​​and fused, and then decoded by a 2D decoder module to obtain the foreground 3D semantic occupancy prediction value.

[0009] Furthermore, the calculation process of the vector quantization module based on Sinkhorn optimal transmission includes: First, BEV feature flattening and cost matrix construction are performed. The mesoscale BEV continuous feature map obtained from the 3D encoder and the large-scale BEV continuous feature map obtained from the 2D encoder are flattened into a set of feature vectors according to their spatial location. , A learnable codebook is set up for each scale. , Calculate the Euclidean distance between the latent feature vector and the code vector to form the cost matrix: Based on cost matrix and parameters Construct the initial allocation matrix : right Perform at least five rounds of Sinkhorn iterations with alternating row and column normalization to obtain an approximately optimal transfer matrix that satisfies the row and column constraints. This is essentially a vector quantization constraint based on Sinkhorn optimal transfer, and the allocation matrix for each scale. , record The matrix at the next iteration is The specific form of each round of alternating normalization is as follows: After multiple iterations, the sum of the weights assigned to each latent feature vector across all code vectors approaches 1; the sum of the weights assigned to each code vector across all latent features approaches the predetermined total weight, thus yielding an approximately optimal transfer matrix that satisfies the row and column constraints. The correspondence between the foreground latent feature vector and the code vector is determined by using the position of the maximum value of each element in the optimal transmission matrix, that is, for the first row... Each foreground latent feature vector is selected from its corresponding row. Take the column index corresponding to the maximum value The code vector is used as the matching code vector.

[0010] Furthermore, during the training process of the vector quantization variational autoencoder network, a one-stage total loss function is employed. End-to-end optimization of the vector quantization variational autoencoder network includes: The first-scale vector quantization loss is obtained by performing vector quantization on the two-scale foreground latent features output by the encoder. Second-scale vector quantization loss And the mean square error loss based on the reconstruction error of the foreground 3D voxel occupancy result. Thus we get: in, , , These are the weighting coefficients for each loss term; First-scale vector quantization loss Second-scale vector quantization loss For the first-scale foreground latent feature set respectively The set of code vectors in the first-scale foreground feature codebook and the second-scale foreground latent feature set The set of code vectors in the second-scale foreground feature codebook Define, where, the first The latent feature vectors and their matching code vectors are denoted as follows: , Then we have: in, Represents the L2 norm. This indicates that the gradient operation is stopped. These are the coefficients used to balance the codebook update terms and commitment terms; Mean square error loss The voxel-level reconstruction error used to measure the foreground 3D semantic occupancy prediction result and the foreground semantic occupancy annotation is as follows: in, This represents the total number of voxels in the training samples. For the first The semantic occupancy annotation values ​​of individual elements are presented in one-hot vector annotation form. This is the corresponding predicted probability vector.

[0011] Furthermore, the deep learning network structure, in which the input is a multi-view pure visual two-dimensional image and the output is a three-dimensional semantic occupancy result, includes: a two-dimensional feature extraction module with ResNet-50 as the backbone network, used to extract multi-scale two-dimensional semantic features and discrete depth estimation; The LSS-based feature transformation module is used to combine the two-dimensional features of each viewpoint with the corresponding discrete depth estimate to obtain the initial features of the BEV bird's-eye view. A feature pyramid network containing global continuous feature branches and foreground feature quantization branches is used to extract continuous foreground features and global continuous features from the initial features of the BEV bird's-eye view, and to quantize the continuous foreground features based on the foreground feature codebook to obtain foreground quantized features. The feature aggregation module aggregates foreground quantization features and global continuous features, inputs them into the 3D voxel decoding network, and through 3D convolution and upsampling operations, maps them into a 3D voxel feature representation containing height dimensions. Finally, it is input into the fully connected layer and outputs the predicted probability distribution of the voxel belonging to each semantic category.

[0012] Furthermore, the feature pyramid network comprising global continuous feature branches and foreground feature quantization branches specifically includes: Initial features The feature pyramid network is fed as input. This is used to construct multi-scale BEV feature representations. The feature pyramid network fuses features from different resolution levels through top-down progressive upsampling and lateral connections, ultimately outputting global feature maps at three spatial resolutions: large-scale, medium-scale, and small-scale. , , : in, , , Together as global continuous features, they converge in the global continuous feature branches to form... same resolution It is used to characterize global geometric structure and semantic context information at multiple scales; For the foreground feature quantization branch, a mesoscale feature map is selected from the three-scale global feature map. and large-scale feature maps As input, they are fed into the foreground feature convolutional units of the corresponding scales. and The foreground feature convolutional unit consists of several layers of two-dimensional convolution, normalization, and ReLU nonlinear activation units. It is used to mine foreground-related patterns and suppress background interference to obtain continuous foreground features at medium and large scales. Subsequently, using a multi-scale foreground feature codebook, the continuous foreground features at the two scales are vector-quantized respectively. Specifically, the feature maps are... and Flattened by spatial location, the foreground latent feature set is obtained, and Euclidean distance is calculated between it and the code vector set in the corresponding scale foreground feature codebook. A quantization strategy based on Sinkhorn optimal transmission is adopted to obtain the BEV foreground quantized feature maps at medium and large scales. and .

[0013] Furthermore, the step of adding the foreground quantized feature map of the BEV bird's-eye view to the supervision signal to train the network includes: Predicting Focal Loss by Occupation Discrete depth estimation loss based on lidar point cloud supervision and prospect quantification consistency loss The two-stage total loss function is constructed by the weighted sum of the following: in, , , These are the weighting coefficients for each loss term. Focal Loss for Occupation Prediction : in voxel number For the number of semantic categories, For the first Individual elements belong to the first One-hot annotation for classes, For the corresponding predicted probability, For category weights, As a focusing factor; For discrete depth estimation loss : in The effective number of pixels for all viewpoints. The depth is the number of discrete layers. For the first The pixel in the first Discrete depth labels on each depth layer For the corresponding predicted probability; For prospect quantification consistency loss : in, , These are the BEV foreground feature maps predicted by the network at medium and large scales. , This is a foreground quantization feature map of a BEV bird's-eye view obtained by training a vector quantization variational autoencoder network based on labels.

[0014] According to another aspect of the specification, a visual three-dimensional semantic occupancy prediction device based on foreground feature quantization is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the visual three-dimensional semantic occupancy prediction method based on foreground feature quantization.

[0015] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned visual three-dimensional semantic occupancy prediction method based on foreground feature quantization.

[0016] The beneficial effects of this invention are: 1. By reconstructing foreground labels and quantizing BEV vectors based on Sinkhorn, a multi-scale foreground feature codebook is learned to decouple foreground and background representations, highlighting the 3D semantic occupancy features of small targets and occluded targets.

[0017] 2. Compared with the original VQ-VAE code vector nearest neighbor allocation, the Sinkhorn-based vector quantization process can effectively improve the codebook utilization during training and alleviate the codebook collapse problem, and can realize the quantization process of BEV feature maps more efficiently.

[0018] 3. A two-stage training framework is adopted. For foreground categories with relatively regular shapes and distribution patterns and high repetition, a discrete foreground feature codebook is introduced to obtain a stable and compact cluster representation. Meanwhile, continuous feature representation is retained for background regions with varied morphologies. This achieves complementarity between discrete foreground quantization and global continuous modeling, thereby improving the accuracy of BEV representation and the robustness of foreground target prediction.

[0019] 4. By jointly optimizing the occupancy prediction Focal Loss, discrete depth estimation loss and foreground quantization consistency loss, the problems of foreground being submerged by background, blurred boundaries and missed detection are mitigated under the condition of relying only on multi-view pure visual input. Attached Figure Description

[0020] Figure 1 A flowchart of a visual 3D semantic occupancy prediction method based on foreground feature quantization is provided in an embodiment of the present invention. Figure 2 A block diagram illustrating the implementation of a vector quantization variational autoencoder network based on foreground features in the first stage of an embodiment of the present invention. Figure 3 A block diagram illustrating the implementation of the second-stage three-dimensional semantic occupancy prediction network provided in this embodiment of the invention; Figure 4 Visual examples of the reasoning process provided in embodiments of the present invention; Figure 5 This is a schematic diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown, this embodiment of the invention provides a visual 3D semantic occupancy prediction method based on foreground feature quantization, including: Step 1, the first stage of training and foreground codebook construction, corresponding to... Figure 1 The S101 and S102 processes mainly include four steps: 1.1 The training dataset with full-category 3D semantic occupancy annotations is labeled, and the foreground categories remain unchanged. The background category and the empty class are merged into an empty class, resulting in foreground semantic occupancy annotations that only include the foreground categories and the empty class. 1.2 Based on the foreground semantic occupancy annotation, a vector quantization variational autoencoder network is constructed to reconstruct the foreground semantic occupancy result. On the two-scale foreground latent features output by the encoder, a BEV vector quantization constraint based on Sinkhorn optimal transmission is introduced to perform vector quantization on the foreground latent features at each scale, thereby obtaining the corresponding two foreground feature codebooks and the two-scale BEV foreground quantization feature maps. 1.3 The BEV foreground quantization feature maps of the two scales are decoded and input into a multilayer perceptron to output the corresponding foreground 3D semantic occupancy prediction value. 1.4 In the first stage of training, the total loss is the weighted sum of the reconstruction loss of the foreground 3D voxel occupancy result and the vector quantization loss. The parameters of the vector quantization variational autoencoder network and the foreground feature codebook are jointly optimized.

[0023] The specific implementation details are as follows: Label transformation is performed on the training dataset with full-category 3D semantic occupancy annotations. The foreground categories remain unchanged, while the background category and the empty category are merged into an empty category, resulting in foreground semantic occupancy annotations that only include the foreground categories and the empty category.

[0024] For example, for the various class labels of the Occ3D-nuScenes dataset, the following classification is performed for label transformation: Table 1 Tag Conversion Table The resulting foreground semantic occupancy labeled voxel map is then used as the input and supervision signal for a Vector Quantization Variational Autoencoder (VQ-VAE). The network is as follows: Figure 2 As shown, it mainly consists of a 3D encoder, a 2D encoder, an upsampling module, a 2D decoder, and a vector quantization module based on Sinkhorn optimal transmission. The specific process is as follows: a) First, the foreground semantic occupancy annotation voxels are used as network input and fed into the 3D encoder. The 3D encoder consists of 8 layers of 3D convolutional, normalized, and non-linear activation units (ReLU). It performs feature extraction and spatial downsampling on the occupancy map to obtain a 3D foreground feature representation, and then aggregates and flattens it in the height dimension to map it onto the BEV plane, resulting in a mesoscale foreground BEV continuous feature map. ).

[0025] b) Input the feature map into the 2D encoder. The 2D encoder uses a 4-layer 2D convolutional, normalized, and non-linear activation unit ReLU structure, and outputs a large-scale foreground BEV continuous feature map after the 2D encoder. ).

[0026] c) To address the continuous characteristics of BEV at the two scales mentioned above, the following is introduced: Figure 2 The vector quantization module based on Sinkhorn optimal transmission shown learnable codebook 1 and learnable codebook 2 respectively, and obtains the corresponding BEV quantization features by combining the codebooks and BEV continuous features.

[0027] Specifically, the implementation process of the BEV vector quantization constraint based on Sinkhorn optimal transmission mainly focuses on the optimal transmission matrix. The solution process will be explained, including: First, BEV feature flattening and cost matrix construction are performed. The continuous feature maps of medium-scale and large-scale BEVs are flattened according to their spatial location into a set of feature vectors. , A learnable codebook is set up for each scale. , Calculate the Euclidean distance between the latent feature vector and the code vector to form the cost matrix: Based on cost matrix and parameters Construct the initial allocation matrix : right Perform at least five rounds of Sinkhorn iterations with alternating row and column normalization to obtain an approximately optimal transfer matrix that satisfies the row and column constraints. This is essentially a vector quantization constraint based on Sinkhorn optimal transfer, and the allocation matrix for each scale. , record The matrix at the next iteration is The specific form of each round of alternating normalization is as follows: After multiple iterations, the sum of the weights assigned to each latent feature vector across all code vectors approaches 1 (row normalization), meaning each latent feature is "fully allocated" overall; the sum of the weights assigned to each code vector across all latent features approaches a predetermined total (column normalization), meaning the usage frequency among different code vectors tends to be balanced, avoiding individual code vectors being idle for a long time or overused. This yields an approximately optimal transfer matrix that satisfies the row and column constraints. Compared to the original VQ-VAE's nearest neighbor code vector allocation, this process effectively improves codebook utilization during training and alleviates the codebook collapse problem. The correspondence between the foreground latent feature vector and the code vector is determined by using the position of the maximum value of each element in the optimal transfer matrix; that is, for the ... Each foreground latent feature vector is selected from its corresponding row. Take the column index corresponding to the maximum value The code vector is used as the matching code vector. Through this strategy, each original latent feature vector is precisely discretized and mapped to a specific code vector in the codebook, thus forming a BEV foreground quantization feature map with discrete and quantized characteristics.

[0028] d) The large-scale BEV foreground quantization feature map is upsampled to a spatial resolution consistent with the mesoscale using an upsampling module, and then fused with the mesoscale BEV foreground quantization feature map to form a multi-scale foreground BEV quantization feature. For example... Figure 2 As shown, the multi-scale foreground BEV quantization features are then input into a two-dimensional decoder, and through multi-layer two-dimensional convolution and upsampling, the resolution is restored to match the input foreground semantic occupancy annotation voxel map. Finally, the height dimension is restored by a multi-layer perceptron and the predicted value of foreground three-dimensional semantic occupancy is output.

[0029] During training, a one-stage total loss function is used. End-to-end optimization of the vector quantization variational autoencoder network includes: The first-scale vector quantization loss is obtained by performing vector quantization on the two-scale foreground latent features output by the encoder. Second-scale vector quantization loss And the mean square error loss based on the reconstruction error of the foreground 3D voxel occupancy result. Thus we get: in, , , These are the weighting coefficients for each loss term.

[0030] First-scale vector quantization loss Second-scale vector quantization loss For the first-scale foreground latent feature set respectively The set of code vectors in the first-scale foreground feature codebook and the second-scale foreground latent feature set The set of code vectors in the second-scale foreground feature codebook Define, where, the first The latent feature vectors and their matching code vectors are denoted as follows: , Then we have: in, Represents the L2 norm. This indicates that the gradient (stop-gradient) operation is stopped. The coefficients used to balance the "codebook update term" and the "commitment term".

[0031] Mean square error loss The voxel-level reconstruction error used to measure the foreground 3D semantic occupancy prediction result and the foreground semantic occupancy annotation is as follows: in, This represents the total number of voxels in the training samples. For the first The semantic occupancy annotation values ​​of individual elements are presented in one-hot vector annotation form. This is the corresponding predicted probability vector.

[0032] Step two, the second stage of training for 3D semantic occupancy prediction based on multi-view pure visual input, corresponding to Figure 1 The S201, S202, and S203 processes mainly include five steps: 2.1 Construct a deep learning network that takes multi-view pure visual 2D images as input and 3D semantic occupancy results as output, and use the 2D image backbone network ResNet-50 to extract 2D image features and discrete depth estimation from each viewpoint; 2.2 The LSS (Lift-Splat-Shoot) feature transformation method is used to map and aggregate the features of the two-dimensional images from each viewpoint into a unified BEV bird's-eye view space to obtain the initial features of the BEV bird's-eye view. 2.3 After the initial features of the BEV bird's-eye view are processed through the feature pyramid network, they are divided into a foreground feature quantization branch and a global continuous feature branch. A multi-scale feature extraction structure is used to extract continuous foreground features at two scales in the foreground feature quantization branch. Vector quantization is performed using the foreground feature codebook obtained in step one, and the two-scale BEV bird's-eye view foreground quantization feature maps obtained in step one are used as one of the supervision signals. 2.4 The global continuous feature branch uses a multi-scale feature extraction structure to extract global continuous features at three scales; 2.5 Aggregate foreground quantization features and global continuous features, output semantic occupancy prediction values ​​through a 3D voxel decoding network and a fully connected layer, and use full-type 3D semantic occupancy annotations to supervise the training of the deep learning network; The specific implementation details are as follows: First, such as Figure 3 As shown in the network structure diagram, for each frame of the original 2D image acquired simultaneously from 6 perspectives, the 6 perspective images are respectively input into the 2D feature extraction module with ResNet-50 as the backbone network. The ResNet-50 removes the final classification layer, retaining only the convolutional layers and residual blocks, which are used to extract multi-scale 2D semantic features. At the same time, during the feature extraction process, a discrete depth estimation branch is combined to predict the probability distribution of each pixel on a predefined depth layer, which is used for subsequent 3D projection.

[0033] The Lift-Splat-Shoot (LSS) feature transformation method is employed to combine the two-dimensional features of each viewpoint with the corresponding discrete depth estimates, achieving uplift from the image plane to a three-dimensional voxel space. In this process, the discrete depth is represented as... The pixel coordinates on each discrete depth layer are... Image features are back-projected from the image plane to three-dimensional voxels using camera intrinsic and extrinsic parameters. : in, For the camera intrinsic parameter matrix, and The features obtained from projections at various viewpoints and depth levels on a unified BEV bird's-eye view plane are accumulated and pooled to form the initial features of the BEV bird's-eye view. .

[0034] Initial features The feature pyramid network is fed as input. This is used to construct multi-scale BEV feature representations. The feature pyramid network fuses features from different resolution levels through top-down progressive upsampling and lateral connections, ultimately outputting global feature maps at three spatial resolutions: large-scale, medium-scale, and small-scale. in, , , Together as global continuous features, such as Figure 3 As shown, it aggregates into global continuous feature branches with same resolution It is used to characterize global geometric structure and semantic context information at multiple scales.

[0035] like Figure 3 As shown, for the foreground feature quantization branch, a mesoscale feature map is selected from the three-scale global feature map. and large-scale feature maps As input, they are fed into the foreground feature convolutional units of the corresponding scales. and The foreground feature convolutional unit consists of several layers of two-dimensional convolution, normalization, and ReLU nonlinear activation units. It is used to further mine foreground-related patterns and suppress background interference, thereby obtaining continuous foreground features at both medium and large scales. Subsequently, using the multi-scale foreground feature codebook learned in step one, vector quantization is performed on the continuous foreground features at the two scales mentioned above. Specifically, the feature maps are... and Flattened by spatial location, the foreground latent feature set is obtained, and Euclidean distance is calculated between it and the code vector set in the corresponding scale foreground feature codebook. Using the same Sinkhorn-based optimal transmission quantization strategy as in the first stage, the BEV foreground quantization feature maps at medium and large scales are obtained. and It is important to note that the two foreground feature codes in this process do not undergo parameter updates based on gradient backpropagation.

[0036] After completing foreground feature quantization and global continuous feature extraction, feature aggregation is performed on both. Specifically, firstly, the large-scale foreground quantization features are aggregated. Upsampling Aligned to a mesoscale resolution, we get: Then, Mesoscale foreground quantification features The concatenation is performed along the channel dimension and then fed into the convolutional unit. The aggregated mesoscale foreground features are obtained as follows: Then, Align again with global continuous features through upsampling Same resolution: and with Channel splicing is performed, and the data is fed into the convolution unit. The final aggregated BEV bird's-eye view feature map is obtained: like Figure 3 As shown, the aggregated BEV bird's-eye view feature map Then input into the 3D voxel decoding network Through 3D convolution and upsampling operations, it is mapped to a 3D voxel feature representation containing height dimension: For each voxel position eigenvectors on A fully connected layer is applied to output the predicted probability distribution of the voxel belonging to each semantic category, thereby generating the 3D semantic occupancy prediction result for the entire scene.

[0037] This training phase uses full-category 3D semantic occupancy annotations as supervision signals, predicting Focal Loss through occupancy. Discrete depth estimation loss based on lidar point cloud supervision and prospect quantification consistency loss The two-stage total loss function is constructed by the weighted sum of the following: in, , , These are the weighting coefficients for each loss term. This represents the Focal Loss for occupancy prediction. : in voxel number For the number of semantic categories, For the first Individual elements belong to the first One-hot annotation for classes, For the corresponding predicted probability, For category weights, This is the focusing factor.

[0038] For discrete depth estimation loss : in This represents the total number of valid pixels across the six viewpoints. The depth is the number of discrete layers. For the first The pixel in the first Discrete depth labels on each depth layer This corresponds to the predicted probability.

[0039] For prospect quantification consistency loss : in, , These are the BEV foreground feature maps predicted by the second-stage network at both the mesoscale and large-scale levels. , The corresponding teacher quantification features obtained from the first stage of label-based training are used to supervise the BEV foreground feature map generated in the second stage.

[0040] Step 3, the three-dimensional semantic occupancy prediction reasoning process, corresponding to Figure 1 The S301 process mainly includes three steps: 3.1 Input the multi-view pure visual two-dimensional images of the scene to be predicted into the deep learning network trained in step two, and use LSS feature transformation to obtain the corresponding initial features of the BEV bird's-eye view. 3.2 Perform feature quantization operation in the foreground feature quantization branch to obtain foreground quantized features, and obtain global continuous features in the global continuous feature branch; 3.3 The foreground quantization features and global continuous features are aggregated. The aggregated BEV bird's-eye view features are input into the 3D voxel decoding network and the fully connected layer. The type with the highest prediction probability in each voxel is taken as the prediction type of this voxel. Finally, the 3D semantic occupancy prediction result of the whole scene is obtained.

[0041] This process does not require LiDAR information or the corresponding teacher quantitative features obtained from the first stage of label-based training. , The achieved visualization effect is as follows: Figure 4 As shown in the example.

[0042] Corresponding to the aforementioned embodiment of a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization, the present invention also provides an embodiment of a visual three-dimensional semantic occupancy prediction device based on foreground feature quantization.

[0043] See Figure 5 The present invention provides a visual three-dimensional semantic occupancy prediction device based on foreground feature quantization, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization in the above embodiment.

[0044] The embodiment of the visual 3D semantic occupancy prediction device based on foreground feature quantization provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a visual 3D semantic occupancy prediction device based on foreground feature quantization provided by the present invention. Except for... Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0045] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0046] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0047] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization as described in the above embodiments.

[0048] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0049] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned visual three-dimensional semantic occupancy prediction method based on foreground feature quantization.

[0050] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0051] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A visual 3D semantic occupancy prediction method based on foreground feature quantization, characterized in that, include: Based on the foreground dataset, a vector quantization variational autoencoder network is constructed and trained. The vector quantization variational autoencoder network reconstructs the foreground semantic occupancy result and performs vector quantization on the foreground latent features at each scale to obtain the foreground feature codebook and the BEV foreground quantization feature map. A deep learning network is constructed that takes multi-view pure visual 2D images as input and outputs 3D semantic occupancy results. The foreground quantization feature map of the BEV bird's-eye view is added to the supervision signal to train the network. The network includes: extracting 2D image features and discrete depth estimation from each viewpoint, mapping and aggregating the 2D image features from each viewpoint to a unified BEV bird's-eye view space, using the feature pyramid module in the network to extract continuous foreground features and global continuous features from the initial features of the BEV bird's-eye view, quantizing the continuous foreground features based on the foreground feature codebook, and outputting semantic occupancy prediction values ​​through a 3D voxel decoding network and a fully connected layer. Using a trained deep learning network, predictions are made on multi-view pure visual 2D images.

2. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 1, characterized in that, The process of constructing the foreground dataset includes: performing label transformation on the training dataset with full-category 3D semantic occupancy annotations, keeping the foreground categories unchanged, merging the background category and the empty class into an empty class, and obtaining foreground semantic occupancy annotations that only include the foreground categories and the empty class.

3. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 1, characterized in that, The vector quantization variational autoencoder network includes a three-dimensional encoder, a two-dimensional encoder, a vector quantization module based on Sinkhorn optimal transmission, an upsampling module, and a two-dimensional decoder. Foreground semantic occupancy annotations are used as network inputs and are encoded sequentially by a 3D encoder and a 2D encoder. The BEV continuous feature maps of different scales output by the two encoders are input into a vector quantization module based on Sinkhorn optimal transmission to obtain foreground quantized feature maps and foreground feature codebooks of different scales. Then, the foreground quantized feature maps of each scale are spliced ​​to the same spatial resolution by an upsampling module, spliced ​​and fused, and then decoded by a 2D decoder module to obtain the foreground 3D semantic occupancy prediction value.

4. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 3, characterized in that, The calculation process of the vector quantization module based on Sinkhorn optimal transmission includes: First, BEV feature flattening and cost matrix construction are performed. The mesoscale BEV continuous feature map obtained from the 3D encoder and the large-scale BEV continuous feature map obtained from the 2D encoder are flattened into a set of feature vectors according to their spatial location. , A learnable codebook is set up for each scale. , Calculate the Euclidean distance between the latent feature vector and the code vector to form the cost matrix: Based on cost matrix and parameters Construct the initial allocation matrix : right Perform at least five rounds of Sinkhorn iterations with alternating row and column normalization to obtain an approximately optimal transfer matrix that satisfies the row and column constraints. This is essentially a vector quantization constraint based on Sinkhorn optimal transfer, and the allocation matrix for each scale. , record The matrix at the next iteration is The specific form of each round of alternating normalization is as follows: After multiple iterations, the sum of the weights assigned to each latent feature vector across all code vectors approaches 1; the sum of the weights assigned to each code vector across all latent features approaches the predetermined total weight, thus yielding an approximately optimal transfer matrix that satisfies the row and column constraints. The correspondence between the foreground latent feature vector and the code vector is determined by using the position of the maximum value of each element in the optimal transmission matrix, that is, for the first row... Each foreground latent feature vector is selected from its corresponding row. Take the column index corresponding to the maximum value The code vector is used as the matching code vector.

5. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 3, characterized in that, During the training process of the vector quantization variational autoencoder network, a one-stage total loss function is used. End-to-end optimization of the vector quantization variational autoencoder network includes: The first-scale vector quantization loss is obtained by performing vector quantization on the two-scale foreground latent features output by the encoder. Second-scale vector quantization loss And the mean square error loss based on the reconstruction error of the foreground 3D voxel occupancy result. Thus we get: in, , , These are the weighting coefficients for each loss term; First-scale vector quantization loss Second-scale vector quantization loss For the first-scale foreground latent feature set respectively The set of code vectors in the first-scale foreground feature codebook and the second-scale foreground latent feature set The set of code vectors in the second-scale foreground feature codebook Define, where, the first The latent feature vectors and their matching code vectors are denoted as follows: , Then we have: in, Represents the L2 norm. This indicates that the gradient operation is stopped. These are the coefficients used to balance the codebook update terms and commitment terms; Mean square error loss The voxel-level reconstruction error used to measure the foreground 3D semantic occupancy prediction result and the foreground semantic occupancy annotation is as follows: in, This represents the total number of voxels in the training samples. For the first The semantic occupancy annotation values ​​of individual elements are presented in one-hot vector annotation form. This is the corresponding predicted probability vector.

6. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 1, characterized in that, The deep learning network structure, in which the input is a multi-view pure visual two-dimensional image and the output is a three-dimensional semantic occupancy result, includes: a two-dimensional feature extraction module with ResNet-50 as the backbone network, used to extract multi-scale two-dimensional semantic features and discrete depth estimation; The LSS-based feature transformation module is used to combine the two-dimensional features of each viewpoint with the corresponding discrete depth estimate to obtain the initial features of the BEV bird's-eye view. A feature pyramid network containing global continuous feature branches and foreground feature quantization branches is used to extract continuous foreground features and global continuous features from the initial features of the BEV bird's-eye view, and to quantize the continuous foreground features based on the foreground feature codebook to obtain foreground quantized features. The feature aggregation module aggregates foreground quantization features and global continuous features, inputs them into the 3D voxel decoding network, and through 3D convolution and upsampling operations, maps them into a 3D voxel feature representation containing height dimensions. Finally, it is input into the fully connected layer and outputs the predicted probability distribution of the voxel belonging to each semantic category.

7. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 6, characterized in that, The feature pyramid network, which includes global continuous feature branches and foreground feature quantization branches, specifically includes: Initial features The feature pyramid network is fed as input. This is used to construct multi-scale BEV feature representations. The feature pyramid network fuses features from different resolution levels through top-down progressive upsampling and lateral connections, ultimately outputting global feature maps at three spatial resolutions: large-scale, medium-scale, and small-scale. , , : in, , , Together as global continuous features, they converge in the global continuous feature branches to form... same resolution It is used to characterize global geometric structure and semantic context information at multiple scales; For the foreground feature quantization branch, a mesoscale feature map is selected from the three-scale global feature map. and large-scale feature maps As input, they are fed into the foreground feature convolutional units of the corresponding scales. and The foreground feature convolutional unit consists of several layers of two-dimensional convolution, normalization, and ReLU nonlinear activation units. It is used to mine foreground-related patterns and suppress background interference to obtain continuous foreground features at medium and large scales. Subsequently, using a multi-scale foreground feature codebook, vector quantization is performed on the continuous foreground features at both scales. Specifically, the feature maps are... and Flattened by spatial location, the foreground latent feature set is obtained, and Euclidean distance is calculated between it and the code vector set in the corresponding scale foreground feature codebook. A quantization strategy based on Sinkhorn optimal transmission is adopted to obtain BEV foreground quantization feature maps at medium and large scales. and .

8. The visual 3D semantic occupancy prediction method based on foreground feature quantization according to claim 6, characterized in that, The step of adding the foreground quantized feature map of the BEV bird's-eye view to the supervision signal for training the network includes: Predicting Focal Loss by Occupation Discrete depth estimation loss based on lidar point cloud supervision and prospect quantification consistency loss The two-stage total loss function is constructed by the weighted sum of the following: in, , , These are the weighting coefficients for each loss term. Focal Loss for Occupation Prediction : in voxel number For the number of semantic categories, For the first Individual elements belong to the first One-hot annotation for classes, For the corresponding predicted probability, For category weights, As a focusing factor; For discrete depth estimation loss : in The effective number of pixels for all viewpoints. The depth is the number of discrete layers. For the first The pixel in the first Discrete depth labels on each depth layer For the corresponding predicted probability; For prospect quantification consistency loss : in, , These are the BEV foreground feature maps predicted by the network at medium and large scales. , This is a foreground quantization feature map of a BEV bird's-eye view obtained by training a vector quantization variational autoencoder network based on labels.

9. A visual 3D semantic occupancy prediction device based on foreground feature quantization, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a visual three-dimensional semantic occupancy prediction method based on foreground feature quantization as described in any one of claims 1-8.

Images