An off-road environment pure vision three-dimensional semantic scene completion method and system
Patent Information
- Application Number
- CN202611133376.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-29
- Publication Date
- 2026-10-23
AI Technical Summary
[0006]本发明旨在解决现有越野环境自动驾驶感知系统中过度依赖高成本激光雷达，以及传统密集网格（体素）表示方法在处理复杂越野地形时存在的计算严重冗余、动态适应性差等技术缺陷，提出了一种越野环境纯视觉三维语义场景补全方法及系统
[0018]本发明通过稀疏三维高斯表示替代传统密集网格，并结合显式语义特征赋值，消除了对高成本激光雷达传感器的实时依赖，实现了纯视觉方案在越野场景下的低成本部署。同时，相较于现有基于密集网格的方法，本发明能够根据场景复杂度动态分配计算资源，显著降低系统内存消耗，提升三维语义占用预测精度，尤其在泥泞道路、灌木、碎石等非结构化类别的识别与重建上表现出较强的鲁棒性。
Smart Images
Figure CN122967273A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving environmental perception technology, specifically to a method and system for completing a pure visual 3D semantic scene in an off-road environment. Background Technology
[0002] In the current field of off-road autonomous driving and environmental perception, mainstream systems rely heavily on LiDAR sensors for environmental perception and 3D spatial modeling. LiDAR, by emitting laser beams and receiving reflected signals, can directly acquire extremely precise 3D absolute position coordinates (i.e., point cloud data) of object surfaces in the environment, and exhibits significant advantages in capturing fine geometric details.
[0003] To reduce hardware deployment costs and simulate the human ability to seamlessly reason about geometry and semantics using vision, the purely visual 3D Semantic Scene Completion (SSC) task has become a research hotspot in recent years. This task aims to jointly infer the 3D geometric structure and corresponding semantic meaning of a scene using only 2D images acquired by low-cost monocular or multi-view cameras. In these technical solutions, traditional existing technologies typically use regular dense meshes (such as 3D voxel grids) as the underlying scene representation to construct the 3D spatial structure; representative methods include MonoScene. These dense voxel methods have explicit 3D geometric representation capabilities. By dividing the continuous 3D space into discrete voxel units, they can intuitively describe the spatial occupancy of objects, driving significant technological progress in structured urban traffic environments with clear lane lines and well-defined traffic signs.
[0004] To further optimize the perception performance and computational efficiency of dense voxel methods, multi-dimensional technological advancements have been undertaken in recent years. On one hand, some methods have introduced coarse-to-fine upsampling strategies or depth-guided voxel filtering techniques (such as VoxFormer) to optimize spatial structure construction; or they have attempted to improve spatial query efficiency by utilizing octree-based hierarchical partitioning and multi-granularity voxel queries (such as OctreeOcc). On the other hand, researchers have attempted to introduce Bird's Eye View (BEV) representations, fusing back-projected BEV features with voxel features (such as FB-OCC), or utilizing multi-scale BEV fusion modules and channel-to-height plug-ins (such as FlashOcc and M²BEV), thereby simplifying the computational process while maintaining a certain level of 3D perception capability. These improvements have, to some extent, enhanced the system's ability to explicitly represent the geometry of complex scenes.
[0005] However, the aforementioned technical solutions have significant limitations in unstructured off-road environments. First, from the perspective of hardware sensors, while LiDAR systems offer high accuracy, their equipment costs are extremely high, and the point cloud data they acquire inherently exhibits sparsity and is highly susceptible to interference from adverse weather conditions. These limitations severely hinder the widespread adoption of LiDAR-based perception solutions in cost-sensitive applications and those requiring large-scale deployment. Second, from the perspective of vision-based algorithm representation, traditional dense mesh (voxel) methods suffer from severe computational redundancy and poor flexibility. Although existing voxel methods employ optimization techniques such as coarse-to-fine upsampling or depth-guided voxel filtering, they remain constrained by predefined fixed partitioning patterns. This rigid mesh representation method struggles to dynamically adapt to the complex situation of multi-scale targets (such as rugged terrain and dynamic obstacles) coexisting in off-road environments. More critically, these methods rely entirely on rigid meshes, failing to eliminate computational redundancy caused by invalid spatial regions in the scene. Summary of the Invention
[0006] This invention aims to address the shortcomings of existing autonomous driving perception systems in off-road environments, such as over-reliance on high-cost LiDAR and the serious computational redundancy and poor dynamic adaptability of traditional dense mesh (voxel) representation methods when dealing with complex off-road terrain. It proposes a pure visual 3D semantic scene completion method and system for off-road environments.
[0007] To achieve the above objectives, this invention provides a method for pure visual 3D semantic scene completion in off-road environments, comprising the following steps: acquiring a single-view RGB image captured by a monocular camera, and extracting multi-scale image features from the RGB image; fusing camera parameters and the multi-scale image features, inputting the fusion into a depth prediction network for depth estimation, and outputting a predicted depth map; mapping the 2D pixel coordinates of the predicted depth map to 3D space based on the predicted depth map and camera parameters, initializing a set of sparse 3D Gaussian points and encapsulating semantic feature information to form an initial 3D semantic Gaussian set, wherein each 3D Gaussian point includes at least the following geometric parameters: mean parameter, scale parameter, and rotation parameter. The process involves: converting parameters; constructing a query vector containing geometrically perceptual information and encoding the geometric parameters into high-dimensional features; using a cross-attention mechanism to fuse the query vector with the multi-scale image features to obtain fused Gaussian features; decoding the fused Gaussian features to obtain the semantic prediction results for each 3D Gaussian point and predicting the update amount of the geometric parameters for each 3D Gaussian point; adjusting the geometric parameters of each 3D Gaussian point in the 3D semantic Gaussian set with the update amount to obtain an optimized 3D semantic Gaussian set; and accumulating the contribution values of each Gaussian function at each discrete position in a preset 3D space based on the optimized 3D semantic Gaussian set to output the 3D semantic occupancy prediction result.
[0008] The process of fusing camera parameters and multi-scale image features includes: concatenating and flattening the camera's intrinsic parameter matrix, rotation matrix, and translation vector, and projecting them through a multilayer perceptron to form a camera parameter feature vector that matches the dimension of the image feature channels; and then fusing the multi-scale image features with the camera parameter feature vector after reweighting them through a channel attention mechanism.
[0009] The initialization of a set of sparse 3D Gaussian points includes: performing a back projection operation using the predicted depth map and camera intrinsic and extrinsic parameters to map the coordinates of each pixel on the predicted depth map to 3D space, generating mean parameters for each of the 3D Gaussian points; and initializing scale parameters and rotation parameters for each of the 3D Gaussian points based on the local depth gradient distribution of the predicted depth map.
[0010] The step of using a cross-attention mechanism to fuse the query vector and the multi-scale image features to obtain fused Gaussian features includes: employing a hierarchical feature aggregation network and using a scaling dot product cross-attention mechanism to calculate the semantic dependency association weights between each query vector and each level of image feature map; and performing weighted aggregation of the multi-scale image features based on the association weights to generate spatially and semantically aligned Gaussian fused features.
[0011] The step of decoding the fused Gaussian features to obtain the semantic prediction results of each 3D Gaussian point and predicting the update amount of the geometric parameters of each 3D Gaussian point, and adjusting the geometric parameters of each 3D Gaussian point in the 3D semantic Gaussian set with the update amount to obtain an optimized 3D semantic Gaussian set includes: inputting the fused Gaussian features into a semantic multilayer perceptron decoder to decode and output the semantic prediction probability distribution corresponding to each 3D Gaussian point; inputting the fused Gaussian features into a geometric parameter residual prediction module to calculate the scale residual increment and the rotation quaternion increment; adjusting the corresponding scale parameter with the scale residual increment and adjusting the corresponding rotation parameter with the rotation quaternion increment to obtain the optimized 3D semantic Gaussian set.
[0012] The scale residual increment is constrained by the tanh function to avoid abrupt scale changes, and the rotated quaternion increment is normalized by L2.
[0013] The step of accumulating the contribution values of each Gaussian function at each discrete location in a preset three-dimensional space based on the optimized three-dimensional semantic Gaussian set, and outputting a three-dimensional semantic occupancy prediction result, includes: obtaining the three-dimensional coordinate center of the voxel grid in the preset three-dimensional space as a discrete location; for each discrete location, calculating the sum of the contribution values of all adjacent Gaussian functions to that discrete location based on the covariance matrix, weight distribution, and semantic prediction probability distribution of each three-dimensional Gaussian function; and outputting a dense three-dimensional semantic occupancy prediction result based on the accumulated contribution values of each discrete location.
[0014] The step of extracting multi-scale image features from the RGB image includes: using ResNet-101 as the backbone network and combining it with a feature pyramid network to extract features from the RGB image and output a multi-scale image feature pyramid.
[0015] In another aspect, this invention provides a pure visual 3D semantic scene completion system for off-road environments, comprising: an image feature extraction module for acquiring a single-view RGB image captured by a monocular camera and extracting multi-scale image features from the RGB image; a depth estimation module for fusing camera parameters and the multi-scale image features, inputting them into a depth prediction network for depth estimation, and outputting a predicted depth map; and a Gaussian initialization module for mapping the 2D pixel coordinates of the predicted depth map to 3D space based on the predicted depth map and camera parameters, initializing a set of sparse 3D Gaussian points and encapsulating semantic feature information to form an initial 3D semantic Gaussian set, wherein each 3D Gaussian point includes at least the following geometric parameters: mean parameter, scale parameter, and rotation parameter; and feature fusion. The system comprises a query vector containing geometrically perceptual information and encoding the geometric parameters into high-dimensional features. It then uses a cross-attention mechanism to fuse the query vector with the multi-scale image features to obtain fused Gaussian features. A collaborative optimization module decodes the fused Gaussian features to obtain the semantic prediction results for each 3D Gaussian point and predicts the update amount of the geometric parameters for each 3D Gaussian point. The updated amount is used to adjust the geometric parameters of each 3D Gaussian point in the 3D semantic Gaussian set to obtain an optimized 3D semantic Gaussian set. Finally, a semantic occupancy output module accumulates the contribution values of each Gaussian function at discrete locations in a preset 3D space based on the optimized 3D semantic Gaussian set, and outputs the 3D semantic occupancy prediction results.
[0016] In another aspect, the present invention provides a computer storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the above-described method for completing a pure visual three-dimensional semantic scene in an off-road environment.
[0017] As can be seen from the above technical solutions, the advantages of the present invention are:
[0018] This invention replaces traditional dense meshes with sparse 3D Gaussian representations and combines this with explicit semantic feature assignment, eliminating the real-time dependence on high-cost LiDAR sensors and enabling low-cost deployment of a pure vision-based solution in off-road scenarios. Furthermore, compared to existing dense mesh-based methods, this invention can dynamically allocate computational resources based on scene complexity, significantly reducing system memory consumption and improving the accuracy of 3D semantic occupancy prediction, especially demonstrating strong robustness in the identification and reconstruction of unstructured categories such as muddy roads, shrubs, and gravel.
[0019] This invention further improves the accuracy of depth estimation, eliminates computational redundancy, achieves geometric and semantic depth alignment, and enhances the model's ability to recognize unstructured objects by using preferred methods such as fusing camera parameters with multi-scale image features for depth estimation, initializing the mean and scale rotation parameters of 3D Gaussian points based on the depth map, using a cross-attention mechanism for feature fusion, and optimizing geometric parameters through residual prediction. Attached Figure Description
[0020] Figure 1 A flowchart (I) of a method for completing a pure visual 3D semantic scene in an off-road environment according to an embodiment of the present invention.
[0021] Figure 2 A schematic diagram (II) of the off-road environment pure visual three-dimensional semantic scene completion method provided in an embodiment of the present invention;
[0022] Figure 3 This is a schematic diagram of the structure of a pure visual 3D semantic scene completion system for off-road environments provided in another embodiment of the present invention;
[0023] In the attached figures, the following labels are used:
[0024] 100-A method for completing a pure visual 3D semantic scene in an off-road environment;
[0025] 200-Off-road environment pure visual 3D semantic scene completion system;
[0026] 20 - Image Feature Extraction Module;
[0027] 21-Depth Estimation Module;
[0028] 22-Gaussian initialization module;
[0029] 23-Feature fusion module;
[0030] 24-Collaborative Optimization Module;
[0031] 25 - Semantic Occupation Output Module;
[0032] Steps S1~S6. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0034] Research has revealed that existing pure vision methods suffer from low prediction accuracy and massive computational costs in off-road scenarios, fundamentally due to the limitations of traditional dense voxel mesh representations and their complex decoding processes. To address this, this invention introduces a lightweight, object-centric "sparse 3D semantic Gaussian representation" to replace the traditional dense mesh. This sparse Gaussian representation method borrows ideas from 3D Gaussian splatting technology for scene representation, but migrates it from rendering tasks to semantic scene completion tasks in off-road environments. This Gaussian representation method can achieve more flexible and accurate modeling of complex off-road environments by simultaneously encoding spatial and semantic information in each Gaussian distribution. Furthermore, this invention finds that accurate depth estimation is crucial in vision-based 3D scene understanding, but monocular depth estimation presents inherent challenges. Therefore, this invention innovatively and explicitly integrates camera parameters to achieve camera-perceived depth estimation when designing the depth prediction network, and cleverly utilizes the real depth generated from LiDAR point cloud data as an intermediate supervision signal. This improvement not only enhances the accuracy of monocular depth prediction but also provides highly accurate spatial anchors for the initialization of subsequent 3D Gaussian models, thereby achieving efficient end-to-end mapping from low-cost image input to dense semantic occupancy output.
[0035] The key technical points of this invention include:
[0036] Key Point 1: Camera-aware Depth Estimation Module. Addressing the inherent challenges of monocular depth estimation, this invention flattens the camera's intrinsic and extrinsic parameters (including intrinsic matrix, rotation matrix, and translation vector) and maps them through a multilayer perceptron (MLP). These are then fused with multi-scale image features and input into the depth prediction network. Simultaneously, during the training phase, a ground-truth depth map generated by LiDAR point cloud projection is introduced as an intermediate supervision signal.
[0037] Technical effects: This design explicitly injects camera geometric priors into the network, helping the depth prediction network to accurately understand the spatial position of image features in its own coordinate system, significantly improving the accuracy of monocular depth prediction, and providing high-precision spatial anchor points for the subsequent initialization of 3D Gaussian.
[0038] Key Point 2: Depth-guided Gaussian Initialization. This invention utilizes the depth map output by the depth prediction network and local depth gradient information to back-project 2D pixels into 3D space, thereby initializing a set of sparse 3D Gaussian point cloud distributions containing mean, scale, and rotation parameters. Each 3D Gaussian distribution not only represents geometric properties but also encapsulates the region of interest and its specific semantic features.
[0039] Technical Results: Employing an object-centric sparse Gaussian representation provides a more resilient alternative to traditional dense rigid meshes. This approach dynamically and adaptively allocates computational and storage resources based on the scale and complexity of different regions in off-road scenarios, effectively eliminating computational redundancy caused by large areas of ineffective space.
[0040] Key Point 3: Dynamic Cross-Attention Fusion and Gaussian Optimization of Multi-Scale Image Features. This invention constructs a hierarchical feature aggregation network, utilizing a scaled dot product cross-attention mechanism to dynamically calculate the association weights between the Gaussian geometric query vector and multi-level two-dimensional (2D) image features, aggregating and generating Gaussian features rich in multimodal semantics. Subsequently, based on this fused feature, a semantic prediction probability distribution is output, and the scale and rotation parameters of the Gaussian are dynamically updated and optimized through a residual network.
[0041] Technical Results: This method achieves deep alignment and collaborative reasoning between geometric structures and multi-scale semantic features in 3D space. It completely eliminates the complex high-dimensional feature decoding steps in traditional voxel methods, realizing efficient end-to-end mapping from images to 3D representations. In particular, it significantly improves the model's recognition accuracy on unstructured multi-scale objects such as dense vegetation (e.g., trees, shrubs).
[0042] like Figures 1 to 2 The flowchart of a pure visual 3D semantic scene completion method 100 for off-road environments provided in an embodiment of the present invention includes:
[0043] S1: Acquire a single-view RGB image captured by a monocular camera and extract multi-scale image features from the RGB image.
[0044] Specifically, it involves acquiring a single-view RGB image captured by a monocular camera. The image feature encoder (e.g., using a ResNet-101 backbone network combined with a feature pyramid network FPN) extracts features from the input image and outputs a multi-scale image feature pyramid. .
[0045] S2: The camera parameters and multi-scale image features are fused and input into the depth prediction network for depth estimation, and the predicted depth map is output.
[0046] Specifically, the camera's intrinsic parameter matrix Rotation matrix Translation vector The images are stitched together and flattened, then projected using a multilayer perceptron (MLP) to form camera parameter feature vectors that match the dimensions of the image feature channels. The extracted 2D multi-scale image features are reweighted using a channel attention mechanism (Squeeze-and-Excitation), combined with the camera parameter feature vector, and then input into the depth prediction network (DepthNet) to output a predicted depth map. During the model training phase of the depth prediction network, a true depth map is generated by back-projecting LiDAR point cloud data onto a 2D image coordinate system. Calculate depth loss This allows for explicit guidance and supervision of the training of deep prediction networks.
[0047] S3: Based on the predicted depth map and camera parameters, map the coordinates of the two-dimensional pixels in the predicted depth map to three-dimensional space, initialize a set of sparse three-dimensional Gaussian points and encapsulate semantic feature information to form an initial three-dimensional semantic Gaussian set. Each of the three-dimensional Gaussian points includes at least the following geometric parameters: mean parameter, scale parameter and rotation parameter.
[0048] The initialization of a set of sparse 3D Gaussian points includes: performing a back projection operation using the predicted depth map and camera intrinsic and extrinsic parameters to map the coordinates of each pixel on the predicted depth map to 3D space, generating the mean parameters of each 3D Gaussian point; and initializing scale parameters and rotation parameters for each 3D Gaussian point based on the local depth gradient distribution of the predicted depth map.
[0049] Specifically, using predicted depth maps Based on the aforementioned camera intrinsic and extrinsic parameters, a back projection operation is performed to map the pixel coordinates on the predicted depth map to 3D space, generating an initial mean position of a set of sparse 3D Gaussian points. Simultaneously, based on the spatial gradient distribution of local depth, the scale parameter is initialized for each 3D Gaussian point. And rotation parameters This allows us to construct the initial 3D semantic Gaussian set.
[0050] S4: Construct a query vector containing geometrically perceptual information and encode the geometric parameters of the 3D Gaussian points into high-dimensional features. Use the cross-attention mechanism to fuse features based on the query vector and multi-scale image features to obtain the fused Gaussian features.
[0051] Specifically, constructing query vectors that include geometric perception information The geometric parameters (position, scale, rotation) of 3D Gaussian points are encoded as high-dimensional features. Then, a hierarchical feature aggregation network (HFAN) is used, employing a scaled dot-product attention mechanism to calculate the semantic dependency weights between the query vector of each 3D Gaussian point and the feature maps of each layer. Finally, a weighted aggregation is used to generate spatially and semantically aligned Gaussian fusion features. .
[0052] S5: Based on the fused Gaussian feature decoding, obtain the semantic prediction results of each 3D Gaussian point, and predict the update amount of the geometric parameters of each 3D Gaussian point. Adjust the geometric parameters of each 3D Gaussian point in the 3D semantic Gaussian set with the update amount to obtain the optimized 3D semantic Gaussian set.
[0053] Specifically, the fused Gaussian features are input into the semantic multilayer perceptron decoder, which decodes and outputs the semantic prediction probability distribution corresponding to each 3D Gaussian point. In this embodiment, the semantic prediction probability distribution The dimension corresponds to the preset number of semantic categories, such as, but not limited to, semantic categories like road, building, vegetation, vehicle, other, and idle. Simultaneously, this fused feature is input into the geometric parameter residual prediction module to calculate the scale residual increment. (Constrained by the tanh function to avoid abrupt scaling) and rotational quaternion increments (After L2 normalization), the original 3D Gaussian geometric parameters are updated and optimized to achieve synergistic optimization of geometric structure and semantic features.
[0054] S6: Based on the optimized 3D semantic Gaussian set, accumulate the contribution values of each Gaussian function at each discrete position in the preset 3D space, and output the 3D semantic occupancy prediction result.
[0055] Step S6 includes: obtaining the three-dimensional coordinate center of the voxel grid in the preset three-dimensional space as a discrete position; for each discrete position, calculating the sum of the contributions of all adjacent Gaussian functions to the discrete position based on the covariance matrix, weight distribution, and semantic prediction probability distribution of each three-dimensional Gaussian function; and outputting a dense three-dimensional semantic occupancy prediction result based on the cumulative result of the contributions of each discrete position.
[0056] Specifically, based on the optimized 3D Gaussian set with explicit semantic information, at each discrete location in the preset target 3D space... (The 3D coordinate center of the voxel mesh), using the covariance matrix and the weight distribution of each Gaussian function, calculate the sum of the contributions of all neighboring Gaussian functions to that location. :
[0057] ;
[0058]
[0059] in, This represents the contribution weight of the i-th Gaussian point at the discrete position p; This represents the mean parameter of the i-th Gaussian point, which is the coordinate of the center position of the Gaussian point in three-dimensional space. This represents the scale parameter of the i-th Gaussian point, which controls the influence range of this Gaussian point in space.
[0060] The final output is the dense 3D semantic occupancy prediction result within this spatial range. Each discrete location p yields a weighted cumulative result, which is the 3D semantic occupancy prediction value for that discrete location.
[0061] This invention completely eliminates the real-time dependence on high-cost LiDAR sensors when analyzing 3D scene structures. Compared to existing dense voxel representation methods, this invention, through explicit semantic feature assignment and sparse Gaussian representation, significantly reduces system memory consumption (down to approximately 3152MB in a specific embodiment) while greatly improving the 3D semantic occupancy prediction accuracy in off-road environments (mIoU index achieves a significant leap, reaching 8.4). Especially in the identification and reconstruction of unstructured off-road categories such as muddy roads, dense bushes, and rubber, this invention demonstrates extremely strong anti-interference capabilities and robustness.
[0062] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0063] like Figure 3 The diagram shown is a structural schematic of a pure visual 3D semantic scene completion system 200 for off-road environments provided in another embodiment of the present invention. It employs the aforementioned pure visual 3D semantic scene completion method 100 for off-road environments and includes:
[0064] The image feature extraction module 20 is used to acquire a single-view RGB image captured by a monocular camera and extract multi-scale image features from the RGB image.
[0065] Depth estimation module 21, connected to image feature extraction module 20, is used to fuse camera parameters and multi-scale image features, input them into depth prediction network for depth estimation, and output a predicted depth map.
[0066] The Gaussian initialization module 22 is connected to the depth estimation module 21. It is used to map the coordinates of two-dimensional pixels in the predicted depth map to three-dimensional space according to the predicted depth map and camera parameters, initialize a set of sparse three-dimensional Gaussian points and encapsulate semantic feature information to form an initial three-dimensional semantic Gaussian set. Each three-dimensional Gaussian point includes at least the following geometric parameters: mean parameter, scale parameter and rotation parameter.
[0067] The feature fusion module 23 is connected to the Gaussian initialization module 23. It is used to construct a query vector containing geometric perception information and encode the geometric parameters into high-dimensional features. It uses a cross-attention mechanism to fuse features based on the query vector and multi-scale image features to obtain the fused Gaussian features.
[0068] The collaborative optimization module 24 is connected to the feature fusion module 23. It is used to obtain the semantic prediction result of each three-dimensional Gaussian point based on the fused Gaussian feature decoding, and predict the update amount of the geometric parameters of each three-dimensional Gaussian point. The update amount is used to adjust the geometric parameters of each three-dimensional Gaussian point in the three-dimensional semantic Gaussian set to obtain the optimized three-dimensional semantic Gaussian set.
[0069] The semantic occupancy output module 25, connected to the collaborative optimization module 24, is used to accumulate the contribution values of each Gaussian function at discrete locations in a preset three-dimensional space based on the optimized three-dimensional semantic Gaussian set, and output the three-dimensional semantic occupancy prediction result.
[0070] Furthermore, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the off-road environment pure visual three-dimensional semantic scene completion system 200 can be referred to the corresponding process in the aforementioned off-road environment pure visual three-dimensional semantic scene completion method 100 embodiment, and will not be repeated here.
[0071] In another embodiment of the present invention, a computer storage medium (not shown) is provided, storing a computer program that, when executed by a processor, implements the above-described off-road environment pure visual 3D semantic scene completion method 100 (e.g. Figure 1 or Figure 2 The steps are as follows.
[0072] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms fall within the scope of protection of the present invention.
Claims
1. A method for completing a purely visual 3D semantic scene in an off-road environment, characterized in that, Includes the following steps: Acquire a single-view RGB image captured by a monocular camera, and extract multi-scale image features from the RGB image; The camera parameters and the multi-scale image features are fused and input into the depth prediction network for depth estimation, and a predicted depth map is output. Based on the predicted depth map and camera parameters, the coordinates of two-dimensional pixels in the predicted depth map are mapped to three-dimensional space. A set of sparse three-dimensional Gaussian points is initialized and semantic feature information is encapsulated to form an initial three-dimensional semantic Gaussian set. Each three-dimensional Gaussian point includes at least the following geometric parameters: mean parameter, scale parameter, and rotation parameter. A query vector containing geometric perception information is constructed and the geometric parameters are encoded as high-dimensional features. A cross-attention mechanism is used to fuse the query vector with the multi-scale image features to obtain the fused Gaussian features. The semantic prediction results of each three-dimensional Gaussian point are obtained based on the fused Gaussian feature decoding, and the update amount of the geometric parameters of each three-dimensional Gaussian point is predicted. The geometric parameters of each three-dimensional Gaussian point in the three-dimensional semantic Gaussian set are adjusted with the update amount to obtain the optimized three-dimensional semantic Gaussian set. Based on the optimized 3D semantic Gaussian set, the contribution values of each Gaussian function are accumulated at each discrete position in the preset 3D space, and the 3D semantic occupancy prediction result is output.
2. The off-road environment pure visual 3D semantic scene completion method according to claim 1, characterized in that, The fusion of camera parameters and multi-scale image features includes: The camera's intrinsic parameter matrix, rotation matrix, and translation vector are concatenated and flattened, and then projected through a multilayer perceptron to form a camera parameter feature vector that matches the dimension of the image feature channels. The multi-scale image features are reweighted using a channel attention mechanism and then fused with the camera parameter feature vector.
3. The off-road environment pure visual 3D semantic scene completion method according to claim 1, characterized in that, The initialization of a sparse set of three-dimensional Gaussian points includes: Using the predicted depth map and camera intrinsic and extrinsic parameters, a back projection operation is performed to map the coordinates of each pixel on the predicted depth map to three-dimensional space, generating the mean parameters of each three-dimensional Gaussian point. Based on the local depth gradient distribution of the predicted depth map, the scale parameters and rotation parameters are initialized for each of the three-dimensional Gaussian points.
4. The off-road environment pure visual 3D semantic scene completion method according to claim 1, characterized in that, The step of using a cross-attention mechanism to fuse features based on the query vector and the multi-scale image features to obtain fused Gaussian features includes: A hierarchical feature aggregation network is adopted, and the semantic dependency association weights between each query vector and the image feature maps of each level are calculated using the scaling dot product cross attention mechanism. The multi-scale image features are weighted and aggregated according to the association weights to generate Gaussian fusion features that are spatially and semantically aligned.
5. The off-road environment pure visual 3D semantic scene completion method according to claim 1, characterized in that, The step of obtaining the semantic prediction result of each 3D Gaussian point based on the fused Gaussian feature decoding, predicting the update amount of the geometric parameters of each 3D Gaussian point, and adjusting the geometric parameters of each 3D Gaussian point in the 3D semantic Gaussian set with the update amount to obtain the optimized 3D semantic Gaussian set includes: The fused Gaussian features are input into the semantic multilayer perceptron decoder, and the semantic prediction probability distribution corresponding to each three-dimensional Gaussian point is decoded and output. The fused Gaussian features are input into the geometric parameter residual prediction module to calculate the scale residual increment and the rotation quaternion increment. The scale parameters are adjusted by the scale residual increment, and the rotation parameters are adjusted by the rotation quaternion increment to obtain the optimized three-dimensional semantic Gaussian set.
6. The off-road environment pure vision three-dimensional semantic scene completion method according to claim 5, characterized in that, The scale residual increment is constrained by the tanh function to avoid abrupt scale changes, and the rotated quaternion increment is normalized by L2.
7. The method according to claim 5, characterized in that, The step of accumulating the contribution values of each Gaussian function at each discrete position in a preset three-dimensional space based on the optimized three-dimensional semantic Gaussian set, and outputting the three-dimensional semantic occupancy prediction result, includes: Obtain the three-dimensional coordinate center of the voxel grid in the preset three-dimensional space as the discrete position; For each discrete position, the sum of the contributions of all adjacent Gaussian functions to that discrete position is calculated based on the covariance matrix, weight distribution, and semantic prediction probability distribution of each three-dimensional Gaussian function. Based on the cumulative contribution values of each discrete location, a dense three-dimensional semantic occupancy prediction result is output.
8. The off-road environment pure visual 3D semantic scene completion method according to claim 1, characterized in that, The extraction of multi-scale image features from the RGB image includes: ResNet-101 is used as the backbone network, and a feature pyramid network is combined to extract features from the RGB image, outputting a multi-scale image feature pyramid.
9. A pure visual 3D semantic scene completion system for off-road environments, characterized in that, include: The image feature extraction module is used to acquire a single-view RGB image captured by a monocular camera and extract multi-scale image features from the RGB image; The depth estimation module is used to fuse camera parameters and multi-scale image features, input them into the depth prediction network for depth estimation, and output a predicted depth map. The Gaussian initialization module is used to map the coordinates of two-dimensional pixels in the predicted depth map to three-dimensional space according to the predicted depth map and camera parameters, initialize a set of sparse three-dimensional Gaussian points and encapsulate semantic feature information to form an initial three-dimensional semantic Gaussian set. Each three-dimensional Gaussian point includes at least the following geometric parameters: mean parameter, scale parameter and rotation parameter. The feature fusion module is used to construct a query vector containing geometric perception information and encode the geometric parameters into high-dimensional features. It uses a cross-attention mechanism to fuse the query vector with the multi-scale image features to obtain the fused Gaussian features. The collaborative optimization module is used to obtain the semantic prediction result of each three-dimensional Gaussian point based on the fused Gaussian feature decoding, and to predict the update amount of the geometric parameters of each three-dimensional Gaussian point. The update amount is used to adjust the geometric parameters of each three-dimensional Gaussian point in the three-dimensional semantic Gaussian set to obtain the optimized three-dimensional semantic Gaussian set. The semantic occupancy output module is used to accumulate the contribution values of each Gaussian function at each discrete position in a preset three-dimensional space based on the optimized three-dimensional semantic Gaussian set, and output the three-dimensional semantic occupancy prediction result.
10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the off-road environment pure visual three-dimensional semantic scene completion method as described in any one of claims 1 to 8.