Cross-country scene three-dimensional semantic occupancy prediction method based on geometric perception state space model

Abstract

The invention belongs to the technical field of automatic driving environment perception, and relates to an off-road scene three-dimensional semantic occupancy prediction method based on a geometric perception state space model. The method comprises the following steps: S1, constructing a three-dimensional semantic occupancy truth value label; step S2, data preprocessing; s3, carrying out refined extraction on double-flow features; step S4, a geometrically guided gating fusion module; step S5, physically sensing the Mama backbone network module; s6, generating a prediction result; and S7, model loss calculation and parameter iteration are carried out. According to the method, a physical perception Mama backbone network is constructed, and continuity and physical consistency of the road surface and the soil slope in three-dimensional space prediction are ensured. According to the method, the problem of cross-country data scarcity is solved, the robustness of the model under the severe weather condition is remarkably improved, and accurate alignment of the multi-modal features on geometric structures and semantic textures is achieved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environment perception technology, and relates to a method for predicting three-dimensional semantic occupancy in off-road scenarios based on a geometric perception state space model. Background Technology

[0002] The core objective of 3D semantic occupancy prediction is to assign an accurate semantic label to each occupied voxel for prediction. In recent years, with the rapid development of deep learning technology, deep learning-based 3D semantic occupancy prediction methods have been widely applied in the field of autonomous driving. The basic principle of this task is to divide the entire spatial region within the perception range into several voxel units, and predict the occupancy status and the specific semantic information contained in each voxel.

[0003] However, most existing occupancy prediction methods focus on urban road environments, while research on the three-dimensional semantic occupancy perception problem in off-road environments is still in its early stages. Off-road scenarios are characterized by significantly uneven ground and a large number of complex and irregularly shaped obstacles (such as fallen trees and hanging branches).

[0004] Furthermore, multimodal fusion is a key means to improve perception performance. However, in off-road and bumpy road conditions, visual features based on view transformations (Lift, Splat, Shoot, LSS algorithms) are prone to generating a large number of false "depth artifacts" in the air. Most existing fusion methods directly stitch together the data, which leads to visual noise compromising the geometric accuracy of the LiDAR, i.e., producing a "negative transfer" phenomenon. At the same time, off-road scenarios lack high-quality 3D semantic annotation data, and traditional model architectures struggle to balance computational efficiency with inference capabilities for hollow objects and smooth terrain.

[0005] Therefore, there is an urgent need for a method for predicting the 3D semantic occupancy of off-road scenes that can automatically generate high-quality labels, effectively suppress visual noise, have long-range geometric reasoning capabilities, and be computationally efficient. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method for predicting the 3D semantic occupancy of off-road scenes based on a geometrically perceptual state space model.

[0007] The technical solution of this invention: A method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptual state-space model, comprising the following steps: Step S1: Construct three-dimensional semantic occupancy truth labels; (1) Two-dimensional semantic perception: First, receive the RGB image. ,in, Image height, 3 represents the image width, corresponding to the RGB three channels; a pre-trained open-vocabulary object detection model is used, loaded with a text prompt vocabulary customized for off-road environments. The semantic tags in the text suggestion dictionary include "path", "tree", "rock" and "grass"; The workflow of the open vocabulary object detection model is as follows: First, the Swin Transformer visual backbone network is used to extract image features from RGB images, while the BERT text encoder is used to extract text features from the text cue vocabulary. Then, the image and text features are input into a cross-modal feature fusion module for interactive alignment, calculating their dot product similarity. Based on the dot product similarity, the image regions that best match the text description are selected, generating a series of object candidate boxes. Next, the segmentation model SAM is used for refined segmentation: the object candidate boxes are used as location cues input to the segmentation model SAM, which uses its pixel-level decoder to infer within the range defined by the object candidate boxes, identifying the object's edge contours and generating a binary mask that distinguishes the object from the background. Finally, the semantic mapping function is executed, utilizing the preset semantic mapping function. The identified text labels Convert to the corresponding physical property category Among them, text labels It is an open vocabulary object detection model based on cross-modal feature alignment logic, which analyzes text prompt vocabularies. The optimal semantic term matched from the data; The mathematical expression for obtaining the physical attribute category is: in, This is a category for areas that are impassable. Classified as a low-risk area; The mathematical expression for a binary mask is: in, It is the rectangular region of the target object identified by Grounding DINO, which is used as a guide for subsequent segmentation; (2) 3D spatiotemporal fusion: The ROLO-SLAM algorithm is run to calculate the pose transformation matrix of the point cloud in the kth frame relative to the world coordinate system using the lidar odometry. The pose transformation matrix It integrates rotation and translation transformation information in three-dimensional space, and is used to transform the k-th frame point cloud of length N within a time sliding window. The coordinates are uniformly transformed to the current world coordinate system; based on this, the projection geometry constructed using camera intrinsic and extrinsic parameters is used to transform the point cloud of the k-th frame. The three-dimensional spatial points are mapped to the two-dimensional image coordinate system; the pixel coordinates of each three-dimensional spatial point on the two-dimensional image plane are checked to see if they fall into the binary mask. Within the coverage area, semantic labels are projected onto three-dimensional spatial points; finally, through spatial alignment and semantic attribute fusion of multi-frame data in the world coordinate system, a complete multi-frame point cloud fusion set with semantic information is calculated. ; in, Represents the point cloud in the k-th frame. This represents the perspective projection function that projects a point in three-dimensional space onto a two-dimensional image plane. This represents the set of original point clouds in the k-th frame within the current time sliding window; Indicates passing through point After the projected coordinates of a point fall into the mask area, a semantic label is assigned to that point in the three-dimensional space. This represents the point cloud in frame k relative to the world coordinate system. This represents the rigid body transformation process that uses rotation and translation transformation information to transform point p from the local coordinate system to the world coordinate system; Finally, construct a resolution of [resolution] in the world coordinate system. The voxel mesh, in which, The physical resolution of the voxel mesh is used to divide a continuous 3D space into discrete cubic geometric units; subsequently, point clouds from multiple frames are fused together to form a point cloud. A voxelization and rasterization voting process is performed. This process maps 3D spatial points to corresponding voxel indices and counts the frequency of semantic categories falling within each voxel grid. Following the majority voting principle, the category that appears most frequently within each voxel grid is selected as the final label for that voxel grid. Finally, a label of size [size missing] is generated. Dense three-dimensional semantics occupy the tag ,in 、 、 These represent the total length of the voxel mesh in height, width, and depth, respectively, and the 3D semantic occupancy label. It represents the spatial occupancy state of the scene and the corresponding semantic category information; Step S2: Data preprocessing; Access the original point cloud from the lidar; To address common interferences in off-road environments such as rain, fog, and dust, a spatiotemporal consistency-based filtering strategy is introduced; for each point acquired by the lidar... Calculate the average reflection intensity in its local neighborhood. Spatial distribution variance Taking advantage of the physical characteristics of rain and fog particles having low reflectivity and discrete spatial distribution, a reflectivity threshold is set. Spatial distribution variance threshold The following filtering formula is constructed to remove noise and retain only valid point clouds. : in, This represents the set of valid point clouds retained only after noise removal; symbol , These are the logical NOT and AND operators, used to construct filtering logic to remove interfering points that do not conform to physical consistency; next, the effective point cloud set... The process is transformed into a regularly arranged voxel mesh, ultimately forming an input voxel mesh that can be processed. ; Step S3: Refined extraction of dual-stream features; In the lidar processing branch, the three-dimensional voxel mesh output in step S2 is used as input; where the three-dimensional voxel mesh is a collection that divides space into multiple independent small squares to extract the shape and position features of the object. In the camera processing branch, semantic features of the RGB image are extracted; in the LiDAR processing branch, a sparse residual network is used to extract features: convolution operations are performed only on the non-empty voxel space, skipping air regions, and after downsampling, sparse geometric features that preserve the geometric topology are extracted. ; The camera processing branch utilizes the LSS view transformation mechanism, and the specific execution process is as follows: First, a ResNet-101 deep residual network is used as the backbone to extract multi-scale features from the RGB image; then, a Feature Pyramid Network (FPN) is used to fuse the multi-scale features, outputting 2D image features containing rich semantic information through top-down paths and lateral connections; second, a Lift operation is performed; using a depth prediction head, for each pixel in the 2D image features... Calculate its probability value at different preset depths to obtain a discrete depth distribution. Simultaneously, the semantic context features of the pixel are extracted using the context network head. Subsequently, an outer product operation is performed to orthogonally combine the semantic context features with the discrete depth probability distribution, "lifting" each pixel into a view frustum feature distributed along the viewing direction. The calculation formula is as follows: Finally, the Splat and Shoot operations are performed; first, based on the camera's intrinsic and extrinsic parameter matrices, the spatial position of each feature point in the frustum feature in the world coordinate system is calculated; where the frustum feature is a set of features formed by "lifting" pixels along the viewing direction; Subsequently, the view frustum features are mapped to a voxel coordinate system aligned with the LiDAR processing branch. This voxel coordinate system is a pre-defined 3D meshed spatial coordinate system designed to achieve cross-modal alignment. Next, a Sum Pooling operation is performed on all feature points falling within the same voxel mesh. This involves accumulating overlapping feature vectors within the voxel mesh to generate dense visual voxel features. This visual voxel feature It maintains geometric consistency with the features output by the LiDAR processing branch in terms of spatial structure and dimension, thereby achieving accurate expression of image semantics in three-dimensional voxel space; Step S4: Geometrically guided gating fusion module; Using the physical existence of the original point cloud from the lidar as a geometric prior, the visual voxel features generated by the camera processing branch are analyzed. Spatial suppression is performed; firstly, a three-dimensional geometric gating network is constructed to suppress the sparse geometric features extracted in step S3. Input into a 3D geometric gating network, through The convolutional layer compresses the feature channels to 1 dimension and then maps them through the Sigmoid activation function to generate a feature vector corresponding to the visual voxel features. Spatial gating diagram with spatial dimensions aligned The probability map of occupancy in three-dimensional space Each value in The confidence level of the existence of physical entities within the 3D voxel space is characterized; subsequently, asymmetric feature filtering based on geometric gating is performed: utilizing the 3D spatial occupancy probability map. As a soft mask, visual voxel features Perform the element-wise Hadamard product operation, the formula is as follows: Under the Hadamard product operation, if the 3D voxel mesh output by the LiDAR processing branch determines that a certain spatial location is empty, then the visual voxel feature corresponding to that spatial location... The image will be forcibly suppressed to 0; if the LiDAR processing branch determines that an object exists, the corresponding RGB image features will be retained; finally, a residual connection method will be used to filter and clean the visual features. Superimposed sparse geometric features In the process, the output is a multimodal fusion feature that integrates precise geometric structure and rich semantic texture. Its final mathematical expression is as follows: in, For the Sigmoid function, For the Hadama product operator; Step S5: Physically sensing the Mamba backbone network module; The multimodal fusion features output from step S4 are input into a physical sensing network based on a linear state space; the specific execution process is as follows: Step S5.1: Discretization of continuous physical processes; Using the zero-order preservation technique, the state transition matrix describing a continuous physical process is... Discretize the input control matrix B to generate discrete parameters that can be called by the physical sensing Mamba backbone network. and Where A represents the evolution trend of the negative state in three-dimensional space, and B represents the influence of the current input feature; obtained through discretization and This enables the physical perception network to have the ability to memorize and recursively process the context of three-dimensional voxel space. Step S5.2: Alternate scanning using gravity and terrain sensing; Z-axis gravity sensing scan: Scan the three-dimensional voxel space sequence formed by unfolding multimodal fusion features along the gravity direction, i.e. from top to bottom and from bottom to top, in order to extract the vertical geometric distribution features; XY-axis terrain sensing scan: Perform serpentine or Hilbert curve path scans along the horizontal plane; utilize the long-range modeling capability of the linear state-space model to smooth terrain abrupt noise and ensure that the predicted road surface and slope remain continuous in the horizontal space; Step S5.3: Recursively update hidden states; During the scanning process, the input features for each three-dimensional voxel spatial location Hidden state The recursive update follows the core physical reasoning formula: in, The system accumulates historical contextual memory along the scanning path; the final output is a refined feature that integrates global 3D contextual information and is constrained by physical laws. ; Step S6: Prediction results are generated; Refined features of the output from step S5 using a cascaded decoding head A gradual spatial resolution restoration is performed; through deconvolution, the size of the visual voxel feature map containing abstract semantic information is restored from low resolution to the original voxel grid size, ensuring that the prediction results cover every tiny spatial unit within the entire receptive range; then, semantic classification and probability calculation are performed; a linear classification layer is used to map each voxel feature after size restoration to... A pre-defined semantic category space is established. Then, the output of the classification layer is normalized using the Softmax function to calculate the probability distribution of each voxel belonging to each semantic category, ensuring the sum of probabilities for all categories is 1. Finally, maximum likelihood decision is performed: for each voxel, the category index with the highest probability value is selected as the final semantic label for that voxel, thus generating a 3D semantic occupancy grid. The mathematical formula for this process is: in, This indicates a cascaded decoding operation; the final output is... The results of 3D occupancy prediction for off-road environments include precise geometric structure and rich semantic information; Step S7: Model loss calculation and parameter iteration; Construct supervised training signals and use the prediction results output in step S6. The three-dimensional semantic occupancy truth labels generated in step S1 Error calculation is performed; a combination of the weighted cross-entropy loss function and the Lovasz-Softmax loss function is used to measure the prediction bias, and the calculation formula is as follows: in, This is the balance coefficient; Weighted Cross-Entropy Loss is used to measure the pixel-by-pixel difference between the predicted probability distribution and the true label. The Lovasz-Softmax loss, based on submodule intersection convex relaxation, is used to optimize the average intersection-union ratio (mIoU) and alleviate the class imbalance problem; the calculated total loss is used as the basis. The gradient is calculated using the backpropagation algorithm, and the parameters of each neural network module in steps S3 to S6 are updated (including...). (Parameter matrix and convolution kernel weights) until the model converges, thereby achieving high-precision prediction of the 3D semantic occupancy of off-road scenes.

[0008] The beneficial effects of this invention are: 1. The method of this invention proposes an automated strategy for constructing ground truth labels for three-dimensional semantic occupancy and for data preprocessing. It generates high-precision labels by using open vocabulary target detection, segmentation models and spatiotemporal fusion technology, which solves the problem of scarce off-road data. At the same time, it introduces a filtering strategy based on spatiotemporal consistency, which effectively removes environmental noise such as rain, fog and dust, and significantly improves the robustness of the model under severe weather conditions. 2. The method of the present invention designs a geometry-guided gated fusion module, which uses the physical existence of LiDAR point clouds as a geometric prior to construct a three-dimensional spatial occupancy probability map, performs spatial gated screening on visual features, effectively suppresses false features generated by camera view transformation, and achieves accurate alignment of multimodal features in geometric structure and semantic texture. 3. The method of this invention constructs a physical perception Mamba backbone network, introduces a linear state space model to replace the traditional convolutional network, and enhances the model's ability to model long-range contexts in off-road scenarios through an alternating scanning strategy of gravity perception and terrain perception, ensuring the continuity and physical consistency of road surface and slope in three-dimensional spatial prediction. Attached Figure Description

[0009] Figure 1 A process diagram for constructing three-dimensional semantic occupancy tags provided in the embodiments of this application; Figure 2 This is a schematic diagram of the overall process of the off-road scene 3D semantic occupancy prediction method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the geometry-guided gating fusion module in the embodiments of this application; Figure 4 This is a schematic diagram of the physical sensing Mamba backbone network principle in the embodiments of this application. Detailed Implementation

[0010] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0011] To verify the effectiveness of the proposed method, model training and comparative experiments were conducted on an off-road scenario dataset. This dataset contains 10,000 frames of strictly time-synchronized LiDAR point cloud and RGB image data, covering typical complex off-road scenarios such as muddy roads, bushes, rocky slopes, shallow water areas, and soft sand, exhibiting rich terrain variations and dynamic disturbances, such as… Figure 1 The process of building the labels shown.

[0012] 1. Dataset Construction and Experiment Setup (1) Data partitioning: The dataset is divided into training set, validation set and test set in a ratio of 7:1:2 to ensure that there is no overlap between training and testing scenarios, so as to objectively evaluate the generalization ability of the model.

[0013] (2) Hardware platform: Model training is performed on a server equipped with four NVIDIA RTX 3090 GPUs, each with 24GB of video memory, using a distributed data parallelism (DDP) strategy.

[0014] (3) Training parameter settings: Optimizer: AdamW was used, with an initial learning rate of 2×10⁻⁶. -4 ; Learning rate scheduling: Cosine annealing is used, which eliminates hot restarts and dynamically decays the learning rate throughout the training process; Weight Decay: Set to 0.01 to enhance model regularization; Batch Size: Set to 2 per card, global Batch Size = 8; Training epochs: A total of 24 epochs are trained, with each epoch traversing the entire training set. The model's performance tends to converge on the validation set.

[0015] 2. Model Training Process (1) Input preprocessing: Spatiotemporal alignment and filtering are performed on the original point cloud to remove motion blur and noise points; image data is normalized to the [0,1] interval and random cropping and color dithering enhancement are performed.

[0016] (2) Network structure implementation: a. The LiDAR branch uses SparseConvNet to extract sparse geometric features, with a convolution kernel size of 3×3×3 and 64 output channels. b. The camera branch uses ResNet-101 as the backbone network and combines it with the FPN structure to output multi-scale feature maps. It generates voxel features in the BEV space through LSS (Lift-Splat-Shoot) view transformation. c. Geometrically guided gating fusion module (e.g.) Figure 3 In the diagram shown, the spatial gating map is generated by a 1×1×1 convolution + Sigmoid, with a gating threshold of 0.1 to suppress invalid visual responses. d. Physical sensing Mamba backbone network (e.g.) Figure 4 As shown, sequence modeling is performed along the Z-axis (height direction) and the XY plane (using Hilbert curve scanning), with state dimensions... =128, discretization step size Δ=0.1.

[0017] (3) Loss function: A composite loss function is used for end-to-end training. in, =1.0, , For weighted cross loss, Lovasz-Softmax loss is used to optimize segmentation performance under class imbalance.

[0018] 3. Testing and Reasoning Process (1) After the model training converges, load the optimal weights for testing.

[0019] (2) Testing phase, such as Figure 2 As shown, the overall process of the model is executed sequentially: a. Data preprocessing module: Performs reflection intensity and variance filtering on the lidar point cloud to remove interference points caused by rain and fog; b. Dual-stream feature extraction: Parallel extraction of radar and image features; c. Feature fusion and sequence modeling: Feature fusion is achieved through geometrically guided gating and Mamba backbone network processing. d. Decoding output: 3D deconvolution (kernel size=2×2×2, stride=2) is used to restore the resolution, and the final output is a 200×200×16 3D semantic occupancy prediction map.

[0020] 4. Evaluation Indicators and Result Analysis The evaluation metric adopted is the mean intersection over union (mIoU), a common metric for 3D semantic segmentation. Its calculation formula is as follows: in, Representing categories The number of true positive, false positive and false negative voxels.

[0021] Experimental results show that the proposed method achieves an mIoU of 19.9% ​​and an occupancy prediction rate IoU of 30.5% on the test set, which is significantly better than existing methods based on pure point clouds or pure images, verifying the effectiveness and robustness of the present invention in complex off-road scenarios.

Claims

1. A method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptual state-space model, characterized in that, The steps are as follows: Step S1: Construct three-dimensional semantic occupancy truth labels; Step S2: Data preprocessing; Step S3: Refined extraction of dual-stream features; Step S4: Geometrically guided gating fusion module; Step S5: Physically sensing the Mamba backbone network module; Step S6: Prediction results are generated; Step S7: Model loss calculation and parameter iteration.

2. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: (1) Two-dimensional semantic perception: First, receive the RGB image. ,in, Image height, 3 represents the image width, corresponding to the RGB three channels; a pre-trained open-vocabulary object detection model is used, loaded with a text prompt vocabulary customized for off-road environments. The semantic tags in the text suggestion dictionary include "path", "tree", "rock" and "grass"; The workflow of the open vocabulary object detection model is as follows: First, the Swin Transformer visual backbone network is used to extract image features from RGB images, while the BERT text encoder is used to extract text features from the text cue vocabulary. Then, the image and text features are input into a cross-modal feature fusion module for interactive alignment, calculating their dot product similarity. Based on the dot product similarity, the image regions that best match the text description are selected, generating a series of object candidate boxes. Next, the segmentation model SAM is used for refined segmentation: the object candidate boxes are used as location cues input to the segmentation model SAM, which uses its pixel-level decoder to infer within the range defined by the object candidate boxes, identifying the object's edge contours and generating a binary mask that distinguishes the object from the background. Finally, the semantic mapping function is executed, utilizing the preset semantic mapping function. The identified text labels Convert to the corresponding physical property category Among them, text labels It is an open vocabulary object detection model based on cross-modal feature alignment logic, which analyzes text prompt vocabularies. The optimal semantic term matched from the data; The mathematical expression for obtaining the physical attribute category is: ； in, This is a category for areas that are impassable. Classified as a low-risk area; The mathematical expression for a binary mask is: in, It is the rectangular region of the target object identified by Grounding DINO, which is used as a guide for subsequent segmentation; (2) 3D spatiotemporal fusion: The ROLO-SLAM algorithm is run to calculate the pose transformation matrix of the point cloud in the kth frame relative to the world coordinate system using the lidar odometry. The pose transformation matrix It integrates rotation and translation transformation information in three-dimensional space, and is used to transform the k-th frame point cloud of length N within a time sliding window. The coordinates are uniformly transformed to the current world coordinate system; based on this, the projection geometry constructed using camera intrinsic and extrinsic parameters is used to transform the point cloud of the k-th frame. The three-dimensional spatial points are mapped to the two-dimensional image coordinate system; the pixel coordinates of each three-dimensional spatial point on the two-dimensional image plane are checked to see if they fall into the binary mask. Within the coverage area, semantic labels are projected onto three-dimensional spatial points; finally, through spatial alignment and semantic attribute fusion of multi-frame data in the world coordinate system, a complete multi-frame point cloud fusion set with semantic information is calculated. ; ； ； in, Represents the point cloud in the k-th frame. This represents the perspective projection function that projects a point in three-dimensional space onto a two-dimensional image plane. This represents the set of original point clouds in the k-th frame within the current time sliding window; Indicates passing through point After the projected coordinates of a point fall into the mask area, a semantic label is assigned to that point in the three-dimensional space. This represents the point cloud in frame k relative to the world coordinate system. This represents the rigid body transformation process that uses rotation and translation transformation information to transform point p from the local coordinate system to the world coordinate system; Finally, construct a resolution of [resolution] in the world coordinate system. The voxel mesh, in which, The physical resolution of the voxel mesh is used to divide a continuous 3D space into discrete cubic geometric units; subsequently, point clouds from multiple frames are fused together to form a point cloud. A voxelization and rasterization voting process is performed. This process maps 3D spatial points to corresponding voxel indices and counts the frequency of semantic categories falling within each voxel grid. Following the majority voting principle, the category that appears most frequently within each voxel grid is selected as the final label for that voxel grid. Finally, a label of size [size missing] is generated. Dense three-dimensional semantics occupy the tag ,in 、 、 These represent the total length of the voxel mesh in height, width, and depth, respectively, and the 3D semantic occupancy label. It represents the spatial occupancy state of the scene and the corresponding semantic category information.

3. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 2, characterized in that, The specific implementation process of step S2 is as follows: Access the original point cloud from the lidar; To address common interferences in off-road environments such as rain, fog, and dust, a spatiotemporal consistency-based filtering strategy is introduced; for each point acquired by the lidar... Calculate the average reflection intensity in its local neighborhood. Spatial distribution variance Taking advantage of the physical characteristics of rain and fog particles having low reflectivity and discrete spatial distribution, a reflectivity threshold is set. Spatial distribution variance threshold The following filtering formula is constructed to remove noise and retain only valid point clouds. : ； in, This represents the set of valid point clouds retained only after noise removal; symbol , These are the logical NOT and logical AND operators, used to construct filtering logic to filter out interfering points that do not conform to physical consistency; next, the effective point cloud set... The process is transformed into a regularly arranged voxel mesh, ultimately forming an input voxel mesh that can be processed. .

4. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 3, characterized in that, The specific implementation process of step S3 is as follows: In the lidar processing branch, the three-dimensional voxel mesh output in step S2 is used as input; where the three-dimensional voxel mesh is a collection that divides space into multiple independent small squares to extract the shape and position features of the object. In the camera processing branch, semantic features of the RGB image are extracted; the LiDAR processing branch uses a sparse residual network to extract features: convolution operations are performed only on the non-empty voxel space, skipping air regions, and after downsampling, sparse geometric features that preserve the geometric topology are extracted. ; The camera processing branch utilizes the LSS view transformation mechanism, and the specific execution process is as follows: First, a ResNet-101 deep residual network is used as the backbone to extract multi-scale features from the RGB image; then, a Feature Pyramid Network (FPN) is used to fuse the multi-scale features, outputting 2D image features containing rich semantic information through top-down paths and lateral connections; second, a Lift operation is performed; using a depth prediction head, for each pixel in the 2D image features... Calculate its probability value at different preset depths to obtain a discrete depth distribution. Simultaneously, the semantic context features of the pixel are extracted using the context network head. Subsequently, an outer product operation is performed to orthogonally combine the semantic context features with the discrete depth probability distribution, "lifting" each pixel into a view frustum feature distributed along the viewing direction. The calculation formula is as follows: ； Finally, the Splat and Shoot operations are performed; first, based on the camera's intrinsic and extrinsic parameter matrices, the spatial position of each feature point in the frustum feature in the world coordinate system is calculated; where the frustum feature is a set of features formed by "lifting" pixels along the viewing direction; Subsequently, the view frustum features are mapped to a voxel coordinate system aligned with the LiDAR processing branch. This voxel coordinate system is a pre-defined 3D meshed spatial coordinate system designed to achieve cross-modal alignment. Next, a Sum Pooling operation is performed on all feature points falling within the same voxel mesh. This involves accumulating overlapping feature vectors within the voxel mesh to generate dense visual voxel features. This visual voxel feature It maintains geometric consistency with the features output by the LiDAR processing branch in terms of spatial structure and dimension, thereby achieving accurate expression of image semantics in three-dimensional voxel space.

5. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 4, characterized in that, The specific implementation process of step S4 is as follows: Using the physical existence of the original point cloud from the lidar as a geometric prior, the visual voxel features generated by the camera processing branch are analyzed. Spatial suppression is performed; firstly, a three-dimensional geometric gating network is constructed to suppress the sparse geometric features extracted in step S3. Input into a 3D geometric gating network, through The convolutional layer compresses the feature channels to 1 dimension and then maps them through the Sigmoid activation function to generate a feature vector corresponding to the visual voxel features. Spatial gating diagram with spatial dimensions aligned ; This three-dimensional space occupancy probability map Each value in The confidence level of the existence of physical entities within the 3D voxel space is characterized; subsequently, asymmetric feature filtering based on geometric gating is performed: utilizing the 3D spatial occupancy probability map. As a soft mask, visual voxel features Perform the element-wise Hadamard product operation, the calculation formula is as follows: ; Under the Hadamard product operation, if the 3D voxel mesh output by the LiDAR processing branch determines that a certain spatial location is empty, then the visual voxel feature corresponding to that spatial location... The image will be forcibly suppressed to 0; if the LiDAR processing branch determines that an object exists, the corresponding RGB image features will be retained; finally, a residual connection method will be used to filter and clean the visual features. Superimposed sparse geometric features In the process, the output is a multimodal fusion feature that integrates precise geometric structure and rich semantic texture. Its final mathematical expression is as follows: ； in, For the Sigmoid function, For the Hadama product operator.

6. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 5, characterized in that, The specific implementation process of step S5 is as follows: The multimodal fusion features output from step S4 are input into a physical sensing network based on a linear state space; the specific execution process is as follows: Step S5.1: Discretization of continuous physical processes; Using the zero-order preservation technique, the state transition matrix describing a continuous physical process is... Discretize the input control matrix B to generate discrete parameters that can be called by the physical sensing Mamba backbone network. and Where A represents the evolution trend of the negative state in three-dimensional space, and B represents the influence of the current input feature; obtained through discretization and This enables the physical perception network to have the ability to memorize and recursively process the context of three-dimensional voxel space. Step S5.2: Alternate scanning using gravity and terrain sensing; Z-axis gravity sensing scan: Scan the three-dimensional voxel space sequence formed by unfolding multimodal fusion features along the gravity direction, i.e. from top to bottom and from bottom to top, in order to extract the vertical geometric distribution features; XY-axis terrain sensing scan: Perform serpentine or Hilbert curve path scans along the horizontal plane; utilize the long-range modeling capability of the linear state-space model to smooth terrain abrupt noise and ensure that the predicted road surface and slope remain continuous in the horizontal space; Step S5.3: Recursive update of hidden states; During the scanning process, the input features for each three-dimensional voxel spatial location Hidden state The recursive update follows the core physical reasoning formula: ； in, The system accumulates historical contextual memory along the scanning path; the final output is a refined feature that integrates global 3D contextual information and is constrained by physical laws. .

7. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 6, characterized in that, The specific implementation process of step S6 is as follows: Refined features of the output from step S5 using a cascaded decoding head A gradual spatial resolution restoration is performed; through deconvolution, the size of the visual voxel feature map containing abstract semantic information is restored from low resolution to the original voxel grid size, ensuring that the prediction results cover every tiny spatial unit within the entire receptive range; then, semantic classification and probability calculation are performed; a linear classification layer is used to map each voxel feature after size restoration to... A predefined semantic category space is defined; then, the output of the classification layer is normalized using the Softmax function to calculate the probability distribution value of each voxel belonging to each semantic category, so that the sum of the probabilities of all categories is 1. Finally, the maximum likelihood decision is performed: for each voxel, the category index with the highest probability value is selected as the final semantic label for that voxel, thereby generating a 3D semantic occupancy grid. The mathematical formula for this process is: ； in, This indicates a cascaded decoding operation; the final output is... This provides 3D occupancy prediction results for off-road environments that incorporate precise geometric structures and rich semantic information.

8. The method for predicting 3D semantic occupancy in off-road scenes based on a geometrically perceptive state-space model according to claim 7, characterized in that, The specific implementation process of step S7 is as follows: Construct supervised training signals and use the prediction results output in step S6. The three-dimensional semantic occupancy truth labels generated in step S1 Error calculation is performed; a combination of the weighted cross-entropy loss function and the Lovasz-Softmax loss function is used to measure the prediction bias, and the calculation formula is as follows: ； in, This is the balance coefficient; The weighted cross-entropy loss is used to measure the pixel-by-pixel difference between the predicted probability distribution and the true label. The Lovasz-Softmax loss, based on submodule intersection convex relaxation, is used to optimize the average intersection-union ratio and alleviate the class imbalance problem; the total loss is calculated based on... The gradient is calculated using the backpropagation algorithm, and the parameters of each module in steps S3 to S6 are updated until convergence, thereby achieving high-precision prediction of the three-dimensional semantic occupancy of the off-road scene.

Images