A columnar height piecewise supervised and local spline decoding three-dimensional semantic occupancy prediction method and system

Abstract

The application provides a columnar height segmented supervision and local spline decoding three-dimensional semantic occupancy prediction method and system, the method comprises the following steps: step 1, input multi-view image, laser radar point cloud and vehicle pose information; step 2, construct bird's eye view BEV column grid, and generate columnar height segmented true value, voxel visibility mask and occlusion soft label; step 3, extract multi-view image features, and perform view cone space conversion to generate bird's eye view BEV features; step 4, predict ordered height segmented parameters based on bird's eye view BEV column, and recover three-dimensional semantic occupancy results by using local spline decoding.The application unifies true value and predicted object into a limited height segment in the same bird's eye view BEV column, so that the supervision target and network output are one-to-one corresponding in expression level, and learning deviation caused by inconsistent expression is reduced.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of computer vision, deep learning, and intelligent transportation systems, and in particular to a method and system for predicting 3D semantic occupancy using columnar height segmentation supervision and local spline decoding. Background Technology

[0002] In autonomous driving systems, vehicles need a stable, continuous, and decision-making-ready three-dimensional representation of their surroundings. Compared to two-dimensional object detection, two-dimensional semantic segmentation, or sparse three-dimensional bounding boxes, three-dimensional semantic occupancy prediction, by predicting the occupancy state and semantic category of each voxel within a predefined three-dimensional space, can more completely represent the spatial distribution relationships of roads, curbs, vehicles, pedestrians, cyclists, buildings, vegetation, and suspended structures. Therefore, it has become an important intermediate representation in autonomous driving perception systems.

[0003] Existing visual 3D semantic occupancy prediction methods typically use multi-view camera images as input. First, image features are extracted using a 2D backbone network. Then, depth distribution estimation, frustum lifting, or differentiable projection are combined to map these features to the bird's-eye view space. Finally, a bird's-eye view BEV encoder or 3D decoder outputs voxel-level semantic results. To balance accuracy and real-time performance, many methods adopt a "2D bird's-eye view BEV encoding + fixed-height layer recovery" approach. This involves first performing efficient convolutions in the bird's-eye view BEV plane, and then recovering the 3D occupancy result through channel rearrangement, 3D convolution, or height-wise layer classification.

[0004] However, the above-mentioned technical approaches still have the following problems in complex autonomous driving scenarios.

[0005] First, existing methods typically classify the semantics of voxels at discrete height levels layer by layer, or directly map the feature channels of BEVs from a bird's-eye view to voxel outputs at fixed height levels. These methods do not explicitly model the starting and ending boundaries and segmentation relationships of targets within the same BEV column from the same bird's-eye view in the height direction. This leads to geometrical breaks, drifts, semantic crosstalk between upper and lower layers, and floating voxel phenomena in the height direction, especially noticeable in areas with complex longitudinal structures such as roadside undulations, vehicle bodies and roofs, tree canopies and ground vegetation, and the underside of elevated roads.

[0006] Second, most existing methods use discrete voxel hard labels as the main supervision signals. However, in real-world autonomous driving scenarios, a large number of voxels are located in occluded areas, blind spots, or sparsely covered point clouds, and the labels in these areas are highly uncertain. If these voxels are still treated together with directly observable voxels as deterministic supervision samples, training noise can easily be introduced, resulting in the network achieving high classification confidence on local voxels but unstable overall 3D topology.

[0007] Third, most existing long-tail class balancing methods only weight samples or losses based on class frequency, failing to distinguish the differences in geometric boundary complexity among different classes. For targets with drastic boundary changes and small spatial scales, such as pedestrians, cyclists, and traffic cones, traditional frequency balancing is insufficient to effectively improve the prediction quality of their height boundaries.

[0008] Furthermore, the main shortcomings of existing technologies are not simply insufficient capabilities of the 2D backbone network, inaccurate depth distribution estimation, or inadequate loss function design, but rather the lack of a unified structural representation foundation among the real spatial structure of 3D occupancy, the representation of training labels, and the decoding form of network output. Existing methods typically represent real scenes as voxel-by-voxel discrete labels, while implementing the prediction process as fixed-height layer classification, channel rearrangement, or 3D convolutional reconstruction. This results in a structural mismatch between the supervised object and the reconstructed object: on the one hand, in real traffic scenarios, the same bird's-eye view BEV column often consists of a small number of continuous height intervals with clear lower and upper boundaries and segmentation relationships; on the other hand, existing models still output results using independent classification methods on a large number of discrete height layers. This inconsistency in representation is the root cause of noise amplification in occluded areas, difficulty in boundary learning, and vertical topological instability. Therefore, a new technical solution is needed to enable the 3D occupancy of the same bird's-eye view BEV column to adopt a consistent structural representation in the three stages of ground truth construction, model prediction, and voxel reconstruction, thereby improving the structural stability and learnability of 3D semantic occupancy prediction for autonomous driving. Summary of the Invention

[0009] Objective of the Invention: This invention provides a column-based height segmentation supervision and local spline decoding method and system for predicting 3D semantic occupancy, addressing the following problems in existing technologies: First, fixed-height-layer classification lacks explicit constraints on the upper and lower boundary relationships within the same bird's-eye view BEV column, leading to unstable topological structures in the height direction; second, discrete voxel hard labels are difficult to accurately characterize occluded areas and areas of observation uncertainty; third, existing voxel recovery methods do not match the continuous variation patterns of real 3D geometry, resulting in inaccurate boundary representation; and fourth, existing class balancing strategies are difficult to simultaneously consider class frequency and boundary complexity.

[0010] To this end, the present invention provides a technical solution using the bird's-eye view BEV column as the smallest modeling unit: during the training phase, a finite number of semantic height segment ground values ​​are constructed for each bird's-eye view BEV column, and visible voxels and occluded uncertain voxels are distinguished; during the prediction phase, multiple height segments satisfying the upper and lower ordered constraints are predicted for each bird's-eye view BEV column, and the semantic response of voxels within each segment is recovered using local spline continuous functions, thereby realizing the generation of three-dimensional semantic occupancy results.

[0011] The method includes the following steps:

[0012] Step 1: Input multi-view images, LiDAR point clouds, and vehicle pose information;

[0013] Step 2: Construct a bird's-eye view BEV column mesh and generate column height segmentation ground truth, voxel visibility mask and occlusion soft label;

[0014] Step 3: Extract multi-view image features and perform view cone space transformation to generate bird's-eye view BEV features;

[0015] Step 4: Predict ordered height segmentation parameters based on the BEV column from a bird's-eye view, and use local spline decoding to recover the 3D semantic occupancy result.

[0016] Step 2 includes: presetting the occupied space range and corresponding voxel resolution, forming a regular voxel grid and a bird's-eye view BEV column grid that corresponds one-to-one with the regular voxel grid on the plane.

[0017] Step 2 also includes: setting the target monitoring time as Select time window Multiple frames of LiDAR point cloud within the time frame, for a period of time The j-th point in the point cloud is subjected to pose transformation, with the time being... Transformation of the j-th point cloud point to the target supervision time In the vehicle's coordinate system:

[0018] ,

[0019] in, Indicates time as The j-th point in the point cloud; Indicates the transformation to the target supervision time. The j-th point in the point cloud; Indicates from time coordinate system to target monitoring time The pose transformation matrix of the coordinate system; represents the point cloud frame time; j represents the point cloud point number.

[0020] For planar grid index Define the planar mesh index The corresponding bird's-eye view BEV column is as follows:

[0021] ,

[0022] in, Indicates the planar grid index is A bird's-eye view of the BEV column; Indicates the grid index in the forward and backward directions; Indicates the grid index in the left and right directions; Indicates the coordinate values ​​in the forward and backward directions; Indicates the coordinate values ​​in the left and right directions; Indicates the coordinate value in the height direction; Indicates the voxel resolution in the forward and backward directions; Indicates the voxel resolution in the left and right directions; Indicates the lower boundary of the space occupied in the front and back directions; Indicates the lower boundary of the space occupied in the left and right directions; Indicates the lower boundary of the occupied space in the height direction; Indicates the boundary of the occupied space in the height direction;

[0023] The compensated multi-frame point clouds and their semantic categories are projected onto the BEV column of each bird's-eye view. For any bird's-eye view BEV column and semantic category index q, the height sample set is defined as follows:

[0024] ,

[0025] in, Indicates the planar grid index is A set of samples with a semantic category index of q; This represents the nth height value in the height sample set; Indicates the total number of height samples;

[0026] Sort the height sample set in ascending order of height, and denote the sorted sequence as:

[0027] ,

[0028] in, This represents the n height values ​​sorted from smallest to largest; the numbers in parentheses indicate their order after sorting.

[0029] The following rule is used for segmentation: when two adjacent height values ​​satisfy... At that time, at the height value and height value The segments are broken apart; among them, Indicates the sorting position index; This represents the height segmentation threshold; in this embodiment, it is taken as... When two adjacent height values and When corresponding to the visible area and the uncertain occlusion area respectively, at the height value and height value Disconnect between segments;

[0030] For each candidate segment, let the number of height samples contained in the candidate segment be . When satisfied When this occurs, it is considered an invalid segment and discarded; among them, Indicates the first The number of height samples contained in each candidate segment; Indicates the candidate segment number; This represents the height sample threshold; in this embodiment, it is taken as... .

[0031] For the retained valid segments, quantiles are used to improve robustness: Let the height samples within each segment be arranged in ascending order. Then the lower and upper boundaries are taken as the 5th percentile and 95th percentile, respectively:

[0032] ,

[0033] ,

[0034] in, This represents the lower boundary of the m-th truth segment with semantic category index q; This represents the upper boundary of the m-th truth segment with semantic category index q; Represents the 5% quantile function; The 95th percentile function; This represents all height sample values ​​in the m-th candidate segment;

[0035] Within each bird's-eye view BEV column, each semantic category retains at most two height segments. If a category obtains more than two valid segments within the same bird's-eye view BEV column, the segments are sorted from largest to smallest by the number of samples, and only the two segments with the largest number of samples are retained, ultimately yielding the columnar height segment truth set:

[0036] ,

[0037] in, Indicates the planar grid index is A columnar height segmented truth set with semantic category index q; This indicates the height segment of the first truth value; This indicates the height segmentation of the second truth value.

[0038] If only one valid segment exists, then only the first truth value height segment is retained; if no valid segment exists, then the category with semantic category index q is recorded in the bird's-eye view BEV column. There are no truth value segments in the middle.

[0039] Step 2 also includes: for each voxel Visibility status is determined based on the laser radar ray penetration relationship; if voxels Located before the point of impact of a certain laser beam, and voxel If the space in which the object is located is actually traversed by the ray, then the voxel will be... Mark as an empty visible voxel; if voxel If the hit location is within two or more adjacent point clouds, then the voxel will be... Marked as occupying a visible voxel; if voxel If the voxel is located behind the hit surface and not directly observed by any rays, then the voxel will be... Marked as an occluded uncertain voxel;

[0040] Define the voxel visibility mask as follows: ,in, Voxel representation Visible voxels; Voxel representation To mask uncertain voxels; Voxel representation The visibility mask;

[0041] For each frame, the time is Point cloud, for voxels The following two counts are statistically analyzed:

[0042] Occupy support count Time is voxels falling in point cloud frames Or fall into voxels The number of points within the neighborhood;

[0043] Free support count Time is In the point cloud frame, the voxel passes from the laser origin to the hit point. The number of laser radar beams;

[0044] The support counts for different time frames are weighted by attenuation; the relative target monitoring time within the time window is set. The time offset is The time-domain weights are defined as follows:

[0045] ,

[0046] in, Indicates time as The temporal weights corresponding to the point cloud frames; Represents the natural exponential function; Indicates the time decay coefficient;

[0047] For occlusion uncertain voxels, the occupancy confidence is defined as:

[0048] ,

[0049] in, Voxel representation The confidence level of occupation; Indicates time as voxels The number of occupied support counts; Indicates time as voxels Free support count; This represents a very small constant to prevent the denominator from being zero;

[0050] For occlusion uncertain voxels, generate a binary soft-label distribution: Let the category corresponding to the occlusion uncertain voxel be q, then define:

[0051] ,

[0052] ,

[0053] in, Voxel representation Semantic category index The soft tag value; Voxel representation The soft label value belonging to the empty voxel category; the superscript "soft" indicates a soft label; the semantic category index 0 indicates the empty voxel category.

[0054] Finally, the following supervision information is output during the training phase: visible voxel hard labels; occluded voxel soft labels; and ground truth values ​​for the BEV column height segments in the bird's-eye view.

[0055] Step 3 includes:

[0056] Assume the number of cameras is The input image set I is:

[0057] ,

[0058] in, This represents the image captured by the i-th camera; i represents the camera number.

[0059] A two-dimensional convolutional backbone network is used to extract image features, with a 50-layer ResNet-50 as the backbone network. Multi-scale features are then fused using a Feature Pyramid Network (FPN) to obtain image features with a uniform number of channels.

[0060] ,

[0061] in, This represents the two-dimensional image feature tensor corresponding to the i-th camera; Represents the real number field; Indicates the number of image feature channels; Indicates the height of image features; Indicates the width of image features;

[0062] For each image feature location, predict the discrete depth distribution, assuming the total number of discrete depth layers is . ,get:

[0063] ,

[0064] in, The image feature coordinates are At the Discrete depth probabilities at each depth layer; Represents the horizontal coordinates of image features; Represents the vertical coordinates of image features; Indicates the depth layer index; Indicates the total number of discrete depth layers; Indicates the depth prediction branch; This represents the soft-maximum normalized activation function; Indicates the first The camera images are at coordinates Image feature values ​​at the location;

[0065] After constructing view frustum features by combining image features with depth distribution, the view frustum features are projected onto the vehicle coordinate system using camera intrinsic and extrinsic parameters. Then, the features within the same grid are weighted and summed according to the bird's-eye view BEV grid to obtain the bird's-eye view BEV feature map. ,in, This represents a bird's-eye view of BEV feature maps; This indicates the number of channels in the BEV feature map from a bird's-eye view. This indicates the height of the BEV feature map from a bird's-eye view. This indicates the width of the BEV feature map from a bird's-eye view.

[0066] The bird's-eye view BEV feature map is input into a two-dimensional convolutional encoder to further extract contextual information, and the encoded bird's-eye view BEV features are output:

[0067] ,

[0068] in, This represents the encoded bird's-eye view BEV features; This represents the number of BEV feature channels in the encoded bird's-eye view.

[0069] Step 4 includes:

[0070] For each bird's-eye view BEV column feature, the network outputs parameters for at most two candidate height segments for each semantic category; for the m-th predicted segment and semantic category index q, the output parameters include: lower boundary offset. Upper boundary offset Segment Existence Score Segmented semantic response coefficient Local spline coefficients ;in, This represents the index of the spline basis function; the total number of spline basis functions is... ;

[0071] The boundary is constructed using cumulative positive value parameterization, and the positive value smoothing function is defined as follows:

[0072] ,

[0073] in, Represents a positive smoothing function; Indicates intermediate parameters; Represents the natural constant;

[0074] For the first predicted segment, the ordered boundary parameterization formula is:

[0075] , ,

[0076] For the second prediction segment:

[0077] , ,

[0078] in, The semantic category index is The lower boundary of the first predicted segment; The semantic category index is The upper boundary of the first predicted segment; The semantic category index is The lower boundary of the second predicted segment; The semantic category index is The upper boundary of the second predicted segment; The semantic category index is The first predicted segment lower boundary offset; The semantic category index is The first predicted segment upper boundary offset; The semantic category index is The second predicted segment lower boundary offset; The semantic category index is The second predicted segment upper boundary offset; Indicates the minimum segment interval;

[0079] Score for segmentation existence Perform Sigmoid normalization; if the segment existence score after normalization is less than a threshold... If the m-th predicted segment is not an invalid segment, then the m-th predicted segment is considered invalid.

[0080] For any height value falling within the predicted segment interval Define height value The locally normalized coordinates are:

[0081] ,

[0082] in, Represents height value In the semantic category index The Local normalized coordinates in each predicted segment;

[0083] For each prediction segment, construct the internal continuous response using B-spline basis functions:

[0084] ,

[0085] in, The semantic category index is The Each predicted segment at height value Local spline continuous response at the location; Indicates the first A cubic B-spline basis function;

[0086] Define a smoothing gate function:

[0087] ,

[0088] in, The semantic category index is The Each predicted segment at height value The gating function value at the location; This represents the Sigmoid activation function; Indicates the boundary steepness coefficient;

[0089] For the height center is For the k-th voxel layer, the response of the m-th predicted segment to the semantic category index q is defined as:

[0090] , ,

[0091] Where k represents the voxel layer index; This represents the height center value of the k-th voxel layer; This represents the response value of the m-th predicted segment with semantic category index q to the k-th voxel layer; This represents the logarithmic response value of the k-th voxel layer belonging to the semantic category index q; This represents the category bias term with semantic category index q;

[0092] Perform Softmax normalization on all semantic categories and the empty voxel category to obtain the voxel semantic probabilities:

[0093] ,

[0094] in, q' represents the probability that the k-th voxel layer belongs to the semantic category index q; q' represents the semantic category index variable in the summation. Represents the total number of semantic categories; semantic category index Indicates the empty voxel category;

[0095] The combination of all bird's-eye view BEV columns and probability tensors on all height layers yields the 3D semantic occupancy prediction results.

[0096] Step 4 also includes:

[0097] For each bird's-eye view BEV column and semantic category index q, a one-to-one matching method based on height order is used:

[0098] When the true value has only one segment, the first predicted valid segment is matched with the unique true value segment;

[0099] When the true value has two segments, matching is performed in ascending order of the lower boundary.

[0100] If the number of predicted segments exceeds the number of true segments, the extra predicted segments are considered invalid segments.

[0101] If the number of true value segments exceeds the number of effective prediction segments, then the missing segments only participate in the existence loss and not in the regression loss.

[0102] Step 4 also includes:

[0103] Near each truth value segment boundary, define a boundary voxel set, for any voxel center height value If the following conditions are met:

[0104] or ,

[0105] The voxel is then called a boundary voxel; where, Indicates the boundary threshold;

[0106] Let the total voxel frequency of the semantic category index q in the training set be... The total number of samples is The boundary voxel number is Then the category weights are defined as follows:

[0107] ,

[0108] in, The semantic category index is Category weights; The semantic category index is The overall voxel frequency; The semantic category index is The total number of samples; The semantic category index is The boundary voxels; Indicates the frequency balance index; This represents the boundary enhancement coefficient.

[0109] Weighted cross-entropy loss is applied to visible voxels:

[0110] ,

[0111] in, This indicates visible voxel supervision loss; Voxel representation The category weight corresponding to the hard tag category; Voxel representation Hard label categories; Voxel representation Predicted as a hard label category The probability of.

[0112] Soft-label monitoring loss is applied to occluded uncertain voxels:

[0113] ,

[0114] in, This indicates the loss of soft supervision due to voxel occlusion; Voxel representation Predicted as a semantic category index The probability of;

[0115] For the prediction segments and ground truth segments of valid matches, the upper and lower boundaries are constrained by the SmoothL1 loss:

[0116] ,

[0117] in, Indicates the boundary regression loss; Indicates the planar grid index is Semantic category index is The segment number is The number of indicators of matching validity; This represents the smoothed L1 loss function;

[0118] Define topological constraint loss:

[0119] ,

[0120] in, Represents the topological constraint loss; This represents the maximum value operation; This represents the relaxation amount of the first type of topological constraint; This represents the relaxation amount of the second type of topological constraint.

[0121] For redundant segments without true counterparts, apply existence suppression loss:

[0122] ,

[0123] in, This indicates an invalid segmentation that suppresses losses; The semantic category index is The The existence score of each predicted segment.

[0124] Total loss function Defined as:

[0125] ,

[0126] in, , , and These represent the loss weight values ​​corresponding to the occlusion voxel soft supervision loss, boundary regression loss, topological constraint loss, and invalid segmentation suppression loss, respectively.

[0127] The present invention also provides a prediction system based on the method, comprising:

[0128] The data acquisition module is used to acquire multi-view images, multi-frame point clouds, and pose information;

[0129] The supervised construction module is used to perform point cloud motion compensation, bird's-eye view BEV column partitioning, column height segmentation ground truth construction, visibility mask construction, and occlusion soft label generation.

[0130] The feature extraction module is used to extract two-dimensional features from multi-view images and generate bird's-eye view BEV features;

[0131] The segmented prediction module is used to predict ordered height segment parameters for the BEV column of each bird's-eye view.

[0132] The local spline decoding module is used to recover voxel-level semantic probabilities based on segmentation parameters and local spline coefficients;

[0133] The training optimization module is used to update network parameters based on the joint loss function;

[0134] The occupancy output module is used to output the three-dimensional semantic occupancy results.

[0135] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0136] The core technical concept of this invention does not lie in simply superimposing several existing modules in the existing three-dimensional semantic occupancy prediction process, but in proposing a structural isomorphic occupancy expression method for BEV columns from a bird's-eye view, and building a unified technical link for supervision, prediction, decoding and optimization around this expression method.

[0137] Specifically, this invention first observes that in autonomous driving scenarios, for any fixed bird's-eye view BEV location, its actual occupancy state along the height direction is usually not composed of a large number of independent discrete voxels randomly, but rather is closer to an ordered structure composed of a small number of continuous occupancy intervals. In other words, the three-dimensional occupancy in a BEV column within the same bird's-eye view is more suitable to be represented as a finite number of height segments with upper and lower boundaries, rather than layer-by-layer independent classification results on a fixed height layer. Based on this understanding, this invention uniformly represents the occupancy structure in a BEV column within the same bird's-eye view as a finite number of ordered height segments, and makes the training label construction, network output, and voxel recovery processes revolve around this segmented representation.

[0138] On the monitoring side, this invention does not directly use the voxel-by-voxel hard label as the unique truth value. Instead, it first extracts a finite number of semantic height segments within each bird's-eye view BEV column based on multi-frame compensated point clouds, forming column-based height segment truth values. These truth values ​​directly describe the lower boundary, upper boundary, and number of segments of each semantic occupier in the same bird's-eye view BEV column, thereby compressing the structural information originally scattered across multiple discrete voxels into a small number of boundary parameters with clear physical meaning.

[0139] On the prediction side, instead of having the network directly output voxel classification results at a fixed height layer, this invention allows the network to directly predict multiple candidate height segments for each bird's-eye view BEV column, and ensures that these segments maintain an ascending order in the height direction through ordered boundary parameterization. Thus, both the network's prediction object and the supervision object are "a finite number of height segments," maintaining structural consistency.

[0140] At the decoding end, this invention does not employ a uniform discrete recovery method for the entire height axis. Instead, for each predicted segment, a continuous response function is constructed within its local height interval, and the semantic response of that segment to each height voxel is generated through local spline decoding. Thus, the discrete voxel result is no longer the raw output of the network's independent classification, but is recovered jointly by the segment structure parameters and their local continuous responses. This decoding method naturally corresponds to the true value of the columnar height segment, making the three stages of "supervision-prediction-recovery" form a unified structural closed loop.

[0141] Furthermore, to ensure stable training of the aforementioned structural closed loop, this invention introduces a supporting mechanism for segment boundary learning at the optimization stage: on the one hand, visibility masks and occlusion confidence are used to distinguish the supervision credibility, avoiding excessive interference from occluded regions on segment ground truth learning; on the other hand, a boundary complexity enhancement term is introduced based on the category frequency weight, making the optimization process more focused on the voxel regions that play a key role in the high segment boundary. Thus, the supervision, prediction, decoding, and optimization mechanisms are not independent of each other, but rather work together to serve the stable learning of the BEV column structured representation from the same bird's-eye view.

[0142] Therefore, the essential difference between this invention and the prior art is not the addition of a single module, but rather the transformation of the three-dimensional semantic occupancy task from a "fixed height layer voxel classification problem" to a "structure prediction and continuous recovery problem of a finite number of ordered height segments within a BEV column from a bird's-eye view", and the establishment of a consistent coupling relationship between label construction, boundary parameterization, local continuous decoding and boundary enhancement optimization around this structure prediction.

[0143] Compared with the prior art, the present invention has the following beneficial effects:

[0144] 1. This invention achieves consistency between supervised and predicted representations. In existing technologies, ground truth values ​​are typically represented as discrete labels per voxel, while model outputs are often obtained through fixed-height layer recovery, 3D convolutional unrolling, or channel rearrangement, resulting in structural inconsistencies. This invention unifies ground truth values ​​and predicted objects into a finite number of height segments within the same bird's-eye view BEV column, thereby ensuring a one-to-one correspondence between the supervised target and the network output at the representational level, reducing learning bias caused by representational inconsistencies.

[0145] 2. The synergistic effect of boundary parameterization and continuous decoding is achieved. The ordered boundary parameterization in this invention is not an independent interval regression method, but a prerequisite for local spline decoding; local spline decoding is not ordinary continuous function fitting, but is used to recover the voxel semantic response within ordered height segments. Together, they constitute the decoding logic of "first determining the segment boundaries, then recovering the continuous structure within the segments," thus improving the topological continuity and boundary stability within the same bird's-eye view BEV column.

[0146] 3. Reduced interference from occlusion noise on structural learning. Since the core learning object in this invention is the height segment boundary and its local response, treating occluded areas as equivalent to visible areas for supervision can easily interfere with boundary learning. Therefore, this invention controls the supervision intensity of occluded areas through visibility masks and soft-label confidence, allowing boundary parameters and segment responses to be driven more by reliable observations, thus improving training stability.

[0147] 4. Improved learning performance for boundary-sensitive categories. Since the prediction targets of this invention are height segments and their boundary locations, samples near the boundaries are more important for model training than ordinary internal samples. Based on this, this invention further combines category frequency weights with the proportion of boundary voxels, providing more sufficient training constraints for small-scale categories with complex boundaries, which helps improve the prediction quality of 3D occupancy boundaries for targets such as pedestrians, cyclists, and traffic cones.

[0148] This invention transforms 3D occupancy prediction from a high-dimensional discrete classification problem into a low-dimensional structure prediction problem. For each bird's-eye view BEV column, this invention predicts only a finite number of segment boundaries and a finite number of local continuous response parameters. Compared to directly classifying all height layers independently, this reduces output redundancy while maintaining structural expressiveness and improves the model's efficiency in representing vertical structures. Attached Figure Description

[0149] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0150] Figure 1 This is a flowchart of the overall method of the present invention.

[0151] Figure 2 This is a flowchart illustrating the construction of truth values ​​for the columnar height segmentation of this invention.

[0152] Figure 3 This is a flowchart of the ordered height segmentation prediction and local spline decoding process of the present invention.

[0153] Figure 4 This is a flowchart of the joint loss training process of the present invention. Detailed Implementation

[0154] Example 1: As Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, this embodiment of the invention provides a columnar height segmentation supervision and local spline decoding method for predicting 3D semantic occupancy, including the following steps:

[0155] Step 1: Input multi-view images, LiDAR point clouds, and vehicle pose information;

[0156] Step 2: Construct a bird's-eye view BEV column mesh and generate column height segmentation ground truth, voxel visibility mask and occlusion soft label;

[0157] Step 3: Extract multi-view image features and perform view cone space transformation to generate bird's-eye view BEV features;

[0158] Step 4: Predict ordered height segmentation parameters based on the BEV column from a bird's-eye view, and use local spline decoding to recover the 3D semantic occupancy result.

[0159] In step 1, multi-view camera images, LiDAR point clouds, and vehicle pose are acquired. In this embodiment, six cameras are used, covering a 360-degree field of view around the vehicle; the LiDAR is a 64-line or 128-line LiDAR; pose information is provided jointly by an inertial measurement unit and an odometer. After time synchronization, the acquired data is input into the supervision construction module and the network training module.

[0160] Step 2 includes: presetting the occupied space range and corresponding voxel resolution, forming a regular voxel grid and a bird's-eye view BEV column grid that corresponds one-to-one with the regular voxel grid on the plane. In this embodiment, the preset occupied space range is set as follows: the front-to-back direction is... The left and right directions are The height direction is .

[0161] The corresponding voxel resolution is set to:

[0162] ,

[0163] in, Indicates the voxel resolution in the forward and backward directions; Indicates the voxel resolution in the left and right directions; This represents the voxel resolution in the height direction. This forms a regular voxel grid and a bird's-eye view BEV column grid that corresponds one-to-one with the regular voxel grid on the plane.

[0164] Let the target monitoring time be Select a time window radius of The time window contains multiple frames of LiDAR point cloud data. In this embodiment, the data is taken as... That is, using a total of 7 frames of point cloud data to construct supervised labels. For time... The A point cloud of points, through pose transformation, time is... The Transformation of point cloud points to target monitoring moment In the vehicle's coordinate system:

[0165] ,

[0166] in, Indicates time as The Point cloud points; Indicates the transformation to the target supervision time. The next Point cloud points; Indicates from time coordinate system to target monitoring time The pose transformation matrix of the coordinate system; Indicates the point cloud frame time; Indicates the moment of target monitoring; Indicates the radius of the time window; This represents the point cloud point number. After the above transformation, a multi-frame compensated point cloud unified to the same coordinate system is obtained.

[0167] For planar grid index Define the planar mesh index The corresponding bird's-eye view BEV column is as follows:

[0168] ,

[0169] in, Indicates the planar grid index is A bird's-eye view of the BEV column; Indicates the grid index in the forward and backward directions; Indicates the grid index in the left and right directions; Indicates the coordinate values ​​in the forward and backward directions; Indicates the coordinate values ​​in the left and right directions; Indicates the coordinate value in the height direction; Indicates the lower boundary of the space occupied in the front and back directions; Indicates the lower boundary of the space occupied in the left and right directions; Indicates the lower boundary of the occupied space in the height direction; It indicates the boundary of the space occupied in the vertical direction.

[0170] The compensated multi-frame point clouds and their semantic categories are projected onto the BEV column of each bird's-eye view. The BEV column and semantic category are indexed for any bird's-eye view. Define the height sample set as:

[0171] ,

[0172] in, Indicates the planar grid index is Semantic category index is A high-altitude sample set; This represents the individual height values ​​in the height sample set; This represents the total number of height samples.

[0173] Sort the height sample set in ascending order of height, and denote the sorted sequence as:

[0174] ,

[0175] in, This represents the height values ​​after sorting by height from smallest to largest; the number in parentheses indicates the sorted order.

[0176] The following rule is used for segmentation: when two adjacent height values ​​satisfy... At that time, at the height value and height value The segments are broken apart; among them, Indicates the sorting position index; This represents the height segmentation threshold. In this embodiment, it is taken as... .

[0177] When two adjacent height values and When corresponding to the "visible area" and the "uncertain occlusion area" respectively, at the height value and height value The segments are broken apart.

[0178] For each candidate segment, let the number of height samples contained in the candidate segment be . When satisfied When this occurs, it is considered an invalid segment and discarded; among them, Indicates the first The number of height samples contained in each candidate segment; Indicates the candidate segment number; This represents the height sample threshold. In this embodiment, it is taken as... .

[0179] For the retained valid segments, instead of directly using extreme values ​​as upper and lower boundaries, quantiles are used to improve robustness. Let the height samples within each segment be arranged in ascending order. Then the lower and upper boundaries are taken as the 5th percentile and 95th percentile, respectively:

[0180] ,

[0181] ,

[0182] in, The semantic category index is The Lower boundary of each truth value segment; The semantic category index is The The upper boundary of each truth value segment; Represents the 5% quantile function; The 95th percentile function; Indicates the first All height sample values ​​in each candidate segment.

[0183] To ensure label stability and training feasibility, a maximum of two height segments are retained for each semantic category within each bird's-eye view BEV column. If a semantic category has more than two valid segments within the same bird's-eye view BEV column, the segments are sorted from largest to smallest number of samples, and only the two segments with the largest number of samples are retained. This yields the true value set of columnar height segments:

[0184] ,

[0185] in, Indicates the planar grid index is Semantic category index is The set of truth values ​​for columnar height segments. If only one valid segment exists, only the first truth value segment is retained; if no valid segment exists, it is denoted as the semantic category index. The category in the bird's-eye view BEV column There are no truth value segments in the middle.

[0186] It should be noted that the columnar height segment ground truth values ​​generated in this embodiment do not exist independently as ordinary auxiliary labels, but directly correspond to the ordered height segment parameters predicted subsequently. That is to say, the basic unit of the ground truth values ​​in this embodiment is not a single discrete voxel, but a finite number of height segment ground truth values ​​in the bird's-eye view BEV column; the subsequent network output also uses the prediction segments corresponding to the finite number of height segment ground truth values ​​as the basic prediction unit.

[0187] For each voxel Voxels are determined based on the penetration relationship of lidar rays. The visibility state. If voxels Located before the impact point of a certain lidar ray, and voxel If the space in question is actually traversed by a lidar ray, then the voxel will be... Mark as an empty visible voxel; if voxel The hit location in the neighboring point cloud of multiple frames will then be the voxel. Marked as occupying a visible voxel; if voxel If the voxel is located behind the hit surface and not directly observed by any lidar beam, then... Marked as an occluded, uncertain voxel. Define the voxel visibility mask as:

[0188] ,

[0189] in, Voxel representation Visible voxels; Voxel representation To mask uncertain voxels; Voxel representation The visibility mask.

[0190] For each frame, the time is Point cloud, for voxels The following two counts are statistically analyzed:

[0191] Occupy support count Indicates time as voxels falling in point cloud frames Or fall into voxels The number of points within the neighborhood;

[0192] Free support count Indicates time as In the point cloud frame, the voxel passes from the laser origin to the hit point. The number of laser radar beams.

[0193] Support counts for different time frames are weighted by attenuation. Let the relative target monitoring time within the time window be... The time offset is The time-domain weights are defined as follows:

[0194] ,

[0195] in, Indicates time as The temporal weights corresponding to the point cloud frames; Represents the natural exponential function; This represents the time decay coefficient. In this embodiment, it is taken as... .

[0196] For occlusion uncertain voxels, the occupancy confidence is defined as:

[0197] ,

[0198] in, Voxel representation The confidence level of occupation; This represents a minimal constant to prevent the denominator from being zero. In this embodiment, it is taken as... .

[0199] For occluded uncertain voxels, instead of using a single hard label, a binary soft label distribution is generated. Let the semantic category index corresponding to the occluded uncertain voxel be... Then the definition is:

[0200] ,

[0201] ,

[0202] in, Voxel representation Semantic category index The soft tag value; Voxel representation Soft label values ​​belonging to the empty voxel category; semantic category index Indicates the empty voxel category.

[0203] Finally, the following supervision information is output during the training phase: visible voxel hard labels, occluded voxel soft labels, and ground truth values ​​of the BEV column height segments in the bird's-eye view. These three are used together as training supervision inputs.

[0204] Step 3 includes:

[0205] 1. Image input.

[0206] Assume the number of cameras is Input image set for:

[0207] ,

[0208] in, Indicates the first Images captured by a camera; This indicates the camera serial number. In this embodiment, .

[0209] 2. Extract two-dimensional features.

[0210] A two-dimensional convolutional backbone network is used to extract image features. Preferably, a 50-layer ResNet-50 residual network is used as the backbone network, and multi-scale features are fused using a Feature Pyramid Network (FPN). After fusion, image features with a uniform number of channels are obtained.

[0211] ,

[0212] in, Indicates the first Two-dimensional image feature tensors corresponding to each camera; Represents the real number field; Indicates the number of image feature channels; Indicates the height of image features; This represents the width of the image feature. In this embodiment, .

[0213] 3. Depth distribution prediction.

[0214] Predict the discrete depth distribution for each image feature location. Let the total number of discrete depth layers be... ,get:

[0215] ,

[0216] in, The image feature coordinates are At the Discrete depth probabilities at each depth layer; Represents the horizontal coordinates of image features; Represents the vertical coordinates of image features; Indicates the depth layer index; Indicates the total number of discrete depth layers; Indicates the depth prediction branch; This represents the soft-maximum normalized activation function; Indicates the first The camera images are at coordinates Image feature values ​​at that location.

[0217] View frustum feature construction and bird's-eye view BEV pooling.

[0218] After constructing view frustum features by combining image features with depth distribution, the view frustum features are projected onto the vehicle coordinate system using camera intrinsic and extrinsic parameters. Then, features within the same grid are weighted and summed according to the bird's-eye view BEV grid to obtain the bird's-eye view BEV feature map.

[0219] ,

[0220] in, This represents a bird's-eye view of BEV feature maps; This indicates the number of channels in the BEV feature map from a bird's-eye view. This indicates the height of the BEV feature map from a bird's-eye view. This represents the width of the BEV feature map from a bird's-eye view. In a preferred embodiment, .

[0221] Next, the bird's-eye view BEV feature map is input into a two-dimensional convolutional encoder to further extract contextual information, and the encoded bird's-eye view BEV features are output:

[0222] ,

[0223] in, This represents the encoded bird's-eye view BEV features; This represents the number of BEV feature channels in the encoded bird's-eye view. In a preferred embodiment... .

[0224] Step 4 includes:

[0225] In this embodiment, ordered height segmentation prediction and local spline continuous decoding need to be used in combination. If only the segment boundaries are predicted without recovering the continuous response within the segments, complete voxel semantic results cannot be obtained; if only continuous functions are used for recovery without first establishing ordered segment boundaries, the continuous functions lack a clear range of action and cannot guarantee the vertical topological relationship within the BEV column of the same bird's-eye view. Therefore, in this embodiment, boundary parameterization and local spline decoding are not arbitrarily parallel, but rather dependent on each other.

[0226] 1. Segmented parameter prediction.

[0227] For each bird's-eye view BEV column feature, the network outputs parameters for at most two candidate height segments for each semantic category. For the ... Predicted segmentation and semantic category indexes The output parameters include: lower boundary offset. Upper boundary offset Segment Existence Score Segmented semantic response coefficient Local spline coefficients .in, This represents the index of the spline basis functions. In a preferred embodiment, the total number of spline basis functions is taken as... .

[0228] 2. Ordered Boundary Parameterization. To ensure that multiple predicted segments within the same semantic category satisfy an ascending order, cumulative positive value parameterization is used to construct the boundaries. The positive value smoothing function is defined as:

[0229] ,

[0230] in, Represents a positive smoothing function; Indicates intermediate parameters; Represents the natural constant.

[0231] For the first predicted segment, the ordered boundary parameterization formula is:

[0232] ,

[0233] ,

[0234] For the second prediction segment:

[0235] ,

[0236] ,

[0237] in, The semantic category index is The lower boundary of the first predicted segment; The semantic category index is The upper boundary of the first predicted segment; The semantic category index is The lower boundary of the second predicted segment; The semantic category index is The upper boundary of the second predicted segment; This represents the minimum segment interval. In this embodiment, .

[0238] Score for segmentation existence Perform Sigmoid normalization; if the segment existence score after normalization is less than a threshold... Then the first Each predicted segment is considered an invalid segment. In this embodiment, .

[0239] 3. Locally normalized coordinates. For any height value falling within the predicted segment interval... Define height value The locally normalized coordinates are:

[0240] ,

[0241] in, Represents height value In the semantic category index The Locally normalized coordinates in each predicted segment.

[0242] 4. Local spline continuous response.

[0243] For each prediction segment, its internal continuous response is constructed using B-spline basis functions:

[0244] ,

[0245] in, The semantic category index is The Each predicted segment at height value Local spline continuous response at the location; Indicates the first A cubic B-spline basis function. In this embodiment, a uniform node vector of order 3 is used.

[0246] 5. Piecewise Gating Function. To ensure that the continuous local spline response only applies within the corresponding prediction piece, a smoothing gating function is defined:

[0247] ,

[0248] in, The semantic category index is The Each predicted segment at height value The gating function value at the location; This represents the Sigmoid activation function; This represents the boundary steepness coefficient. In this embodiment, .

[0249] 6. Piecewise response and voxel semantic logarithmic values. For height-centered... The Individual stratum, defined as the first Each predicted segment is indexed against the semantic category. The response is:

[0250] ,

[0251] ,

[0252] in, Indicates the voxel layer index; Indicates the first The height center value of the individual's pithelial layer; The semantic category index is The The predicted segment for the first... The response value of the individual pithelial layer; Indicates the first Individual element layers belong to semantic category indexes as The logarithmic response value; The semantic category index is Category bias.

[0253] 7. Voxel semantic probabilities. Softmax normalization is performed on all semantic categories and the empty voxel category to obtain the voxel semantic probabilities:

[0254] ,

[0255] in, Indicates the first Individual primitive layers belong to semantic category indexes The probability of; q' represents the natural exponential function; q' represents the semantic category index variable in the summation; Represents the total number of semantic categories; semantic category index This represents the empty voxel category. The 3D semantic occupancy prediction result is obtained by combining the BEV columns of all bird's-eye views and the probability tensors on all height layers.

[0256] 8. Matching Predicted Segments with Ground Value Segments. To train the boundary regression loss, it is necessary to establish a correspondence between predicted segments and ground value segments. This is done for each bird's-eye view BEV column and semantic category index. The matching is performed one-to-one according to height order: when there is only one segment of the true value, the first predicted valid segment is matched with the unique true value segment; when there are two segments of the true value, the matching is performed in order from low to high according to the lower boundary; if the number of predicted segments is greater than the number of true value segments, the extra predicted segments are recorded as invalid segments; if the number of true value segments is greater than the number of valid predicted segments, the missing segments only participate in the existence loss and do not participate in the regression loss.

[0257] Define a set of boundary voxels near the boundary of each truth segment. For any voxel center height value... If satisfied or ,

[0258] The voxel is then called a boundary voxel; where, This represents the boundary threshold. In this embodiment, .

[0259] Set up a semantic category index The overall voxel frequency in the training set is The total number of samples is The boundary voxel number is Then the category weights are defined as follows:

[0260] ,

[0261] in, The semantic category index is Category weights; The semantic category index is The overall voxel frequency; The semantic category index is The total number of samples; The semantic category index is The boundary voxels; Indicates the frequency balance index; This represents the boundary enhancement coefficient. In this embodiment, , .

[0262] Weighted cross-entropy loss is applied to visible voxels:

[0263] ,

[0264] in, This indicates visible voxel supervision loss; Voxel representation The category weight corresponding to the hard tag category; Voxel representation Hard label categories; Voxel representation Predicted as a hard label category The probability of.

[0265] Soft-label monitoring loss is applied to occluded uncertain voxels:

[0266] ,

[0267] in, This indicates the loss of soft supervision due to voxel occlusion; Voxel representation Predicted as a semantic category index The probability of.

[0268] For the prediction segments and ground truth segments of valid matches, the upper and lower boundaries are constrained by the SmoothL1 loss:

[0269] ,

[0270] in, Indicates the boundary regression loss; Indicates the planar grid index is Semantic category index is The segment number is The number of indicators of matching validity; This represents the smoothed L1 loss function.

[0271] Define topological constraint loss:

[0272] ,

[0273] in, Represents the topological constraint loss; This represents the maximum value operation; This represents the relaxation amount of the first type of topological constraint; This represents the relaxation amount of the second type of topological constraint.

[0274] For redundant segments without true counterparts, apply existence suppression loss:

[0275] ,

[0276] in, This indicates an invalid segmentation that suppresses losses; The semantic category index is The The existence score of each predicted segment.

[0277] Total loss function Defined as:

[0278] ,

[0279] in, , , and These represent the loss weights corresponding to the occlusion voxel soft supervision loss, boundary regression loss, topological constraint loss, and ineffective segmentation suppression loss, respectively. In this embodiment, , , , .

[0280] 9. Training methods

[0281] The AdamW optimizer was used for training, with the initial learning rate set as follows: ;

[0282] The weight decay coefficient is set as follows: ;

[0283] The training consisted of 24 epochs with a batch size of 8. In the later stages of training, exponential moving average weights were used for inference to improve model stability.

[0284] Example 2: Specific implementation of a scenario where two layers occupy a city road;

[0285] To illustrate the practical application of this invention, this embodiment selects an urban road scenario for explanation. In this scenario, there is a green belt and trees to the right front of the vehicle, with shrubs at lower levels and tree canopies at higher levels; meanwhile, a passenger car is parked to the left of the green belt, with part of the area behind the vehicle obscured. This scenario simultaneously includes low-position occupied areas, high-position occupied areas, intermediate empty areas, and uncertain occupancy areas, reflecting the typical challenges in 3D semantic occupancy prediction for autonomous driving.

[0286] 1. Input data and parameter settings

[0287] In this embodiment, there are 6 cameras and a 64-line LiDAR. The target monitoring time is recorded as follows: The time window radius is taken as That is, the supervision information is constructed using point clouds from 7 frames before and after the target supervision time. The 3D occupied space range is set as follows: forward and backward directions. left and right directions Altitude direction The voxel resolution is set to: , , The height segmentation threshold is set as follows: The minimum number of samples for candidate segments is set to a threshold. The total number of spline basis functions is taken Boundary threshold is taken .

[0288] 2. Results obtained in step 1: Acquire multi-view images, multi-frame point clouds, and vehicle pose information.

[0289] At the moment of target supervision The system acquires 6 synchronized images, 7 frames of point clouds, and corresponding pose information. After pose compensation of the 7 frames of point clouds, the position of static objects in the same coordinate system is more consistent, reducing point cloud misalignment caused by vehicle motion and providing stable input for subsequent construction of column height segmentation ground truth.

[0290] 3. Results obtained in step 2: Construct a bird's-eye view BEV column mesh, generate column height segmentation ground truth, voxel visibility mask, and occlusion soft label.

[0291] (1) Example of truth value for segmented height expression

[0292] Select a BEV column from a bird's-eye view, with its forward and backward range being... The range in the left and right directions is In this bird's-eye view BEV column, the height samples for vegetation categories are as follows:

[0293] ,

[0294] Due to height value and The difference between them is greater than Therefore, it is divided into two effective segments. Using the 5th and 95th quantiles, we obtain: the first truth segment is approximately... The second truth segment is approximately .

[0295] The results indicate that the vegetation in the BEV column at the same bird's-eye view is not continuously occupied, but rather has a two-layer structure of "low-lying shrubs occupying - middle empty space - high-lying canopy occupying".

[0296] (2) Example of covering soft labels

[0297] Then select an undefined voxel located behind the parked vehicle. In the 7-frame point cloud, the occupancy support count sequence of this voxel is as follows:

[0298] ,

[0299] The free support counting sequence is:

[0300] ,

[0301] Take the time decay coefficient Then, the occupancy confidence of the voxel can be calculated using the occupancy confidence formula. The calculation results show that this voxel is more likely to be in the state of "occupied but occluded". Therefore, this voxel is used with a soft label during training, rather than being treated as a hard label for an empty voxel or an occupied voxel.

[0302] (3) Technical problems solved in this step

[0303] This step addresses two practical problems. First, in scenarios with multi-layered structures within a BEV column from the same bird's-eye view, traditional volumetric real values ​​struggle to directly express the hierarchical relationship of "low-position occupied—middle-position idle—high-position occupied," while this invention can explicitly express this structure through column-height segmented real values. Second, for vehicle-occupied areas, traditional hard-label supervision is prone to introducing noise, while this invention reduces the impact of erroneous supervision by using soft-label occlusion.

[0304] 4. Results obtained in step three: Extract multi-view image features and perform view frustum space transformation to generate bird's-eye view BEV features.

[0305] In this embodiment, a 50-layer residual convolutional network is used to extract image features, and a feature pyramid network is used for fusion. The dimension of the image feature tensor for each path after fusion is:

[0306] ,

[0307] The total number of discrete depth layers is taken After view frustum transformation and bird's-eye view BEV pooling, the bird's-eye view BEV feature map is obtained:

[0308] ,

[0309] After further encoding, we get:

[0310] ,

[0311] The result of this step is that the two-dimensional texture information in the original multi-view image is uniformly mapped to the bird's-eye view BEV space, providing a feature basis for subsequent column-by-column height segmentation prediction.

[0312] 5. Results obtained in step 4: Predicted ordered height segmentation parameters based on the BEV column from a bird's-eye view, and recovered the 3D semantic occupancy results using local spline decoding.

[0313] For the aforementioned BEV column in the bird's-eye view of double-layer vegetation, the network outputs two prediction segments. An example prediction result is: the first prediction segment is approximately... The second prediction segment is approximately It can be seen that the predicted result is basically consistent with the true value segmentation position, and the intermediate empty area is preserved.

[0314] Subsequently, continuous responses are recovered within each prediction segment using local spline functions and mapped onto each height voxel layer. The final results show that vegetation category probabilities are higher in the lower voxel layers; empty voxel categories are higher in the middle height layers; and vegetation category probabilities increase again in the higher voxel layers. In other words, the final 3D semantic occupancy result preserves two separate occupancy regions, rather than incorrectly connecting them into a single continuous occupancy volume.

[0315] 6. Final results, technical problems solved, and advantages

[0316] The final result of this embodiment is that for the bird's-eye view BEV column with a two-layer structure, the final output three-dimensional semantic occupancy result can correctly retain the structural relationship of "low-position occupied area - middle empty area - high-position occupied area"; for occluded areas, the final result will not be directly misjudged as empty due to the lack of observation in a single frame.

[0317] The practical technical problem addressed by this embodiment is that in complex scenarios such as tree canopies, shrubs, the bottom of elevated structures, and vehicle obstructions, existing technologies are prone to issues such as misfilling of intermediate empty areas, adhesion between upper and lower structures, and mismonitoring of obstructed areas.

[0318] Compared with the prior art, the advantages of the present invention are reflected in the following aspects: the structural expression is more accurate, and it can directly represent the multi-layer occupied structure in the BEV column from the same bird's-eye view; the middle empty area is easier to retain, and it is not easy to connect two real and separate occupied structures at the top and bottom incorrectly; the occlusion supervision is more robust, and the noise in the occlusion area is reduced by soft labels; the boundary prediction is more stable, and the height boundary recovery capability is improved by boundary segmentation and local spline decoding.

[0319] Therefore, this embodiment shows that the present invention can more effectively adapt to multi-layer occupancy structures and occlusion uncertainties in autonomous driving scenarios, and has good engineering application value.

[0320] Corresponding to the method described above, the present invention also provides an autonomous driving three-dimensional semantic occupancy prediction system, comprising:

[0321] 1. Data acquisition module, used to acquire multi-view images, multi-frame point clouds, and pose information;

[0322] 2. Supervision construction module, used to perform point cloud motion compensation, bird's-eye view BEV column division, column height segmentation ground truth construction, visibility mask construction, and occlusion soft label generation;

[0323] 3. Feature extraction module, used to extract two-dimensional features from multi-view images and generate bird's-eye view BEV features;

[0324] 4. Segmented prediction module, used to predict ordered height segment parameters for each bird's-eye view BEV column;

[0325] 5. Local spline decoding module, used to recover voxel-level semantic probabilities based on segmentation parameters and local spline coefficients;

[0326] 6. Training optimization module, used to update network parameters based on the joint loss function;

[0327] 7. Occupation output module, used to output the three-dimensional semantic occupancy result.

[0328] In a preferred embodiment, the above modules can be deployed on the same computing device and executed by the processor calling program instructions in the memory; or they can be distributed and deployed in a training server and an in-vehicle inference terminal, wherein the training server performs supervised construction and model training, and the in-vehicle inference terminal performs forward prediction.

[0329] The present invention employs a fixed number of small segments, a clear segment extraction threshold, a fixed quantile boundary, a finite number of spline basis functions, and a defined prediction segment matching rule, thereby ensuring that the technical solution can be directly implemented by those skilled in the art.

[0330] Specifically: the generation of the columnar height segmentation ground truth depends on multi-frame motion-compensated point clouds and a fixed threshold. And quantile boundary rules, with clearly defined operational steps;

[0331] The soft label for the occluded area is directly calculated from visibility determination, occupied support count, and free support count, and has a clear statistical caliber;

[0332] The number of parameters output by segmented prediction is fixed, with a maximum of two segments per class, which facilitates engineering implementation;

[0333] The basis function forms, number of basis functions, and boundary parameterization methods for local spline decoding have all been clearly given;

[0334] The loss function, weight coefficients, and matching rules during the training process are clearly defined and can be implemented directly.

[0335] Therefore, this invention does not merely propose an abstract objective, but rather provides specific technical means that can be implemented by those skilled in the art.

[0336] This invention proposes a columnar height segmentation supervision and local spline decoding method and system for 3D semantic occupancy prediction. The method uses the BEV column from a bird's-eye view as the smallest modeling unit, rewriting the 3D occupancy representation from a fixed-height-layer discrete classification into a finite number of ordered height segment parameters and locally continuous responses within each segment. Supervision is then performed using the corresponding columnar height segment ground truth values, thereby improving structural consistency, boundary accuracy, and training stability in the 3D semantic occupancy prediction task.

[0337] This invention provides a method and system for predicting 3D semantic occupancy using columnar height segmentation supervision and local spline decoding. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for predicting 3D semantic occupancy using columnar height segmentation supervision and local spline decoding, characterized in that, Includes the following steps: Step 1: Input multi-view images, LiDAR point clouds, and vehicle pose information; Step 2: Construct a bird's-eye view BEV column mesh and generate column height segmentation ground truth, voxel visibility mask and occlusion soft label; Step 3: Extract multi-view image features and perform view cone space transformation to generate bird's-eye view BEV features; Step 4: Predict ordered height segmentation parameters based on the BEV column from a bird's-eye view, and use local spline decoding to recover the 3D semantic occupancy result.

2. The method according to claim 1, characterized in that, Step 2 includes: presetting the occupied space range and corresponding voxel resolution, forming a regular voxel grid and a bird's-eye view BEV column grid that corresponds one-to-one with the regular voxel grid on the plane.

3. The method according to claim 2, characterized in that, Step 2 also includes: setting the target monitoring time as Select time window Multiple frames of LiDAR point cloud within the time frame, for a period of time The j-th point in the point cloud is subjected to pose transformation, with the time being... Transformation of the j-th point cloud point to the target supervision time In the vehicle's coordinate system: ， in, Indicates time as The j-th point in the point cloud; Indicates the transformation to the target supervision time. The j-th point in the point cloud; Indicates from time coordinate system to target monitoring time The pose transformation matrix of the coordinate system; represents the point cloud frame time; j represents the point cloud point number. For planar grid index Define the planar mesh index The corresponding bird's-eye view BEV column is as follows: ， in, Indicates the planar grid index is A bird's-eye view of the BEV column; Indicates the grid index in the forward and backward directions; Indicates the grid index in the left and right directions; Indicates the coordinate values ​​in the forward and backward directions; Indicates the coordinate values ​​in the left and right directions; Indicates the coordinate value in the height direction; Indicates the voxel resolution in the forward and backward directions; Indicates the voxel resolution in the left and right directions; Indicates the lower boundary of the space occupied in the front and back directions; Indicates the lower boundary of the space occupied in the left and right directions; Indicates the lower boundary of the occupied space in the height direction; Indicates the boundary of the occupied space in the height direction; The compensated multi-frame point clouds and their semantic categories are projected onto the BEV column of each bird's-eye view. For any bird's-eye view BEV column and semantic category index q, the height sample set is defined as follows: ， in, Indicates the planar grid index is A set of samples with a semantic category index of q; This represents the nth height value in the height sample set; Indicates the total number of height samples; Sort the height sample set in ascending order of height, and denote the sorted sequence as: ， in, This represents the n height values ​​sorted from smallest to largest; the numbers in parentheses indicate their order after sorting. The following rule is used for segmentation: when two adjacent height values ​​satisfy... At that time, at the height value and height value The segments are broken apart; among them, Indicates the sorting position index; Indicates the height segmentation threshold; when two adjacent height values and When corresponding to the visible area and the uncertain occlusion area respectively, at the height value and height value Disconnect between segments; For each candidate segment, let the number of height samples contained in the candidate segment be . When satisfied When this occurs, it is considered an invalid segment and discarded; among them, Indicates the first The number of height samples contained in each candidate segment; Indicates the candidate segment number; Indicates the height sample threshold; For the retained valid segments, quantiles are used to improve robustness: Let the height samples within each segment be arranged in ascending order. Then the lower and upper boundaries are taken as the 5th percentile and 95th percentile, respectively: ， ， in, This represents the lower boundary of the m-th truth segment with semantic category index q; This represents the upper boundary of the m-th truth segment with semantic category index q; Represents the 5% quantile function; The 95th percentile function; This represents all height sample values ​​in the m-th candidate segment; Within each bird's-eye view BEV column, each semantic category retains at most two height segments. If a category obtains more than two valid segments within the same bird's-eye view BEV column, the segments are sorted from largest to smallest by the number of samples, and only the two segments with the largest number of samples are retained, ultimately yielding the columnar height segment truth set: ， in, Indicates the planar grid index is A columnar height segmented truth set with semantic category index q; This indicates the height segment of the first truth value; This indicates the height segmentation of the second truth value; If only one valid segment exists, then only the first truth value height segment is retained; if no valid segment exists, then the category with semantic category index q is recorded in the bird's-eye view BEV column. There are no truth value segments in the middle.

4. The method according to claim 3, characterized in that, Step 2 also includes: for each voxel Visibility status is determined based on the laser radar ray penetration relationship; if voxels Located before the point of impact of a certain laser beam, and voxel If the space in which the object is located is actually traversed by the ray, then the voxel will be... Mark as an empty visible voxel; if voxel If the hit location is within two or more adjacent point clouds, then the voxel will be... Marked as occupying a visible voxel; if voxel If the voxel is located behind the hit surface and not directly observed by any rays, then the voxel will be... Marked as an occluded uncertain voxel; Define the voxel visibility mask as follows: ,in, Voxel representation Visible voxels; Voxel representation To mask uncertain voxels; Voxel representation The visibility mask; For each frame, the time is Point cloud, for voxels The following two counts are statistically analyzed: Occupy support count Time is voxels falling in point cloud frames Or fall into voxels The number of points within the neighborhood; Free support count Time is In the point cloud frame, the voxel passes from the laser origin to the hit point. The number of laser radar beams; The support counts for different time frames are weighted by attenuation; the relative target monitoring time within the time window is set. The time offset is The time-domain weights are defined as follows: ， in, Indicates time as The temporal weights corresponding to the point cloud frames; Represents the natural exponential function; Indicates the time decay coefficient; For occlusion uncertain voxels, the occupancy confidence is defined as: ， in, Voxel representation The confidence level of occupation; Indicates time as voxels The number of occupied support counts; Indicates time as voxels Free support count; This represents a very small constant to prevent the denominator from being zero; For occlusion uncertain voxels, generate a binary soft-label distribution: Let the category corresponding to the occlusion uncertain voxel be q, then define: ， ， in, Voxel representation Semantic category index The soft tag value; Voxel representation The soft label value belonging to the empty voxel category; the superscript "soft" indicates a soft label; the semantic category index 0 indicates the empty voxel category; Finally, the following supervision information is output during the training phase: visible voxel hard labels; occluded voxel soft labels; and ground truth values ​​for the BEV column height segments in the bird's-eye view.

5. The method according to claim 4, characterized in that, Step 3 includes: Assume the number of cameras is The input image set I is: ， in, This represents the image captured by the i-th camera; A two-dimensional convolutional backbone network is used to extract image features, with a 50-layer ResNet-50 as the backbone network. Multi-scale features are then fused using a Feature Pyramid Network (FPN) to obtain image features with a uniform number of channels. ， in, This represents the two-dimensional image feature tensor corresponding to the i-th camera; Represents the real number field; Indicates the number of image feature channels; Indicates the height of image features; Indicates the width of image features; For each image feature location, predict the discrete depth distribution, assuming the total number of discrete depth layers is . ,get: , in, The image feature coordinates are At the Discrete depth probabilities at each depth layer; Represents the horizontal coordinates of image features; Represents the vertical coordinates of image features; Indicates the depth layer index; Indicates the total number of discrete depth layers; Indicates the depth prediction branch; This represents the soft-maximum normalized activation function; Indicates the first The camera images are at coordinates Image feature values ​​at the location; After constructing view frustum features by combining image features with depth distribution, the view frustum features are projected onto the vehicle coordinate system using camera intrinsic and extrinsic parameters. Then, the features within the same grid are weighted and summed according to the bird's-eye view BEV grid to obtain the bird's-eye view BEV feature map. ,in, This represents a bird's-eye view of BEV feature maps; This indicates the number of channels in the BEV feature map from a bird's-eye view. This indicates the height of the BEV feature map from a bird's-eye view. This indicates the width of the BEV feature map from a bird's-eye view. The bird's-eye view BEV feature map is input into a two-dimensional convolutional encoder to further extract contextual information, and the encoded bird's-eye view BEV features are output: ， in, This represents the encoded bird's-eye view BEV features; This represents the number of BEV feature channels in the encoded bird's-eye view.

6. The method according to claim 5, characterized in that, Step 4 includes: For each bird's-eye view BEV column feature, the network outputs parameters for at most two candidate height segments for each semantic category; for the m-th predicted segment and semantic category index q, the output parameters include: lower boundary offset. Upper boundary offset Segment Existence Score Segmented semantic response coefficient Local spline coefficients ;in, This represents the index of the spline basis function; the total number of spline basis functions is... ; The boundary is constructed using cumulative positive value parameterization, and the positive value smoothing function is defined as follows: ， in, Represents a positive smoothing function; Indicates intermediate parameters; Represents the natural constant; For the first predicted segment, the ordered boundary parameterization formula is: ， ， For the second prediction segment: ， ， in, The semantic category index is The lower boundary of the first predicted segment; The semantic category index is The upper boundary of the first predicted segment; The semantic category index is The lower boundary of the second predicted segment; The semantic category index is The upper boundary of the second predicted segment; The semantic category index is The first predicted segment lower boundary offset; The semantic category index is The first predicted segment upper boundary offset; The semantic category index is The second predicted segment lower boundary offset; The semantic category index is The second predicted segment upper boundary offset; Indicates the minimum segment interval; Score for segmentation existence Perform Sigmoid normalization; if the segment existence score after normalization is less than a threshold... If the m-th predicted segment is not an invalid segment, then the m-th predicted segment is considered invalid. For any height value falling within the predicted segment interval Define height value The locally normalized coordinates are: ， in, Represents height value In the semantic category index The Local normalized coordinates in each predicted segment; For each prediction segment, construct the internal continuous response using B-spline basis functions: ， in, The semantic category index is The Each predicted segment at height value Local spline continuous response at the location; Indicates the first A cubic B-spline basis function; Define a smoothing gate function: ， in, The semantic category index is The Each predicted segment at height value The gating function value at the location; This represents the Sigmoid activation function; Indicates the boundary steepness coefficient; For the height center is For the k-th voxel layer, the response of the m-th predicted segment to the semantic category index q is defined as: ， ， Where k represents the voxel layer index; This represents the height center value of the k-th voxel layer; This represents the response value of the m-th predicted segment with semantic category index q to the k-th voxel layer; This represents the logarithmic response value of the k-th voxel layer belonging to the semantic category index q; This represents the category bias term with semantic category index q; Perform Softmax normalization on all semantic categories and the empty voxel category to obtain the voxel semantic probabilities: ， in, q' represents the probability that the k-th voxel layer belongs to the semantic category index q; q' represents the semantic category index variable in the summation. Represents the total number of semantic categories; semantic category index Indicates the empty voxel category; The combination of all bird's-eye view BEV columns and probability tensors on all height layers yields the 3D semantic occupancy prediction results.

7. The method according to claim 6, characterized in that, Step 4 also includes: For each bird's-eye view BEV column and semantic category index q, a one-to-one matching method based on height order is used: When the true value has only one segment, the first predicted valid segment is matched with the unique true value segment; When the true value has two segments, matching is performed in ascending order of the lower boundary. If the number of predicted segments exceeds the number of true segments, the extra predicted segments are considered invalid segments. If the number of true value segments exceeds the number of effective prediction segments, then the missing segments only participate in the existence loss and not in the regression loss.

8. The method according to claim 7, characterized in that, Step 4 also includes: Near each truth value segment boundary, define a boundary voxel set, for any voxel center height value If the following conditions are met: or , The voxel is then called a boundary voxel; where, Indicates the boundary threshold; Let the total voxel frequency of the semantic category index q in the training set be... The total number of samples is The boundary voxel number is Then the category weights are defined as follows: ， in, The semantic category index is Category weights; The semantic category index is The overall voxel frequency; The semantic category index is The total number of samples; The semantic category index is The boundary voxels; Indicates the frequency balance index; Indicates the boundary enhancement coefficient; Weighted cross-entropy loss is applied to visible voxels: ， in, This indicates visible voxel supervision loss; Voxel representation The category weight corresponding to the hard tag category; Voxel representation Hard label categories; Voxel representation Predicted as a hard label category The probability of; Soft-label monitoring loss is applied to occluded uncertain voxels: ， in, This indicates the loss of soft supervision due to voxel occlusion; Voxel representation Predicted as a semantic category index The probability of; For the prediction segments and ground truth segments of valid matches, the upper and lower boundaries are constrained by the SmoothL1 loss: ， in, Indicates the boundary regression loss; Indicates the planar grid index is Semantic category index is The segment number is The indicator of matching validity; This represents the smoothed L1 loss function; Define topological constraint loss: ， in, Represents the topological constraint loss; This represents the maximum value operation; This represents the relaxation amount of the first type of topological constraint; This represents the relaxation amount of the second type of topological constraint; For redundant segments without true counterparts, apply existence suppression loss: ， in, This indicates that the loss is suppressed by invalid segmentation; The semantic category index is The Existence score of each predicted segment; Total loss function Defined as: ， in, , , and These represent the loss weight values ​​corresponding to the occlusion voxel soft supervision loss, boundary regression loss, topological constraint loss, and invalid segmentation suppression loss, respectively.

9. A prediction system based on the method described in any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to acquire multi-view images, multi-frame point clouds, and pose information; The supervised construction module is used to perform point cloud motion compensation, bird's-eye view BEV column partitioning, column height segmentation ground truth construction, visibility mask construction, and occlusion soft label generation. The feature extraction module is used to extract two-dimensional features from multi-view images and generate bird's-eye view BEV features; The segmented prediction module is used to predict ordered height segment parameters for the BEV column of each bird's-eye view. The local spline decoding module is used to recover voxel-level semantic probabilities based on segmentation parameters and local spline coefficients; The training optimization module is used to update network parameters based on the joint loss function; The occupancy output module is used to output the three-dimensional semantic occupancy results.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images