A panoramic 3D occupancy prediction method based on 3D Gaussian sputtering

Abstract

The present invention discloses a method for predicting 3D occupancy based on 3D Gaussian sputtering. The method comprises the following steps: extracting 2D features from multi-view RBG images using a 2D image feature extraction network; converting the 2D image features into 3D features through a depth prediction network; extracting the 3D features using a 3D feature extraction network; performing 3D Gaussian sputtering on the 3D features, and then rendering them to a final 2D image; setting a loss function and using 2D true values ​​for supervision; and constructing a dense 2D semantic and depth true value data set. The present application realizes effective supervision of 3D occupancy prediction, and improves training speed and prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a surround 3D occupancy prediction method based on 3D Gaussian sputtering. Background Art

[0002] Current 3D occupancy prediction methods mainly use 3D voxel truth values ​​to supervise the learning of the model. However, labeling 3D voxels is costly and challenging. Related studies have shown that after using pre-labeled LiDAR data to generate preliminary 3D voxel truth values, it still takes about 4,000 hours of manual labor to complete the post-processing of these truth values. These problems limit the availability and scalability of 3D occupancy prediction methods. Given the cost advantage of panoramic cameras and the characteristics of images with rich semantic information, methods that use 2D truth values ​​to supervise 3D occupancy prediction models are gradually gaining more attention.

[0003] In recent studies, 3D occupancy prediction methods based on 2D true value supervision mostly use NeRF's volume rendering technology to render 3D occupancy prediction results into 2D semantic and depth maps. This type of method optimizes the model by minimizing the loss of the rendered image and the 2D true value.

[0004] However, this supervision method also inherits the limitations of NeRF volume rendering. Each 2D pixel truth value only corresponds to a single ray starting from the camera light and passing through the center of the pixel. The 3D occupancy farther from the camera optical center is smaller in scale, and the NeRF model has fewer sampling points in this area, resulting in the inability to effectively supervise it. In addition, NeRF uses a uniform sampling strategy, and the detail resolution of distant objects is far less than that of nearby objects, which makes the model perform poorly in scenes with large scale changes. In addition, each true value pixel corresponds to a ray, so the computational overhead of volume rendering is related to the true value pixel density, and it is usually necessary to set an upper limit on the number of rays, which affects the true value utilization and model performance.

[0005] When training a 3D occupancy prediction model at the same time, an intuitive method is to directly convert the multi-frame LiDAR point cloud into a unified coordinate system for encryption, and then project the obtained dense point cloud to the camera perspective to generate a dense 2D true value. This simple solution is only suitable for static scenes, while vehicles, pedestrians, etc. in the scene are often dynamically changing, and their positions in different frames will be different, which will cause the foreground target true value to overlap, blur or other errors, and thus produce erroneous 2D true values.

[0006] Therefore, there is still no effective technical solution for the above-mentioned technical problems. Summary of the invention

[0007] The present invention aims to provide a method for predicting 3D occupancy based on 3D Gaussian sputtering, which improves the prediction accuracy and the training speed.

[0008] In order to achieve the purpose of the present invention, the present invention provides a method for predicting 3D occupancy based on 3D Gaussian sputtering, comprising the following steps:

[0009] Step 1: Use a 2D image feature extraction network to extract 2D features from multi-view RBG images;

[0010] Step 2: Convert the 2D image features into 3D features through a depth prediction network;

[0011] Step 3: extracting the 3D features using a 3D feature extraction network;

[0012] Step 4, performing 3D Gaussian sputtering on the 3D features, and then rendering them into a final 2D image;

[0013] Step 4.1: Set parameters and convert 3D features into 3D Gaussian representation: Use two MLPs to complete the prediction of the opacity and semantics of the 3D Gaussian, and set the scaling matrix parameters, rotation matrix parameters and center coordinate parameters of the 3D Gaussian;

[0014] Step 4.2: Project the 3D Gaussian into a 2D Gaussian on the image plane to obtain the information required for 2D rendering:

[0015] Step 4.2.1: Project the 3D Gaussian center point to the pixel coordinate system;

[0016] Step 4.2.2: Calculate the 3D covariance matrix using the scaling matrix and the rotation matrix;

[0017] Step 4.2.3: Convert the 3D covariance matrix into a 2D covariance matrix, and finally obtain the information required for the 2D rendering;

[0018] Step 4.3: Render the 2D Gaussian to a 2D image:

[0019] Step 4.3.1: Generate interaction IDs and sort the tiles covered by 2D Gaussian:

[0020] Step 4.3.2: Rendering based on opacity and color.

[0021] Step 5: Set the loss function and use the 2D true value for supervision: the loss function includes the cross entropy loss function and the SILog loss function;

[0022] Step 6: Build a dense 2D semantic and depth truth dataset. By combining the existing 3D detection boxes and 3D lidar point cloud semantic segmentation labels, generate dense 2D semantic and depth truth values ​​for training the 3D occupancy prediction model.

[0023] Step 6.1: Aggregate the current scene point cloud for the foreground and the background respectively to obtain a foreground point cloud dictionary and a background dense point cloud of the scene;

[0024] Step 6.2: Distribute the foreground point cloud dictionary and the background dense point cloud to the 3D dense point cloud generated by each frame: project the dense point cloud of the current frame onto the image plane using the camera internal and external parameters to obtain a dense 2D true value;

[0025] Step 6.3: Disambiguate the 2D ground truth.

[0026] Compared with the prior art, the significant improvements of the present invention are: (1) the present invention uses 3D Gaussian sputtering to supervise 3D occupancy prediction when only 2D true values ​​are used, thereby improving the training speed and prediction accuracy; (2) the present invention also proposes a method for constructing a corresponding dense 2D semantic and depth truth data set. The data generated by this method is denser and more suitable for the panoramic 3D occupancy prediction method based on 3D Gaussian sputtering proposed in the present invention.

[0027] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 It is the 3D Gaussian projection process of the present invention;

[0031] Figure 3 It is the rendering process of 3D Gaussian sputtering of the present invention;

[0032] Figure 4 It is a dense 2D truth value construction flow chart of the present invention;

[0033] Figure 5 is a schematic diagram of the visual cone of the present invention;

[0034] Figure 6 is a flow chart of the 3D feature extraction steps of the present invention;

[0035] Figure 7 It is an example diagram of dense 2D truth value visualization of the present invention;

[0036] Figure 8 is the dense 2D truth value for which the present invention is ambiguous;

[0037] Fig. 9 It is a flow chart of projecting a 3D annotation frame to a 2D annotation frame of the present invention;

[0038] Fig.10 It is the dense 2D true value after processing by the present invention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] The present invention is a method for predicting 3D occupancy based on 3D Gaussian sputtering, combined with Figure 1 , including the following steps:

[0041] Step 1: Use a 2D image feature extraction network to extract 2D features from multi-view RBG images: Specifically, Swin Transformer can be used to extract image features to obtain 2D image features; the Swin Transformer performs hierarchical feature extraction on the input image, and applies different step sizes at each stage to extract higher-level features, and adds the past two frames of data as input to introduce time series data; the feature layer will be divided into The block, =4, each block is used as the input of the linear embedding layer, and is flattened and mapped to a vector space of fixed dimension, namely the embedding vector. The formula is as follows:

[0042] ;

[0043] in, For the flattened block, for dimensional projection matrix, is the number of channels, is the embedding dimension, Set to 128, is the bias term;

[0044] The formula of the multi-head self-attention mechanism is as follows:

[0045]

[0046] Among them, MHA represents the multi-head self-attention result, is the input feature, For different queries, is the key, For the value, for Projected into different query, key, and value spaces, is the dot product of the keys, is the square root of the dimension of the key vector, is the weight matrix.

[0047] Step 2: Convert the 2D image features into 3D features through a depth prediction network; specifically, convert the 2D image features into 3D features through an LSS network;

[0048] The 2D image features will first be fed into the depth prediction network to predict the depth distribution vector for each pixel. , =41, defined as starting from 4 meters away from the optical center of the camera, there is a characteristic plane every 1 meter, a total of 41 planes, and the depth range is [4m, 45m]. Figure 5 , which is represented as a viewing cone in the camera coordinate system;

[0049] In the process of depth distribution prediction, we first need to determine the center of the depth search and range , these two parameters are initialized based on the depth prediction of a single frame, ensuring that the search process focuses on the most likely depth area, in order to fine-tune the depth center and range Parameters, the model uses the expectation maximization (EM) algorithm, the depth center The formula is as follows:

[0050] ;

[0051] in, Indicates Deep candidates, represents the probability of the depth candidate;

[0052] According to the depth distribution predicted by the model, the 2D feature values ​​are converted into 3D frustum features;

[0053] At the same time, the coordinate system of the 3D voxel feature is usually the vehicle coordinate system, so the conversion of the cone feature to the 3D feature also needs to be completed.

[0054] Step 3: Extract the 3D features using a 3D feature extraction network; Figure 6Specifically, a 3D residual network may be used to extract 3D features. The 3D residual network processes 3D features by introducing a 3D convolutional layer: multi-scale hierarchical features are used to extract features and construct a feature pyramid to obtain context information. The feature pyramid encodes multi-level information from input data by combining pooling and upsampling.

[0055] The convolution kernel used in the 3D convolution is a cube or voxel. When performing 3D convolution, the convolution kernel slides on the three dimensions of the input data to calculate the weighted sum in the coverage area. The formula is as follows:

[0056] ;

[0057] in To output data, For input data, is the 3D convolution kernel, is the coordinate of the voxel, depth, height, and width represent the three dimensions of the input data.

[0058] Step 4, performing 3D Gaussian sputtering on the 3D features, and then rendering them into a final 2D image;

[0059] Step 4.1: Set parameters and convert 3D features into 3D Gaussian representation: Use two MLPs to complete the prediction of the opacity and semantics of the 3D Gaussian, and set the scaling matrix parameters, rotation matrix parameters and center coordinate parameters of the 3D Gaussian; the voxel size is predefined as 0.4m, and the scaling matrix of the 3D Gaussian is set The values ​​in the three dimensions are set to 0.15m; the center point of the 3D Gaussian Fixed to the center coordinates of each voxel in the ego-vehicle coordinate system; 3D Gaussian rotation matrix It will be fixed without optimization; then 3D Gaussian sputtering will be performed according to the 3D Gaussian parameters in the scene;

[0060] Step 4.2: Project the 3D Gaussian into a 2D Gaussian on the image plane to obtain the information required for 2D rendering, combined with Figure 2 ;

[0061] Step 4.2.1: Project the 3D Gaussian center point to the pixel coordinate system;

[0062]

[0063] in, is the external parameter of the camera, is the camera’s intrinsic parameter matrix, is the normalized pixel coordinate, is a four-dimensional vector, where Need to be transformed to 1 to maintain homogeneity, is the actual pixel coordinate, is the point coordinate after the external parameter changes, is the center point of the 3D Gaussian, is the output image width, is the output image height, ( , ) are the principal point coordinates.

[0064] Step 4.2.2: Calculate the 3D covariance matrix using the scaling matrix and the rotation matrix;

[0065] 3D Gaussian is represented by quaternion, thus avoiding the gimbal lock problem in Euler angle representation; the rotation matrix represented by the quaternion (a, b, c, d) The formula is as follows:

[0066] ;

[0067] Compute the covariance matrix:

[0068] ;

[0069] in, is the scaling matrix, which is a 3x3 diagonal matrix. The elements on the main diagonal represent the scaling parameters in three dimensions. for The transpose of for The transpose of .

[0070] Step 4.2.3: Convert the 3D covariance matrix into a 2D covariance matrix, and finally obtain the information required for the 2D rendering;

[0071] To obtain a reasonable Gaussian approximation on the image plane, set the affine transformation matrix :

[0072] ;

[0073] in, and The camera is and The focal length in the direction, is through the transformation matrix and the projection matrix Homogeneous coordinates of a point from the world coordinate system to the camera coordinate system;

[0074] 2D covariance matrix The formula is as follows:

[0075] ;

[0076] in, is the extrinsic rotation matrix, is the 3D covariance matrix, for The transpose of for The transpose of .

[0077] Step 4.3: Render the 2D Gaussian to a 2D image, combined with Figure 3 ;

[0078] Step 4.3.1: Generate a unique interaction ID for all the blocks covered by each 2D Gaussian and sort them;

[0079] The interaction ID is a 64-bit integer obtained by combining the block number and the depth of the 2D Gaussian center point, i.e., the block ID. Specifically, the block ID is shifted left by 32 bits to leave the lower 32 bits for the depth value, and then the block ID and the depth value are bitwise ORed to form a 64-bit integer.

[0080] During the rendering process, the interaction ID can be used to quickly locate the specific block covered by the 2D Gaussian. At the same time, by performing ascending sorting on the interaction ID, the same or similar blocks can be grouped together, which helps to perform data merging and batch processing operations during graphics processing or rendering, reducing processing time and improving efficiency. More importantly, it also ensures that the image can be rendered in the correct visual depth order.

[0081] Step 4.3.2: Rendering based on opacity and color;

[0082] For each pixel in the image, the colors of all 3D Gaussians in its corresponding light cone need to be weighted as the rendering result. Calculate the color of pixel i The formula is as follows:

[0083] ;

[0084] in, is the color of the vth 2D Gaussian, N is the number of 2D Gaussians in the corresponding light cone, The actual opacity of its corresponding rendering. is the cumulative effect of all related Gaussians on the current Gaussian transparency, and its formula is as follows:

[0085] ;

[0086] in, represents the opacity of the mth Gaussian;

[0087] The actual opacity of the render The calculation formula is as follows:

[0088] ;

[0089] in, is the covariance matrix according to the 2D Gaussian and the offset between the pixel center and the Gaussian center The calculated attenuation term, the attenuation term The relative distance between the pixel and the center of the Gaussian is taken into account; when the distance is farther, the attenuation term The bigger, The smaller the value, the lower the opacity, that is, the more transparent the 2D Gaussian is visually at the pixel;

[0090] Finally, according to the opacity , the depth of the Gaussian related to the pixel point is weighted and summed to get the final depth rendering result. The formula is as follows:

[0091] ;

[0092] in, is the depth of the vth 3D Gaussian, is the cumulative effect of all related Gaussians on the transparency of the current Gaussian.

[0093] Step 5: Set the loss function and use the 2D true value for supervision: the loss function includes the cross entropy loss function and the SILog loss function;

[0094] The SILog loss function formula is as follows:

[0095] ;

[0096] in, For the The predicted depth of pixels, For the The true depth of pixels, is the total number of pixels.

[0097] Step 6: Construct a dense 2D semantic and depth truth dataset. A method for constructing a dense 2D semantic and depth truth dataset based on the nuScenes dataset is proposed. By combining the existing 3D detection boxes and 3D lidar point cloud semantic segmentation labels, dense 2D semantics and depth truth are generated for training the 3D occupancy prediction model. Figure 4 , split the foreground target and the background target, and splice multiple frames separately, while also dealing with the occlusion problem of the foreground target;

[0098] Step 6.1: Aggregate the current scene point cloud for the foreground and background respectively to obtain all foreground target point clouds and all background point clouds of the scene;

[0099] Step 6.1.1: Extract the first frame of sample data from the current scene, including LiDAR data, LiDAR semantic labels, vehicle location information, LiDAR sensor calibration data, and the index of the next frame of samples;

[0100] Step 6.1.2: Create an initialization list LIST_static and a dictionary LIST_dym. After entering the loop, continue to process the LiDAR data in the current scene. In the loop, first obtain the true value of the 3D annotation box in the current frame, then extract the point cloud in the annotation box, and assign the LiDAR semantic label to the point cloud. This part of the data is then used to create a mask to separate the background point cloud and the point cloud of the foreground target. The background point cloud is converted to the LiDAR coordinate system of the first frame sample and added to the initialization list LIST_static. The foreground target point cloud will be stored according to its unique identifier (token) in this scene and added to the dictionary LIST_dym. If the current frame has a subsequent frame, load the next frame to continue processing; otherwise, end the loop.

[0101] Step 6.1.3: Aggregate the background point cloud list to obtain a background dense point cloud; finally obtain the foreground point cloud dictionary and the background dense point cloud of the scene, providing a data basis for the subsequent generation of dense truth values ​​for each frame.

[0102] Step 6.2: Distribute the point cloud to the 3D dense point cloud generated by each frame: Use the camera internal and external parameters to project the dense point cloud of the current frame onto the image plane to obtain a dense 2D true value;

[0103] Step 6.2.1: convert the background dense point cloud from the first frame LiDAR coordinate system of the scene to the current frame LiDAR coordinate system, and extract the corresponding point cloud from the foreground point cloud dictionary according to the identification of the foreground target in the current frame;

[0104] Step 6.2.2: Fuse the foreground point cloud with the background point cloud to obtain the true value of the dense point cloud of the current frame, and remove points that do not belong to the predefined voxel space range, that is, no more than 40 meters in the horizontal direction and -1m-5.4m in the vertical direction;

[0105] Step 6.2.3: Use the camera's internal and external parameters to project the dense point cloud of the current frame onto the image plane to obtain a dense 2D true value. Specifically, the dense point cloud is first converted from the LiDAR coordinate system to the camera coordinate system, and then the dense point cloud under the camera coordinate is projected onto the image plane through the camera's internal parameter matrix. Figure 7; This mapping process simultaneously performs depth information and semantic label information, combining Figure 8 .

[0106] Step 6.3: Combine Fig. 9 , disambiguating the 2D truth value;

[0107] Step 6.3.1: Use the camera internal and external parameter matrix to project the eight vertices of the 3D annotation box from the world coordinate system to the 2D image plane, where the z coordinates of the vertices are positive;

[0108] Step 6.3.2: After the world coordinate system projection is completed, an irregular polygon is obtained. To simplify its representation on the image, the convex hull of the 2D points in the polygon is calculated, that is, the minimum convex polygon containing all the projected points;

[0109] Step 6.3.3: Calculate the intersection of the convex hull and the image boundary to determine which parts of the 3D box are actually visible, and then calculate the boundary of the intersection to obtain the 2D annotation box;

[0110] Step 6.3.4: Combine Fig.10 , the background pixels in the 2D annotation box are filtered using the 2D annotation box to obtain unambiguous dense 2D true values.

[0111] The number of true value points on the newly obtained frame of 6 camera images ranges from 200,000 to 400,000 points. For the model output size of 900 For the 1600-pixel rendering, the total number of true value pixels accounts for 13.89% to 27.78%. At the same time, this application uses the true value of 7 adjacent frames as supervision during the training process. After testing, the total number of 2D true value pixels of 7 frames is about 2.5 million on average. The difference in specific numbers is due to the different number of valid point clouds in each frame sample.

[0112] Compared with the baseline model RenderOcc, which also uses 2D truth values, the framework proposed in this application has improved the average IoU indicator by 7.7%, of which the IoU indicator of the car category has increased by 35.5%. There is a 40.7% improvement in the IoU indicator of small target categories, such as pedestrians, which shows that pixel-level 2D semantic truth values ​​can achieve better semantic supervision. There is also a significant improvement of 43.9% in the IoU indicator of the bus semantic category. This shows that this application has a relatively better effect on the 3D occupancy prediction of dynamic targets and can supervise more scene details.

[0113] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0114] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A panoramic 3D occupancy prediction method based on 3D Gaussian sputtering, characterized in that: The following steps are involved: Step 1: Use a 2D image feature extraction network to extract 2D features from multi-view RBG images; Step 2: Convert the 2D image features into 3D features through a depth prediction network; Step 3: extracting the 3D features using a 3D feature extraction network; Step 4, performing 3D Gaussian sputtering on the 3D features, and then rendering them into a final 2D image; Step 4.1: Set parameters and convert 3D features into 3D Gaussian representation: Use two MLPs to complete the prediction of the opacity and semantics of the 3D Gaussian, and set the scaling matrix parameters, rotation matrix parameters and center coordinate parameters of the 3D Gaussian; Step 4.2: Project the 3D Gaussian into a 2D Gaussian on the image plane to obtain the information required for 2D rendering: Step 4.2.1: Project the 3D Gaussian center point to the pixel coordinate system; Step 4.2.2: Calculate the 3D covariance matrix using the scaling matrix and the rotation matrix; Step 4.2.3: Convert the 3D covariance matrix into a 2D covariance matrix, and finally obtain the information required for the 2D rendering; Step 4.3: Rendering the 2D Gaussian to a 2D image; Step 4.3.1: Generate interaction IDs for the blocks covered by 2D Gaussian and sort them; Step 4.3.2: Render according to opacity and color; Step 5: Set the loss function and use the 2D true value for supervision: the loss function includes the cross entropy loss function and the SILog loss function; Step 6: Build a dense 2D semantic and depth truth dataset. By combining the existing 3D detection boxes and 3D lidar point cloud semantic segmentation labels, generate dense 2D semantic and depth truth for training the 3D occupancy prediction model.

2. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 1, characterized in that: The step 4.2.1 is: ;in, is the external parameter of the camera, is the camera’s intrinsic parameter matrix, is the normalized pixel coordinate, is a four-dimensional vector, where Need to be transformed to 1 to maintain homogeneity, is the actual pixel coordinate, is the point coordinate after the external parameter changes, is the center point of the 3D Gaussian, is the output image width, is the output image height, ( , ) are the principal point coordinates.

3. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 2, characterized in that: The step 4.2.2 is: 3D Gaussian is represented by quaternion, thus avoiding the gimbal lock problem in Euler angle representation; the rotation matrix represented by the quaternion (a, b, c, d) The formula is as follows: ； Compute the 3D covariance matrix: ； in, is the scaling matrix, for The transpose of for The transpose of .

4. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 3, characterized in that: The step 4.2.3 is: Set the affine transformation matrix : ； in, and The camera is and The focal length in the direction, By transforming the matrix and the projection matrix Homogeneous coordinates of a point from the world coordinate system to the camera coordinate system; 2D covariance matrix The formula is as follows: ； in, is the extrinsic rotation matrix, is the 3D covariance matrix, for The transpose of for The transpose of .

5. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 4, characterized in that: The step 4.3.1 is: the interaction ID is a 64-bit integer obtained by combining the block number and the depth of the 2D Gaussian center point, i.e., the block ID. Specifically, the block ID is shifted left by 32 bits to leave the lower 32 bits for the depth value, and then the block ID and the depth value are bit-wise ORed to form a 64-bit integer.

6. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 5, characterized in that: The step 4.3.2 is: for each pixel in the image, the colors of all 3D Gaussians in the corresponding light cone need to be weighted to serve as the rendering result; Calculate the color of pixel i The formula is as follows: ； in, is the color of the vth 2D Gaussian, N is the number of 2D Gaussians in the corresponding light cone, The actual opacity of its corresponding rendering. is the cumulative effect of all related Gaussians on the transparency of the current Gaussian. The formula is as follows: ； in, represents the opacity of the mth Gaussian; The actual opacity of the render The calculation formula is as follows: ； in, is the covariance matrix according to the 2D Gaussian and the offset between the pixel center and the Gaussian center The calculated attenuation term, the attenuation term The relative distance between the pixel and the center of the Gaussian is taken into account; when the distance is farther, the attenuation term The bigger, The smaller the value, the lower the opacity, that is, the more transparent the 2D Gaussian is visually at the pixel; Finally, according to the opacity , the depth of the Gaussian related to the pixel point is weighted and summed to get the final depth rendering result. The formula is as follows: ； in, is the depth of the vth 3D Gaussian, is the cumulative effect of all related Gaussians on the transparency of the current Gaussian.

7. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 1, characterized in that: Step 6 is a method for constructing a dense 2D semantic and depth truth dataset based on the nuScenes dataset: Step 6.1: Aggregate the current scene point cloud for the foreground and background respectively to obtain a foreground point cloud dictionary and a background dense point cloud of the scene; Step 6.2: Distribute the foreground point cloud dictionary and the background dense point cloud to the 3D dense point cloud generated by each frame: project the dense point cloud of the current frame onto the image plane using the camera internal and external parameters to obtain a dense 2D true value; Step 6.3: Disambiguate the 2D ground truth.

8. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 7, characterized in that: The step 6.1 comprises the following steps: Step 6.1.1: Extract the first frame of sample data from the current scene, including LiDAR data, LiDAR semantic labels, vehicle location information, LiDAR sensor calibration data, and the index of the next frame of samples; Step 6.1.2: Create an initialization list and dictionary, and after entering the loop, continue to process the LiDAR data in the current scene; in the loop, first obtain the true value of the 3D annotation box in the current frame, then extract the point cloud in the annotation box, and assign the LiDAR semantic label to the point cloud; convert the background point cloud to the LiDAR coordinate system of the first frame sample and add it to the initialization list; the foreground target point cloud will be stored according to its identification in the current scene and added to the dictionary; if the current frame has a subsequent frame, load the next frame to continue processing; otherwise, end the loop; Step 6.1.3: Aggregate the background point cloud list to obtain a background dense point cloud; and finally obtain the foreground target dictionary and the background dense point cloud of the scene.

9. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 7, characterized in that: The step 6.2 comprises the following steps: Step 6.2.1: convert the background dense point cloud from the first frame LiDAR coordinate system of the scene to the current frame LiDAR coordinate system, and extract the corresponding point cloud from the foreground point cloud dictionary according to the identification of the foreground target in the current frame; Step 6.2.2: Fuse the foreground point cloud with the background point cloud to obtain the true value of the dense point cloud of the current frame, and remove points that do not belong to the predefined voxel space range; Step 6.2.3: Use the camera's internal and external parameters to project the dense point cloud of the current frame onto the image plane to obtain a dense 2D true value. Specifically: the dense point cloud is first converted from the LiDAR coordinate system to the camera coordinate system, and then the dense point cloud in the camera coordinate system is projected onto the image plane through the camera's internal parameter matrix; this mapping process is performed on both depth information and semantic label information.

10. The method for predicting 3D occupancy based on 3D Gaussian sputtering according to claim 7, characterized in that: The step 6.3 comprises the following steps: Step 6.3.1: Use the camera internal and external parameter matrix to project the eight vertices of the 3D annotation box from the world coordinate system to the 2D image plane, where the z coordinates of the vertices are positive; Step 6.3.2: After the world coordinate system projection is completed, an irregular polygon is obtained. To simplify its representation on the image, the convex hull of the 2D points in the polygon is calculated; Step 6.3.3: Calculate the intersection of the convex hull and the image boundary, and then calculate the boundary of the intersection to obtain a 2D annotation box; Step 6.3.4: Use the 2D annotation box to filter the background pixels in the box to obtain unambiguous dense 2D true values.

Images