Mine area 3D occupancy prediction method, device, vehicle, equipment, medium and chip

Abstract

The present application provides a method, apparatus, vehicle, equipment, medium, and chip for predicting 3D occupancy in a mining area, and relates to the field of autonomous driving and modeling in mining areas. The method comprises: obtaining 3D image features based on a 2D image of the mining area, and obtaining 3D point cloud features based on a 3D point cloud of the mining area; fusing the 3D image features and the 3D point cloud features to obtain fused features; performing occupancy prediction on the fused features using a 3D occupancy prediction head, and using 3D occupancy true values ​​for supervision to obtain a 3D occupancy distribution prediction value; converting the 3D occupancy distribution prediction value into a 2D semantic segmentation prediction value and a depth prediction value using a 3D Gaussian splattering technique, and using the 2D semantic segmentation true value to supervise the 2D semantic segmentation prediction value, and using the depth true value to supervise the depth prediction value.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving and modeling in mining areas, and in particular to a method, device, computer equipment, readable storage medium and chip for predicting 3D occupancy in mining areas. Background Art

[0002] Compared to ordinary urban roads and highways, mining environments are more complex and varied, characterized by rugged terrain, dusty conditions, complex lighting conditions, and frequent dynamic obstacles. Therefore, ensuring the safety and efficiency of autonomous driving systems in mining environments is a core research issue.

[0003] As an important environmental modeling method, 3D occupancy grids can accurately model the surrounding environment through a grid-based approach, providing occupancy status and probability information for each grid cell, thereby providing reliable data support for path planning and obstacle avoidance decisions. In mining environments, the use of 3D occupancy grid technology can more effectively model irregular terrain, dynamic obstacles (such as mining cars and pedestrians), and unpredictable environmental changes (such as landslides and falling rocks), thereby improving the adaptability of autonomous driving systems.

[0004] However, the 3D occupancy prediction scheme in related technologies only uses the true 3D occupancy value for supervision, and the true 3D occupancy value data is often sparse in mining scenarios, so it cannot fully guide the training of the model. Summary of the Invention

[0005] In view of this, the present application provides a method, device, vehicle, computer equipment, readable storage medium and chip for predicting 3D occupancy in mining areas, which solves the problem that the 3D occupancy prediction scheme in the related art only uses the true 3D occupancy value for supervision, resulting in low training accuracy.

[0006] In a first aspect, an embodiment of the present application provides a method for predicting 3D mining area occupancy, comprising:

[0007] Obtaining 3D image features based on the 2D image of the mining area, and obtaining 3D point cloud features based on the 3D point cloud of the mining area;

[0008] Performing feature fusion on the 3D image features and the 3D point cloud features to obtain fused features;

[0009] Performing occupancy prediction on the fused features through a 3D occupancy prediction head and using the 3D occupancy true value for supervision to obtain a 3D occupancy distribution prediction value;

[0010] The 3D occupancy distribution prediction value is converted into a 2D semantic segmentation prediction value and a depth prediction value through the 3D Gaussian splashing technology, and the 2D semantic segmentation prediction value is supervised by the 2D semantic segmentation true value, and the depth prediction value is supervised by the depth true value.

[0011] In a second aspect, an embodiment of the present application provides a mining area 3D occupancy prediction device, comprising:

[0012] An image feature acquisition module is used to obtain 3D image features based on the 2D image of the mining area;

[0013] Point cloud feature acquisition module, used to obtain 3D point cloud features based on the mining area 3D point cloud;

[0014] A feature fusion module, configured to fuse the 3D image features and the 3D point cloud features to obtain fused features;

[0015] A prediction module is used to perform occupancy prediction on the fused features through a 3D occupancy prediction head and use the 3D occupancy true value for supervision to obtain a 3D occupancy distribution prediction value;

[0016] The first supervision module is used to convert the 3D occupancy distribution prediction value into a 2D semantic segmentation prediction value and a depth prediction value through a 3D Gaussian splashing technology, and supervise the 2D semantic segmentation prediction value using a 2D semantic segmentation true value, and supervise the depth prediction value using a depth true value.

[0017] In a third aspect, an embodiment of the present application provides a vehicle, comprising the device according to the second aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer device comprising a first processor and a first memory, wherein the first memory stores a program or instruction running on the first processor, and when the program or instruction is executed by the first processor, the steps of the method of the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0020] In a sixth aspect, an embodiment of the present application provides a chip comprising at least one second processor and a communication interface, wherein the communication interface is coupled to the at least one second processor, and the at least one second processor is used to run programs or instructions to implement the method of the first aspect.

[0021] The embodiments of the present application provide a method, device, vehicle, computer equipment, readable storage medium, and chip for predicting 3D occupancy in a mining area. The method, device, vehicle, computer equipment, readable storage medium, and chip are used to obtain 3D image features based on a 2D image of the mining area, and 3D point cloud features based on a 3D point cloud of the mining area. The 3D image features and 3D point cloud features are fused to obtain fused features. The fused features are processed for occupancy prediction, and the output results are supervised using 3D occupancy true values ​​to obtain 3D occupancy distribution prediction values. The 3D occupancy distribution prediction values ​​are converted into 2D semantic segmentation prediction values ​​and depth prediction values ​​through 3D Gaussian splattering technology, and the 2D semantic segmentation prediction values ​​are supervised using 2D semantic segmentation true values, and the depth prediction values ​​are supervised using depth true values.

[0022] In this embodiment, 2D semantics and depth supervision are introduced through 3D Gaussian splatting on top of 3D ground truth occupancy supervision, achieving multimodal, multi-dimensional signal supervision for 3D occupancy prediction. This enhances the model's ability to perceive sparse objects due to their distributional advantages in 2D semantic segmentation. Furthermore, since 2D semantic segmentation signals are denser than 3D ground truth occupancy signals, they can effectively supplement 3D ground truth occupancy signals to enhance model prediction performance.

[0023] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A schematic diagram showing the flow of a mining area 3D occupancy prediction method according to an embodiment of the present application is shown;

[0026] Figure 2 A logical diagram of the mining area 3D occupancy model training phase in an embodiment of the present application is shown;

[0027] Figure 3 A logical diagram of the mining area 3D occupancy inference stage in an embodiment of the present application is shown;

[0028] Figure 4 A structural block diagram of a mining area 3D occupancy prediction device according to an embodiment of the present application is shown;

[0029] Figure 5 A schematic structural block diagram of a vehicle according to an embodiment of the present application is shown;

[0030] Figure 6 A schematic block diagram showing the structure of a computer device according to an embodiment of the present application is shown;

[0031] Figure 7 A schematic block diagram showing the structure of a computer-readable storage medium according to an embodiment of the present application is shown;

[0032] Figure 8 A schematic block diagram of the structure of a chip according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0034] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0035] Autonomous driving in mining areas relies on high-precision 3D occupancy prediction to provide fine-grained environmental perception to ensure driving safety and improve work efficiency. However, relying solely on image input in an unstructured, complex terrain environment such as a mining area can easily lead to occupancy prediction errors, affecting the stability and reliability of the system. In addition, since the occupancy data of mining scenes are sparsely and unevenly distributed, the methods in related technologies are difficult to provide complete and accurate environmental modeling. The mining area 3D occupancy prediction method, device, computer equipment, readable storage medium and chip provided in the embodiments of the present application integrate multimodal inputs to make up for the limitations of a single sensor, and combine multi-dimensional supervisory signals. Since sparse targets or small targets have a larger semantic proportion in 2D space than in 3D space, the introduction of 2D semantic signal supervision can alleviate the sparsity of sparse target or small target occupancy data, effectively improve the accuracy and robustness of 3D occupancy prediction, and enable the autonomous driving system to have stronger adaptability in the complex environment of mining areas. Below, in conjunction with the accompanying drawings, the mining area 3D occupancy prediction method, device, vehicle, computer equipment, readable storage medium and chip provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios. Unless there is a conflict, the following embodiments and features in the embodiments can be combined with each other.

[0036] The present application embodiment provides a method for predicting 3D occupancy of a mining area, such as Figure 1 and Figure 2 As shown, the method includes:

[0037] Step 101: obtaining 3D image features based on a 2D image of the mining area, and obtaining 3D point cloud features based on a 3D point cloud of the mining area.

[0038] In this step, a 2D image of the mining area is acquired using the vehicle's image acquisition device (e.g., a monocular camera) and converted into 3D image features. Furthermore, a 3D point cloud of the mining area is acquired using the vehicle's LiDAR, and 3D point cloud features are derived from the 3D point cloud.

[0039] Vehicles include pickup trucks, mini trucks, light trucks, micro buses, dump trucks, trucks, tractors, trailers, special vehicles, mining trucks, wide-body trucks, articulated trucks, excavators, electric shovels, bulldozers, etc. within the mining area.

[0040] In one embodiment of the present application, 3D image features are obtained based on a 2D image of a mining area, including: performing 2D feature extraction on the 2D image of the mining area through a 2D image feature extraction network to obtain 2D image features; predicting the depth distribution vector of the 2D image of the mining area through a depth prediction network, and converting the 2D image features into 3D image features using the depth distribution vector.

[0041] In this embodiment, the 2D image feature extraction network includes ResNet (Residual Network) and FPN (Feature Pyramid Network). ResNet can be ResNet50, ResNet-101, ResNet-152, etc. The 2D image of the mining area is sequentially subjected to feature extraction using ResNet and FPN to obtain 2D image features. The obtained 2D image features are input into the depth prediction network to predict a depth distribution vector for each pixel. The depth distribution vector is supervised by the true depth value to ensure the accuracy of the predicted depth distribution vector. The predicted depth distribution vector is then used to convert the 2D image features into 3D image features based on the LSS (Lift-Splat-Shoot) method. LSS (Lift-Splat-Shoot) is a method for three-dimensional object detection, particularly suitable for scenarios such as autonomous driving that require the perception of three-dimensional spatial information from two-dimensional images. Specifically, based on the predicted depth distribution vector, the 2D image features are mapped to voxels or pillars in the three-dimensional space, and the features in each voxel or pillar are aggregated and fused in the three-dimensional space to obtain 3D image features.

[0042] In one embodiment of the present application, obtaining 3D point cloud features based on a 3D point cloud of a mining area includes: grouping the 3D point cloud of the mining area according to voxels of a preset size to obtain a plurality of voxel groups; performing a coordinate mean operation on the 3D point cloud of the mining area contained in each voxel group to obtain point cloud data corresponding to the voxel group;

[0043] Sparse convolution is performed on the point cloud data of multiple voxel groups to obtain 3D point cloud features.

[0044] In this embodiment, the 3D point cloud of the mining area is grouped according to voxels of preset size. Within each voxel group, the coordinate mean operation of the point cloud contained therein is performed to obtain the point cloud data corresponding to the voxel group, and then the point cloud data is subjected to sparse convolution to obtain the 3D point cloud features.

[0045] In the embodiment of the present application, the input is multimodal information including images and point clouds, which can make up for the limitations of a single sensor and improve the accuracy of 3D occupancy prediction.

[0046] Step 102: Fusing the 3D image features and the 3D point cloud features to obtain fused features.

[0047] In this step, the 3D image features and 3D point cloud features are fused through a feature fusion network. For example, the 3D image features and 3D point cloud features can be concatenated at the feature channel and then reduced to the original dimension using an MLP (Multilayer Perceptron) network.

[0048] In step 103 , the 3D occupancy prediction head is used to perform occupancy prediction on the fused features, and the 3D occupancy true value is used for supervision to obtain a 3D occupancy distribution prediction value.

[0049] In this step, the fused features are processed for occupancy prediction, and the output results are supervised by the 3D occupancy true value to obtain the 3D occupancy distribution prediction value.

[0050] In one embodiment of the present application, occupancy prediction is performed on the fused features by a 3D occupancy prediction head, and the 3D occupancy true value is used for supervision to obtain a 3D occupancy distribution prediction value, including:

[0051] Enhance the fusion features through 3D feature pyramid network;

[0052] The enhanced fusion features are input to the 3D occupancy prediction head, which outputs the initial 3D occupancy distribution prediction value;

[0053] The 3D occupancy true value is used to supervise the initial 3D occupancy distribution prediction value to obtain the 3D occupancy distribution prediction value.

[0054] In this embodiment, the fused features are enhanced by a 3D feature pyramid network, and then the 3D occupancy prediction head outputs an initial 3D occupancy distribution prediction. This initial 3D occupancy distribution prediction is supervised based on the true 3D occupancy values, which are pre-acquired training data. Specifically, a loss is calculated based on the true 3D occupancy values ​​and the initial 3D occupancy distribution prediction to verify the prediction results.

[0055] In the embodiment of the present application, 3D occupancy prediction is performed after fusion feature enhancement processing, which can improve the accuracy of 3D occupancy prediction.

[0056] In step 104 , the 3D occupancy distribution prediction value is converted into a 2D semantic segmentation prediction value and a depth prediction value by using the 3D Gaussian splattering technique, and the 2D semantic segmentation prediction value is supervised by the 2D semantic segmentation true value, and the depth prediction value is supervised by the depth true value.

[0057] While using the true 3D occupancy value to supervise 3D occupancy prediction, this application also adds 2D semantic and depth supervision through 3D Gaussian splashing, thereby realizing multi-dimensional signal supervision of 3D occupancy prediction.

[0058] In this step, after performing 3D occupancy prediction to obtain 3D occupancy distribution prediction values, the 3D occupancy distribution prediction values ​​are converted into 2D semantic segmentation prediction values ​​and depth prediction values ​​through 3D Gaussian splatting technology. 3D Gaussian splatting is a technology that diffuses discrete 3D data into a continuous space using a Gaussian kernel function. Its core idea is to project the features of each 3D point (such as position, color, depth, semantics, etc.) onto the 2D image plane in the form of a Gaussian distribution, generating a smooth and continuous distribution.

[0059] In the embodiment of the present application, the 3D point cloud feature of each 3D occupancy distribution prediction value corresponds to a voxel in space, and the center of the voxel is used as the center of the Gaussian kernel of the Gaussian splash. The rotation matrix and the scale matrix are both set to fixed values. For example, the rotation matrix can be set to 0, that is, no rotation, and the scale matrix is ​​set to a fixed value and no stretching. The opacity is calculated by the predicted value of the empty class according to the following formula:

[0060]

[0061] in, The desired opacity for 3D Gaussian rendering, The predicted value for the null category.

[0062] Through the rotation matrix and scale matrix, the geometric transformation of 3D data is achieved to ensure that the projection matches the observation perspective and scale. The overlay effect of the projection is controlled by opacity to enhance the realism and readability of the scene.

[0063] Furthermore, the 2D semantic segmentation predictions and depth predictions are supervised using the 2D semantic segmentation ground truth and depth ground truth, respectively. The 2D semantic segmentation ground truth and depth ground truth are pre-acquired training data. The depth ground truth can be obtained by projecting a single or multi-frame 3D point cloud of the mining area, while the 2D semantic segmentation ground truth can be obtained from the input image using a large semantic segmentation model. For example, the loss is calculated using the 2D semantic segmentation ground truth and 2D semantic segmentation predictions, and the loss is calculated using the depth ground truth and depth predictions to verify the 2D semantic and depth prediction results.

[0064] The embodiment of the present application provides a method for predicting 3D occupancy in mining areas based on multimodal fusion and multidimensional signal supervision using 3D Gaussian splashing. This method introduces 2D semantics and depth supervision through 3D Gaussian splashing on the basis of 3D occupancy truth supervision, thus realizing multimodal and multidimensional signal supervision for 3D occupancy prediction. Based on the distribution advantage of sparse targets in 2D semantic segmentation, the model's ability to perceive sparse targets is enhanced. At the same time, since 2D semantic segmentation signals are denser than 3D occupancy truth signals, they can serve as an effective supplement to 3D occupancy truth values ​​to improve the model's prediction effect.

[0065] When using 3D occupancy grids to predict 3D occupancy of the mining environment and then modeling the mining environment, the reasoning process is as follows: Figure 3 As shown, 3D image features are obtained based on the image, 3D point cloud features are obtained based on the point cloud, and then the 3D image features and 3D point cloud features are fused to obtain 3D fused features. After feature enhancement processing is performed on the 3D fused features, 3D occupancy prediction is performed.

[0066] Since the prediction target of 3D occupancy is limited to a local cube close to the vehicle body, which is a small area relative to the mine scene within the field of view, the projection of the 3D occupancy truth value can only reflect the occupancy projection of the elements within the cube. The 2D semantic segmentation directly corresponding to the input image corresponds to the projection of all occupied voxels in the camera's field of view, which means that the projections of voxels that are far away from the vehicle body are also included, which is different from the projection of the 3D occupancy truth value. Therefore, the 2D semantic segmentation truth value obtained in the training data is not suitable for direct supervision of the 2D semantic segmentation prediction value. In view of this problem, it is necessary to first calculate a mask for the 2D segmentation result based on the distance between the object and the image acquisition device. In one embodiment of the present application, the method further includes: generating a mask of the near-field area involved in the 3D occupancy based on the depth truth value obtained from the 3D point cloud of the mining area, and obtaining the masked 2D semantic segmentation truth value through the mask;

[0067] The 2D semantic segmentation true value is used to supervise the 2D semantic segmentation prediction value, including: using the masked 2D semantic segmentation true value to supervise the 2D semantic segmentation prediction value.

[0068] In this embodiment, the depth truth value obtained from the 3D point cloud of the mining area, that is, the sparse depth map, is used to extract the foreground area from the sparse depth map, generate a mask of the foreground area, and then obtain the 2D semantic segmentation truth value aligned with the mask, excluding the distant pixels to obtain the masked 2D semantic segmentation truth value, thereby reducing the difference between the projection of the 2D semantic segmentation and the 3D occupancy truth value, avoiding misleading predictions caused by objects at a distance, and then using the masked 2D semantic segmentation truth value to supervise the 2D semantic segmentation prediction value, thereby improving the adaptability of the data and improving the reliability of supervision.

[0069] In one embodiment of the present application, masking is performed on the regional depth map to obtain the masked 2D semantic segmentation truth value, including:

[0070] Remove target points that do not meet the preset 3D occupancy grid depth range from the regional depth map;

[0071] After removing the target point, obtain the connected region in the regional depth map and calculate the connected area of ​​the connected region;

[0072] If the connected area of ​​the connected region is greater than the preset threshold, the connected region is retained and the mask value of the connected region is set to "1"; if the connected area of ​​the connected region is less than or equal to the preset threshold, the mask value of the connected region is set to "0";

[0073] The connected areas with mask value “1” are morphologically expanded to fill the gaps and obtain the masked 2D semantic segmentation truth value.

[0074] In this embodiment, the regional depth map obtained by lidar projection is often sparse and cannot be used directly as a mask. The mask is first calculated for the 2D segmentation result based on the distance. First, the target points that do not conform to the preset 3D occupied grid depth interval are removed from the regional depth map. These target points that are not in the preset 3D occupied grid depth interval, that is, points without depth information, belong to points that are farther away. For example, points with a distance greater than 80 meters are excluded. The 3D occupied grid depth interval is pre-set and is related to the training data set. Different training data sets can correspond to different 3D occupied grid depth intervals.

[0075] Furthermore, all connected regions in the depth map of the region are obtained to connect sparse points into regions and calculate the connected area. A threshold is set. When the connected area of ​​a connected region is greater than the threshold, it is retained and its mask value is set to "1", that is, True. When the connected area of ​​a connected region is less than or equal to the threshold, it is not retained and its mask value is set to "0". Then, the retained connected regions are morphologically expanded to fill small gaps, and finally the masked depth-based 2D semantic segmentation true value is obtained. The masked depth-based 2D semantic segmentation true value is the same size as the image. The masked 2D semantic segmentation true value is then used to supervise the 2D semantic segmentation prediction value. The pixels with a mask value of "1" in the masked 2D semantic segmentation true value are calculated with the 2D semantic segmentation prediction value, and the pixels with a mask value of "0" are not calculated with the 2D semantic segmentation prediction value, thereby improving the reliability of supervision.

[0076] In the image branch, when converting 2D image features into 3D image features, depth information is obtained from the LiDAR input for feature enhancement. However, in related technologies, semantic information is not involved in the feature extraction process of the LiDAR branch, resulting in the lack of semantic information enhancement in the features extracted by the LiDAR branch. In one embodiment of the present application, before performing feature fusion on the 3D image features and the 3D point cloud features to obtain the fused features, the method further includes:

[0077] The 3D point cloud features are converted into 2D semantic segmentation prediction values ​​through the 3D Gaussian splashing technology, and the masked 2D semantic segmentation true values ​​are used to supervise the 2D semantic segmentation prediction values.

[0078] In this embodiment, before feature fusion of 3D image features and 3D point cloud features, the 3D point cloud features are converted into 2D semantic segmentation prediction values ​​through 3D Gaussian splashing technology, and then the 2D semantic segmentation prediction values ​​are supervised by the 2D semantic segmentation true values. In particular, the 2D semantic segmentation prediction values ​​can be supervised by the masked 2D semantic segmentation true values.

[0079] Feature fusion solutions in related technologies do not provide semantic information guidance to the radar branch before fusion. However, in the embodiments of the present application, the radar branch before feature fusion renders 2D semantic segmentation from point cloud features and uses masked 2D ground truth for supervision, which can improve the semantic consistency of radar features, align radar features with image features, enhance feature quality before fusion, and make it better suited for multimodal perception tasks.

[0080] In one embodiment of the present application, 3D point cloud features are converted into 2D semantic segmentation prediction values ​​using a 3D Gaussian splattering technique, including:

[0081] Through the 3D convolutional network, semantic information is extracted from the 3D point cloud features to obtain the 3D semantic distribution prediction value;

[0082] The 3D semantic distribution prediction value is projected into 2D space through the 3D Gaussian splashing technology to obtain the 2D semantic segmentation prediction value.

[0083] In this embodiment, the structural information in the 3D point cloud features can be used to generate semantic predictions. First, semantic information is extracted from the 3D point cloud features through a two-layer 3×3 3D convolutional network to obtain a 3D semantic distribution prediction value based on the sparse radar feature. Then, the 3D semantic distribution prediction value is projected into the 2D space through a 3D Gaussian splash rendering method to obtain a 2D semantic segmentation prediction value. The masked 2D semantic segmentation true value is then used for supervision to help the radar feature branch obtain semantic information. Among them, the 3D point cloud feature of each 3D occupancy distribution prediction value corresponds to a voxel in space, and the center of the voxel is used as the Gaussian kernel center of the Gaussian splash. The rotation matrix and the scale matrix are both set to fixed values. For example, the rotation matrix can be set to 0, that is, no rotation, and the scale matrix is ​​set to a fixed value and no stretching. The opacity is calculated by the prediction value of the empty category according to the following formula:

[0084]

[0085] in, The desired opacity for 3D Gaussian rendering, The predicted value for the null category.

[0086] In the related art, when performing multimodal feature fusion, the fusion effect is not supervised, so the fusion effect cannot be effectively guaranteed. In one embodiment of the present application, before performing occupancy prediction on the fused features through the 3D occupancy prediction head, the method further includes:

[0087] Perform dimensionality reduction on the fused features and output RGB prediction values ​​and opacity prediction values;

[0088] The RGB prediction value and opacity prediction value are converted into 2D RGB prediction value through 3D Gaussian splashing technology, and the 2D RGB prediction value is supervised by the RGB true value.

[0089] In this embodiment, the fused 3D fusion features should have both semantic context information and structural information. The fused features are processed by MLP network or 1×1 3D convolution network for dimensionality reduction, and RGB prediction values ​​and opacity prediction values ​​are output. Then, the RGB prediction values ​​and opacity prediction values ​​are converted into 2D RGB prediction values ​​through 3D Gaussian splashing, and the 2D RGB prediction values ​​are supervised by the RGB true value to ensure the fusion effect of the fused features. It should be noted that the RGB true value here can also be the masked RGB true value.

[0090] In this application, after fusion of features, 2D RGB prediction values ​​are output and supervised through 3D Gaussian splashing to enhance the geometric consistency and semantic consistency of the fused features, helping the model learn more consistent cross-modal representations.

[0091] As a specific implementation of the above-mentioned mining area 3D occupancy prediction method, the embodiment of the present application provides a mining area 3D occupancy prediction device. Figure 4 As shown, the mining area 3D occupancy prediction device 400 includes: an image feature acquisition module 401, a point cloud feature acquisition module 402, a feature fusion module 403, a prediction module 404 and a first supervision module 405.

[0092] The image feature acquisition module 401 is used to obtain 3D image features based on the 2D image of the mining area;

[0093] Point cloud feature acquisition module 402, used to obtain 3D point cloud features based on the mining area 3D point cloud;

[0094] A feature fusion module 403 is used to fuse 3D image features and 3D point cloud features to obtain fused features;

[0095] Prediction module 404, configured to perform occupancy prediction on the fused features using a 3D occupancy prediction head and use the 3D occupancy ground truth for supervision to obtain a 3D occupancy distribution prediction value;

[0096] The first supervision module 405 is used to convert the 3D occupancy distribution prediction value into a 2D semantic segmentation prediction value and a depth prediction value through the 3D Gaussian splashing technology, and supervise the 2D semantic segmentation prediction value using the 2D semantic segmentation true value, and supervise the depth prediction value using the depth true value.

[0097] Furthermore, the image feature acquisition module 401 is specifically configured to:

[0098] 2D image features are extracted from the 2D image of the mining area through a 2D image feature extraction network to obtain 2D image features;

[0099] The depth distribution vector of the 2D image of the mining area is predicted through the depth prediction network, and the depth distribution vector is used to convert the 2D image features into 3D image features.

[0100] Furthermore, the point cloud feature acquisition module 402 is specifically configured to:

[0101] The mining area 3D point cloud is grouped according to voxels of preset sizes to obtain multiple voxel groups;

[0102] Perform coordinate mean calculation on the 3D point cloud of the mining area contained in each voxel group to obtain the point cloud data corresponding to the voxel group;

[0103] Sparse convolution is performed on the point cloud data of multiple voxel groups to obtain 3D point cloud features.

[0104] Furthermore, the prediction module 404 is specifically configured to:

[0105] Enhance the fusion features through 3D feature pyramid network;

[0106] The enhanced fusion features are input to the 3D occupancy prediction head, which outputs the initial 3D occupancy distribution prediction value;

[0107] The 3D occupancy true value is used to supervise the initial 3D occupancy distribution prediction value to obtain the 3D occupancy distribution prediction value.

[0108] Furthermore, the device further includes: a mask processing module, configured to: generate a mask of a near-view area involved in 3D occupancy based on a depth truth value obtained from the 3D point cloud of the mining area, and obtain a masked 2D semantic segmentation truth value through the mask;

[0109] The first supervision module 405 is specifically configured to supervise the 2D semantic segmentation prediction value using the masked 2D semantic segmentation true value.

[0110] Furthermore, the mask processing module is specifically configured to:

[0111] Remove target points that do not meet the preset 3D occupancy grid depth range from the regional depth map;

[0112] After removing the target point, obtain the connected region in the regional depth map and calculate the connected area of ​​the connected region;

[0113] If the connected area of ​​the connected region is greater than the preset threshold, the connected region is retained and the mask value of the connected region is set to "1"; if the connected area of ​​the connected region is less than or equal to the preset threshold, the mask value of the connected region is set to "0";

[0114] The connected areas with mask value “1” are morphologically expanded to fill the gaps and obtain the masked 2D semantic segmentation truth value.

[0115] Furthermore, the device further includes: a second monitoring module, configured to:

[0116] The 3D image features and 3D point cloud features are fused. Before obtaining the fused features, the 3D point cloud features are converted into 2D semantic segmentation prediction values ​​through the 3D Gaussian splashing technology, and the masked 2D semantic segmentation true values ​​are used to supervise the 2D semantic segmentation prediction values.

[0117] Furthermore, the second supervision module is specifically configured to:

[0118] Through the 3D convolutional network, semantic information is extracted from the 3D point cloud features to obtain the 3D semantic distribution prediction value;

[0119] The 3D semantic distribution prediction value is projected into 2D space through the 3D Gaussian splashing technology to obtain the 2D semantic segmentation prediction value.

[0120] Furthermore, the device further includes: a third monitoring module, configured to:

[0121] Before the fusion feature is used for occupancy prediction through the 3D occupancy prediction head, the fusion feature is reduced in dimension to output the RGB prediction value and the opacity prediction value;

[0122] The RGB prediction value and opacity prediction value are converted into 2D RGB prediction value through 3D Gaussian splashing technology, and the 2D RGB prediction value is supervised by the RGB true value.

[0123] The mining area 3D occupancy prediction device 400 in the embodiment of the present application can be a computer device, or a component in a computer device, such as an integrated circuit or a chip. The mining area 3D occupancy prediction device 400 provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the embodiment of the mining area 3D occupancy prediction method are not described here.

[0124] The present application also provides a vehicle, such as Figure 5 As shown, the vehicle 500 includes the above-mentioned mining area 3D occupancy prediction device 400.

[0125] The vehicle 500 can execute the mining area 3D occupancy prediction method described in the above embodiment through the mining area 3D occupancy prediction device 400. It is understood that the implementation method of the vehicle 500 controlling the mining area 3D occupancy prediction device 400 can be set according to the actual application scenario and is not specifically limited in the embodiment of the present application.

[0126] The vehicle 500 is provided with a terminal, which includes but is not limited to: an on-board terminal, an on-board controller, an on-board module, an on-board module, an on-board component, an on-board chip, an on-board unit, an on-board radar or an on-board camera and other sensors. The vehicle can implement the method provided in this application through the on-board terminal, on-board controller, on-board module, on-board module, on-board component, on-board chip, on-board unit, on-board radar or camera. The vehicles in this application include passenger cars and commercial vehicles. Common models of commercial vehicles include but are not limited to: pickup trucks, micro trucks, light trucks, micro buses, dump trucks, trucks, tractors, trailers, special vehicles and mining vehicles. Mining vehicles include but are not limited to mining trucks, wide-body trucks, articulated trucks, excavators, electric shovels, bulldozers, etc. This application does not further limit the type of smart cars, and any type of vehicle is within the scope of protection of this application.

[0127] The present application also provides a computer device, such as Figure 6As shown, the computer device 600 includes a first processor 601 and a first memory 602. The first memory 602 stores programs or instructions that can be run on the first processor 601. When the program or instruction is executed by the first processor 601, the various steps of the above-mentioned mining area 3D occupancy prediction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0128] The first memory 602 can be used to store software programs and various data. The first memory 602 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store an operating system, applications or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the first memory 602 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 602 in the embodiment of the present application includes, but is not limited to, these and any other suitable types of memory.

[0129] The first processor 601 may include one or more processing units. Optionally, the first processor 601 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the first processor 601.

[0130] The present application also provides a readable storage medium. Figure 7 As shown, a program or instruction 701 is stored on the readable storage medium 700. When the program or instruction 701 is executed by the processor, each process of the above-mentioned mining area 3D occupancy prediction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0131] The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. The computer-readable storage medium 700 may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one location to another. The storage medium may be any target medium that can be accessed by a computer.

[0132] As one possible design, computer-readable storage medium 700 may include a Compact Disc Read-Only Memory (CD-ROM), RAM, ROM, EEPROM, or other optical disk storage; computer-readable storage media may include magnetic disk storage or other magnetic disk storage devices. Furthermore, any connection line may also be appropriately referred to as a computer-readable storage medium. For example, if software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL (Digital Subscriber Line), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers.

[0133] The embodiment of the present application also provides a chip, such as Figure 8 As shown, the chip 800 includes at least one processor (for example, a second processor 801) and a communication interface 802. The communication interface 802 is coupled to the second processor 801. The second processor 801 is used to run programs or instructions to implement the various processes of the above-mentioned mining area 3D occupancy prediction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0134] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0135] Preferably, the chip 800 further includes a memory, such as a second memory 803 , which stores the following elements: executable modules or data structures, or subsets thereof, or extended sets thereof.

[0136] In this embodiment of the present application, the second memory 803 may include a read-only memory and a random access memory, and provide instructions and data to the second processor 801. A portion of the second memory 803 may also include a non-volatile random access memory (NVRAM).

[0137] In the embodiment of the present application, the second processor 801, the communication interface 802 and the second memory 803 are coupled together via a bus system 804. In addition to the data bus, the bus system 804 may also include a power bus, a control bus and a status signal bus. Figure 8 Various buses are labeled as bus system 804 .

[0138] The mining area 3D occupancy prediction method described in the above embodiment of the present application can be applied to the second processor 801, or implemented by the second processor 801. The second processor 801 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by a hardware integrated logic circuit in the second processor 801 or by instructions in the form of software. The above-mentioned second processor 801 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. The second processor 801 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention.

[0139] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0140] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A mining area 3D occupancy prediction method, characterized in that: include: Obtaining 3D image features based on the 2D image of the mining area, and obtaining 3D point cloud features based on the 3D point cloud of the mining area; Performing feature fusion on the 3D image features and the 3D point cloud features to obtain fused features; Performing occupancy prediction on the fused features through a 3D occupancy prediction head and using the 3D occupancy true value for supervision to obtain a 3D occupancy distribution prediction value; Converting the 3D occupancy distribution prediction value into a 2D semantic segmentation prediction value and a depth prediction value by using a 3D Gaussian splattering technique, and supervising the 2D semantic segmentation prediction value using a 2D semantic segmentation true value, and supervising the depth prediction value using a depth true value; Before performing occupancy prediction on the fused features by the 3D occupancy prediction head, the method further includes: Performing dimensionality reduction processing on the fused features and outputting RGB prediction values ​​and opacity prediction values; The RGB prediction value and the opacity prediction value are converted into 2D RGB prediction values ​​through 3D Gaussian splattering technology, and the 2D RGB prediction value is supervised by using the RGB true value.

2. The method according to claim 1, characterized in that The obtaining of 3D image features based on the 2D image of the mining area includes: Performing 2D feature extraction on the 2D image of the mining area through a 2D image feature extraction network to obtain 2D image features; The depth distribution vector of the 2D image of the mining area is predicted by a depth prediction network, and the 2D image features are converted into 3D image features using the depth distribution vector.

3. The method according to claim 1, characterized in that The 3D point cloud features obtained based on the mining area 3D point cloud include: Grouping the mining area 3D point cloud according to voxels of preset sizes to obtain a plurality of voxel groups; Performing coordinate mean operation on the mining area 3D point cloud contained in each voxel group to obtain point cloud data corresponding to the voxel group; Sparse convolution is performed on the point cloud data of the multiple voxel groups to obtain the 3D point cloud features.

4. The method according to claim 1, wherein The 3D occupancy prediction head performs occupancy prediction on the fused features and uses the 3D occupancy true value for supervision to obtain a 3D occupancy distribution prediction value, including: Enhance the fused features through a 3D feature pyramid network; Inputting the enhanced fusion features into the 3D occupancy prediction head and outputting an initial 3D occupancy distribution prediction value; The initial 3D occupancy distribution prediction value is supervised by using the 3D occupancy true value to obtain a 3D occupancy distribution prediction value.

5. The method according to claim 1, wherein The method further comprises: Based on the depth truth value obtained from the 3D point cloud of the mining area, a mask of the near-view area involved in the 3D occupancy is generated, and the masked 2D semantic segmentation truth value is obtained through the mask; The using the 2D semantic segmentation true value to supervise the 2D semantic segmentation prediction value includes: The masked 2D semantic segmentation true value is used to supervise the 2D semantic segmentation prediction value.

6. The method according to claim 5, characterized in that Methods for obtaining the true value of the masked 2D semantic segmentation include: Remove target points that do not meet the preset 3D occupancy grid depth range from the regional depth map; After removing the target point, obtaining a connected region in the depth map of the region, and calculating a connected area of ​​the connected region; If the connected area of ​​the connected region is greater than the preset threshold, the connected region is retained and the mask value of the connected region is set to "1"; if the connected area of ​​the connected region is less than or equal to the preset threshold, the mask value of the connected region is set to "0"; The connected area with a mask value of "1" is subjected to morphological dilation processing to fill the gaps and obtain the masked 2D semantic segmentation true value.

7. The method according to claim 5, characterized in that Before fusing the 3D image features and the 3D point cloud features to obtain fused features, the method further includes: The 3D point cloud features are converted into 2D semantic segmentation prediction values ​​through the 3D Gaussian splattering technology, and the masked 2D semantic segmentation true values ​​are used to supervise the 2D semantic segmentation prediction values.

8. The method according to claim 7, characterized in that The converting of the 3D point cloud features into 2D semantic segmentation prediction values ​​by the 3D Gaussian splatting technique includes: Extracting semantic information from the 3D point cloud features through a 3D convolutional network to obtain a 3D semantic distribution prediction value; The 3D semantic distribution prediction value is projected into the 2D space by the 3D Gaussian splattering technique to obtain the 2D semantic segmentation prediction value.

9. A mining area 3D occupancy prediction device, characterized in that: include: An image feature acquisition module is used to obtain 3D image features based on the 2D image of the mining area; Point cloud feature acquisition module, used to obtain 3D point cloud features based on the mining area 3D point cloud; A feature fusion module, configured to fuse the 3D image features and the 3D point cloud features to obtain fused features; A prediction module is used to perform occupancy prediction on the fused features through a 3D occupancy prediction head and use the 3D occupancy true value for supervision to obtain a 3D occupancy distribution prediction value; A first supervision module is configured to convert the 3D occupancy distribution prediction value into a 2D semantic segmentation prediction value and a depth prediction value by using a 3D Gaussian splattering technique, and supervise the 2D semantic segmentation prediction value using a 2D semantic segmentation true value, and supervise the depth prediction value using a depth true value; The device further includes a third monitoring module, configured to: Performing dimensionality reduction processing on the fused features and outputting RGB prediction values ​​and opacity prediction values; The RGB prediction value and the opacity prediction value are converted into 2D RGB prediction values ​​through 3D Gaussian splattering technology, and the 2D RGB prediction value is supervised by using the RGB true value.

10. A vehicle, characterized in that: The vehicle includes the mining area 3D occupancy prediction device according to claim 9.

11. A computer device, characterized in that: It includes a first processor and a first memory, the first memory stores a program or instruction running on the first processor, and when the program or instruction is executed by the first processor, the steps of the mining area 3D occupancy prediction method as described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the mining area 3D occupancy prediction method according to any one of claims 1 to 8 are implemented.

13. A chip, characterized in that: The chip includes at least one second processor and a communication interface, the communication interface is coupled to the at least one second processor, and the at least one second processor is used to run programs or instructions to implement the steps of the mining area 3D occupancy prediction method as described in any one of claims 1 to 8.

Images