Open scene-oriented three-dimensional occupancy prediction and quantitative deployment method

Abstract

The invention provides an open scene-oriented three-dimensional occupancy prediction and quantitative deployment method, which is applied to an intelligent driving vehicle and comprises the following steps: step 1, acquiring image information, laser radar point cloud data and visual depth information; 2, constructing a training set and a test set, and generating a three-dimensional occupation dense label; 3, obtaining a multi-scale aerial view feature with historical information through an aerial view encoder, and reconstructing the aerial view feature into a three-dimensional voxel feature by adopting space-time alignment and feature fusion to construct a three-dimensional occupation prediction model; 4, training testing and parameter debugging are conducted on the model through the training set and the testing set, and when the performance of the model reaches the optimal performance, training debugging of the model is completed; and step 5, quantifying the trained model, converting the three-dimensional occupancy prediction model into an inference engine, and deploying the inference engine to an intelligent driving vehicle so as to carry out target detection.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a three-dimensional occupancy prediction and quantization deployment method for open scenarios. Background Art

[0002] As a basic component of the autonomous driving perception system, 3D object detection can provide the vehicle with precise stereo space perception ability. It can accurately identify the three-dimensional dimensions, positions, and motion states of obstacles (including vehicles, pedestrians, traffic signs, etc.), and then provide key inputs for path planning, obstacle avoidance decision-making, and motion control. However, vision-centered 3D object detection still faces inherent limitations in complex open scenarios: (1) Existing 3D bounding box-based representation methods have significant limitations in the fine-grained modeling of object geometric shapes. This coarse-grained representation paradigm of 3D bounding boxes is difficult to accurately analyze complex geometric topologies and has insufficient semantic association modeling ability for background regions. (2) The detected categories are limited. Limited by the supervised learning paradigm, existing detectors are difficult to effectively identify unstructured obstacles with long-tail distribution characteristics and unknown category targets in open scenarios. Although the obstacle detection method based on three-dimensional occupancy grids can provide more detailed scene information, due to the need for dense grid division during three-dimensional voxel modeling and the use of high-parameter three-dimensional convolution operators to extract three-dimensional voxel features, this method consumes a large amount of memory space and computing resources and is not conducive to in-vehicle deployment. Summary of the Invention

[0003] In view of this, the present invention provides a quantization deployment of a three-dimensional occupancy prediction model based on an open scenario to solve the technical problem that the prior art does not fully consider the motion state of dynamic objects and cannot capture key kinematic parameters such as object speed and acceleration.

[0004] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios, which is applied to intelligent driving vehicles. The method includes: Step 1, obtaining image information, lidar point cloud data, and visual depth information through a multi-sensor synchronous acquisition device; Step 2, constructing a training set and a test set according to the information obtained by the multi-sensor synchronous acquisition device, and generating three-dimensional occupancy dense labels; Step 3, obtaining multi-frame historical image information through the multi-sensor synchronous acquisition device, obtaining multi-scale bird's-eye features with historical information through a bird's-eye view encoder, and reconstructing the bird's-eye features into three-dimensional voxel features by spatio-temporal alignment and feature fusion to construct a three-dimensional occupancy prediction model; Step 4, training, testing, and parameter debugging the model through the training set and the test set. When the performance of the model reaches the optimal, the model completes training and debugging; Step 5, quantizing the trained model through a deep learning inference optimizer, converting the three-dimensional occupancy prediction model into an inference engine, and then deploying it to an intelligent driving vehicle for object detection.

[0005] Furthermore, the multi-sensor synchronous acquisition device includes a lidar, a panoramic camera, and a high-precision inertial measurement unit.

[0006] Furthermore, the method for generating a three-dimensional occupancy dense label includes: Step 21, forming a dense three-dimensional scene from multiple frames of lidar point cloud data and visual depth information acquired by the multi-sensor synchronous acquisition device; Step 22, using the Poisson reconstruction algorithm to fill in structural voids within a preset range; Step 23, discretizing the reconstructed scene into a voxel grid with a preset resolution, so as to later represent the position and semantics of each grid using a three-dimensional occupancy dense label; Step 24, using the Laplacian smoothing and QEM grid optimization algorithms to optimize the voxel grid to ensure surface continuity and geometric accuracy; Step 25, determining the voxel semantic label through a multi-frame point cloud cumulative voting mechanism and k-nearest neighbor semantic statistics; Step 26, using a conditional random field to process the voxel semantic label to ensure spatial consistency and the accuracy of semantic boundaries, and generating a three-dimensional occupancy dense label.

[0007] Furthermore, Step 3 includes: Step 31, cropping, scaling, and stitching the acquired multiple frames of historical images to obtain multiple frames of two-dimensional images; Step 32, performing multi-scale image feature extraction on the multiple frames of two-dimensional images through a residual network and a feature pyramid network; Step 33, performing expansion, projection, and aggregation processing on the multi-scale image features through a depth prediction network to obtain a bird's-eye view feature and acquire the bird's-eye feature; Step 34, using spatio-temporal alignment to transform, correct, and supplement the bird's-eye feature to enhance the perception ability of dynamic objects or attributes; Step 35, using a multi-scale fusion strategy to perform feature weighted aggregation on the current bird's-eye feature and the corrected bird's-eye feature to obtain the final bird's-eye feature; Step 36, through parametric modeling, constructing a three-dimensional occupancy prediction model based on the final bird's-eye feature.

[0008] Furthermore, Step 32 includes: Step 321, extracting feature maps P of different scales from multiple frames of two-dimensional images through a residual network k , where k represents the scale; Step 322, using the feature pyramid network to perform upsampling through a top-down path, and the high-level feature map is upsampled to the resolution of the low-level feature map; Step 323, fusing the upsampled feature map with the original high-level feature map, and after obtaining the fused multi-scale feature map, performing multi-scale image feature extraction, where the fusion formula is:

[0009] where, let the high-level feature map be P k+1 , the low-level feature map be P k , Upsample represents the upsampling operation, is the fused multi-scale feature map.

[0010] Further, step 33 includes: step 331, predicting the discrete depth of the multi-scale feature map through a depth prediction network; step 332, expanding the 2D image features of the multi-scale feature map into 3D spatial features according to the discrete depth; step 333, projecting the point cloud features in the 3D spatial features onto a bird's-eye view grid to obtain bird's-eye view features; step 334, performing feature aggregation on the bird's-eye view features to obtain bird's-eye features.

[0011] Further, step 34 includes: step 341, performing spatio-temporal coordinate transformation on the bird's-eye features at each historical moment using the ego-vehicle motion information; step 342, correcting the pose difference caused by vehicle motion to obtain the corrected bird's-eye features; step 343, dynamically adjusting the spatial position of the features through deformable convolution to compensate for the alignment error caused by sensor noise, and the feature adjustment formula is: where is the bird's-eye feature after deformable convolution, DeformConv is the deformable convolution, is the corrected bird's-eye feature.

[0012] Further, the formula for weighted aggregation is: where w i is the weight of each scale feature, N is the number of historical frames, MultiScaleFusion is multi-scale fusion, is the final bird's-eye feature after weighted fusion, F BEV (t) is the current bird's-eye feature, is the bird's-eye feature after deformable convolution.

[0013] Further, step 36 includes: step 361, converting the bird's-eye view spatial features into fine-grained voxel features F Voxel through parametric modeling; step 362, converting the 3D coordinates of the voxels into high-dimensional vectors through an embedding layer; step 363, combining the embedded high-dimensional vectors with the encoded bird's-eye features and inputting them into a multi-layer perceptron to generate voxel features; step 364, decoding the three-dimensional voxel features with temporal information into a velocity vector V t , thus completing the construction of the three-dimensional occupancy prediction model.

[0014] Further, the step 5 includes: step 51, converting the trained three-dimensional occupancy prediction model into an engine file supported by a deep learning inference optimizer; step 52, converting the model with the engine file into an efficient inference engine suitable for embedded devices through the deep learning inference optimizer; step 53, configuring an inference environment on the embedded device, integrating the generated engine file into the C++ environment, and using the deep learning inference optimizer for engine deserialization, thereby completing the conversion of the three-dimensional occupancy prediction model into an inference engine.

[0015] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios. Existing technologies mostly focus on the voxel occupancy probability prediction of static scenarios, and do not fully consider the motion states of dynamic objects, and cannot capture key kinematic parameters such as object speed and acceleration. In view of the existing technical defects, the technical solution provided by this application uses a feature pyramid to extract 2D fine-grained image features to maintain the accuracy of the model. At the same time, a lightweight three-dimensional feature reconstruction strategy is adopted. Through parametric modeling, the efficient representation in the BEV (Bird's Eye View) space is converted into fine-grained voxel prediction, avoiding the direct calculation load of high-dimensional voxel data in traditional methods, significantly reducing memory occupancy, and deploying dual-task prediction heads in parallel on the BEV feature layer to perform occupancy prediction and speed prediction respectively, so as to solve the problem that the decision-making reliability of the traditional technical solution in downstream tasks such as emergency obstacle avoidance and trajectory planning is restricted due to the lack of speed prediction ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic flowchart of a three-dimensional occupancy prediction and quantization deployment method for open scenarios provided by the present invention;

[0017] Figure 2 is a schematic flowchart of a method for generating three-dimensional occupancy dense labels provided by the present invention;

[0018] Figure 3 is a schematic flowchart of a method for model quantization and conversion into an inference engine provided by the present invention;

[0019] Figure 4 is a schematic flowchart of a method for reconstructing three-dimensional voxel features provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] Example 1

[0022] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios, which is applied to vehicles with intelligent driving functions. The method reads dynamic perception information of the external environment through an in-vehicle high-definition camera and transmits it to an embedded development platform through a communication device, and deploys and infers on the embedded platform to obtain detection results based on a three-dimensional occupancy grid map. As Figure 1 shown, the method includes the following steps.

[0023] Step 1, obtain image information, lidar point cloud data, and visual depth information through a multi-sensor synchronization acquisition device;

[0024] Step 2, construct a training set and a test set according to the information obtained by the multi-sensor synchronization acquisition device, and generate three-dimensional occupancy dense labels;

[0025] Step 3, according to the multi-frame historical image information obtained by the multi-sensor synchronization acquisition device, obtain multi-scale bird's-eye features with historical information through a bird's-eye view encoder, and reconstruct the bird's-eye features into three-dimensional voxel features by using spatio-temporal alignment and feature fusion to construct a three-dimensional occupancy prediction model;

[0026] Step 4, through the training set and the test set, train, test and debug the parameters of the model. When the performance of the model reaches the optimal, the model completes training and debugging;

[0027] Step 5, through a deep learning inference optimizer, quantize the trained model, convert the three-dimensional occupancy prediction model into an inference engine, and then deploy it to an intelligent driving vehicle for object detection.

[0028] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios. Existing technologies mostly focus on the voxel occupancy probability prediction of static scenarios, do not fully consider the motion states of dynamic objects, and cannot capture key kinematic parameters such as object speed and acceleration. Aiming at the existing technical defects, the technical solution provided by this application uses a feature pyramid to extract 2D fine-grained image features to maintain the accuracy of the model. At the same time, it adopts a lightweight three-dimensional feature reconstruction strategy, and converts the efficient representation in the BEV (Bird's Eye View) space into fine-grained voxel prediction through parametric modeling, avoiding the direct calculation load of high-dimensional voxel data in traditional methods, significantly reducing memory occupancy, and deploying dual-task prediction heads in parallel on the BEV feature layer to perform occupancy prediction and speed prediction respectively, so as to solve the problem that the decision-making reliability of the traditional technical solution in downstream tasks such as emergency obstacle avoidance and trajectory planning is restricted due to the lack of speed prediction ability.

[0029] Example 2

[0030] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios, as Figure 1 shown, the method includes the following steps.

[0031] Step 1, through a multi-sensor synchronous acquisition device, obtain image information, lidar point cloud data, and visual depth information;

[0032] In the vehicle data acquisition and calibration stage, construct a multi-sensor synchronous acquisition device, and focus on obtaining six types of parameters: geometric shape features (truncation degree, occlusion rate), spatial relationship features (observation angle, ego-vehicle coordinate system pose), and multi-modal sensing data (lidar point cloud, surround-view visual image). The hardware deployment of this device adopts a networking scheme of a high-precision inertial measurement unit (IMU), a 64-line lidar, and a panoramic camera group, that is, the multi-sensor synchronous acquisition device includes a lidar, a panoramic camera, and a high-precision inertial measurement unit.

[0033] Step 2, according to the information obtained by the multi-sensor synchronous acquisition device, construct a training set and a test set, and generate three-dimensional occupancy dense labels;

[0034] Based on the collected real vehicle data, referring to the NuScenes dataset (a large-scale open-source dataset dedicated to the research and development in the field of autonomous driving), construct a training set and a test set. In addition, since the goal of the three-dimensional occupancy prediction task is to perform voxel-level modeling on the three-dimensional scene, in terms of data labels, different from traditional 3D object detection, what the three-dimensional occupancy prediction task requires for labels is three-dimensional occupancy voxel labels, which represent the position and semantics of each grid, rather than box frames. Therefore, the model training also needs to generate three-dimensional occupancy dense labels. As Figure 2 shown, the method for generating three-dimensional occupancy dense labels includes:

[0035] Step 21, form a dense three-dimensional scene from multiple frames of lidar point cloud data and visual depth information obtained by the multi-sensor synchronous acquisition device;

[0036] As can be seen from Step 1, the multi-sensor synchronous acquisition device can obtain multiple frames of lidar point cloud data and visual depth information.

[0037] Step 22, use the Poisson reconstruction algorithm to fill in the structural holes within a preset range;

[0038] The structural holes within the preset range usually refer to filling in the structural holes within 5 meters using the Poisson reconstruction algorithm.

[0039] Step 23, discretize the reconstructed scene into a voxel grid with a preset resolution, so as to later use three-dimensional occupancy dense labels to represent the position and semantics of each grid;

[0040] The preset resolution usually refers to a resolution of 0.5 meters.

[0041] Step 24: Use the Laplacian smoothing and QEM grid optimization algorithms to optimize the voxel grid to ensure surface continuity and geometric accuracy.

[0042] Among them, Laplacian smoothing, also known as Laplacian smoothing, was proposed by the French mathematician Laplace and is a method to solve the zero-probability problem. In application scenarios such as text classification, Laplacian smoothing can correct probability estimates and improve prediction accuracy. Quadic Error Metrics (QEM) grid simplification is an optimization algorithm based on quadratic metric error.

[0043] Step 25: Determine the voxel semantic labels through the multi-frame point cloud cumulative voting mechanism and k-nearest neighbor semantic statistics.

[0044] The multi-frame point cloud cumulative voting mechanism refers to a mechanism method that draws on the cumulative voting system in the field of computer vision or 3D reconstruction to process the fusion and decision-making of multi-frame point cloud data. The k-nearest neighbor (KNN), that is, the nearest neighbor algorithm, means that if most of the k nearest samples of a sample in the feature space belong to a certain category, then the sample also belongs to this category and has the characteristics of the samples in this category. That is, each sample can be represented by its k closest neighbors. The k-nearest neighbor algorithm is suitable for classification and also for regression, and is widely used in recommendation systems, semantic search, and anomaly detection.

[0045] Step 26: Use the conditional random field to process the voxel semantic labels to ensure spatial consistency and the accuracy of semantic boundaries, and generate a three-dimensional occupancy dense label.

[0046] The conditional random field (CRF) is a discriminative probabilistic undirected graph model, commonly used in sequence labeling tasks in natural language processing. CRF, that is, Conditional Random Fields, is a conditional probability distribution model of another set of output random variables given a set of input random variables. It is a discriminative probabilistic undirected graph model.

[0047] Step 3: According to the multi-frame historical image information obtained by the multi-sensor synchronous acquisition device, obtain multi-scale bird's-eye view features with historical information through the bird's-eye view encoder, and reconstruct the bird's-eye view features into three-dimensional voxel features by spatio-temporal alignment and feature fusion to construct a three-dimensional occupancy prediction model.

[0048] Step 4: Through the training set and the test set, train, test, and debug the parameters of the model. When the performance of the model reaches the optimal level, the model completes the training and debugging.

[0049] After the model training and testing in step 3, after designing the 3D occupancy prediction model, the model is trained and tested on the dataset made in step 2, and the parameters are debugged to make the performance of the model reach the optimal, and at the same time, the optimal weights are saved.

[0050] Step 5, through the deep learning inference optimizer, quantize the trained model, convert the 3D occupancy prediction model into an inference engine, and then deploy it on the intelligent driving vehicle for object detection.

[0051] After the model training is completed, in order to apply the model to the real vehicle environment, the present invention performs model quantization through TensorRT, converts the deep learning model into an inference engine on the embedded system, and deploys it on the embedded device to achieve the inference acceleration of the 3D occupancy prediction engine. TensorRT is a high-performance deep learning inference optimizer that can provide low-latency and high-throughput deployment inference for deep learning applications, and can be used to accelerate the inference of large-scale data centers, embedded platforms or autonomous driving platforms. TensorRT is a C++ library that provides a C++ API (API is a set of subroutine definitions, protocols and tools for building application software. Generally speaking, this is a set of clearly defined communication methods between various software components), mainly used for high-performance inference acceleration for NVIDIA GPUs. As Figure 3 shown, step 5 includes:

[0052] Step 51, convert the trained 3D occupancy prediction model into an engine file supported by the deep learning inference optimizer;

[0053] The present invention first exports the trained model in the ONNX format (the ONNX (Open Neural Network Exchange) file format is a standard for representing deep learning models, which allows the model to be transferred between different AI frameworks), and verifies the correctness and consistency of the exported model in the inference stage. During the export process, for the custom operators in the model, convert them into the form supported by ONNX, ensure the consistency of the operator functions, and at the same time enable the operator to be correctly mapped to the ONNX operator and seamlessly combined with other ONNX operators for inference.

[0054] Step 52, through the deep learning inference optimizer, convert the model with the engine file into an efficient inference engine suitable for the embedded device;

[0055] During the model conversion process, enable the quantization mode to convert the floating-point parameters in the model into integers, thereby significantly reducing memory occupancy and computational requirements. In addition, based on the in-vehicle collected dataset, construct a quantization calibration dataset and use this dataset to generate quantization parameters to ensure a significant improvement in inference speed while the inference accuracy of the model does not decrease significantly. For custom operators that TensorRT does not natively support, write custom plugins to ensure the efficient execution of these operators in TensorRT. At the same time, to further optimize the inference speed, fuse the convolutional layer with the BatchNorm (Batchnorm is an algorithm often used in deep networks to accelerate neural network training, accelerate the convergence speed, and improve stability) layer to reduce the computational amount and memory overhead, thereby further accelerating the inference process and reducing the inference latency.

[0056] Step 53, configure the inference environment on the embedded device, integrate the generated engine file into the C++ environment, and use the deep learning inference optimizer to deserialize the engine, thereby completing the conversion from the three-dimensional occupancy prediction model to the inference engine.

[0057] TensorRT will automatically identify the hardware resources and optimize the scheduling of the inference tasks to make full use of the hardware acceleration. At the same time, with the help of TensorRT's memory optimization technology, specify the video memory for the input and output bound to each engine, avoid repeated video memory applications, reduce memory occupancy, and improve the resource reuse efficiency. By enabling the INT8 quantization (INT8 quantization is a technology that converts floating-point model parameters and activation values into 8-bit integers) mode, reduce the computational complexity, further improve the inference speed, and optimize the utilization of device resources.

[0058] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios. Existing technologies mostly focus on the voxel occupancy probability prediction of static scenarios, do not fully consider the motion states of dynamic objects, and cannot capture key kinematic parameters such as object speed and acceleration. In view of the existing technical deficiencies, the technical solution provided by this application uses a feature pyramid to extract 2D fine-grained image features to maintain the accuracy of the model. At the same time, it adopts a lightweight three-dimensional feature reconstruction strategy, and through parametric modeling, converts the efficient representation in the BEV (Bird's Eye View) space into fine-grained voxel predictions, avoiding the direct computational load of high-dimensional voxel data, significantly reducing memory occupancy, and deploying a dual-task prediction head in parallel on the BEV feature layer to perform occupancy prediction and speed prediction respectively, so as to solve the problem that the decision-making reliability of the traditional technical solution in downstream tasks such as emergency obstacle avoidance and trajectory planning in the autonomous driving system is restricted due to the lack of speed prediction ability.

[0059] Embodiment III

[0060] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios, as Figure 1 shown, the method includes the following steps.

[0061] Step 1, obtain image information, lidar point cloud data, and visual depth information through a multi-sensor synchronization acquisition device;

[0062] Step 2, construct a training set and a test set according to the information obtained by the multi-sensor synchronization acquisition device, and generate three-dimensional occupancy dense labels;

[0063] Step 3, according to the multi-frame historical image information obtained by the multi-sensor synchronization acquisition device, obtain multi-scale bird's-eye view features with historical information through a bird's-eye view encoder, and reconstruct the bird's-eye view features into three-dimensional voxel features by using spatio-temporal alignment and feature fusion to construct a three-dimensional occupancy prediction model;

[0064] As Figure 4 shown, this step is based on the input multi-frame historical image information, obtains multi-scale BEV features with historical information through a BEV encoder based on a feature pyramid network, and reconstructs the BEV features into three-dimensional voxel features by using spatio-temporal alignment and feature fusion, thereby constructing a three-dimensional occupancy grid prediction model. The step 3 includes the following steps.

[0065] Step 31, crop, scale, and splice the obtained multi-frame historical images to obtain multi-frame two-dimensional images;

[0066] Step 32, perform multi-scale image feature extraction on the multi-frame two-dimensional images through a residual network and a feature pyramid network;

[0067] In this step, according to the multi-frame two-dimensional images obtained by pre-processing, a residual network and a feature pyramid network are used to extract multi-scale image features. The step 32 includes the following steps.

[0068] Step 321, extract feature maps P of different scales from the multi-frame two-dimensional images through a residual network k , where k represents the scale;

[0069] Step 322, perform upsampling through the feature pyramid network using a top-down path, and the high-level feature maps are upsampled to the resolution of the low-level feature maps;

[0070] Step 323, fuse the upsampled feature maps with the original high-level feature maps, and after obtaining the fused multi-scale feature maps, perform multi-scale image feature extraction, where the fusion formula is: where, let the high-level feature map be P k+1 , and the low-level feature map be P k, Upsample represents the upsampling operation, is the fused multi-scale feature map.

[0071] Step 33: Expand, project, and aggregate the multi-scale image features through a depth prediction network to obtain the bird's-eye view features and acquire the bird's-eye features;

[0072] The said step 33 includes:

[0073] Step 331: Predict the discrete depth of the multi-scale feature map through a depth prediction network;

[0074] Step 332: Expand the 2D image features of the multi-scale feature map into 3D spatial features according to the discrete depth;

[0075] Step 333: Project the point cloud features in the 3D spatial features onto the bird's-eye view grid to obtain the bird's-eye view features;

[0076] Step 334: Aggregate the features of the bird's-eye view features to obtain the bird's-eye features.

[0077] Step 34: Perform transformation, correction, and supplementation on the bird's-eye features using spatio-temporal alignment to enhance the perception ability of dynamic objects or attributes;

[0078] The said step 34 includes:

[0079] Step 341: Perform spatio-temporal coordinate transformation on the bird's-eye features at each historical moment using the ego-vehicle motion information;

[0080] Step 342: Correct the pose differences caused by vehicle motion to obtain the corrected bird's-eye features;

[0081] Step 343: Dynamically adjust the spatial position of the features through deformable convolution to compensate for the alignment error caused by sensor noise. The feature adjustment formula is: where, is the bird's-eye feature after deformable convolution, DeformConv is the deformable convolution, is the corrected bird's-eye feature, F BEV (t) is the current bird's-eye feature, is the bird's-eye feature after deformable convolution.

[0082] Step 35: Adopt a multi-scale fusion strategy to perform feature weighted aggregation on the current bird's-eye feature and the corrected bird's-eye feature to obtain the final bird's-eye feature;

[0083] To further enhance the perception ability of dynamic objects or attributes, this application also adopts feature fusion. Through a multi-scale fusion strategy, feature weighted aggregation is performed to highlight the motion significant regions. The formula for the weighted aggregation is as follows: where w i is the weight of each scale feature, N is the number of historical frames, MultiScaleFusion is multi-scale fusion, is the final bird's-eye view feature after weighted fusion.

[0084] Step 36, through parametric modeling, construct a three-dimensional occupancy prediction model according to the final bird's-eye view feature.

[0085] To reduce memory consumption and improve computational efficiency, in step 36, a three-dimensional occupancy prediction model is constructed through parametric modeling. The specific steps include:

[0086] Step 361, convert the bird's-eye view spatial feature into a fine-grained voxel feature F Voxel ;

[0087] Step 362, convert the 3D coordinates of the voxels into high-dimensional vectors through an embedding layer;

[0088] Step 363, combine the embedded high-dimensional vectors with the encoded bird's-eye view features and input them into a multi-layer perceptron to generate voxel features;

[0089] Step 364, decode the three-dimensional voxel feature containing temporal information into a velocity vector V t , thus completing the construction of the three-dimensional occupancy prediction model.

[0090] By adopting a lightweight three-dimensional feature reconstruction strategy, this application avoids complex three-dimensional convolution calculations, significantly reduces the video memory occupancy and computational complexity. At the same time, by directly decoding the BEV feature into three-dimensional voxel features through the embedded coordinate vectors, the efficiency and resource consumption of three-dimensional voxel modeling are further optimized.

[0091] Step 4, through the training set and test set, train, test and debug the parameters of the model. When the performance of the model reaches the optimal, the model completes the training and debugging;

[0092] Step 5, through a deep learning inference optimizer, quantize the trained model, convert the three-dimensional occupancy prediction model into an inference engine, and then deploy it to an intelligent driving vehicle for object detection.

[0093] The present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios. Existing technologies mostly focus on predicting the voxel occupancy probability of static scenarios, without fully considering the motion states of dynamic objects and unable to capture key kinematic parameters such as object speed and acceleration. In view of the existing technical defects, the technical solution provided by this application uses a feature pyramid to extract 2D fine-grained image features to maintain the accuracy of the model. At the same time, it adopts a lightweight three-dimensional feature reconstruction strategy. Through parametric modeling, the efficient representation in the BEV (Bird's Eye View) space is transformed into fine-grained voxel prediction, avoiding the direct computational load of high-dimensional voxel data in traditional methods and significantly reducing memory occupancy. A dual-task prediction head is deployed in parallel on the BEV feature layer to perform occupancy prediction and speed prediction respectively, so as to solve the problem that the decision reliability of the traditional technical solution in downstream tasks such as emergency obstacle avoidance and trajectory planning is restricted due to the lack of speed prediction ability in the traditional technical solution.

[0094] In summary, the embodiment of the present invention provides a three-dimensional occupancy prediction and quantization deployment method for open scenarios, which is a lightweight and easily deployable three-dimensional occupancy grid detection method for open driving scenarios. Through sparse three-dimensional voxel modeling, the three-dimensional space scene is transformed into sparse voxel units, and each voxel unit corresponds to certain spatial occupancy probability and corresponding voxel semantic categories, spatial positions, depths and other attributes, realizing the extraction of target semantic information and the reconstruction of spatial positions in the three-dimensional space. Since in complex and dynamic open driving scenarios, vision-centered 3D object detection faces problems such as insufficient fine-grained modeling ability of object geometric forms and limited detection categories, and the voxel modeling method has problems of high video memory occupancy and high computational complexity, the present invention designs a lightweight and easily deployable three-dimensional occupancy grid detection method. On the premise of maintaining accuracy, it adopts a lightweight three-dimensional feature reconstruction strategy to avoid complex 3D convolution calculations. By embedding coordinate vectors, the BEV features are directly decoded into three-dimensional voxel features, reducing the computational burden and memory occupancy of three-dimensional voxel modeling. At the same time, a spatio-temporal alignment module and a feature fusion module are introduced to enhance the perception ability of dynamic objects. Three-dimensional grid modeling and speed prediction are performed through a dual-task prediction head, improving the detection accuracy and practicability. Finally, the model is quantized and deployed on an embedded platform to achieve quantization acceleration of the model, so as to meet the dual requirements of real-time performance and accuracy in real vehicle scenarios.

[0095] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A three-dimensional occupancy prediction and quantization deployment method for open scenarios, applied to intelligent driving vehicles, characterized in that, The method includes: Step 1, obtaining image information, lidar point cloud data, and visual depth information through a multi-sensor synchronous acquisition device; Step 2, constructing a training set and a test set based on the information obtained by the multi-sensor synchronous acquisition device, and generating a three-dimensional occupancy dense label; Step 3, according to the multi-frame historical image information obtained by the multi-sensor synchronous acquisition device, obtaining multi-scale bird's-eye features with historical information through a bird's-eye view encoder, and reconstructing the bird's-eye features into three-dimensional voxel features by using spatio-temporal alignment and feature fusion to construct a three-dimensional occupancy prediction model; Step 4, training, testing, and parameter debugging the model through the training set and the test set. When the performance of the model reaches the optimal, the model completes the training and debugging; Step 5, quantifying the trained model through a deep learning inference optimizer, converting the three-dimensional occupancy prediction model into an inference engine, and then deploying it to an intelligent driving vehicle for object detection.

2. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 1, characterized in that The multi-sensor synchronous acquisition device includes a lidar, a panoramic camera, and a high-precision inertial measurement unit.

3. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 1, wherein The method for generating a three-dimensional occupancy dense label includes: Step 21, forming a dense three-dimensional scene from the multi-frame lidar point cloud data and visual depth information obtained by the multi-sensor synchronous acquisition device; Step 22, using the Poisson reconstruction algorithm to fill in the structural holes within a preset range; Step 23, discretizing the reconstructed scene into a voxel grid with a preset resolution, so as to represent the position and semantics of each grid with a three-dimensional occupancy dense label in the later stage; Step 24, using the Laplacian smoothing and QEM grid optimization algorithm to optimize the voxel grid to ensure surface continuity and geometric accuracy; Step 25, determining the voxel semantic label through a multi-frame point cloud cumulative voting mechanism and k-nearest neighbor semantic statistics; Step 26, processing the voxel semantic label using a conditional random field to ensure spatial consistency and the accuracy of semantic boundaries, and generating a three-dimensional occupancy dense label.

4. A three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 1, characterized in that The said Step 3 includes: Step 31, performing cropping, scaling, and stitching on the obtained multi-frame historical images to obtain multi-frame two-dimensional images; Step 32, performing multi-scale image feature extraction on the multi-frame two-dimensional images through a residual network and a feature pyramid network; Step 33, performing expansion, projection, and aggregation processing on the multi-scale image features through a depth prediction network to obtain bird's-eye view features and acquire the bird's-eye features; Step 34, using spatio-temporal alignment to transform, correct, and supplement the bird's-eye features to enhance the perception ability of dynamic objects or attributes; Step 35, adopting a multi-scale fusion strategy to perform feature weighted aggregation on the current bird's-eye features and the corrected bird's-eye features to obtain the final bird's-eye features; Step 36, constructing a three-dimensional occupancy prediction model according to the final bird's-eye features through parametric modeling.

5. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 4, wherein The said Step 32 includes: Step 321, extract feature maps \(P\) of different scales from multiple frames of two-dimensional images through a residual network, where \(k\) represents the scale; k ​ Step 322, performing upsampling through the feature pyramid network using a top-down path, and the high-level feature map is upsampled to the resolution of the low-level feature map; Step 323, fuse the enhanced feature map with the original high-level feature map. After obtaining the fused multi-scale feature map, perform multi-scale image feature extraction. The fusion formula is as follows: where the high-level feature map is denoted as P k+1 , the low-level feature map is denoted as P k , Upsample represents the upsampling operation, is the fused multi-scale feature map.

6. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 4, wherein The said Step 33 includes: Step 331, predicting the discrete depth of the multi-scale feature map through a depth prediction network; Step 332, expand the 2D image features of the multi-scale feature map into 3D spatial features according to the discrete depth; Step 333: Project the point cloud features in the 3D spatial features onto the bird's-eye view grid to obtain the bird's-eye view features; Step 334: Aggregate the features of the bird's-eye view to obtain the bird's-eye features.

7. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 4, wherein The said step 34 includes: Step 341: Perform spatio-temporal coordinate transformation on the bird's-eye features at each historical moment using the ego-vehicle motion information; Step 342: Correct the pose differences caused by vehicle motion to obtain the corrected bird's-eye features; Step 343, dynamically adjust the spatial position of the features through deformable convolution to compensate for the alignment error caused by sensor noise. The feature adjustment formula is as follows: where is the bird's-eye view feature after deformable convolution, DeformConv is the deformable convolution, is the corrected bird's-eye view feature.

8. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 4, characterized in that The formula for the weighted aggregation is as follows: Among them, w i is the weight of each scale feature, N is the number of historical frames, MultiScaleFusion is multi-scale fusion, is the final bird's-eye view feature after weighted fusion, F BEV (t) is the current bird's-eye view feature, is the bird's-eye view feature after deformable convolution.

9. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 4, wherein The said step 36 includes: Step 361, convert the bird's-eye view spatial features through parametric modeling into fine-grained voxel features F Voxel ; Step 362: Convert the 3D coordinates of the voxel into a high-dimensional vector through an embedding layer; Step 363: Combine the embedded high-dimensional vector with the encoded bird's-eye features and input them into a multi-layer perceptron to generate voxel features; Step 364, decode the three-dimensional voxel feature containing temporal information into a velocity vector V t , thus completing the construction of the three-dimensional occupancy prediction model.

10. The three-dimensional occupancy prediction and quantization deployment method for an open scenario according to claim 1, wherein The said step 5 includes: Step 51: Convert the trained three-dimensional occupancy prediction model into an engine file supported by the deep learning inference optimizer; Step 52: Through the deep learning inference optimizer, convert the model with the engine file into an efficient inference engine suitable for embedded devices; Step 53: Configure the inference environment on the embedded device, integrate the generated engine file into the C++ environment, and use the deep learning inference optimizer for engine deserialization, thereby completing the conversion of the three-dimensional occupancy prediction model to the inference engine.

Images