Event camera assisted multi-modal vehicle-mounted three-dimensional occupancy prediction method

Abstract

The invention discloses an event camera-assisted multi-modal vehicle-mounted three-dimensional occupancy prediction method, which comprises the following steps of: acquiring continuously acquired laser radar point cloud data, images, events and IMU measurement data, performing synchronization of spatial dimensions and time dimensions on various data, and determining the occupancy of a vehicle-mounted vehicle on the basis of the synchronized laser radar point cloud data and IMU measurement data, so as to predict the occupancy of the vehicle-mounted vehicle-mounted vehicle-mounted occupancy of the vehicle-mounted vehicle-mounted vehicle-mounted vehicle-mounted vehicle-mounted vehicle-mounted image. Dense point cloud data is obtained; performing space-time alignment on the dense point cloud, the image and the event, performing semantic annotation on the image after the space-time alignment, performing dynamic and static separation on the dense point cloud after the space-time alignment through cross-modal semantic projection, and performing static scene optimization and dynamic scene fusion to obtain an occupied voxel true value; training an occupancy prediction model based on the laser radar point cloud data and the occupied voxel truth value; and acquiring laser radar point cloud data and image data of a scene and event data in image exposure time, and realizing three-dimensional occupation prediction by using the trained occupation prediction model.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving and computer vision technology, and in particular relates to an event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction method. Background Art

[0002] Three-dimensional occupancy prediction, also known as scene semantic completion, is a core technology in autonomous driving environmental perception. It achieves a fine-grained understanding of complex scenes through three-dimensional voxel modeling. Three-dimensional occupancy prediction divides the environment into a voxel grid and predicts whether each voxel is occupied and its semantic category (such as vehicle, pedestrian, building, etc.). Compared to traditional 3D object detection, which focuses only on specific objects, 3D occupancy prediction can fully describe the scene's geometric structure and semantic information, including occluded areas and irregular obstacles.

[0003] Current in-vehicle environmental perception systems primarily rely on traditional frame cameras and LiDAR for three-dimensional occupancy prediction. For example, traditional methods use LiDAR to capture the sparse geometric structure of the real world and further generate dense, high-precision three-dimensional semantic voxels. Current vision-based solutions estimate the geometric structure and semantic information of a scene from images at a lower cost, but compared to LiDAR solutions, it is more difficult to directly obtain the scene's geometric information. Both LiDAR-based and traditional frame camera-based methods are limited by sensor performance and degrade under high-speed motion and adverse weather conditions, resulting in reduced perception accuracy. In terms of data annotation, manual annotation of point cloud semantic information is often used through interactive pages.

[0004] An event camera is a new type of image sensor that detects pixel-level brightness changes and outputs an asynchronous event stream, rather than the frame images of traditional cameras. This gives events the advantages of high dynamic range, low latency, high timeliness, and low redundant information. This offers great potential for resolving the "ghosting" problem in autonomous driving scenarios. In high-speed motion and inclement weather conditions, where lidar and traditional frame cameras can degrade, event cameras can output stable information, improving autonomous driving safety. Currently, there are no in-vehicle systems that combine event cameras, lidar, and traditional frame cameras for 3D occupancy prediction.

[0005] Existing 3D occupancy prediction technology has significant drawbacks. First, it has limitations in dynamic scene perception: Traditional frame cameras are prone to motion blur in high-speed motion scenes, resulting in distorted 3D occupancy predictions for dynamic objects (such as suddenly appearing pedestrians). Second, it has limitations in adverse scene perception: In extreme weather conditions such as fog and poor lighting conditions, traditional frame cameras can lose information due to low illumination, overexposure, and underexposure.

[0006] Events offer great potential for resolving these challenges due to their high dynamic range, low latency, high timeliness, and low redundant information. However, there is currently no research on multimodal, on-vehicle 3D occupancy prediction systems assisted by event cameras. Furthermore, the efficiency of labeling 3D occupancy prediction data is limited. Traditional 3D labeling requires annotating point cloud data frame by frame, which is time-consuming and costly. Furthermore, there is still no mature and operational solution for 4D labeling, specifically for handling dynamic objects. Summary of the Invention

[0007] In order to overcome the dynamic scene perception limitations and harsh scene perception limitations of traditional three-dimensional occupancy prediction systems, as well as the low efficiency of 4D annotation of three-dimensional occupancy prediction data, the present invention proposes an event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction method.

[0008] According to a first aspect of an embodiment of the present application, a multimodal vehicle-mounted three-dimensional occupancy prediction method assisted by an event camera is provided, comprising:

[0009] Step S1: Acquire continuously collected lidar point cloud data, images, events, and IMU measurement data, synchronize the spatial and temporal dimensions of each type of data, and obtain dense point cloud data based on the synchronized lidar point cloud data and IMU measurement data;

[0010] Step S2: Perform spatiotemporal alignment on the dense point cloud, image, and event, perform semantic annotation on the spatiotemporal aligned image, perform dynamic and static separation on the spatiotemporal aligned dense point cloud through cross-modal semantic projection, perform static scene optimization and dynamic scene fusion, and obtain the true value of occupied voxels;

[0011] Step S3: training an occupancy prediction model based on the lidar point cloud data obtained in step S1 and the ground truth values ​​of occupied voxels obtained in step S2;

[0012] Step S4: Obtain the lidar point cloud data, image data, and event data within the image exposure time of the scene, and use the trained occupancy prediction model to achieve three-dimensional occupancy prediction.

[0013] Furthermore, in step S1, the dense point cloud data is obtained through the following process:

[0014] Using GNSS and IMU combined positioning, the extended Kalman filter fuses global position and inertial measurement data to directly output 6-DoF pose. If there is a GPS signal obstruction problem, positioning degradation compensation must be performed.

[0015] Based on the SLAM framework, the GPS / IMU pose is combined with the laser point cloud to build a globally consistent high-precision map. At the same time, the accumulated error is corrected through loop detection to obtain dense point cloud data.

[0016] Furthermore, step S2 includes:

[0017] S2-1: For dense point clouds, images, and events, temporal alignment is performed based on the timestamps of the images, and spatial alignment is performed based on the field of view of the event camera.

[0018] S2-2: Use the Grounded-SAM (Grounded-SAM) semantic segmentation model to perform pixel-level semantic annotation on the spatiotemporally aligned images to obtain two-dimensional semantic labels.

[0019] S2-3: Based on the external parameters of the lidar and RGB camera, the dense point cloud is separated into static and dynamic maps through cross-modal semantic projection to obtain static and dynamic maps;

[0020] S2-4: Spatiotemporal refinement of dynamic objects: Based on the dynamic map, perform instance annotation of dynamic target tails in the BEV view and perform spatiotemporal interpolation optimization;

[0021] S2-5: Probabilistically guided voxel refinement, static scene optimization through point cloud clustering, dynamic scene fusion through voting mechanism and obtaining the ground truth of occupied voxels.

[0022] Furthermore, step S2-1 includes

[0023] Time alignment is performed based on the timestamp of the image: the timestamp t of each frame of image is img , find the timestamp t of the point cloud frame with the smallest time difference in the point cloud sequence pc , if |t img -t pc |≤t threshold , it is considered synchronized, otherwise the image frame is discarded, where t threshold is the timestamp difference threshold;

[0024] Spatial alignment based on the field of view of the event camera: Image distortion correction and resampling are performed to ensure that the image content matches the field of view of the event camera.

[0025] Furthermore, step S2-3 includes:

[0026] Cross-modal semantic projection: Based on the external parameter matrix of the lidar and RGB camera, the dense point cloud data is cross-modally projected to the camera coordinate system, the dense point cloud data in the camera coordinate system is transformed into a semantic map through perspective transformation, and the corresponding two-dimensional semantic label I is obtained according to the pixel position kabek , thus obtaining the point cloud semantic label P label ;

[0027] Dynamic and static separation strategy: According to the point cloud semantic label P label, dynamic targets are separated from static backgrounds in dense point cloud data, and static maps and dynamic maps are constructed using lidar mapping methods. In the dynamic map, the motion trajectory of the dynamic target is modeled as a three-dimensional dynamic motion pipeline.

[0028] Furthermore, step S2-4 includes:

[0029] BEV interactive annotation: projecting the three-dimensional dynamic motion pipeline into a two-dimensional bird's-eye view coordinate system for interactive calibration;

[0030] Spatiotemporal interpolation optimization: Based on the trajectory vertex coordinates of the point cloud timestamp and the orientation angle of the 3D dynamic motion pipeline, linear interpolation is used to calculate the intermediate position coordinates and orientation angles lost between frames due to the lack of lidar scanning.

[0031] Furthermore, step S2-5 includes

[0032] Static scene optimization: Based on the RANSAC plane fitting algorithm, the ground detection and elimination of static point clouds are performed, and the k-means clustering unit is constructed based on the xy plane coordinate difference of non-ground point clouds, combined with the point cloud semantic label P label Statistical distribution probability, assigning the semantic category with the largest distribution probability to the corresponding point cloud cluster;

[0033] Dynamic scene fusion: The dynamic and static point clouds are segmented by spatiotemporal bounding boxes, divided into voxel grids, and the occupancy probability of dynamic targets in each voxel is calculated through a voting mechanism;

[0034] Among them, the i-th voxel V i The semantic label L that appears most frequently in i for:

[0035]

[0036] Among them, L i,j For V i The jth point P in i,j The associated semantic label, δ(L i,j ,l) is the Kronecker delta function, l is the semantic label of the distribution.

[0037] Furthermore, the occupancy prediction model in step S3 includes a voxel query suggestion generation module based on LMSCNet, two encoders, a self-attention module, a deformable attention module and a segmentation head. The voxel query suggestion generation module based on LMSCNet receives lidar point cloud data and generates sparse voxel query suggestions; the first encoder receives rasterized events and calculates the keys and values ​​of the events; the second encoder receives images and calculates the keys and values ​​of the images; the keys and values ​​of the events and images are added and fused respectively and then input into the self-attention module for feature enhancement to obtain fused features; the deformable attention module generates three-dimensional voxel features based on the sparse voxel query suggestions and fused features; the segmentation head converts the three-dimensional voxel features into three-dimensional occupancy prediction results.

[0038] Furthermore, the specific process of event rasterization is as follows:

[0039] Get event stream data, which includes all events within the exposure time of each frame. Each event is represented by (x, y, polarity, timestamp), where x and y are the pixel coordinates of the event, polarity represents the polarity of the event, and timestamp is the timestamp of the event.

[0040] According to the polarity of the event, the events are divided into positive events and negative events. At each pixel position, the number of positive events and negative events and the timestamp of the last positive event and negative event are counted, and each channel of each pixel is normalized separately.

[0041] According to a second aspect of an embodiment of the present application, there is provided an event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction device, comprising:

[0042] Dense point cloud acquisition module: acquires continuously collected lidar point cloud data, images, events, and IMU measurement data, synchronizes the spatial and temporal dimensions of various data, and obtains dense point cloud data based on the synchronized lidar point cloud data and IMU measurement data;

[0043] Occupancy voxel truth acquisition module: This module performs spatiotemporal alignment of dense point clouds, images, and events, semantically annotates the aligned images, separates static and dynamic features of the aligned dense point clouds through cross-modal semantic projection, and performs static scene optimization and dynamic scene fusion to obtain the occupied voxel truth value.

[0044] Model training module: trains the occupancy prediction model based on the lidar point cloud data obtained by the dense point cloud acquisition module and the occupied voxel true value obtained by the occupied voxel true value acquisition module;

[0045] 3D occupancy prediction module: obtains the lidar point cloud data, image data and event data within the image exposure time of the scene, and uses the trained occupancy prediction model to realize 3D occupancy prediction.

[0046] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0047] Compared to traditional sensor-based 3D occupancy prediction techniques, this solution fundamentally overcomes the inherent limitations of traditional technologies in temporal resolution by innovatively leveraging event stream data to fill in the information gaps within the 10ms time window. This approach significantly improves dynamic scene perception capabilities. This technical approach addresses the underlying imaging mechanism that causes motion blur in traditional frame cameras during high-speed motion (e.g., turning). It constructs an optimization system consisting of data processing algorithms, feature extraction models, and prediction models to achieve high-precision 3D occupancy prediction for dynamic objects (e.g., peeking pedestrians). Compared to existing event camera-assisted visual 3D occupancy prediction methods, this invention incorporates a cross-modal data calibration and complementary enhancement mechanism to leverage the synergistic advantages of LiDAR's precise 3D structural information and the high dynamic response characteristics of event stream data. Furthermore, the event camera's high dynamic range, which covers illumination environments ranging from 0.1 to 10 klux, significantly enhances perception robustness in extreme scenarios such as fog and nighttime. This can effectively enhance the safety of autonomous driving. Furthermore, to address the current shortage and difficulty in acquiring 3D occupancy prediction datasets, this application proposes a semi-automatic 4D dataset annotation solution. This effectively reduces the cost of point cloud annotation, addresses the annotation redundancy issue in traditional methods, and improves the efficiency of dynamic object annotation. This not only provides real-world training data and enriches autonomous driving datasets, but is also of great significance for the advancement of autonomous driving.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 The figure is a schematic diagram showing an event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction method according to an exemplary embodiment.

[0051] Figure 2 This is a schematic diagram of the 3D occupancy prediction data annotation method.

[0052] Figure 3 It is a voxel construction display diagram.

[0053] Figure 4 It is the network structure diagram of the three-dimensional occupancy prediction model.

[0054] Figure 5 This is a three-dimensional occupancy prediction effect diagram.

[0055] Figure 6 The present invention is a block diagram of an event camera-assisted multi-modal vehicle-mounted three-dimensional occupancy prediction device according to an exemplary embodiment.

[0056] Figure 7 The figure is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0058] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0059] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0060] This application provides an event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction method, such as Figure 1 As shown, it includes:

[0061] Step S1: Acquire continuously collected lidar point cloud data, images, events, and IMU measurement data, synchronize all types of data, calibrate the intrinsic and extrinsic parameters of each sensor, and obtain dense point cloud data based on the synchronized lidar point cloud data and IMU measurement data;

[0062] Specifically, continuous data collection is performed through a lidar, RGB camera, event camera, and IMU installed on the top of the vehicle, where the event camera is equipped with a wide-angle lens to expand its field of view.

[0063] In one embodiment, the following sensor models are used: the Velodyne HDL-32E lidar has a measurement range of 1 to 70 meters, 32 scan lines, angular resolution of 0.16° horizontally at 600 rpm and 1.33° vertically, and operates at 10 fps. The 9-axis CH104M IMU and the Realsense D455i RGB-D camera offer a depth error of <2% at 4 meters, a field of view of 87° by 58°, a baseline of 9.5 cm, and 30 fps. The Prophesee EVK4 event camera offers a resolution of 1280 x 720 pixels, a pixel size of 4.86 μm x 4.86 μm, and a maximum bandwidth of 1.6 Gbps.

[0064] In specific implementations, since event cameras are sensitive to infrared light, an infrared filter needs to be installed in front of the lens to filter out interference signals generated by lidar (such as 905nm / 1550nm wavelengths) and RGB-D sensors (such as active infrared light sources). This measure can avoid false triggering due to infrared noise in event stream data and improve the accuracy of dynamic target detection. In addition, installing the lidar at the highest point of the vehicle (such as the roof bracket) can reduce the obstruction of the scanning field of view by the vehicle structure (such as the body and rearview mirrors), ensuring 360° point cloud coverage. This layout strategy reduces the impact of blind spots on map integrity by maximizing the vertical field of view.

[0065] After obtaining the data of the multimodal sensor, the various data are aligned in time and space. The spatial coordinates of the multimodal sensor are unified by jointly calibrating the intrinsic and extrinsic matrix of the lidar and each camera. Specifically, the intrinsic parameter matrix I of the event camera and the RGB camera is obtained in advance. RGB with I event Obtain multi-sensor external parameters in advance through calibration: the transformation matrix T from the LiDAR coordinate system to the event camera coordinate system LiDAR2RGB , the transformation matrix T from the RGB camera coordinate system (RGB Camera) to the event camera coordinate system RGB2Event Based on the ROS (Robot Operating System) framework, the system integrates various sensor SDKs (Software Development Kits) to synchronously acquire RGB images, depth maps, LiDAR point clouds, and event stream data under low-light conditions, achieving time synchronization.

[0066] Regarding the marking of the internal and external parameters of the sensor, this embodiment is achieved by the following method:

[0067] (1) Labeling the intrinsic parameters of the event camera and the extrinsic parameters of the event camera and RGB-D camera: First, convert the asynchronous events into frames, shoot the flickering checkerboard and use E2VID to reconstruct the frames, and then directly label the event frames and images using the OpenCV labeling toolbox.

[0068] (2) External parameter annotation of camera and lidar: Using the open source autonomous driving software platform Autoware, the key points of calibration are: a sufficiently open scene is required so that the calibration plate is easily identifiable in the point cloud; the checkerboard needs to be deflected at various angles and placed at different positions on the left, center, and right in front of the camera. A total of 20 to 30 frames. Through this process, we can obtain the internal parameters I of the event camera and RGB camera. RGB with I event , the transformation matrix T from the LiDAR coordinate system to the EventCamera coordinate system LiDAR2RGB , the transformation matrix T from the RGB camera coordinate system (RGB Camera) to the event camera coordinate system RGB2Event .

[0069] The above-mentioned camera and lidar extrinsic parameter annotation methods involve lidar mapping methods. Therefore, the following two solutions can be given for sensor construction:

[0070] (i) Mapping based on Fast-LIO2 (requires IMU assistance): If the lidar (such as Velodyne-32) does not have a built-in IMU, an independent IMU module must be installed and aligned with the lidar data through hardware synchronization to provide an initial pose estimate.

[0071] (ii) GPS / IMU fusion mapping (no Fast-LIO2 required): GNSS (such as RTK-GPS) and high-precision IMU (such as NovAtel SPAN-IGM) are used for positioning. The global position and inertial measurement data are fused through the extended Kalman filter (EKF) to directly output the 6-DoF pose. This method is suitable for open scenes, but attention should be paid to compensating for positioning degradation under GPS signal occlusion (such as tunnels and urban canyons). Then, based on the SLAM framework (such as LIO-SAM), the GPS / IMU pose is combined with the laser point cloud to construct a globally consistent high-precision map. At the same time, the accumulated error is corrected through loop closure detection, thus obtaining dense point cloud data.

[0072] After the above processing, the spatiotemporally aligned multimodal data (dense point cloud, image, event) is encapsulated through the ROS workspace to output the joint perception data stream required for the 3D occupancy prediction task.

[0073] Step S2: Perform spatiotemporal alignment on the dense point cloud, image, and event, perform semantic annotation on the spatiotemporal aligned image, perform dynamic and static separation on the spatiotemporal aligned dense point cloud through cross-modal semantic projection, perform static scene optimization and dynamic scene fusion, and obtain the true value of occupied voxels;

[0074] Specifically, this step can realize the construction from raw sensor data to 3D occupied voxels, including 2D semantic-guided mapping, spatiotemporal refinement of dynamic objects, and probability-guided voxel refinement. After the point cloud obtains semantics through projection, the dynamic target processing branch and the static background processing branch are separated. This step includes the following sub-steps:

[0075] S2-1: For dense point clouds, images, and events, temporal alignment is performed based on the timestamps of the images, and spatial alignment is performed based on the field of view of the event camera.

[0076] In one embodiment, the specific content of step S2-1 is:

[0077] (1.1) Time alignment is based on the timestamp of the image. For each frame of image, the timestamp t img , find the timestamp t of the point cloud frame with the smallest time difference in the point cloud sequence pc , if |t img -t pc |≤t threshold , it is considered synchronized, otherwise the image frame is discarded. In one embodiment, the timestamp difference threshold t threshold =25ms.

[0078] (1.2) Spatial alignment is based on the field of view of the event camera. Image distortion correction and resampling are performed to ensure that the image content strictly matches the field of view of the event camera.

[0079] S2-2: Use the Grounded-SAM (Grounded-SAM) semantic segmentation model to perform pixel-level semantic annotation on the spatiotemporally aligned images to obtain two-dimensional semantic labels.

[0080] Specifically, the spatiotemporally aligned RGB image is input into the Grounded-SAM model, and the pre-acquired text description (such as "vehicle, pedestrian, traffic sign") is converted into a semantic vector through the text encoder of the multimodal large model, and the association mapping between text and visual features is established, that is, the two-dimensional semantic label I label .

[0081] S2-3: Based on the external parameters of the lidar and RGB camera, the dense point cloud is separated into static and dynamic maps through cross-modal semantic projection to obtain static and dynamic maps;

[0082] In one embodiment, the specific content of step S2-3 is:

[0083] (3.1) Cross-modal semantic projection: Based on the extrinsic matrix of the lidar and RGB camera, the dense point cloud data is cross-modally projected into the camera coordinate system. The dense point cloud data in the camera coordinate system is transformed into a semantic map through perspective transformation, and the corresponding two-dimensional semantic label I is obtained according to the pixel position. label , thus obtaining the point cloud semantic label P label , to achieve accurate weighting of point cloud semantic attributes;

[0084]

[0085] in, is the coordinate of the pixel in the image in the camera coordinate system, is the coordinate of the laser radar point cloud data in the laser radar coordinate system, Z cam is the normalization coefficient, K is the camera intrinsic parameter matrix, and the extrinsic parameter matrix includes the rotation matrix R and the displacement vector t.

[0086] (3.2) Dynamic and static separation strategy: According to the point cloud semantic label P label , separate dynamic targets from static background in dense point cloud data (in one embodiment, point clouds labeled as cars, pedestrians, etc. are dynamic targets, and point clouds corresponding to other labels are considered static background). A LiDAR mapping method is used to construct static and dynamic maps, where the motion trajectories of dynamic targets are modeled as three-dimensional dynamic motion pipelines.

[0087] S2-4: Spatiotemporal refinement of dynamic objects: Based on the dynamic map, perform instance annotation of dynamic target tails in the BEV view and perform spatiotemporal interpolation optimization;

[0088] In a specific implementation, the specific content of step S2-4 is:

[0089] (4.1) BEV interactive annotation: The three-dimensional dynamic motion pipeline is projected into the two-dimensional bird's-eye view coordinate system and interactively calibrated, that is, left-click on the visualization interface to establish the direction vector, and right-click to confirm and automatically construct the bounding box.

[0090] (4.2) Spatiotemporal interpolation optimization: Based on the trajectory vertex coordinates of the point cloud timestamp and the orientation angle of the 3D dynamic motion pipeline, the intermediate position coordinates and orientation angles lost between frames due to the lack of lidar scanning are calculated through linear interpolation, eliminating the mutation problem caused by discrete sampling. This method effectively solves the tailing problem of dynamic targets.

[0091] S2-5: Probabilistically guided voxel refinement: Static scene optimization via point cloud clustering, dynamic scene fusion via voting mechanism and obtaining the ground truth of occupied voxels.

[0092] In one embodiment, the specific content of step S2-5 is:

[0093] (5.1) Static scene optimization: Ground detection and elimination are performed on static point clouds based on the RANSAC plane fitting algorithm. K-means clustering units are constructed for non-ground point clouds based on xy plane coordinate differences. The point cloud semantic labels P are combined with the label Statistical distribution probability, the semantic category with the highest probability (such as "vegetation" and "traffic sign") is assigned to the corresponding point cloud cluster.

[0094] (5.2) Dynamic scene fusion: The dynamic and static point clouds are segmented by spatiotemporal bounding boxes and divided into voxel grids. The occupancy probability of the semantic label in each voxel is calculated through a voting mechanism to obtain the ground truth of the occupied voxel.

[0095] Specifically, the occupancy probability of the semantic label in each voxel is calculated to obtain the i-th voxel V i The semantic label L with the highest probability i :

[0096]

[0097] Among them, L i,j For V i The jth point P in i,j The associated semantic label, δ(L i,j , l) is the Kronecker delta function, l is the semantic label of the distribution (in one embodiment, l is a number from 0 to 15, representing 16 semantic labels); when L i,j =1, the value is 1, otherwise it is 0. After voting, if Figure 3 As shown, high-quality occupied voxels can be obtained.

[0098] Step S3: training an occupancy prediction model based on the lidar point cloud data obtained in step S1 and the ground truth values ​​of occupied voxels obtained in step S2;

[0099] like Figure 4As shown, the occupancy prediction model includes a voxel query suggestion generation module based on LMSCNet, two encoders, a self-attention module, a deformable attention module and a segmentation head. In the occupancy prediction model: the voxel query suggestion generation module based on LMSCNet receives lidar point cloud data and generates sparse voxel query suggestions; the first encoder receives rasterized events and calculates the keys and values ​​of the events; the second encoder receives images and calculates the keys and values ​​of the images; the keys and values ​​of the events and images are added and fused respectively and then input into the self-attention module for feature enhancement to obtain fused features; the deformable attention module generates three-dimensional voxel features based on the sparse voxel query suggestions and fused features; the segmentation head converts the three-dimensional voxel features into three-dimensional occupancy prediction results.

[0100] In the specific implementation, the specific contents of the voxel query suggestion generation module in LMSCNet are:

[0101] 1.1) LiDAR point cloud data is input into LMSCNet. LMSCNet has a 3DUNet structure and extracts features of LiDAR point cloud data layer by layer through convolution operations at multiple scales. The feature map at each scale reflects contextual information at a different range. Furthermore, spatial details are gradually restored through multi-scale feature summation and upsampling operations, mapping the abstract features into a sparse voxel representation.

[0102] 1.2) Generate sparse voxel query suggestions Q based on sparse voxel representation LiDAR Query suggestions are indices of key voxels in the scene that contain important geometric information about the environment.

[0103] In specific implementation, the specific process of event rasterization is:

[0104] 2.1) Acquire event stream data, which includes all events within the exposure time of each frame. Each event is typically represented by four values: (x, y, polarity, timestamp), where x and y are the pixel coordinates of the event. Polarity represents the polarity of the event, typically +1 (positive event) or -1 (negative event). Timestamp is the timestamp of the event.

[0105] 2.2) Events are classified into positive events (polarity = +1) and negative events (polarity = -1) based on their polarity. At each pixel, the number of positive and negative events, as well as the timestamps of the last positive and negative events, are counted for a total of four channels. To eliminate uneven data distribution, 3σ normalization is performed on each channel.

[0106] In one embodiment, the first encoder and the second encoder both use ResNet-50 and feature pyramid. ResNet-50 extracts event and image features respectively, while taking into account the depth, width and input resolution of the network, effectively extracting shallow edge information and spatial semantic information. And effectively avoid phenomena such as gradient explosion and gradient disappearance. Combined with the feature pyramid, through multi-scale feature fusion, the model is more comprehensive in capturing local details and global context information, and enhances the model's perception of spatial context. The key vector and value vector of the event and image are obtained through the Transformer encoding layer. Taking the value vector as an example, the eigenvalue vector of the image modality is The eigenvalue vector of the event mode is expressed as After adding the value vectors, we get Apply the self-attention mechanism to calculate the weight After adaptive weighted fusion, the fusion feature value vector of event and image is obtained

[0107]

[0108] in is the weight.

[0109] Similarly, the fusion feature key vector of event and image can be obtained Where β is the weight.

[0110] The fused feature value vector / key vector is matched with the sparse voxel query suggestions obtained by LiDAR. A deformable attention mechanism is used to adjust the matching results, focusing on relevant spatial information, to generate fused 3D voxel features. After passing through the segmentation head, the final 3D occupancy prediction result is obtained.

[0111] During the model training process, the ground truth of occupied voxels is used as the three-dimensional occupancy prediction result for training.

[0112] Step S4: Obtaining the lidar point cloud data, image data, and event data within the image exposure time of the scene, and using the trained occupancy prediction model to achieve three-dimensional occupancy prediction;

[0113] like Figure 5 As shown in the figure, in a high dynamic range scene at night, this method can successfully complete the three-dimensional occupancy prediction task and can also smoothly perceive dynamic targets.

[0114] Corresponding to the aforementioned embodiment of the event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction method, the present application also provides an embodiment of an event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction device.

[0115] Figure 6FIG1 is a block diagram of an event camera-assisted multi-modal vehicle-mounted three-dimensional occupancy prediction device according to an exemplary embodiment. Figure 6 , the device may include

[0116] Dense point cloud acquisition module 21: acquires continuously collected lidar point cloud data, images, events, and IMU measurement data, synchronizes the spatial and temporal dimensions of various data, and obtains dense point cloud data based on the synchronized lidar point cloud data and IMU measurement data;

[0117] Occupancy voxel true value acquisition module 22: performs spatiotemporal alignment of dense point clouds, images, and events, semantically annotates the spatiotemporally aligned images, separates the spatiotemporally aligned dense point clouds into static and dynamic parts through cross-modal semantic projection, and performs static scene optimization and dynamic scene fusion to obtain the occupied voxel true value;

[0118] Model training module 23: training the occupancy prediction model based on the lidar point cloud data acquired by the dense point cloud acquisition module and the occupied voxel true value obtained by the occupied voxel true value acquisition module;

[0119] The three-dimensional occupancy prediction module 24 obtains the lidar point cloud data, image data and event data within the image exposure time of the scene, and uses the trained occupancy prediction model to realize three-dimensional occupancy prediction.

[0120] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0121] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0122] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned event camera-assisted multi-modal vehicle-mounted three-dimensional occupancy prediction method. Figure 7As shown in the figure, a hardware structure diagram of an event camera-assisted multi-modal vehicle-mounted three-dimensional occupancy prediction system provided by an embodiment of the present invention is provided in any device with data processing capabilities, except Figure 7 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0123] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the event camera-assisted multi-modal vehicle-mounted three-dimensional occupancy prediction method as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0124] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

Claims

1. An event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction method, characterized in that: include: Step S1: Acquire continuously collected lidar point cloud data, images, events, and IMU measurement data, synchronize the spatial and temporal dimensions of each type of data, and obtain dense point cloud data based on the synchronized lidar point cloud data and IMU measurement data; Step S2: Perform spatiotemporal alignment on the dense point cloud, image, and event, perform semantic annotation on the spatiotemporal aligned image, perform dynamic and static separation on the spatiotemporal aligned dense point cloud through cross-modal semantic projection, perform static scene optimization and dynamic scene fusion, and obtain the true value of occupied voxels; Step S3: training an occupancy prediction model based on the lidar point cloud data obtained in step S1 and the ground truth values ​​of occupied voxels obtained in step S2; Step S4: Obtain the lidar point cloud data, image data, and event data within the image exposure time of the scene, and use the trained occupancy prediction model to achieve three-dimensional occupancy prediction.

2. The method according to claim 1, characterized in that In step S1, the dense point cloud data is obtained through the following process: Using GNSS and IMU combined positioning, the extended Kalman filter fuses global position and inertial measurement data to directly output 6-DoF pose. If there is a GPS signal obstruction problem, positioning degradation compensation must be performed. Based on the SLAM framework, the GPS / IMU pose is combined with the laser point cloud to build a globally consistent high-precision map. At the same time, the accumulated error is corrected through loop detection to obtain dense point cloud data.

3. The method according to claim 1, characterized in that Step S2 includes: S2-1: For dense point clouds, images, and events, temporal alignment is performed based on the timestamps of the images, and spatial alignment is performed based on the field of view of the event camera. S2-2: Use the Grounded-SAM (Grounded-SAM) semantic segmentation model to perform pixel-level semantic annotation on the spatiotemporally aligned images to obtain two-dimensional semantic labels. S2-3: Based on the external parameters of the lidar and RGB camera, the dense point cloud is separated into static and dynamic maps through cross-modal semantic projection to obtain static and dynamic maps; S2-4: Spatiotemporal refinement of dynamic objects: Based on the dynamic map, perform instance annotation of dynamic target tails in the BEV view and perform spatiotemporal interpolation optimization; S2-5: Probabilistically guided voxel refinement, static scene optimization through point cloud clustering, dynamic scene fusion through voting mechanism and obtaining the ground truth of occupied voxels.

4. The method according to claim 3, characterized in that Step S2-1 includes Time alignment is performed based on the timestamp of the image: the timestamp t of each frame of image is img , find the timestamp t of the point cloud frame with the smallest time difference in the point cloud sequence pc , if |t img -t pc |≤t threshold , it is considered synchronized, otherwise the image frame is discarded, where t threshold is the timestamp difference threshold; Spatial alignment based on the field of view of the event camera: Image distortion correction and resampling are performed to ensure that the image content matches the field of view of the event camera.

5. The method according to claim 3, characterized in that Step S2-3 includes: Cross-modal semantic projection: Based on the external parameter matrix of the lidar and RGB camera, the dense point cloud data is cross-modally projected to the camera coordinate system, the dense point cloud data in the camera coordinate system is transformed into a semantic map through perspective transformation, and the corresponding two-dimensional semantic label I is obtained according to the pixel position label , thus obtaining the point cloud semantic label P label ; Dynamic and static separation strategy: According to the point cloud semantic label P label , dynamic targets are separated from static backgrounds in dense point cloud data, and static maps and dynamic maps are constructed using lidar mapping methods. In the dynamic map, the motion trajectory of the dynamic target is modeled as a three-dimensional dynamic motion pipeline.

6. The method according to claim 3, characterized in that Step S2-4 includes: BEV interactive annotation: projecting the three-dimensional dynamic motion pipeline into a two-dimensional bird's-eye view coordinate system for interactive calibration; Spatiotemporal interpolation optimization: Based on the trajectory vertex coordinates of the point cloud timestamp and the orientation angle of the 3D dynamic motion pipeline, linear interpolation is used to calculate the intermediate position coordinates and orientation angles lost between frames due to the lack of lidar scanning.

7. The method according to claim 3, characterized in that Step S2-5 includes Static scene optimization: Based on the RANSAC plane fitting algorithm, the ground detection and elimination of static point clouds are performed, and the k-means clustering unit is constructed based on the xy plane coordinate difference of non-ground point clouds, combined with the point cloud semantic label P label Statistical distribution probability, assigning the semantic category with the largest distribution probability to the corresponding point cloud cluster; Dynamic scene fusion: The dynamic and static point clouds are segmented by spatiotemporal bounding boxes, divided into voxel grids, and the occupancy probability of dynamic targets in each voxel is calculated through a voting mechanism; Among them, the i-th voxel V i The semantic label L that appears most frequently in i for: Among them, L i,j For V i The jth point P in i,j The associated semantic label, δ(L i,j ,l) is the Kronecker delta function, l is the semantic label of the distribution.

8. The method according to claim 1, characterized in that The occupancy prediction model in step S3 includes a LMSCNet-based voxel query suggestion generation module, two encoders, a self-attention module, a deformable attention module, and a segmentation head. The LMSCNet-based voxel query suggestion generation module receives lidar point cloud data and generates sparse voxel query suggestions. The first encoder receives the rasterized events and calculates the key and value of the events; The second encoder receives the image and calculates the key and value of the image; the key and value of the event and image are added and fused respectively, and then input into the self-attention module for feature enhancement to obtain the fused feature; the deformable attention module generates 3D voxel features based on the sparse voxel query suggestion and the fused feature; The segmentation head converts the 3D voxel features into 3D occupancy prediction results.

9. The method according to claim 8, characterized in that The specific process of event rasterization is: Get event stream data, which includes all events within the exposure time of each frame. Each event is represented by (x, y, polarity, timestamp), where x and y are the pixel coordinates of the event, polarity represents the polarity of the event, and timestamp is the timestamp of the event. According to the polarity of the event, the events are divided into positive events and negative events. At each pixel position, the number of positive events and negative events and the timestamp of the last positive event and negative event are counted, and each channel of each pixel is normalized separately.

10. An event camera-assisted multimodal vehicle-mounted three-dimensional occupancy prediction device, characterized in that: include: Dense point cloud acquisition module: acquires continuously collected lidar point cloud data, images, events, and IMU measurement data, synchronizes the spatial and temporal dimensions of various data, and obtains dense point cloud data based on the synchronized lidar point cloud data and IMU measurement data; Occupancy voxel truth acquisition module: This module performs spatiotemporal alignment of dense point clouds, images, and events, semantically annotates the aligned images, separates static and dynamic features of the aligned dense point clouds through cross-modal semantic projection, and performs static scene optimization and dynamic scene fusion to obtain the occupied voxel truth value. Model training module: trains the occupancy prediction model based on the lidar point cloud data obtained by the dense point cloud acquisition module and the occupied voxel true value obtained by the occupied voxel true value acquisition module; 3D occupancy prediction module: obtains the lidar point cloud data, image data and event data within the image exposure time of the scene, and uses the trained occupancy prediction model to realize 3D occupancy prediction.

Images