Fusion perception method and device of event camera and lidar, and electronic equipment

Abstract

The application relates to the technical field of computer vision, and particularly discloses a fusion perception method and device of an event camera and a laser radar and electronic equipment. The fusion perception device of the event camera and the laser radar acquires event stream data collected by the event camera and point cloud data collected by the laser radar; the event stream data and the point cloud data are subjected to spatiotemporal coupling calibration processing to obtain calibrated data; wherein the spatiotemporal coupling calibration processing is used for realizing joint calibration of the event stream data and the point cloud data in the time dimension and the space dimension; first features corresponding to the event stream data and second features corresponding to the point cloud data in the calibrated data are extracted; the first features and the second features are subjected to feature association to obtain a feature association pair; feature fusion is performed based on the feature association pair to obtain fused features; target perception processing is performed based on the fused features to obtain a perception result, so that the precision and effect of environmental perception can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a fusion perception method, device, and electronic device of an event camera and a lidar. Background Technology

[0002] Currently, scenarios such as autonomous driving and robot navigation place extremely high demands on the accuracy, real-time performance, and robustness of environmental perception. Existing perception solutions largely rely on single sensors or loosely fused multi-sensor data: for single-sensor solutions, while LiDAR can provide high-precision 3D distance information, its low sampling frequency leads to delayed response to dynamic targets, and point cloud noise increases in adverse weather conditions; event cameras, with their microsecond-level temporal resolution and high dynamic range, can quickly capture target motion but lack distance information, resulting in sparse event stream data; while multi-sensor fusion solutions are mostly a loose "independent processing + post-fusion" model, suffering from insufficient spatiotemporal calibration accuracy, inadequate data complementarity mining, and poor real-time performance, making it difficult to meet the perception needs of complex scenarios. Therefore, how to achieve a perception method that enables high-precision joint calibration and deep fusion of multi-sensor data to fully leverage the performance advantages of different sensors and improve the overall performance of the perception system is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a fusion sensing method, apparatus, and electronic device for event cameras and LiDAR, which can fully leverage the performance advantages of event cameras and LiDAR to effectively improve the accuracy and effectiveness of environmental perception.

[0004] To achieve the above objectives, this application provides the following technical solution:

[0005] In a first aspect, embodiments of this application provide a fusion sensing method using an event camera and a LiDAR, comprising:

[0006] Acquire event stream data collected by the event camera and point cloud data collected by LiDAR;

[0007] Spatiotemporal coupling calibration processing is performed on event stream data and point cloud data to obtain calibrated data; wherein, the spatiotemporal coupling calibration processing is used to achieve joint calibration of event stream data and point cloud data in the time and spatial dimensions;

[0008] Extract the first feature corresponding to the event stream data and the second feature corresponding to the point cloud data from the calibration data;

[0009] Perform feature association on the first feature and the second feature to obtain feature association pairs;

[0010] Feature fusion is performed based on feature association pairs to obtain fused features;

[0011] The target perception process is performed based on the fused features to obtain the perception result.

[0012] In some embodiments of this application, spatiotemporal coupling calibration processing is performed on event stream data and point cloud data to obtain calibrated data, including:

[0013] Synchronize the event stream data and point cloud data in time to obtain time-aligned event stream data and point cloud data;

[0014] Spatial registration is performed on time-aligned event stream data and point cloud data to obtain calibrated data.

[0015] In some embodiments of this application, time synchronization is performed on event stream data and point cloud data to obtain time-aligned event stream data and point cloud data, including:

[0016] Extract the event timestamp sequence from the event stream data and the point cloud timestamp sequence from the point cloud data;

[0017] Determine the cumulative distance matrix based on event timestamp sequences and point cloud timestamp sequences;

[0018] Based on the minimum path of the cumulative distance matrix, time-aligned event stream data and point cloud data are determined; wherein, time-aligned event stream data and point cloud data are used to indicate the event stream time window corresponding to each point cloud frame in the point cloud data.

[0019] In some embodiments of this application, spatial registration is performed on time-aligned event stream data and point cloud data to obtain calibrated data, including:

[0020] The time-aligned event stream data is transformed to the 3D camera coordinate system based on the extrinsic parameter matrix to obtain the first transformed event stream data;

[0021] The first transformation event stream data is transformed into the lidar coordinate system to obtain the calibration data.

[0022] In some embodiments of this application, extracting a first feature corresponding to event stream data and a second feature corresponding to point cloud data from the calibration data includes:

[0023] The event streams in the calibrated data are accumulated into event images, and dynamic features in the event images are extracted using a convolutional neural network to obtain the first feature;

[0024] The point cloud in the calibration data is divided into columnar sections, and the three-dimensional structural features of the columnar sections are extracted to obtain the second feature.

[0025] In some embodiments of this application, feature association is performed on the first feature and the second feature to obtain feature association pairs, including:

[0026] Determine the similarity between each pixel feature in the first feature and each column feature in the second feature;

[0027] Pixel features and columnar features with similarity exceeding a preset similarity threshold are identified as feature association pairs.

[0028] In some embodiments of this application, feature fusion is performed based on feature association pairs to obtain fused features, including:

[0029] Feature weights are determined based on the similarity of feature association pairs;

[0030] The feature association pairs and feature weights are weighted and fused to obtain the fused features.

[0031] In some embodiments of this application, target perception processing is performed based on the fused features to obtain perception results, including:

[0032] The target detection algorithm is used to identify the target based on the fused features, and the target identification result is obtained. The target identification result includes the target category, three-dimensional coordinates and target size.

[0033] Target tracking is performed based on a time-series tracking algorithm and target recognition results to obtain tracking results, which include target velocity and target acceleration.

[0034] The perception result is determined based on the target recognition and tracking results.

[0035] Secondly, embodiments of this application provide a fusion sensing device for an event camera and a lidar, including a processor and a memory storing executable instructions of the processor; when the instructions are executed by the processor, the aforementioned fusion sensing method for an event camera and a lidar is implemented.

[0036] Thirdly, embodiments of this application provide an electronic device, including an event camera, a lidar, and a fusion sensing device for the event camera and lidar; the fusion sensing device for the event camera and lidar is used to execute a fusion sensing method for the event camera and lidar. Attached Figure Description

[0037] To more intuitively illustrate the prior art and this application, several exemplary figures are provided below. It should be understood that the specific shapes and structures shown in the figures should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary figures, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0038] Figure 1A schematic diagram illustrating the implementation process of the event camera and lidar fusion sensing method provided in the embodiments of this application;

[0039] Figure 2 A schematic diagram of a spatiotemporally tightly coupled fusion sensing system of an event camera and lidar provided in an embodiment of this application;

[0040] Figure 3 A schematic diagram illustrating the implementation logic of the event camera and lidar fusion sensing method provided in this application embodiment;

[0041] Figure 4 A schematic diagram of the composition structure of the event camera and lidar fusion sensing device provided in the embodiments of this application;

[0042] Figure 5 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application.

[0043] Figure label:

[0044] The event camera and lidar fusion sensing device 1, processor 11, memory 12, communication interface 13, bus 14;

[0045] Electronic devices 0, event camera 2, LiDAR 3. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Any combination of different embodiments is possible.

[0047] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0048] In intelligent systems such as autonomous driving and robot navigation, environmental perception is a key core technology, and its performance directly determines the safety and reliability of the system. Current mainstream environmental perception solutions mainly rely on single sensors or simple loose fusion of multiple sensors, which presents significant technical bottlenecks.

[0049] In existing technologies, lidar, with its high-precision distance measurement capabilities and 3D spatial modeling advantages, can accurately acquire information such as the position and size of obstacles in the environment. However, it suffers from problems such as low data sampling frequency (typically 10~20Hz), delayed response to rapid changes in dynamic targets, and increased point cloud noise under strong light or inclement weather. While using a single lidar for environmental perception improves data accuracy through point cloud filtering algorithms, it cannot solve the problem of insufficient rapid capture of dynamic targets, and its robustness decreases under complex lighting conditions.

[0050] Event cameras, as a novel type of visual sensor, output asynchronous event streams only when pixel brightness changes significantly. They offer advantages such as microsecond-level temporal resolution, high dynamic range (140dB), and low power consumption, enabling rapid response to the motion state of dynamic targets. However, they lack distance information, and the event stream data is sparse, making it difficult to perform 3D environment modeling and precise obstacle localization when used alone. Current proposed dynamic target detection methods based on event cameras, while capable of quickly capturing target motion, cannot acquire target distance and 3D contour information, failing to meet the comprehensive environmental perception requirements of intelligent systems.

[0051] To address the limitations of single-sensor systems, existing multi-sensor fusion solutions often employ a loosely coupled approach. This involves processing LiDAR point clouds and event camera event streams independently before performing data-level or decision-level fusion. Specifically, LiDAR first acquires 3D point clouds, while event cameras capture dynamic events, followed by decision-level fusion based on Kalman filtering. However, this approach suffers from several issues: 1) Low spatiotemporal calibration accuracy: Time synchronization between LiDAR and event cameras relies on external hardware triggers or simple timestamp alignment. Spatial calibration is based solely on a fixed extrinsic parameter matrix, failing to consider spatiotemporal shifts caused by sensor installation errors and environmental disturbances. 2) Insufficient data complementarity mining: The dynamic response advantage of event streams and the 3D distance advantage of LiDAR are not fully utilized, resulting in high redundancy and low effective information utilization after fusion. 3) Poor real-time performance: The independent processing followed by fusion increases data processing latency, making it difficult to meet the millisecond-level perception response requirements of scenarios such as autonomous driving.

[0052] In summary, existing technologies generally suffer from insufficient spatiotemporal alignment accuracy, excessively high fusion levels, and difficulty in fully utilizing the high temporal resolution advantage of event streams. There is an urgent need for a fusion perception method that can achieve tightly coupled modeling and joint optimization at both the temporal and spatial levels.

[0053] This application provides a fusion sensing method using an event camera and a LiDAR, such as... Figure 1 As shown, the fusion sensing method of event camera and lidar of electronic device may include the following steps:

[0054] Step 101: Acquire event stream data collected by the event camera and point cloud data collected by the LiDAR.

[0055] In the embodiments of this application, the event camera and lidar fusion sensing device can first acquire event stream data collected by the event camera and point cloud data collected by the lidar.

[0056] In the embodiments of this application, the event camera can be used to collect asynchronous event stream data of brightness changes in the environment in real time. Each event may include pixel coordinates (x, y), microsecond-level timestamp t, and brightness change polarity p (+1 indicates brightening, -1 indicates darkening).

[0057] In the embodiments of this application, the lidar can collect three-dimensional point cloud data at a certain sampling frequency. Each point cloud can include three-dimensional coordinates (x, y, z), reflection intensity I, and timestamp t.

[0058] In the embodiments of this application, the event camera and lidar fusion sensing device can receive event stream data and point cloud data through a data bus.

[0059] Step 102: Perform spatiotemporal coupling calibration on the event stream data and point cloud data to obtain calibrated data; wherein, the spatiotemporal coupling calibration is used to achieve joint calibration of event stream data and point cloud data in the time and spatial dimensions.

[0060] In the embodiments of this application, the event camera and lidar fusion sensing device can perform spatiotemporal coupling calibration processing on the event stream data collected by the event camera and the point cloud data collected by the lidar after acquiring the event stream data and the point cloud data to obtain calibrated data; wherein, the spatiotemporal coupling calibration processing is used to realize the joint calibration of the event stream data and the point cloud data in the time dimension and the spatial dimension.

[0061] In the embodiments of this application, the calibration of the time dimension can adopt the Dynamic Time Warping (DTW) algorithm to achieve sub-microsecond time alignment; the calibration of the spatial dimension is based on the optimal extrinsic parameter matrix, which transforms the pixel coordinates of the event stream data to the lidar coordinate system to achieve coordinate system unification.

[0062] In some embodiments of this application, the event camera and lidar fusion sensing device can perform spatiotemporal coupling calibration processing on event stream data and point cloud data to obtain calibrated data, and then perform time synchronization on the event stream data and point cloud data to obtain time-aligned event stream data and point cloud data; and perform spatial registration on the time-aligned event stream data and point cloud data to obtain calibrated data.

[0063] In some embodiments of this application, time synchronization can be achieved based on a dynamic time warping algorithm, which establishes a nonlinear time mapping relationship by calculating the minimum cumulative distance between the event stream and the point cloud timestamp sequence.

[0064] In some embodiments of this application, spatial registration can be achieved based on a preset optimal extrinsic parameter matrix (including rotation matrix R + translation vector T) to transform the time-aligned event stream pixel coordinates to the lidar coordinate system.

[0065] In some embodiments of this application, when the event camera and lidar fusion sensing device performs time synchronization on event stream data and point cloud data to obtain time-aligned event stream data and point cloud data, it can extract the event timestamp sequence of the event stream data and the point cloud timestamp sequence of the point cloud data; determine the cumulative distance matrix based on the event timestamp sequence and the point cloud timestamp sequence; and determine the time-aligned event stream data and point cloud data based on the minimum path of the cumulative distance matrix; wherein, the time-aligned event stream data and point cloud data are used to indicate the event stream time window corresponding to each point cloud frame in the point cloud data.

[0066] For example, an event timestamp sequence of length n is represented as: A point cloud timestamp sequence of length m is represented as The DTW algorithm is used to perform nonlinear matching between the event timestamp sequence and the point cloud timestamp sequence to determine the cumulative distance matrix between the two timestamp sequences.

[0067] In the embodiments of this application, the event timestamp sequence is a timestamp sequence of all events in the event stream data arranged in chronological order of occurrence, such as the microsecond-level timestamps of each of events 1 to n; the point cloud timestamp sequence is a timestamp sequence of each point cloud frame output by the lidar arranged in chronological order of acquisition, such as the timestamps of each of point cloud frames 1 to m.

[0068] In some embodiments of this application, when determining the cumulative distance matrix based on the event timestamp sequence and the point cloud timestamp sequence, a distance matrix between the event timestamp sequence and the point cloud timestamp sequence can be constructed first, and then the cumulative distance matrix can be determined based on the distance matrix; wherein, the distance matrix represents the Euclidean distance between the timestamps in the event timestamp sequence and the timestamps in the point cloud timestamp sequence.

[0069] In some embodiments of this application, the calculation of the cumulative distance matrix starts from the starting point of the distance matrix (i.e., the position corresponding to the first timestamp in the event timestamp sequence and the first timestamp in the point cloud timestamp sequence). The value of each position consists of two parts: one is the distance matrix value corresponding to the current position, and the other is the smallest cumulative distance matrix value among the three adjacent positions (above, left, and upper left). The sum of these two parts is the cumulative distance matrix value of the current position. Through this step-by-step recursive method, the complete cumulative distance matrix is ​​obtained.

[0070] In some embodiments of this application, the minimum path refers to tracing back from the last position of the cumulative distance matrix (i.e., the position corresponding to the last timestamp of the event and the last timestamp of the point cloud), selecting the path with the smallest cumulative distance value at each step. The final path obtained is the "optimal matching relationship" between the event timestamp and the point cloud timestamp. Through this optimal matching relationship, the event stream time window corresponding to each point cloud frame can be determined. For example, taking the timestamp of each point cloud frame as the center, taking the interval 5 milliseconds before and 5 milliseconds after that timestamp, all events within this interval are the event stream data aligned with the time of that point cloud frame.

[0071] For example, the elements in the cumulative distance matrix may include the minimum cumulative distance from the starting point (1,1) to the current point (i,j), and its recursive formula can be expressed as the following formula:

[0072] (1);

[0073] in, This represents the minimum cumulative distance from the first timestamp of the event and the first timestamp of the point cloud (i.e., the starting point of the cumulative distance matrix) to the first timestamp of the event and the j-th timestamp of the point cloud. Let the distance matrix be the Euclidean distance between two timestamps, where the element in the i-th row and j-th column is the Euclidean distance between them. This can be represented as: , It is a timestamp in the event timestamp sequence. It is a timestamp in the point cloud timestamp sequence. This represents the smallest value among three adjacent elements in the cumulative distance matrix; This represents the element in the (i-1)th row and j-th column of the cumulative distance matrix, which is the minimum cumulative distance between the (i-1)th timestamp of the event and the j-th timestamp of the point cloud. This represents the element in the i-th row and j-1-th column of the cumulative distance matrix; This represents the element in the (i-1)th row and (j-1)th column of the cumulative distance matrix. By tracing back the path to the minimum value in the cumulative distance matrix, the optimal matching relationship between the event timestamp sequence and the point cloud timestamp sequence can be obtained, thereby determining the event stream time window corresponding to each point cloud frame, that is, obtaining a segment of event stream data that matches its time for each point cloud frame.

[0074] For example, the event stream time window can be represented as , =5ms.

[0075] In some embodiments of this application, when the event camera and lidar fusion sensing device performs spatial registration on time-aligned event stream data and point cloud data to obtain calibrated data, it can transform the time-aligned event stream data to the three-dimensional camera coordinate system based on the extrinsic parameter matrix to obtain first transformed event stream data; and transform the first transformed event stream data to the lidar coordinate system to obtain calibrated data.

[0076] In some embodiments of this application, since the time-aligned event stream data was originally two-dimensional pixel coordinates output by the event camera (containing only the x and y positions of the pixel plane), it cannot be directly matched with the three-dimensional point cloud data space. Therefore, this application uses the camera intrinsic parameter matrix and extrinsic parameter matrix to transform the time-aligned event stream data to the three-dimensional camera coordinate system to obtain the first transformed event stream data.

[0077] For example, the two-dimensional pixel coordinates of the event are first converted into ray directions in the camera coordinate system using the camera intrinsic parameter matrix; then, the three-dimensional camera coordinates (containing the three spatial dimensions X, Y, and Z) corresponding to the event are calculated by combining the parameters related to camera depth in the extrinsic parameter matrix. The extrinsic parameter matrix is ​​a pre-stored optimal extrinsic parameter matrix (R, T).

[0078] For example, the conversion can be expressed as the following formulas (2) and (3):

[0079] (2);

[0080] in, This represents the depth value from the target point to the camera's imaging plane in the 3D camera coordinate system. It is a homogeneous coordinate representation of two-dimensional pixel coordinates; It is the camera intrinsic parameter matrix. These are the spatial coordinates of the target point in the 3D camera coordinate system; camera intrinsic parameter matrix. The standard form can be expressed as , This represents the equivalent focal length of the camera in the x-direction of the pixel plane. This represents the equivalent focal length of the camera along the y-axis in the pixel plane. and This represents the coordinates of the principal point on the camera's imaging plane.

[0081] (3);

[0082] in, The rotation matrix in the extrinsic parameter matrix is ​​a 3×3 orthogonal matrix used to describe the relative attitude of the camera and the LiDAR. This represents the translation vector, which is a 3×1 vector used to describe the relative position of the camera and the LiDAR. This represents the three-dimensional coordinates of the target point in the lidar coordinate system.

[0083] For example, the extrinsic parameter matrix is ​​obtained by calibration using a calibration plate, wherein the rotation matrix R is a 3×3 orthogonal matrix and the translation vector T is a 3×1 vector; finally, after converting the first transformed event stream data to the lidar coordinate system, the transformed event stream data in the same coordinate system as the point cloud data can be obtained.

[0084] In the embodiments of this application, spatial registration of event stream data is achieved through a two-step coordinate transformation: first, the data is transformed to the 3D camera coordinate system, and then mapped to the lidar coordinate system. The transformation logic is rigorous. Based on the optimal extrinsic parameter matrix obtained by calibration, the spatial transformation error is ensured to be less than 0.5 pixels, achieving high-precision spatial alignment between event stream and point cloud data, and providing spatially consistent data support for feature association and fusion.

[0085] Step 103: Extract the first feature corresponding to the event stream data and the second feature corresponding to the point cloud data from the calibration data.

[0086] In the embodiments of this application, the event camera and lidar fusion sensing device can perform spatiotemporal coupling calibration processing on event stream data and point cloud data, and after obtaining calibrated data, extract the first feature corresponding to the event stream data and the second feature corresponding to the point cloud data from the calibrated data.

[0087] In embodiments of this application, the first feature may include dynamic features corresponding to event stream data, used to characterize temporal information such as the target's motion direction and speed; the second feature may include three-dimensional structural features corresponding to point cloud data, used to characterize spatial information such as the target's position, size, and contour.

[0088] In some embodiments of this application, a convolutional neural network (CNN) can be used to extract the first feature, and a point pillar algorithm can be used to extract the second feature. The two types of features can have the same dimension, such as 256 dimensions, to ensure compatibility of subsequent association and fusion.

[0089] In some embodiments of this application, when the event camera and lidar fusion sensing device extracts the first feature corresponding to the event stream data and the second feature corresponding to the point cloud data in the calibration data, it can accumulate the event stream in the calibration data into an event image, and use a convolutional neural network to extract the dynamic features in the event image to obtain the first feature; divide the point cloud in the calibration data into columns, and extract the three-dimensional structural features of the columns to obtain the second feature; wherein, the second feature is used to characterize spatial information.

[0090] In some embodiments of this application, the event stream accumulation can adopt the time window accumulation method, which accumulates the events within the 5ms event stream time window corresponding to each point cloud frame into an event image according to pixel coordinates and polarity. The pixel value is determined by the event polarity and quantity (for example, positive events accumulate by 1, and negative events accumulate by 1).

[0091] In some embodiments of this application, the convolutional neural network can be a lightweight structure including 3 convolutional layers and 2 pooling layers, taking an event image with a resolution of 320×240 as input and outputting 256-dimensional dynamic features. These features can be used to capture temporal information such as the direction of motion and speed of a target, such as the motion trend features of a vehicle changing lanes.

[0092] In some embodiments of this application, the size of the columnar division can be set, for example, the size can be set to 0.2m×0.2m×4m (x-axis×y-axis×z-axis), dividing the point cloud data in the lidar coordinate system into multiple non-overlapping columnar divisions, each columnar division containing several point clouds.

[0093] In some embodiments of this application, the three-dimensional structural features of each column can be extracted by a column feature network, including the number of point clouds, the mean of reflection intensity, the variance of coordinate distribution, etc., to obtain a 256-dimensional second feature. The second feature can be used to characterize the spatial information of the target, such as its position, size, and contour.

[0094] For example, assuming that the fusion perception device of event camera and LiDAR is used for perception in the autonomous driving scenario of the vehicle, the extracted second feature may include the height of the pedestrian and the length, width and height features of the vehicle.

[0095] In the embodiments of this application, differentiated feature extraction schemes are designed for the characteristics of the two types of data. Event streams are accumulated to adapt convolutional neural networks to event images to efficiently extract dynamic features; point clouds are precisely extracted in three-dimensional structure through columnar partitioning and feature networks. The two types of features have the same dimension and complementary information, providing high-quality input for subsequent feature association and fusion, and ensuring the effectiveness of fused features.

[0096] Step 104: Perform feature association on the first feature and the second feature to obtain feature association pairs.

[0097] In the embodiments of this application, the event camera and lidar fusion sensing device can extract the first feature corresponding to the event stream data and the second feature corresponding to the point cloud data in the calibration data, and then perform feature association on the first feature and the second feature to obtain feature association pairs.

[0098] In embodiments of this application, feature association pairs can be used to indicate feature pairs that have a high degree of correlation between the first feature and the second feature.

[0099] In some embodiments of this application, when the event camera and lidar fusion sensing device performs feature association on the first feature and the second feature to obtain feature association pairs, it can determine the similarity of each pixel feature in the first feature and each columnar feature in the second feature; and determine the pixel features and columnar features with similarity exceeding a preset similarity threshold as feature association pairs.

[0100] In some embodiments of this application, a similarity calculation model can be constructed based on an attention mechanism. Each pixel feature in the first feature and each column feature in the second feature are input into the model, and the cosine similarity is calculated through vector dot product.

[0101] In some embodiments of this application, after determining the similarity between each pixel feature in the first feature and each columnar feature in the second feature, a similarity matrix can be obtained, which includes the similarity between every two features.

[0102] In some embodiments of this application, the preset similarity threshold can be set to 0.7. This threshold is determined based on a large amount of calibration data and can balance the accuracy of association with the number of effective association pairs.

[0103] In some embodiments of this application, pixel features with a similarity greater than 0.7 are combined with columnar features to form feature association pairs; for example, “vehicle motion pixel features and vehicle three-dimensional structure features” and “pedestrian motion pixel features and pedestrian three-dimensional structure features” can ensure that each association pair corresponds to complementary information of the same target.

[0104] In the embodiments of this application, the similarity between pixel features and columnar features is calculated through an attention mechanism to achieve refined feature association; effective association pairs are filtered by a preset similarity threshold, and feature combinations that are unrelated or weakly related are eliminated to avoid invalid information interfering with the fusion effect, and the complementary relationship between event flow dynamic information and LiDAR spatial information is fully explored, laying the foundation for high-quality feature fusion.

[0105] Step 105: Perform feature fusion based on feature association pairs to obtain fused features.

[0106] In the embodiments of this application, the event camera and lidar fusion sensing device can perform feature association on the first feature and the second feature to obtain feature association pairs, and then perform feature fusion based on the feature association pairs to obtain fused features.

[0107] In some embodiments of this application, when the event camera and lidar fusion sensing device performs feature fusion based on feature association pairs to obtain fused features, it can determine feature weights based on the similarity of feature association pairs; and perform weighted fusion on feature association pairs and feature weights to obtain fused features.

[0108] In embodiments of this application, the similarity of feature association pairs may include motion similarity and positional similarity; feature weights may include weights of dynamic features and weights of three-dimensional structural features; feature association pairs may include dynamic feature vectors and three-dimensional structural feature vectors.

[0109] In the embodiments of this application, motion similarity characterizes the degree of matching between the dynamic features of the event flow and the actual motion state of the target. The dynamic features are mainly used to capture the temporal information such as the target's motion direction and speed. If the motion similarity is higher, it means that the dynamic features can more accurately reflect the target's true motion trend and have greater value in supplementing the dynamic information of the fused features. Therefore, the weight of the dynamic features is positively correlated with the motion similarity. Position similarity characterizes the degree of fit between the three-dimensional structural features of the lidar and the actual spatial position of the target. The three-dimensional structural features are mainly used to provide spatial information such as the target's position and size. If the position similarity is higher, it means that the three-dimensional structural features can more accurately anchor the target's spatial contour and have a stronger supporting effect on the spatial information of the fused features. Therefore, the weight of the three-dimensional structural features is positively correlated with the position similarity.

[0110] In some embodiments of this application, when determining feature weights based on the similarity of feature association pairs, the similarity of feature association pairs can first be decomposed into motion similarity and position similarity. The motion similarity is multiplied by a preset motion adaptation coefficient to obtain the weighted score of the motion dimension; the position similarity is multiplied by a preset position adaptation coefficient to obtain the weighted score of the position dimension; the sum of the weighted scores of the motion dimension and the position dimension is determined; the weight of the dynamic feature is obtained by dividing the weighted score of the motion dimension by the sum of the weighted scores; and the weight of the three-dimensional structural feature is obtained by dividing the weighted score of the position dimension by the sum of the weighted scores.

[0111] In some embodiments of this application, weighted fusion can employ a vector weighted summation algorithm; the product of the dynamic feature vector and the weights of the dynamic feature can be added to the product of the three-dimensional structural feature vector and the weights of the three-dimensional structural feature to obtain the fused features.

[0112] Step 106: Perform target perception processing based on the fused features to obtain the perception result.

[0113] In the embodiments of this application, the event camera and lidar fusion sensing device can perform feature fusion based on feature association pairs to obtain fused features, and then perform target perception processing based on the fused features to obtain the perception result.

[0114] In the embodiments of this application, high-precision joint alignment of two types of data is achieved through spatiotemporal coupling calibration, avoiding the accumulation of errors from single-dimensional calibration; through feature association and fusion, the dynamic advantages of event flow and the three-dimensional structural advantages of LiDAR are fully explored to make up for the information defects of a single sensor; the overall process logic is coherent, forming end-to-end processing from data acquisition to perception output, which greatly improves the accuracy, real-time performance and robustness of perception, and adapts to the needs of complex scenarios such as autonomous driving.

[0115] In some embodiments of this application, when the event camera and lidar fusion sensing device performs target perception processing based on the fused features to obtain the perception result, it can use a target detection algorithm to perform target recognition on the fused features to obtain the target recognition result; wherein, the target recognition result includes the target category, three-dimensional coordinates, and target size; target tracking is performed based on the temporal tracking algorithm and the target recognition result to obtain the tracking result; wherein, the tracking result includes the target velocity and the target acceleration; and the perception result is determined based on the target recognition result and the tracking result.

[0116] In the embodiments of this application, the target detection algorithm can use the YOLOv8 lightweight network to adapt to the real-time perception requirements, input 256-dimensional fusion features, perform multi-scale feature fusion through a feature pyramid network, and output the target recognition result.

[0117] For example, in the context of environmental perception in autonomous driving, the target category may include common road obstacles such as vehicles, pedestrians, and cyclists; the three-dimensional coordinates may be coordinates in the LiDAR coordinate system; the target size may include length, width, and height, such as the three-dimensional coordinates of a vehicle (5.3m, 1.2m, 0.9m) and its size (4.8m×1.8m×1.5m).

[0118] In the embodiments of this application, the time-series tracking algorithm can employ the Kalman filter algorithm to predict the target's position at the next moment based on the three-dimensional coordinates in the target recognition result, and update the target state by combining the recognition results of subsequent frames.

[0119] In the embodiments of this application, the target velocity in the tracking result can be a three-dimensional velocity vector, and the acceleration can be a three-dimensional acceleration vector, so as to accurately reflect the target motion state.

[0120] In some embodiments of this application, the category, three-dimensional coordinates, and size of the target recognition result can be integrated with the speed and acceleration of the tracking result, while supplementing road contour information (based on point cloud data extraction) to form a complete perception result.

[0121] In some embodiments of this application, the perception results can be output in a standardized format, adapted to a CAN bus or Ethernet interface, and can be directly transmitted to the autonomous driving decision control system to provide data support for path planning and obstacle avoidance decisions.

[0122] In the embodiments of this application, the YOLOv8 algorithm is used to achieve efficient and accurate target recognition, and the Kalman filter algorithm is used to ensure stable tracking of dynamic targets. The combination of the two makes the perception results include both the static attributes of the target and the dynamic motion state. The integrated perception results are comprehensive and in a standard format, which can directly serve the downstream decision-making system and meet the perception requirements of real-time performance and accuracy in autonomous driving scenarios.

[0123] Based on the above embodiments, in another embodiment of this application, exemplarily, as follows: Figure 2 As shown, a spatiotemporally tightly coupled fusion sensing system of an event camera and a lidar is provided. The system includes an event camera module, a lidar module, a spatiotemporal calibration module, a tightly coupled fusion module, a sensing output module, and a data bus; each module realizes bidirectional data transmission through the data bus.

[0124] The event camera module includes an image sensor, a data acquisition unit, and a timestamp encoding unit. The structural parameters of the event camera module are: resolution 640×480 pixels, dynamic range 140dB, event trigger threshold ±0.5% brightness change, and timestamp accuracy ±1μs. The event camera module's function is to acquire asynchronous event stream data of brightness changes in the environment in real time. Each event includes Boakie pixel coordinates (x, y), timestamp t, and polarity p (brightness increase / decrease), and is output to the spatiotemporal calibration module via a data bus.

[0125] The lidar module can be a 128-line mechanical rotating lidar, comprising a laser emitting unit, an echo receiving unit, a distance calculation unit, and a timestamp encoding unit. The structural parameters of the lidar module include a detection range of 0.1~200m, a ranging accuracy of ±2cm, a sampling frequency of 15Hz, a horizontal field of view of 360°, a vertical field of view of -25°~+15°, and a timestamp accuracy of ±10μs. The lidar module's function is to acquire real-time environmental 3D point cloud data. Each point cloud includes 3D coordinates (x, y, z), reflection intensity I, and a timestamp t, and outputs this data to the spatiotemporal calibration module via a data bus.

[0126] The spatiotemporal calibration module includes a time synchronization unit, a spatial registration unit, and a calibration data storage unit. The time synchronization unit is connected to the event camera module and the LiDAR module via a data bus to receive event streams and point clouds. The spatial registration unit is bidirectionally connected to the calibration data storage unit to read pre-stored calibration data. The outputs of both the time synchronization unit and the spatial registration unit are connected to a tightly coupled fusion module. The time synchronization unit uses a dynamic time warping algorithm to achieve sub-microsecond time alignment by calculating the minimum distance between the event stream timestamp sequence and the point cloud timestamp sequence. The spatial registration unit uses an iterative optimization algorithm to solve for the optimal extrinsic parameter matrix (rotation matrix R, translation vector T) based on multiple sets of calibration data collected from the calibration board (calibration board corner points captured by the event camera and calibration board 3D point clouds scanned by the LiDAR), achieving accurate spatial coordinate transformation. The calibration data storage unit uses high-speed flash memory to store the initial values ​​of extrinsic parameters generated during the calibration process, intermediate results of iterative optimization, and final calibration parameters.

[0127] For example, such as Figure 3 As shown, the external input data includes calibration data, event streams, and point clouds collected based on the calibration board; both the event streams and point clouds have timestamps; the extrinsic parameter matrix can be determined based on the collected calibration data, and subsequent spatial registration can be performed based on the extrinsic parameter matrix; the time synchronization unit can synchronize the event streams and point clouds through a dynamic time warping algorithm to output aligned data, and then perform spatial registration in combination with the extrinsic parameter matrix, followed by tight coupling fusion of features, and finally, perception is performed based on the fused features to output the final perception result.

[0128] The tightly coupled fusion module is based on a heterogeneous computing architecture of Field Programmable Gate Array (FPGA) and Graphics Processing Unit (GPU), and includes an event stream feature extraction unit, a point cloud feature extraction unit, a feature association matching unit, and a fusion feature reconstruction unit. The input end of the tightly coupled fusion module is connected to the spatiotemporal calibration module to receive the calibrated event stream and point cloud, and the output end is connected to the perception output module. The event stream feature extraction unit employs three convolutional layers and two pooling layers from a convolutional neural network (CNN) to extract dynamic feature maps (including target motion direction, velocity, and other information) from the calibrated input event stream. The resulting feature map has a resolution of 320×240. The point cloud feature extraction unit uses the PointPillars algorithm to divide the point cloud into pillars and extracts three-dimensional structural features (including target position, size, contour, and other information) through a pillar feature network. The feature association matching unit, based on an attention mechanism, calculates the similarity matrix between the dynamic feature map and the three-dimensional structural features to achieve pixel-level feature association and explore the complementary relationship between the dynamic information of the event stream and the distance information of the point cloud. The fusion feature reconstruction unit uses a weighted fusion algorithm to assign feature weights based on association similarity and reconstruct fusion features containing spatiotemporal information, three-dimensional structural information, and dynamic information.

[0129] The perception output module includes a result processing unit and a data interface unit. The result processing unit is used to perform target detection (using the YOLOv8 algorithm) and tracking (using the Kalman filter algorithm) on the fused features, and outputs the three-dimensional coordinates, velocity, acceleration, size, category and road contour information of the obstacles. The data interface unit outputs the perception results to the autonomous driving decision control system through a CAN bus or Ethernet interface.

[0130] The data bus can use a PCIe 4.0 bus with a transmission bandwidth of 16GB / s. The data bus is used to realize low-latency data transmission between modules, ensure the real-time transmission of event streams and point cloud data, and enable bidirectional interaction of calibration parameters, fusion features, and perception results.

[0131] In some embodiments of this application, the event camera module and the lidar module can be fixedly installed at the top center of the autonomous vehicle, with the distance between them controlled within 20cm, the optical axis orientation aligned (parallel to the vehicle's driving direction), and the horizontal error of the installation plane not exceeding ±0.5°, to ensure that the collected environmental data covers the same field of view.

[0132] For example, a 9×6 checkerboard calibration board (square size 5cm×5cm) is used, and the calibration board is fixed at different positions (at least 5 positions) 1~5m away from the sensor to ensure that the calibration board appears simultaneously in the field of view of the event camera and the scanning range of the LiDAR; the event camera and LiDAR are controlled to simultaneously collect event stream and point cloud data at each calibration board position and store them in the calibration data storage unit; the spatial registration unit reads the calibrated data, uses the corner points of the calibration board as a reference, and iteratively optimizes the rotation matrix R and translation vector T using the least squares method until the reprojection error is less than 0.5 pixels, determines the optimal extrinsic parameter matrix and stores it; the time synchronization unit uses the DTW algorithm to perform timestamp sequence matching on the calibrated data, establishes the time mapping relationship between the event stream and the point cloud, and determines the time synchronization compensation parameters.

[0133] For example, in a robot navigation scenario, a smaller event camera can be used, such as one with a resolution of 346×260. A solid-state LiDAR can be used, with a detection range of 0.1~100m and a sampling frequency of 20Hz. The installation position is adjusted to the top of the robot, facing the direction of robot movement. The distance of the calibration board can be adjusted again to 0.5~3m for calibration, and a preset reprojection error threshold, such as 0.8 pixels, can be used to determine whether the external parameter calibration error is less than 0.8 pixels. If it is less, the reprojection error verification is considered successful. Furthermore, the height of obstacles and the slope of the ground can be increased to adapt to the terrain perception requirements of robot navigation.

[0134] For example, in an intelligent monitoring scenario, an event camera and a LiDAR are fixedly mounted on a monitoring pole, facing the monitoring area. The event camera uses a global shutter mode, and the LiDAR's sampling frequency can be 10Hz. During calibration, the distance to the calibration board can be set to 0.5~10m. Considering the impact of monitoring pole vibration, a dynamic calibration mechanism is added (automatically calibrating external parameters once per hour). Finally, target intrusion detection information, boundary crossing alarm information, and target trajectory tracking information can be output.

[0135] For example, taking an autonomous driving scenario, during the data acquisition phase, the event camera module collects a real-time event stream of brightness changes in the road environment ahead. For each pixel whose brightness change exceeds a threshold, it outputs an event containing (x, y, t, p). The LiDAR module rotates and scans at a frequency of 15Hz, outputting a 3D point cloud containing (x, y, z, l, t) in real time. The event stream and point cloud are synchronously transmitted to the spatiotemporal calibration module via a data bus, with a transmission delay of less than 1ms. During the spatiotemporal calibration phase, a dynamic time warping algorithm is used to perform nonlinear matching on the timestamp sequences of the event stream and point cloud. The core is to calculate the minimum distance between the two time sequences. Assuming the event timestamp sequence is... The point cloud timestamp sequence is Define the distance matrix The elements in this matrix represent the Euclidean distance between two timestamps; then, a cumulative distance matrix is ​​defined. Its element is the minimum cumulative distance from the starting point (1,1) to the current point (i,j). Its recursive formula is as shown in the aforementioned formula (1). By backtracking the minimum path in the cumulative distance matrix, the optimal matching relationship between the event timestamp sequence and the point cloud timestamp sequence can be obtained, thereby determining the event stream time window corresponding to each point cloud frame.

[0136] For example, the spatial registration unit calls the pre-stored optimal extrinsic matrix (R, T), converts the two-dimensional pixel coordinates of the event stream into three-dimensional camera coordinates through the aforementioned formula (2), and then converts them into three-dimensional coordinates in the lidar system through the aforementioned formula (3).

[0137] In some embodiments of this application, the event flow feature extraction unit of the tightly coupled fusion module performs feature extraction on the calibrated event flow: the event flow within the time window is accumulated into an event image, and dynamic feature maps are extracted through a CNN network, focusing on capturing dynamic information such as the direction and speed of the obstacle's movement; the point cloud feature extraction unit uses the PointPillars algorithm to divide the calibrated point cloud into columns and extract the three-dimensional structural features of each column, including information such as the location, size, and reflection intensity distribution of the obstacle.

[0138] In some embodiments of this application, the feature association matching unit calculates the similarity between each pixel feature in the dynamic feature map and each columnar feature in the three-dimensional structural features based on the attention mechanism, and constructs a similarity matrix; it filters valid association pairs according to the similarity threshold (set to 0.7) to achieve pixel-level association between dynamic features and three-dimensional structural features. For example, the dynamic features reflecting vehicle motion in the event stream are associated with the three-dimensional structural features corresponding to the vehicle in the lidar, and the complementary information between the two is mined.

[0139] In some embodiments of this application, the fusion feature reconstruction unit assigns feature weights based on correlation similarity: dynamic feature weights are positively correlated with motion similarity, and three-dimensional structural feature weights are positively correlated with position similarity; the fusion features are reconstructed through a weighted summation algorithm, and the fusion features simultaneously contain dynamic motion information and three-dimensional structural information, effectively compensating for the information deficiencies of a single sensor.

[0140] In some embodiments of this application, the result processing unit of the perception output module performs target detection and tracking on the fused features: it uses the YOLOv8 algorithm to identify obstacle categories (vehicles, pedestrians, cyclists, etc.) from the fused features and outputs static information such as three-dimensional coordinates and dimensions; it uses the Kalman filter algorithm to track the target motion state and outputs dynamic information such as speed and acceleration in combination with dynamic features; the data interface unit outputs the perception results to the autonomous driving decision control system in real time through the CAN bus, with an output delay of less than 20ms, which meets the real-time requirements of autonomous driving.

[0141] In the embodiments of this application, the spatiotemporal calibration module adopts the Dynamic Time Warping (DTW) algorithm and the iterative optimization extrinsic parameter solving algorithm. Compared with the existing fixed timestamp alignment and static extrinsic parameter calibration methods, the time synchronization accuracy is improved from the millisecond level to the sub-microsecond level, and the spatial registration and reprojection error is reduced. The collaborative calibration mechanism of time synchronization and spatial registration effectively eliminates the perception deviation caused by sensor installation errors, environmental disturbances (such as vehicle vibration and monitoring pole swaying) and time asynchrony, ensuring high-precision spatiotemporal alignment between event stream and point cloud data. The overall spatiotemporal coupling calibration provides a high-quality data foundation for tightly coupled fusion, which effectively improves the accuracy of subsequent feature extraction and fusion reconstruction.

[0142] In the embodiments of this application, the event flow feature extraction unit focuses on dynamic information capture, and the point cloud feature extraction unit focuses on 3D structural information extraction. The two respectively leverage the advantages of the high temporal resolution of the event camera and the high-precision ranging of the lidar. The feature association matching unit achieves pixel-level feature association through an attention mechanism, and the fusion feature reconstruction unit is based on similarity-weighted fusion, so that the fused features simultaneously contain dynamic motion information and 3D structural information, effectively compensating for the information deficiencies of a single sensor. The detection response time for dynamic targets is shortened, the obstacle position measurement error is reduced, and the category recognition accuracy is improved. Especially in complex scenarios such as high-speed driving and rapid lane changes of targets, the perception accuracy is significantly better than that of existing loosely coupled schemes.

[0143] In the embodiments of this application, an FPGA+GPU heterogeneous computing architecture is adopted. The FPGA is responsible for real-time preprocessing and spatiotemporal calibration of event streams and point cloud data, while the GPU is responsible for feature extraction and fusion reconstruction. Compared with the traditional CPU processing architecture, the data processing speed is improved. The tightly coupled fusion architecture eliminates the redundant steps of independent processing and post-fusion in the existing solutions, realizing end-to-end low-latency processing from data acquisition, spatiotemporal calibration to fusion perception. The overall system processing latency is reduced, meeting the millisecond-level perception response requirements of scenarios such as autonomous driving and robot navigation.

[0144] In the embodiments of this application, the high dynamic range of the event camera enables it to stably output event streams even under extreme lighting conditions such as strong light, backlight, and weak light. The distance measurement of the lidar is unaffected by lighting conditions, and the performance of the two complements each other. The tightly coupled fusion mechanism allows the system to maintain perception performance through information compensation from another sensor when the performance of a single sensor degrades. For example, when the noise of the lidar point cloud increases under strong light, the dynamic information of the event stream can help filter effective point clouds; when the event stream is sparse under weak light, the three-dimensional point cloud of the lidar can help complete the target outline. In harsh environments such as heavy rain, fog, strong light, and weak light, the system's perception accuracy remains high. Compared with existing single-sensor or loosely coupled solutions, the robustness is greatly improved, ensuring the safe and stable operation of the intelligent system.

[0145] In the embodiments of this application, on the one hand, a calibration system that coordinates time and space dimensions is constructed. In the time dimension, by extracting the timestamp sequences of event streams and point clouds, a cumulative distance matrix is ​​constructed and the minimum path is traced to accurately match the event stream time window corresponding to each point cloud frame, ensuring high alignment of the two types of data in the time dimension and avoiding asynchronous deviations caused by simple timestamp alignment. In the spatial dimension, a coordinate system is completed based on the extrinsic parameter matrix. The time-aligned event stream data is first transformed to the 3D camera coordinate system and then mapped to the LiDAR coordinate system to achieve spatial position matching of the two types of data, effectively eliminating spatial offsets caused by sensor installation errors and environmental disturbances. This joint calibration mechanism provides high-quality basic data with spatiotemporal consistency for subsequent data fusion, solving the core pain point of insufficient spatiotemporal alignment accuracy in existing technologies. On the other hand, with feature differentiation extraction, accurate association, and dynamic weighted fusion as the core, deep complementarity between event camera and LiDAR data is achieved. In the feature extraction stage, differentiated processing is applied to the data characteristics of the two types of sensors: event streams are accumulated into event images to extract dynamic features reflecting the target's motion state, and point clouds are divided into columns to extract 3D structural features representing the target's position and size, leveraging the advantages of the event camera's high temporal resolution and the lidar's high-precision ranging. In the feature association stage, the similarity between pixel features and columnar features is calculated, and valid association pairs exceeding a preset threshold are selected to ensure that the associated features correspond to complementary information about the same target. In the feature fusion stage, feature weights are dynamically assigned based on the similarity of the association pairs, and the associated features are weighted and fused so that the fused features simultaneously contain dynamic motion information and 3D structural information, fully exploiting the complementary value of the two types of data. This architecture differs from the traditional loose mode of "independent processing + post-fusion," achieving end-to-end tight coupling from the feature level to the fusion result, significantly improving the accuracy and robustness of environmental perception.

[0146] In summary, this application breaks through the limitations of existing loosely coupled fusion of multiple sensors, establishes a deep spatiotemporal correlation between the event camera and LiDAR, achieves "tight coupling" at the data level through high-precision spatiotemporal calibration, and achieves "tight coupling" at the feature level through pixel-level feature correlation and weighted fusion, ultimately realizing the collaborative perception of dynamic information and 3D structural information, improving the system's accuracy, real-time performance, and robustness. Specifically, the spatiotemporally tight coupling calibration mechanism employs the Dynamic Time Warping (DTW) algorithm to achieve sub-microsecond time synchronization between the event stream and point cloud, combined with an iterative optimization extrinsic parameter solving algorithm based on a calibration board, to achieve high precision. Spatial registration solves the problems of low accuracy and weak anti-interference ability of existing calibration methods; Pixel-level feature association fusion model: Based on the attention mechanism, a similarity matching model of event flow dynamic features and LiDAR 3D structural features is constructed. The fusion features containing spatiotemporal, structural and dynamic information are reconstructed through weighted fusion algorithm, which fully explores the data complementarity of the two sensors; Heterogeneous tight-coupled processing architecture of FPGA and GPU: The spatiotemporal calibration, feature extraction and fusion reconstruction and other functional modules are processed end-to-end through heterogeneous computing architecture, eliminating redundant calculation steps, greatly reducing processing latency and meeting real-time requirements. The core differences between this application and existing related technologies include: different calibration methods, where most related technologies use fixed timestamp alignment and static extrinsic parameter calibration, while this application uses dynamic time warping and iterative optimization of extrinsic parameter solving to achieve higher precision spatiotemporal synchronization and registration; different fusion architectures, where most related technologies use a loosely coupled architecture of independent processing and post-fusion, while this application uses an end-to-end tightly coupled architecture of data calibration, feature association, and fusion reconstruction, resulting in stronger data complementarity; in addition, this application achieves effective improvements in sub-microsecond time synchronization, latency processing, and perception accuracy through heterogeneous computing architecture and optimization algorithms, with overall performance significantly superior to existing technologies.

[0147] Based on the above embodiments, another embodiment of this application provides a fusion sensing device for an event camera and a lidar, such as... Figure 4 As shown, the event camera and lidar fusion sensing device 1 proposed in this application embodiment may include a processor 11 and a memory 12 storing executable instructions of the processor 11; further, the electronic device 1 may also include a communication interface 13 and a bus 14 for connecting the processor 11, the memory 12 and the communication interface 13.

[0148] In the embodiments of this application, the processor 11 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), FPGA, Central Processing Unit (CPU), controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and this application embodiment does not specifically limit it. The memory 12 can be connected to the processor 11, wherein the memory 12 is used to store executable program code, which includes computer operation instructions. The memory 12 may include high-speed RAM memory, and may also include non-volatile memory, such as at least two disk drives.

[0149] In embodiments of this application, bus 14 is used to connect communication interface 13, processor 11 and memory 12 to enable communication between these devices.

[0150] In embodiments of this application, memory 12 is used to store instructions and data.

[0151] In practical applications, the aforementioned memory 12 can be volatile memory, such as random-access memory (RAM), or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 11.

[0152] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0153] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment.

[0154] Specifically, the program instructions corresponding to the event camera and LiDAR fusion sensing method in this embodiment can be stored on storage media such as optical discs and hard disks. When the program instructions corresponding to the event camera and LiDAR fusion sensing method in the storage media are read or executed by an electronic device, the following steps are included:

[0155] Acquire event stream data collected by the event camera and point cloud data collected by LiDAR;

[0156] The event stream data and the point cloud data are subjected to spatiotemporal coupling calibration processing to obtain calibrated data; wherein, the spatiotemporal coupling calibration processing is used to achieve joint calibration of the event stream data and the point cloud data in the time dimension and the spatial dimension.

[0157] Extract the first feature corresponding to the event stream data and the second feature corresponding to the point cloud data from the calibration data;

[0158] Perform feature association on the first feature and the second feature to obtain feature association pairs;

[0159] Based on the aforementioned feature associations, feature fusion is performed to obtain the fused features;

[0160] Based on the fused features, target perception processing is performed to obtain the perception result.

[0161] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, the electronic device 0 includes an event camera and lidar fusion sensing device 1, an event camera 2, and a lidar 3; the event camera and lidar fusion sensing device can be used to execute the event camera and lidar fusion sensing method.

[0162] Event cameras can be used to collect event stream data in the sensing environment; lidar can be used to collect point cloud data in the sensing environment.

[0163] For example, the electronic device may be a vehicle or a robot equipped with an event camera, a lidar, and a fusion perception device of the event camera and lidar; the vehicle may perform real-time environmental perception during driving based on the event camera, lidar, and the fusion perception device of the event camera and lidar, and then achieve autonomous driving based on the perception results; the robot may perform real-time environmental perception during movement based on the event camera, lidar, and the fusion perception device of the event camera and lidar, and then perform obstacle avoidance and other operations based on the perception results.

[0164] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0165] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each step and / or block in the schematic and / or block diagrams, as well as combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more steps of the schematic and / or one or more blocks of the block diagrams.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. A fusion sensing method using an event camera and a lidar, characterized in that, The method includes: Acquire event stream data collected by the event camera and point cloud data collected by LiDAR; Extract the event timestamp sequence from the event stream data and the point cloud timestamp sequence from the point cloud data; The cumulative distance matrix between the event timestamp sequence and the point cloud timestamp sequence is determined based on the dynamic time warping algorithm; Based on the minimum path of the cumulative distance matrix, time-aligned event stream data and point cloud data are determined; wherein, the time-aligned event stream data and point cloud data are used to indicate the event stream time window corresponding to each point cloud frame in the point cloud data; Spatial registration is performed on the time-aligned event stream data and point cloud data to obtain calibrated data; the spatial registration characterizes the process of transforming the pixel coordinates of the event stream data to the lidar coordinate system based on the extrinsic parameter matrix. The event stream in the calibrated data is accumulated into an event image, and a convolutional neural network is used to extract the dynamic features in the event image to obtain the first feature; The point cloud in the calibration data is divided into columnar sections, and the three-dimensional structural features of the columnar sections are extracted to obtain the second feature; Determine the similarity between each pixel feature in the first feature and each column feature in the second feature; The pixel features and the column features whose similarity exceeds a preset similarity threshold are identified as feature association pairs; Based on the aforementioned feature associations, feature fusion is performed to obtain the fused features; Based on the fused features, target perception processing is performed to obtain the perception result.

2. The event camera and lidar fusion sensing method according to claim 1, characterized in that, The spatial registration of the time-aligned event stream data and point cloud data to obtain calibrated data includes: The time-aligned event stream data is transformed to the 3D camera coordinate system based on the extrinsic parameter matrix to obtain the first transformed event stream data; The first conversion event stream data is converted to the lidar coordinate system to obtain the calibration data.

3. The event camera and lidar fusion sensing method according to claim 1, characterized in that, The feature fusion based on the feature association pairs to obtain the fused features includes: Feature weights are determined based on the similarity of the feature association pairs; The feature association pairs and the feature weights are weighted and fused to obtain the fused features.

4. The event camera and lidar fusion sensing method according to claim 1, characterized in that, The target perception processing based on the fused features to obtain the perception result includes: The fused features are analyzed using a target detection algorithm to identify the target, resulting in a target identification result. The target identification result includes the target category, three-dimensional coordinates, and target size. Target tracking is performed based on the time-series tracking algorithm and the target recognition results to obtain tracking results; wherein, the tracking results include target velocity and target acceleration; The perception result is determined based on the target recognition result and the tracking result.

5. A fusion sensing device combining an event camera and a lidar, characterized in that, The method includes a processor and a memory storing processor-executable instructions; when the instructions are executed by the processor, the method described in any one of claims 1 to 4 is implemented.

6. An electronic device, characterized in that, It includes an event camera, a lidar, and a fusion sensing device for the event camera and lidar; the fusion sensing device for the event camera and lidar is used to perform the fusion sensing method for the event camera and lidar as described in any one of claims 1 to 4.

Images