A Depth Estimation Method Based on the Fusion of LiDAR and Event Camera

Abstract

The present invention relates to a depth estimation method based on the fusion of lidar and event camera. It includes: S1. Data preprocessing; S2. Extracting the point cloud generated by the lidar according to the perspective of the event camera to obtain the target point cloud; S3. Based on event-based spatial scanning, back-projecting the event stream data to obtain a three-dimensional representation of the event stream data; S4. Using density clustering based on weighted Euclidean distance for the point cloud data and then performing three-dimensional target modeling; S5. Using the Bowyer-Watson algorithm to perform reasonable triangulation on all spatial coordinate points in the clustering clusters that construct the stereo surface model; S6. Performing intersection detection on the surface of the modeled target object and the three-dimensionalized events and realizing spatial coordinate point reconstruction to complete event depth estimation; S7. Reconstructing spatial coordinate points for the events in the three-dimensional space, and performing depth filling on the locally sparse point cloud through depth expansion and hole filling. By fusing the lidar and the event camera, dense depth information is obtained.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and more specifically, to a depth estimation method based on the fusion of laser radar and event camera. Background Art

[0002] In recent years, driverless technology has gradually become a research hotspot for universities and enterprises at home and abroad, and has attracted public attention due to its frequent appearance in commercial application tests. Among them, the information perception system is the key foundation for the autonomous driving of driverless vehicles, and is the premise for ensuring the safety, stability, and normal driving of driverless vehicles without collisions. With the improvement of the accuracy and integration of on-board perception sensors in recent years, more and more sensors are deployed on driverless vehicle platforms to perceive the external environment in detail. Among them, laser radar has gradually become one of the main sensors on driverless autonomous vehicles with its high-precision distance detection, rich information, and high reliability that is insensitive to light changes compared to visual cameras. However, due to the complexity of the environment and the high accuracy requirements of perception tasks, high-resolution dense point cloud data is often required. High-beam laser radars often have high costs, which greatly limits their widespread application and deployment in the industry. How to obtain dense point cloud data and depth information based on low-beam laser radar is a key problem that needs to be solved urgently.

[0003] Compared with traditional visual cameras, event cameras are a new type of low-latency, low-redundancy sensor with high dynamic range and high resolution. They do not produce motion blur when autonomous vehicles are moving at high speed. Therefore, high-resolution event cameras can provide dense environmental information for point clouds. However, event cameras cannot directly obtain depth information, so point cloud data is required to provide prior depth information for event streams. In summary, the effective fusion of low-beam lidar and event cameras is a potential solution.

[0004] So far, many studies have focused on combining different sensors for data fusion to obtain high-quality perceptual information, thereby realizing environmental perception tasks in scenarios related to autonomous driving, including object detection, classification, tracking, etc. Since the three-dimensional lidar point cloud obtained by using a low-beam lidar is very sparse, when the three-dimensional lidar point cloud is projected onto a two-dimensional plane, the actual distance between objects will be distorted. At the same time, the three-dimensional lidar point cloud is random and scattered in space distribution, and the spatial coordinate points cannot be obtained through direct address indexing or iterators. The event camera only responds to the gradient of environmental light changes, and the data is output in the form of a stream. In actual scenarios, it is often the geometric edges of objects in the environment that trigger events, and the event stream cannot cover the entire surface of the object. At the same time, many noises will be caused by other small light changes in the environment. Therefore, there are still many problems to be solved on how to use multi-sensors for data fusion to obtain high-quality perceptual information. Summary of the Invention

[0005] To overcome the above-mentioned defects in the prior art, the present invention provides a depth estimation method based on the fusion of lidar and event camera, which realizes depth estimation in complex scenarios such as outdoor high-speed movement of autonomous vehicles.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: a depth estimation method based on the fusion of lidar and event camera, including the following steps:

[0007] S1. Obtain three-dimensional point cloud data through lidar and obtain event stream data through event camera; for the point cloud data, perform denoising operation; for the event stream data generated by the event camera, perform time slice truncation, event stream noise reduction, and event stream dilation operations;

[0008] S2. Extract the point cloud generated by the lidar according to the perspective of the event camera to obtain the target point cloud;

[0009] S3. Based on event-based spatial scanning, realize the back-projection of the event stream data to obtain the three-dimensional representation of the event stream data;

[0010] S4. Use density clustering based on weighted Euclidean distance for the point cloud data and then perform three-dimensional target modeling;

[0011] S5. Use the Bowyer-Watson algorithm to perform reasonable triangulation on all spatial coordinate points in the clustering clusters that construct the solid surface model;

[0012] S6. Perform fast intersection detection on the surface of the modeled target object and the three-dimensionalized event and realize the reconstruction of spatial coordinate points to complete event depth estimation;

[0013] S7. Reconstruct the spatial coordinate points of the event in three-dimensional space, complete the front-end fusion of event information and point cloud, and finally perform depth filling on the locally sparse point cloud through the methods of depth dilation and hole filling.

[0014] The present invention only uses the fusion of an event camera and a lidar, and does not need to rely on positioning information and pose estimation. By fully utilizing the characteristics of the event camera and lidar sensors, depth estimation in complex scenarios such as the outdoor high-speed movement of autonomous vehicles can be achieved. Compared with the depth estimation methods that rely on a single sensor, the present invention has obtained satisfactory results and performance improvements.

[0015] Further, the specific process of performing time segment interception includes: First, based on the timestamp of the lidar, intercept the event stream with a set time length; assume that the frequency of the lidar outputting point cloud data is f, and assume that the timestamp of a certain frame of three-dimensional lidar point cloud is T. Then the timestamp of the previous frame of lidar point cloud is T - 1 / f seconds. In a very small time window Δt close to the moment T, the environmental geometric information corresponding to the event stream generated by the event camera is consistent with the model established by the three-dimensional lidar point cloud obtained at the moment T. Then, use the time window Δt to truncate the event stream and project it onto the two-dimensional pixel plane. Based on the moment T, select the event with the response time closest to the moment T as the final output event at this position; the event stream E T is represented as a set of quadruples (u, v, t, p), that is:

[0016] E T ={(u, v, t, p)|T - Δt < t < T, 0 < Δt < 1 / fs} (1)

[0017] In the formula, (u, v) represents the pixel position, t represents the timestamp, and p represents the polarity of the light intensity change.

[0018] Further, the specific process of event stream noise reduction includes: S uv is a filter window with the event point coordinates (u, v) and size M×N. Calculate the mean value of each event point e t within the local range S uv ; perform convolution operation using the absolute value |p| of the polarity p; according to the sparse characteristics of noise on the two-dimensional plane, introduce a threshold e th , and the improved mean filter formula is as follows:

[0019]

[0020] When the mean value within the local filter window exceeds e th , then retain the event information at this position; otherwise, it is considered noise and removed; g(s, t) represents S uvThe event polarity in

[0021] Further, the event stream dilation specifically includes: enhancing the event stream information by using the dilation operation in digital image processing; defining a structural element S, and moving the structural element B on the two-dimensional planar event E T When the structural element S moves to a certain pixel coordinate position (x, y), if the intersection of the structural element S and the two-dimensional planar event E T is not an empty set, then modify the information at the pixel coordinate position (x, y) into an event (x, y, T, 1); by controlling the number of times the structural element S moves on the two-dimensional planar event E T constrain the range of the event information dilation to obtain the dilated event stream E d :

[0022]

[0023] Further, the step S2 specifically includes: by projecting the laser point cloud in the three-dimensional space onto the two-dimensional pixel plane of the event camera, using the threshold of the pixel coordinates for constraint to screen the laser point cloud that meets the requirements; adopting the pinhole imaging model, using the provided internal and external parameters of the event camera to calculate the two-dimensional pixel plane coordinates of the three-dimensional laser point cloud, and retaining the spatial coordinate points whose pixel coordinates fall within the pixel plane of the event camera; at the same time, based on the Random Sample Consensus (RANSAC) algorithm, extract the ground point cloud from the three-dimensional laser point cloud and remove the influence of the arc.

[0024] Further, the step S3 specifically includes: performing back-projection on the event stream information on the two-dimensional plane to achieve spatial scanning, and then performing inverse operations according to the established pinhole imaging model of the event camera to obtain the three-dimensional normalized coordinates in space corresponding to a single event; then, combined with the center coordinate point of the event camera, calculate the azimuth vector of the event.

[0025] Further, the step S4 specifically includes:

[0026] S41. Three-dimensional point cloud density clustering based on weighted Euclidean distance; using the DBSCAN density clustering algorithm to divide the three-dimensional laser point cloud into point sets. When performing three-dimensional laser point cloud density clustering, instead of directly using the Euclidean distance, different weights of the horizontal distance and the vertical distance are considered on the basis of the Euclidean distance. The specific distance formula is as follows:

[0027]

[0028] In the formula, a and b are different spatial coordinate points, are the weights of different coordinate axes in a three-dimensional Cartesian coordinate system; in order to ensure that spatial coordinate points at different heights at the same position can be classified into the same clustering cluster during the clustering process, the set weight condition is:

[0029] w x = w y > w z ; (5)

[0030] S42. Three-dimensional target modeling; using a stereoscopic plane to estimate the depth of an event, that is, obtaining the corresponding three-dimensional coordinates by calculating the intersection point of the azimuth vector of the event and the plane in space; using each clustering cluster to find its convex hull to obtain a rough representation of the outermost surface of the target object. In order to facilitate and quickly perform the intersection detection calculation of the azimuth vector and the plane, the convex hull of the clustering cluster is triangulated and represented by non-intersecting triangular patches. Thus, any position on the surface of the target object can be calculated using a unique minimum triangular patch.

[0031] Furthermore, in the step S42, in order to enable the constructed target surface model to fully represent the actual target surface, when constructing the surface model, some virtual point clouds for assisting in constructing the surface model are added to the point cloud clustering cluster. Their x and y coordinates are the same as those of the actual point cloud, but the z coordinate needs to be recalculated; the specific calculation method includes:

[0032] Assume that a certain point cloud clustering cluster is P cluster = {(x i , y i , z i ) | i = 1, 2, …, m}. For the actual coordinate point p i (x i , y i , z i ) ∈ P cluster , the distance from this point to the lidar is d = x i ; using trigonometric functions, the known installation height H of the lidar and the angular resolution α of the lidar, calculate the height h cluster from the highest point cloud beam in the clustering cluster P 4 to the upper laser beam:

[0033]

[0034] Similarly, calculate the height h cluster from the lowest point cloud beam in the clustering cluster P 1 to the lower laser beam:

[0035]

[0036] To ensure the spatial coverage in the vertical direction and avoid overlapping, when adding virtual point clouds, half of the angular resolution of the lidar is used for calculation, and the height of the target object in the road environment is considered, with upper and lower limit constraints added; therefore, the final height calculation formula is as follows:

[0037]

[0038]

[0039] Furthermore, the method of fast intersection detection in step S6 specifically includes:

[0040] Starting from the center of the event camera, scanning is performed in the three-dimensional space environment in the direction of the normalized vector from the center of the event camera to the event on the image plane;

[0041] During the spatial scanning process, it is necessary to judge and calculate whether there is an intersection between the azimuth vector and the target surface in the space environment; on the premise of using triangulation to model the surface of the target object, the vertex (V 0 ,V 1 ,V 2 ) of the triangular surface is used to construct the parametric equation of the triangular surface; therefore, for any given target point coordinate P in the three-dimensional space, if the point falls within a certain triangular surface, it is represented using the barycentric coordinate system:

[0042] P = a(V 1 - V 0 ) + b(V 2 - V 0 ) + V 0

[0043] = (1 - a - b)V 0 + aV 1 + bV 2 (10)

[0044] In the formula, a > 0, b > 0, a + b < 1; formula (10) represents any point in the triangular surface, which is represented by the weighted sum of the adjacent two sides starting from any vertex as vectors;

[0045] The event projection formula is expressed as:

[0046] R CP = C + Dt (11)

[0047] In the formula, C represents the camera center coordinate, D is the normalized vector, and t is the vector length to be solved;

[0048] Therefore, the problem of determining and calculating whether the azimuth vector intersects a certain triangular surface is converted into solving the simultaneous equations of formula (10) and formula (11):

[0049] C + Dt = (1 - a - b)V 0 + aV 1 + bV 2 (12)

[0050] By transposing and arranging the formula into matrix form, the final solution equation can be obtained as follows:

[0051]

[0052] In the formula, E 1 = V 1 - V 0 , E 2 = V 2 - V 0 , E 3 = C - V 0 .

[0053] Furthermore, step S7 specifically includes:

[0054] S71. Spatial coordinate point reconstruction: First, use the azimuth vector of the event and the surface model of the target object for depth estimation, and it is necessary to accurately determine the original target surface that causes the event response; by using the spatial range of the laser point cloud clustering cluster to constrain the azimuth vector of the event, filter out the surface model of the target object that is closest to the lidar and the event camera and intersects the azimuth vector of the event, and then perform depth calculation and reconstruct the three-dimensional space point; for the spatial constraint of depth, the specific constraint conditions are as follows:

[0055]

[0056] In the formula, A q is the clustering cluster that intersects the three-dimensional event ray;

[0057] After filtering out the surface model of the target object and performing preliminary depth calculation, project the three-dimensional laser point cloud of the target object into two dimensions, and then use Delaunay triangulation to triangulate the three-dimensional laser point cloud after two-dimensional projection, and further use the algorithm for detecting the intersection of the three-dimensional space ray and the triangular surface to perform depth calculation and reconstruct the three-dimensional space point;

[0058] S72. Depth expansion: First, the blank pixels near the point cloud with depth information are most likely to have a depth close to or even the same as that of the point cloud. Therefore, the existing depth information is used to perform depth expansion on these blank pixels. According to the sparse characteristics after the two-dimensional projection of the point cloud and the scanning characteristics of the lidar beam, by designing a custom kernel and using the dilation operation in digital image processing technology, depth estimation is performed on the pixels near the two-dimensional point cloud.

[0059] S73. Hole filling: After depth expansion, due to the cautious operation of expansion, depth filling is not carried out on a large scale, and there are still many blank pixels between the point cloud fields of view in the depth map. For the smaller holes in the depth map, depth estimation is performed by connecting the depth information of the target object edges near the blank pixels, and the closing operation in morphology is used to close the smaller holes. For the larger holes in the depth map, depth filling is performed using a kernel of a set size.

[0060] Compared with the prior art, the beneficial effects are:

[0061] 1. Different from the existing work that focuses on solving indoor environments or close-range 3D reconstruction, this method focuses on outdoor open road scenes, solves the problems of medium and long-range depth estimation and point cloud densification, uses only a monocular event camera and a single lidar, and does not rely on precise positioning information and pose estimation, but completes depth estimation according to the sensor characteristics and the collected data.

[0062] 2. The present invention proposes to use the principle of ray casting, and through back-projecting the events, realizes event-based spatial scanning. Since the event information comes from the change of the environmental light brightness gradient, the event-based spatial scanning can still work effectively in low-light environments.

[0063] 3. The present invention makes full use of the characteristics of the lidar with accurate ranging and the characteristics of the event camera to capture brightness change information, and performs depth estimation on two-dimensional events in three-dimensional space. In order to accurately trigger the target surface of the event in three-dimensional space, the present invention designs a weighted Euclidean distance based on the density clustering algorithm, constructs a three-dimensional point cloud convex hull using the clustered point cloud clusters, and determines the smallest triangular patches through Delaunay triangulation, so as to accurately estimate the depth of the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a schematic flowchart of the method of the present invention.

[0065] Figure 2 is a schematic diagram of the lidar beam projection of the present invention.

[0066] Figure 3 is a schematic diagram of the intersection detection between the three-dimensional space ray and the triangular surface of the present invention.

[0067] Figure 4 This is the flowchart of the densification of sparse point clouds in the present invention.

[0068] Figure 5 This is the dense depth map in the embodiment of the present invention. Detailed implementation manners

[0069] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention; for better illustrating this embodiment, some components in the accompanying drawings will be omitted, enlarged or reduced, which do not represent the dimensions of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the accompanying drawings may be omitted. The positional relationships described in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention.

[0070] As Figure 1 shown, this embodiment provides a depth estimation method based on the fusion of lidar and event camera. The data sources are mainly two parts: the three-dimensional point cloud obtained from the lidar, which contains the three-dimensional geometric information of environmental objects; the event stream obtained from the event camera. For the three-dimensional point cloud data of the lidar, it is clustered by an improved density clustering algorithm, and a three-dimensional solid surface model of each cluster is constructed. For the event stream information of the event camera, according to the projection principle of light rays and the camera model, the data is back-projected to achieve spatial scanning, and the spatial coordinate points of the event are reconstructed in combination with the solid surface model; then the spatial coordinate system is transformed and unified into the lidar coordinate system to complete the densification of the point cloud; finally, a dense depth map is obtained by using depth expansion and hole filling. The specific steps are as follows:

[0071] Step 1. For the point cloud data, perform denoising operations; for the event stream data generated by the event camera, perform time slice truncation, event stream noise reduction, and event stream dilation operations.

[0072] The data preprocessing is mainly for the event stream data, and only some conventional denoising operations are performed on the point cloud data. For the event camera, due to various interference factors such as ambient light, there are often many noises in the event stream data. During the preprocessing process, we truncate the event stream according to time slices and convert it into the form of frames for noise reduction, and then through event registration, the events generated during movement are transformed into the key frames, so as to fill and enrich the event information and improve the signal-to-noise ratio.

[0073] 1.1 Truncate the event stream according to time slices

[0074] Since the forms and structures of event cameras and point cloud data are different, in order to achieve effective fusion of heterogeneous data. First, based on the timestamp of the lidar, the event stream is intercepted with a certain time length. The frequency of the lidar outputting point cloud data is 20Hz. Assuming the timestamp of a certain frame of three-dimensional lidar point cloud is T, the timestamp of the previous frame of lidar point cloud can be obtained as T - 0.05s. It can be reasonably considered that within a very small time window Δt close to time T, the environmental geometric information corresponding to the event stream generated by the event camera is consistent with the model established by the three-dimensional lidar point cloud obtained at time T.

[0075] This method uses the time window Δt to truncate the event stream and project it onto the two-dimensional pixel plane. Considering that within this time window, the photosensitive components at the same position in the two-dimensional pixel plane may output events with different polarities. Therefore, this method selects the event with the response time closest to time T as the final output event at this position based on time T. The event stream E T can be represented as a set of quadruples (u, v, t, p), that is:

[0076] E T ={(u, v, t, p)|T - Δt < t < T, 0 < Δt < 0.05s} (1)

[0077] where (u, v) represents the pixel position, t represents the timestamp, and p represents the polarity of the light intensity change.

[0078] 1.2 Event stream noise reduction

[0079] In the actual scenario, the event noise on the two-dimensional pixel plane is often isolated, and there are no more event points in the local range. S uv is a filter window with the event point coordinates (u, v) and size M×N. Calculate the mean value of each event point e t within the local range S uv . In order to avoid the cancellation of the positive and negative polarities p of the event points during the convolution operation, resulting in the filtering of normal event points, this method uses the absolute value |p| of the polarity p for the convolution operation. The mean filter may generate floating-point numbers during the calculation process, while the event data polarities generated by the event camera are only 1 and -1, and it will also cause non-zero values to appear at pixel positions where no event response was originally generated, which may lead to incorrect depth estimation and thus the formation of obstacles that do not exist in the three-dimensional space. Therefore, according to the sparse characteristics of the noise on the two-dimensional plane, this method introduces a threshold eth, and the improved mean filter formula is as follows:

[0080]

[0081] When the mean value within the local filter window exceeds eth , then the event information at that position is retained; otherwise, it is considered noise and removed. g(s, t) represents the event polarity in S uv . Since the polarity of an event is only 1 and -1, and the absolute value is used in the filtering calculation, formula (2) is equivalent to counting the number of events in the local filtering window and retaining the event information that is still dense within the local range.

[0082] 1.3 Event stream dilation

[0083] During the movement of the event camera or the environmental target object, the relative movement between the event camera and the three-dimensional space environment causes the source of the event response generated by the event camera to be often the geometric edge of the environmental target object. If only relying on the point cloud timestamp for event matching, only a small amount of effective event information can be obtained. To reasonably enhance the event stream information, this method uses the dilation operation in digital image processing to enhance the event stream information. A structure element S is defined, and the structure element B is moved on the two-dimensional planarized event E T . When the structure element S moves to a certain pixel coordinate position (x, y), if the intersection of the structure element S and the two-dimensional planarized event E T is not an empty set, then the information at the pixel coordinate position (x, y) is modified into an event (x, y, T, 1). By controlling the number of times the structure element S moves on the two-dimensional planarized event E T , the range of the event information dilation is constrained, and the dilated event stream E d is obtained:

[0084]

[0085] Although the dilation operation will cause the range of the target area represented by the event stream information to become larger, making the target boundary expand outwards. However, due to the characteristics of the event information, the dilation operation will merge the hollow target area in contact with the target edge into the target object, and finally can fill the target area holes in the event information and effectively enhance the event stream information.

[0086] Step 2. According to the perspective of the event camera, extract the point cloud generated by the lidar to obtain the target point cloud.

[0087] The horizontal viewing angle of the lidar is 360°, and it can obtain environmental data within 360° around the sensor. However, the horizontal viewing angle of the event camera in the dataset used in this method is 65°, which is much smaller than that of the lidar. Therefore, the overlapping part of the 3D lidar points and the event stream information in space is only 65°, and the remaining 3D lidar points do not fall within the field of view of the event camera. In order to extract the lidar points within the visible range of the event camera, this method projects the lidar points in the 3D space onto the 2D pixel plane of the event camera, and uses the threshold of the pixel coordinates for constraint to screen the lidar points that meet the requirements. Adopting the pinhole imaging model, using the provided internal and external parameters of the event camera, the 2D pixel plane coordinates of the 3D lidar points are calculated, and the spatial coordinate points whose pixel coordinates fall on the pixel plane of the event camera are retained. In the specific implementation, the effective ranging distance of the lidar in the urban road environment and the actual height of the road obstacles are comprehensively considered, and the constraints of depth and height are added to the extraction of the 3D lidar points.

[0088] In addition, the focus of this method is on the depth estimation of obstacles in the 3D environment and does not pay attention to the depth information of the road surface. Due to the principle characteristics of the lidar, there may be multiple discrete arcs projected on the road surface in the 3D lidar points. When using the density clustering algorithm to cluster the 3D lidar points, these arcs will have a very large impact on the clustering results, resulting in relatively large errors in the clustering results and affecting the subsequent 3D surface reconstruction. Therefore, this method extracts the ground lidar points from the 3D lidar points based on the Random Sample Consensus (RANSAC) algorithm and removes the influence of the arcs.

[0089] Step 3. Event-based spatial scanning is performed to back-project the event stream data to obtain a 3D representation of the event stream data.

[0090] Since the denoised event stream data belongs to the information in the 2D space and cannot be directly fused with the data of the 3D lidar points. Therefore, it is necessary to back-project the 2D data to obtain a 3D representation of the event stream data. The event stream information on the 2D plane is back-projected to achieve spatial scanning, and then the inverse operation is performed according to the pinhole imaging model established for the event camera to obtain the 3D spatial normalized coordinates corresponding to a single event. Combining with the camera center coordinate point, the azimuth vector of the event is calculated.

[0091] It should be noted that the pixel plane coordinates are the coordinates obtained after quantization of the image plane, and the optical center can be used to calculate the three-dimensional coordinates in the lidar coordinate system through the external parameters of the camera. Therefore, this method calculates the azimuth vector by calculating the three-dimensional normalized coordinates of the event, rather than directly using the pixel plane coordinates of the event. Therefore, we have realized the back-projection and spatial scanning of the event, obtained a three-dimensional representation of the event, and unified the reference coordinate systems of the event and the three-dimensional lidar point cloud.

[0092] Step 4. Perform three-dimensional target modeling on the point cloud data after density clustering based on weighted Euclidean distance.

[0093] This step is mainly for further processing of the processed three-dimensional lidar point cloud. After the three-dimensional lidar point cloud is extracted through the target point cloud, it still belongs to a discrete point set in three-dimensional space, and there may not necessarily be a unique corresponding relationship with the event information. That is to say, a certain point in the three-dimensional lidar point cloud is very likely not to be represented by any of the azimuth vectors obtained in the previous step. Similarly, not all azimuth vectors necessarily point to a definite spatial coordinate point in the three-dimensional lidar point cloud. Therefore, further solid geometry surface modeling is carried out using the three-dimensional lidar point cloud to obtain a solid plane representation of the target obstacle in the three-dimensional environment, and the depth estimation is finally completed through the intersection detection between the three-dimensional target surface and the azimuth vector. It is specifically divided into the following steps:

[0094] 4.1 Three-dimensional point cloud density clustering based on weighted Euclidean distance

[0095] When using discrete 3D laser point clouds for solid surface modeling, it is necessary to divide the spatial coordinate points belonging to different planes and determine the spatial correlation relationships of the coordinate points in 3D space. This method uses the DBSCAN density clustering algorithm to divide the 3D laser point cloud. It should be noted that the 3D laser point cloud is dense in the horizontal direction, but the density in the vertical direction depends on the number of beams of the lidar. The point cloud in the vertical direction is very sparse compared to the horizontal direction. If this is not taken into account in the density clustering algorithm, the clustering result may be discontinuous in the vertical direction, that is, the same object perpendicular to the ground may be divided into multiple parts in the vertical direction, which is not conducive to solid plane modeling. In addition, in the road environment of the autonomous driving scenario, most obstacles are on the ground rather than suspended in the air. Spatial coordinate points at the same position but different heights often belong to the same object. Based on this assumption, in order to divide different target objects as much as possible in the horizontal distance and ensure that the point cloud in a clustering cluster belongs to the same object as much as possible. When performing 3D laser point cloud density clustering in this method, the Euclidean distance is not directly used, but different weights of the horizontal distance and the vertical distance are considered on the basis of the Euclidean distance. The specific distance formula is as follows, and this formula is a custom weighted Euclidean distance:

[0096]

[0097] In formula (4), a and b are different spatial coordinate points, are the weights of different coordinate axes in the 3D Cartesian coordinate system. In the work of this method, in order to enable different height spatial coordinate points at the same position to be classified into the same clustering cluster during the clustering process, the set weight conditions are:

[0098] w x = w y > w z (5)

[0099] 4.2 3D Object Modeling

[0100] Each clustering cluster obtained after 3D laser point cloud density clustering, the 3D spatial points in it can only independently represent the coordinates of a certain local position of the target object. In the work of this method, it is necessary to use a solid plane for depth estimation of events, that is, to obtain the corresponding 3D coordinates by calculating the intersection point of the azimuth vector of the event and the plane in space. Therefore, this method uses each clustering cluster to find its convex hull to obtain a rough representation of the outermost surface of the target object. In order to facilitate and quickly perform the intersection detection calculation of the azimuth vector and the plane, this method triangulates the convex hull of the clustering cluster and represents it using non-intersecting triangular patches. Thus, any position on the surface of the target object can be calculated using a unique minimum triangular patch.

[0101] However, in actual situations, due to the beam limitations of lidar, the three-dimensional lidar is very sparse in the vertical direction. The lidar beams often only project into the surface of the target object, resulting in the object surface covered by the beams being only a part of the actual object surface. For a distant target object, it is very likely that only one lidar beam of a 16-line lidar projects onto its surface. After three-dimensional lidar point cloud clustering, only a lidar point cloud clustering cluster with one beam is obtained. It is difficult to construct a three-dimensional geometric surface in the vertical direction for such a clustering cluster. In order to enable the constructed target surface model to fully represent the actual target surface, when constructing the surface model, this method adds some virtual point clouds to assist in constructing the surface model in the point cloud clustering cluster. The x and y coordinates of these virtual point clouds are the same as those of the actual point cloud, but the z coordinate needs to be recalculated.

[0102] As Figure 2 shown, when adding virtual point clouds to the point cloud clustering cluster, the occlusion problem of other lidar beams needs to be considered. If the added virtual point clouds are too high or too low, it will cause points that do not belong to the target object to be misestimated in spatial scanning and depth estimation, violating the lidar beam projection situation in reality. In Figure 2 , beam 2 and beam 3 are the lowest and highest beams in the actual point cloud clustering cluster, corresponding to heights h 2 and h 3 ; beam 1 and beam 4 are the lowest and highest beams of the added virtual point clouds, corresponding to heights h 1 and h 4 .

[0103] Suppose a point cloud clustering cluster is P cluster ={(x i ,y i ,z i )|i = 1, 2, …, m}. For the actual coordinate point p i (x i ,y i ,z i ) ∈ P cluster , the distance from this point to the lidar is d = x i . Using trigonometric functions, the known lidar installation height H, and the angular resolution α of the lidar, the height h cluster from the highest point cloud beam in the clustering cluster P 4 to the upper lidar beam can be calculated as follows:

[0104]

[0105] Similarly, the height h cluster from the lowest point cloud beam in the clustering cluster P1 :

[0106]

[0107] However, if the height calculated directly using Formula (6-10) and Formula (6-11) when adding virtual point clouds, the surface models constructed using the clustered point clouds after augmentation will overlap with each other in the vertical direction, resulting in distant events being misestimated onto nearby target objects. Therefore, to ensure the spatial coverage in the vertical direction and avoid overlapping, when this method implements adding virtual point clouds, it calculates using half of the angular resolution of the lidar and considers the height of the target objects in the road environment, adding upper and lower constraints. So, the final height calculation formula is as follows:

[0108]

[0109]

[0110] Step 5. Use the Bowyer-Watson algorithm to perform reasonable triangulation on all the spatial coordinate points in the clusters that construct the three-dimensional surface model.

[0111] After completing the three-dimensional surface modeling, it is possible to determine the matching of a certain event with the target objects in space by combining event-based spatial scanning and the three-dimensional surface model. However, this can only roughly determine the approximate orientation of the event in three-dimensional space and cannot directly and accurately obtain the depth of the event because the sparse three-dimensional lidar point cloud has lost this part of the depth information. Therefore, it is necessary to estimate the depth by relying on the existing spatial coordinate points. To make full use of the existing spatial coordinate points, this method needs to perform reasonable triangulation on all the spatial coordinate points in the clusters that construct the three-dimensional surface model, dividing the surface of the target object into as many and as small patches as possible. In addition, since the side of the target object close to the lidar reflects the laser beam in the three-dimensional space environment and the triangulation in three-dimensional space will pass through the interior of the target object. Therefore, this method projects the three-dimensional spatial coordinate points into two-dimensional space for triangulation. In the triangular mesh obtained by Delaunay triangulation, the circumcircle of any triangle is empty, that is, there are no other vertices, and the minimum interior angle of the triangle is maximized. This characteristic makes Delaunay triangulation not obtain long and narrow triangles, avoiding large errors in depth estimation. In addition, during the process of constructing triangles by Delaunay triangulation, the three closest vertices are used, ensuring that the sides of all triangles do not intersect each other. Thus, after the event back-projection, if there is a triangular surface that triggers the event, then this triangular surface is unique. Finally, Delaunay triangulation can form a convex polygon at the outermost boundary, which enables the constructed model to reflect as large a target surface as possible.

[0112] In the operation of this method, the Bowyer-Watson algorithm is selected for implementation. The Bowyer-Watson algorithm is a classic algorithm for implementing Delaunay triangulation, independently proposed by Bowyer and Watson in 1981 respectively.

[0113] Step 6. Perform fast intersection detection on the surface of the modeled target object and the three-dimensionalized event, and implement the reconstruction of spatial coordinate points to complete the event depth estimation.

[0114] After point cloud clustering, the geometric surface obtained by modeling the clustering clusters does not necessarily exist in the current environment. That is to say, it is also impossible to determine whether the patches in the convex hull represent the real physical surface. The patches in the convex hull can only roughly estimate the approximate range of the target points and cannot accurately estimate the depth. Therefore, this method needs to use the surface of the modeled target object and the three-dimensionalized event to perform intersection detection and implement the reconstruction of spatial coordinate points to complete the event depth estimation.

[0115] A more detailed introduction to the fast intersection detection algorithm is as follows:

[0116] When using the pinhole imaging model to perform spatial scanning on the event back-projection, an azimuth vector with an unknown length and the camera center as the origin can be calculated for each event. Referring to the principle of ray casting, the light reflected from the surface of the target object in space passes through the image plane of the camera in a straight line and causes the generation of this event. Therefore, it can be reasonably considered that the camera center, the event on the image plane, and the light reflection point on the target object plane are on the same straight line. The azimuth vector obtained by event back-projection also lies on this straight line, with the direction opposite to the direction of ray casting, and the end point falls on the light reflection point on the target object plane, which is the three-dimensional space coordinate that needs to be restored in the depth estimation and sparse point cloud densification in the operation of this method. During the process of event back-projection and depth estimation, it is necessary to quickly and accurately judge and calculate the end point of the azimuth vector. By combining the surface model of the target object after triangulation, this method simplifies the event depth estimation problem into an intersection detection problem between a three-dimensional space ray and a solid triangular patch, and completes the depth estimation of the event by calculating the intersection points between the event space scanning process and the object surface model.

[0117] The depth estimation algorithm implemented by this method is based on The method proposed by Tomas et al. in 1997 is applied to determine whether a ray in space intersects a certain triangle and find the intersection point. This algorithm is very suitable for use in triangular meshes. It only requires the three vertices of the triangle and does not need to dynamically calculate or read the saved plane equation, and can quickly make judgments and calculations. Starting from the camera center, scanning is carried out in a three-dimensional space environment with the normalized vector from the camera center to the event on the image plane as the direction.

[0118] As Figure 3 shown, during the space scanning process, it is necessary to judge and calculate whether there is an intersection point between the azimuth vector and the target surface in the space environment. On the premise that this method works by using triangulation to model the surface of the target object, the vertices (V 0 , V 1 , V 2 ) of the triangular surface can be used to construct the parametric equation of the triangular surface. Therefore, for any given target point coordinate P in three-dimensional space, if this point falls within a certain triangular surface, it is represented using the barycentric coordinate system:

[0119] P = a(V 1 - V 0 ) + b(V 2 - V 0 ) + V 0

[0120] = (1 - a - b)V 0 + aV 1 + bV 2 (10)

[0121] where a > 0, b > 0, a + b < 1. Equation (6 - 13) indicates that any point on the triangular surface can be represented as the weighted sum of two adjacent sides starting from any vertex as vectors.

[0122] The event projection formula is expressed as:

[0123] R CP = C + Dt (11)

[0124] where C represents the camera center coordinate, D is the normalized vector, and t is the vector length to be solved.

[0125] Therefore, the problem of judging and calculating whether the azimuth vector intersects a certain triangular surface is converted into solving the simultaneous equations of Equation (10) and Equation (11):

[0126] C + Dt = (1 - a - b)V 0 + aV 1 + bV 2 (12)

[0127] Transpose the formula and organize it into matrix form. Finally, the equation to be solved can be obtained as follows:

[0128]

[0129] where E 1 = V 1 - V 0 , E 2 = V 2 - V 0 , E 3 = C - V 0 .

[0130] According to the final formula (13), this method can quickly determine whether a spatial ray intersects with the target surface and find the intersection point only by using the vertices of the triangular surface and the expression of the azimuth vector, without the need to solve the plane equation of the target surface.

[0131] Step 7 performs spatial coordinate point reconstruction on the event in three-dimensional space to complete the front-end fusion of event information and point cloud. Finally, through the methods of depth dilation and hole filling, depth filling is performed on the locally sparse point cloud.

[0132] After completing the data preprocessing, event space scanning, and depth estimation algorithm, the event information and point cloud information are still independent heterogeneous data. This method completes the front-end fusion of event information and point cloud by performing spatial coordinate point reconstruction on the event in three-dimensional space. Finally, through the methods of depth dilation and hole filling, depth filling is performed on the locally sparse point cloud. It mainly includes three steps: spatial coordinate point reconstruction, depth dilation, and hole filling.

[0133] 7.1 Spatial Coordinate Point Reconstruction

[0134] First, use the azimuth vector of the event and the surface model of the target object to estimate the depth, and it is necessary to accurately determine the original target surface that caused the event response. However, the azimuth vector of the event represents a ray, and there may be multiple target surfaces intersecting with this ray in three-dimensional space. According to the principles of light reflection and projection, assuming that the target objects in the road environment of the autonomous driving scenario can reflect laser well, this method works by using the spatial range of the laser point cloud clustering cluster to constrain the azimuth vector of the event, screening out the target object surface model that is closest to the lidar and the event camera and intersects with the azimuth vector of the event, and then performing depth calculation and reconstructing the three-dimensional space point. For the spatial constraint of depth, the specific constraint conditions are as follows:

[0135]

[0136] where A qTo find the clustering clusters that intersect with the three-dimensional event ray.

[0137] However, considering the existence of target objects with special shapes such as rings in three-dimensional space, in order to obtain more accurate spatial coordinate points in three-dimensional coordinates and achieve more accurate depth estimation, this method works after screening out the surface model of the target object and performing preliminary depth calculation. It projects the three-dimensional laser point cloud of the target object into two dimensions, and then uses Delaunay triangulation to triangulate the three-dimensional laser point cloud after two-dimensional projection. Further, an algorithm for detecting the intersection of three-dimensional space rays and triangular faces is used for depth calculation and reconstruction of the three-dimensional space points. The flow chart is as Figure 4 shown.

[0138] In the flow chart, the azimuth vector set De of the event stream and the set of each clustering cluster are used as inputs. First, for the azimuth vector de of each event, traverse the set of clustering clusters obtained after clustering the density of all three-dimensional laser point clouds. For each clustering cluster point cloud among them, calculate the coordinate mean value to obtain the centroid coordinates of the clustering cluster. Then, through the depth of the centroid coordinates, use formula (11) to calculate the spatial coordinate points of the azimuth vector de at this depth. If this spatial coordinate point falls within the current clustering cluster (quickly judged using the maximum and minimum values of the coordinate point set), then for the surface model established in this clustering cluster, use formula (14) to detect the intersection of the azimuth vector de and each patch in the surface model. If there is a patch that intersects with the azimuth vector, that is, the intersection point exists, then project this patch onto the pixel plane and perform Delaunay triangulation, return the corresponding three-dimensional space patch, recalculate the intersection point of the azimuth vector de and the patch after triangulation, and save the spatial coordinate point closest to the sensor. Finally, obtain the spatial coordinate points corresponding to the finally calculated azimuth vector de.

[0139] 7.2 Depth expansion and hole filling.

[0140] After completing the reconstruction of the spatial coordinate points of the event, the effective boundary information of the event has been incorporated into the sparse point cloud. However, due to the relative sparsity of the event in the two-dimensional plane, there are many gaps in the space between the point clouds, the known depth information is not fully utilized, the local area of the point cloud is relatively dense, and overall it is relatively sparse. In order to make full use of the prior depth information and obtain a dense depth map, the method of depth expansion and hole filling is used to fill the blank pixels with depth based on the existing depth information to achieve depth completion.

[0141] First, the blank pixels near the point cloud with depth information are most likely to have a depth close to or even the same as that of the point cloud. Therefore, the existing depth information can be used to perform depth dilation on these blank pixels. According to the sparse characteristics after the two-dimensional projection of the point cloud and the scanning characteristics of the lidar beam, this method designs a custom kernel and uses the dilation operation in digital image processing technology to estimate the depth of the pixels near the two-dimensional point cloud.

[0142] After depth dilation, due to the cautious operation of dilation, depth filling is not carried out on a large scale, and there are still many blank pixels between the fields of view of the point cloud in the depth map. At this time, the blank pixels are far from the effective point cloud depth and are not suitable for directly filling the depth. For the small holes in the depth map, considering the structure of the target objects in the traffic environment, the depth can be estimated by connecting the depth information of the edges of the target objects near the blank pixels. Therefore, this method uses the closing operation in morphology to close the small holes, and a 5×5 kernel is used. The closing operation retains the existing depth edge information of the target objects and can also well estimate the depth of the pixels in the small holes. For the large holes in the depth map, since they are far from the point cloud with accurate depth, a larger kernel needs to be used for filling. This method uses a 7×7 kernel for depth filling. It should be noted that the operations of depth dilation and hole filling only perform depth filling on the blank pixels, keeping the existing depth information unchanged to avoid increasing the depth error. In addition, the larger blank areas in the depth map often belong to the range of ultra-distance, such as the unobstructed foreground or sky area. For this part of the depth blank area, this method does not perform further operations to avoid bringing incorrect depth estimates.

[0143] Experimental platform parameter information and publicly available dataset information for the experimental work

[0144] 1.1 Experimental environment and platform

[0145] The configuration parameters of the experimental platform used in the experimental work of this method are shown in Table 1.

[0146] Table 1 Configuration parameter table

[0147]

[0148] ROS is an open-source robot development system framework that provides services similar to those of an operating system, facilitating robot development for researchers or engineers. The main functions of ROS include low-level driver management, hardware abstraction description, execution of common functions, program package management, and inter-program message passing. Developing and researching robots on ROS can greatly improve the code reuse rate. Additionally, ROS features distributed processing, leveraging a low-coupling and loose runtime architecture that enables individual module executable files to run independently, and even allows individual modules to run independently across different devices and networks.

[0149] 1.2 Experimental Datasets

[0150] The open-source dataset used in the work of this method is the Multi-Vehicle Stereo Event Camera dataset (MVSEC). This dataset was released by researchers at the University of Pennsylvania in 2018 and was designed and collected for the research and development of 3D perception algorithms based on event cameras. Data from sensors such as lidar, IMU, GPS, and event cameras were collected from multiple platforms, including cars, motorcycles, hexacopter drones, and handheld platforms, in scenarios both during the day and at night, indoors and outdoors, and the data were fused to obtain ground truth data and depth images. For event generation, the MVSEC dataset configures and installs two DAVIS 346B cameras aligned along the X-axis with a baseline of 10 cm, and synchronizes the timestamps of the cameras by transmitting a synchronization pulse to the right camera (slave camera) through an external wire using the trigger signal generated from the left camera (master camera). The resolution of each camera is 346×260, equipped with a 4nm lens and a 65° vertical field of view. For the acquisition of 3D lidar point clouds, considering that the lidar needs to provide accurate depth information within the visual range of other sensors, complete overlap was considered between the smaller vertical field of view of the lidar and the vertical field of view of the stereo event camera during installation.

[0151] In the MVSEC dataset, the sensor parameters and characteristics of the lidar and event cameras for collecting 3D lidar point clouds and event streams are shown in Table 2 below.

[0152] Table 2 Some Sensors and Their Characteristics in the MVSEC Dataset

[0153]

[0154]

[0155] This method works by conducting experiments using the original data of lidar and event cameras collected on a vehicle platform provided in this dataset, and using the corresponding ground truth data for testing. The scenarios are outdoor road environments under daytime and nighttime conditions.

[0156] 1.3 Experimental performance

[0157] Using a rainbow color bar, that is, the color thresholds from red to purple represent the distance from near to far. The experimental results are as Figure 5 shown. It can be seen that after data fusion, adding an event camera can well handle the problem of being difficult to recognize due to local overexposure or local darkness in the ambient light conditions. Moreover, vehicles and pedestrians that were previously difficult to recognize due to the overly sparse point cloud are well restored in the dense depth map.

[0158] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0159] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention and are not limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or alterations can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A depth estimation method based on the fusion of lidar and event camera, characterized in that, it includes the following steps: S1. Obtain three-dimensional point cloud data through lidar and obtain event stream data through an event camera; for the point cloud data, perform denoising operations; for the event stream data generated by the event camera, perform time slice interception, event stream noise reduction, and event stream dilation operations; S2. Extract the point cloud generated by the lidar according to the perspective of the event camera to obtain the target point cloud; S3. Based on event-based spatial scanning, perform back-projection of the event stream data to obtain a three-dimensional representation of the event stream data; S4. Use density clustering based on weighted Euclidean distance for the point cloud data and then perform three-dimensional target modeling; S5. Use the Bowyer-Watson algorithm to perform reasonable triangulation on all spatial coordinate points in the clustering clusters that construct the solid surface model; S6. Perform fast intersection detection on the surface of the modeled target object and the three-dimensionalized event and implement spatial coordinate point reconstruction to complete event depth estimation; S7. Perform spatial coordinate point reconstruction on the event in three-dimensional space to complete the front-end fusion of event information and point cloud. Finally, through depth dilation and hole filling methods, perform depth filling on the locally sparse point cloud.

2. The depth estimation method based on the fusion of lidar and event camera according to claim 1, characterized in that, The specific process of time segment truncation includes: First, based on the timestamp of the lidar, the event stream is truncated with a set time length; assuming that the frequency of the lidar outputting point cloud data is f, and assuming that the timestamp of a certain frame of three-dimensional lidar point cloud is T, and the timestamp of the previous frame of lidar point cloud is T - 1 / f seconds, then within a very small time window Δt close to time T, the environmental geometric information corresponding to the event stream generated by the event camera is consistent with the model established by the three-dimensional lidar point cloud obtained at time T; then, the time window Δt is used to truncate the event stream and project it onto the two-dimensional pixel plane, and based on time T, the event with the response time closest to time T is selected as the final output event at this position; the event stream E T is represented as a set of quadruples (u, v, t, p), that is: E T = {(u, v, t, p) | T - Δt < t < T, 0 < Δt < 1 / f} (1) In the formula, (u, v) represents the pixel position, t represents the timestamp, and p represents the polarity of the light intensity change.

3. The depth estimation method based on the fusion of lidar and event camera according to claim 1, characterized in that, The event stream noise reduction specifically includes: S uv For a filter window with event point coordinates (u, v) and size M×N, calculate each event point e t within the local range S uv for the mean value; perform convolution operation using the absolute value |p| of the polarity p; according to the sparse characteristic of noise on the two-dimensional plane, introduce a threshold e th , and the proposed improved mean filter formula is as follows: When the mean value within the local filter window exceeds e th , then the event information at that position is retained; otherwise, it is considered noise and removed; g(s, t) represents the event polarity in S uv .

4. The depth estimation method based on the fusion of lidar and event camera according to claim 1, characterized in that, The event stream dilation specifically includes: enhancing the event stream information by using the dilation operation in digital image processing; defining a structural element S, and moving the structural element B on the two-dimensional planarized event E T When the structural element S moves to a certain pixel coordinate position (x, y), if the intersection of the structural element S and the two-dimensional planarized event E T is not an empty set, then modify the information at the pixel coordinate position (x, y) into an event (x, y, T, 1); by controlling the number of times the structural element S moves on the two-dimensional planarized event E T constrain the range of the event information dilation, and obtain the dilated event stream E d :

5. The depth estimation method based on the fusion of lidar and event camera according to claim 1, characterized in that, The specific steps of step S2 include: by projecting the lidar point cloud in three-dimensional space onto the two-dimensional pixel plane of the event camera, using the threshold of the pixel coordinates for constraint to screen the lidar point cloud that meets the requirements; adopting the pinhole imaging model, using the internal parameters and external parameters of the provided event camera to calculate the two-dimensional pixel plane coordinates of the three-dimensional lidar point cloud, and retaining the spatial coordinate points whose pixel coordinates fall on the pixel plane of the event camera; at the same time, based on the random sample consensus (RANSAC) algorithm, extract the ground point cloud from the three-dimensional lidar point cloud and remove the influence of arcs.

6. The depth estimation method based on the fusion of lidar and event camera according to claim 1, characterized in that, The specific steps of step S3 include: perform back-projection on the event stream information on the two-dimensional plane to achieve spatial scanning, and then perform inverse operations according to the pinhole imaging model established for the event camera to obtain the spatial three-dimensional normalized coordinates corresponding to a single event; then combine the center coordinate point of the event camera to calculate the azimuth vector of the event.

7. The depth estimation method based on the fusion of lidar and event camera according to claim 1, characterized in that, The specific steps of step S4 include: S41. 3D point cloud density clustering based on weighted Euclidean distance; Use the DBSCAN density clustering algorithm to divide the point set of the 3D lidar point cloud. When performing 3D lidar point cloud density clustering, the Euclidean distance is not directly used, but the different weights of the horizontal distance and the vertical distance are considered on the basis of the Euclidean distance. The specific distance formula is as follows: where a and b are different spatial coordinate points, and w x , w y , w z are the weights of different coordinate axes in a three-dimensional Cartesian coordinate system; in order to ensure that spatial coordinate points at different heights at the same position can be classified into the same cluster during the clustering process, the set weight condition is: w x = w y > w z ;(5) S42. 3D object modeling; Use the stereo plane to estimate the depth of the event, that is, obtain the corresponding 3D coordinates by calculating the intersection point of the azimuth vector of the event and the plane in space; Use the convex hull of each clustering cluster to obtain a rough representation of the outermost surface of the target object. In order to facilitate and quickly perform the intersection detection calculation of the azimuth vector and the plane, triangulate the convex hull of the clustering cluster and represent it with non-intersecting triangular patches. Thus, any position on the surface of the target object can be calculated using the unique smallest triangular patch.

8. The depth estimation method based on the fusion of lidar and event camera according to claim 7, characterized in that, in the step S42, in order to make the constructed target surface model fully represent the actual target surface, when constructing the surface model, some virtual point clouds for assisting in constructing the surface model are added to the point cloud clustering cluster. Its x and y coordinates are the same as those of the actual point cloud, but the z coordinate needs to be recalculated; The specific calculation method includes: Suppose a point cloud clustering cluster is P cluster ={(x i ,y i ,z i )|i = 1, 2, …, m}. For the actual coordinate point p i (x i ,y i ,z i ) ∈ P cluster , the distance from this point to the lidar is d = x i ; Using trigonometric functions, the known lidar installation height H, and the angular resolution α of the lidar, calculate the height h cluster from the highest point cloud beam in the clustering cluster P 4 to the upper laser beam: Similarly, calculate the clustering cluster P cluster The height h of the lowest point cloud beam in 1 : where h 2 is the height of the wire harness 2 in the actual point cloud clustering cluster; h 3 is the height of the wire harness 3 in the actual point cloud clustering cluster; In order to ensure the coverage of the space in the vertical direction and avoid overlapping with each other, when adding virtual point clouds, calculate using half of the lidar angular resolution, and consider the height of the target object in the road environment, and add upper and lower constraints; Therefore, the final height calculation formula is as follows:

9. The depth estimation method based on the fusion of lidar and event camera according to claim 8, characterized in that, the method of rapid intersection detection in the step S6 specifically includes: Starting from the center of the event camera, scan in the 3D space environment in the direction of the normalized vector from the center of the event camera to the event on the image plane; During the space scanning process, it is necessary to judge and calculate whether there is an intersection point between the azimuth vector and the target surface in the space environment; On the premise of using triangulation to model the surface of the target object, use the vertices (V0, V1, V2) of the triangular surface to construct the parametric equation of the triangular surface; Therefore, given an arbitrary target point coordinate P in the 3D space, if the point falls within a certain triangular surface, it is represented using the barycentric coordinate system: P = a(V 1 - V 0 ) + b(V 2 - V 0 ) + V 0 =(1 - a - b)V 0 +aV 1 +bV 2 (10) where a>0, b>0, a + b < 1; Equation (10) represents any point in the triangular surface, which is represented as the weighted sum of two adjacent sides starting from any vertex as vectors; The event projection formula is expressed as: R CP = C + Dt (11) where C represents the camera center coordinate, D is the normalized vector, and t is the vector length to be solved; Therefore, the problem of judging and calculating whether the azimuth vector intersects with a certain triangular surface is converted into solving the simultaneous equations of formula (10) and formula (11): C + Dt = (1 - a - b)V 0 + aV 1 + bV 2 (12) Move the formula and organize it into matrix form, and finally the solution equation can be obtained as: where E 1 = V 1 - V 0 , E 2 = V 2 - V 0 , E 3 = C - V 0 .

10. The depth estimation method based on the fusion of lidar and event camera according to claim 9, characterized in that, the step S7 specifically includes: S71. Spatial coordinate point reconstruction: First, depth estimation is performed using the azimuth vector of the event and the surface model of the target object, and it is necessary to accurately judge the original target surface that caused the event response; by using the spatial range of the lidar point cloud clustering cluster to constrain the azimuth vector of the event, the surface model of the target object that is closest to the lidar and event camera and intersects with the azimuth vector of the event is screened out, and then depth calculation and reconstruction of the three-dimensional space point are performed; for the spatial constraint of depth, the specific constraint conditions are as follows: Where, A q is to find the clustering clusters intersecting with the three-dimensional event rays; p r (x r , y r , z r ) is the coordinate point of the r-th point cloud; After screening out the surface model of the target object and performing preliminary depth calculation, the three-dimensional lidar point cloud of the target object is projected two-dimensionally, and then Delaunay triangulation is used to triangulate the three-dimensional lidar point cloud after two-dimensional projection, and further depth calculation and reconstruction of the three-dimensional space point are performed using the algorithm of intersection detection between the three-dimensional space ray and the triangular surface; S72. Depth expansion: First, the blank pixels near the point cloud with depth information are most likely to have a depth close to or even the same as the point cloud depth, so depth expansion is performed on these blank pixels by means of the existing depth information; according to the sparse characteristics after the two-dimensional projection of the point cloud and the scanning characteristics of the lidar beam, by designing a custom kernel and using the dilation operation in digital image processing technology, depth estimation is performed on the pixels near the two-dimensional point cloud; S73. Hole filling: After depth expansion, due to the cautious operation of expansion, depth filling will not be carried out on a large scale, and there are still many blank pixels between the point cloud fields of view in the depth map; for the smaller holes in the depth map, depth estimation is performed by connecting the depth information of the target object edge near the blank pixels, and the smaller holes are closed using the closing operation in morphology; for the larger holes in the depth map, depth filling is performed using a kernel of a set size.

Images