Sparse event point-oriented spatio-temporal clustering small target detection method

Abstract

This invention discloses a spatiotemporal clustering method for small target detection based on sparse event points. The method involves preprocessing raw data captured by an event camera to obtain preprocessed event data; traversing all event points in the event data, assigning a weight value to each event point, and sorting all event points in descending order according to their weight values; using the average weight of the bottom M% of event points as a clustering threshold, and performing nearest neighbor clustering on the event data based on this threshold to obtain preliminary detection results; and finally, performing point cloud filtering on the preliminary detection results to obtain the final detection results. This invention can directly extract small target features from event data for detection. Under static backgrounds, it can effectively filter out event point clutter triggered by thermal noise and interference event points generated by the movement of large objects, achieving high detection accuracy.

Description

Technical Field

[0001] This invention relates to small target detection technology in point clouds, and in particular to a spatiotemporal clustering method for small target detection based on sparse event points. Background Technology

[0002] Existing techniques detect small targets by reconstructing intensity images from event camera data. While this method has achieved some success, it sacrifices the temporal resolution advantage of event cameras. Furthermore, some techniques project event camera data to adapt to image-based detection methods, but these techniques primarily utilize edge information rather than spatiotemporal information from the event data, leading to missed detections due to the lack of spatiotemporal information.

[0003] Most techniques for object detection using event cameras employ convolutional neural networks, training network models with large amounts of labeled data to extract temporal and spatial features from event camera data. However, due to the scarcity of event datasets and the high computational demands, these techniques typically require extensive preparation. In recent years, methods using spiking neural networks to process event data for object detection have achieved some success in extracting temporal features while reducing computational load. However, currently available datasets for sparse event points in object detection are few and insufficient in size, and there are no public datasets for small object detection. Existing techniques have not yet solved the problem of dataset scarcity.

[0004] Existing target detection techniques in event cameras lack specific algorithms for small targets. Algorithms for sparse event point data are mostly effective for tangible targets such as pedestrians, buildings, and cars, performing poorly and with low accuracy for intangible targets. While image-based methods can detect target locations, they can only detect horizontal positions within a certain time range based on frame rate, sacrificing temporal precision. Existing techniques detect edge features such as planes and curved surfaces in event point clouds, which are effective for tangible targets. However, intangible targets in event point clouds often exist as straight lines or curves, making it difficult for existing methods to solve the problem of detecting intangible targets.

[0005] The terms used in this invention are explained as follows:

[0006] An event camera, also known as a neuromorphic vision sensor or a biomimetic silicon-based vision sensor, is a device with a novel imaging mechanism. Each pixel operates independently, outputting asynchronous spatiotemporal pulse signals when brightness changes. The raw data format of an event camera is as follows:

[0007] Z i = (x, y, t, p);

[0008] Among them, Z i Let (x, y) be the output of the i-th pixel sensor, (x, y) be the two-dimensional position coordinates of the imaging pixel i, t be the imaging time (which varies between each pixel sensor), and p be the event polarity (positive or negative polarity corresponding to an increase or decrease in light intensity).

[0009] Compared to traditional cameras, event cameras offer microsecond-level temporal resolution, excelling at capturing fast-moving targets and overcoming motion blur issues common with traditional cameras. Furthermore, event cameras boast an extremely high dynamic range, enabling them to function effectively even in challenging lighting conditions.

[0010] Event point clouds are obtained by removing polarity from raw data captured by event cameras. They possess two-dimensional spatial and temporal dimensions, but lack depth information compared to traditional LiDAR point clouds. Traditional point clouds consist of points in the three directions (x, y, z) at the same moment, while event point clouds are formed by the change in light intensity in two-dimensional space (x, y) as time t increases.

[0011] Small targets generally refer to targets smaller than 32*32 pixels in image resolution or whose pixel percentage is less than 2% of the entire image. There is currently no clear definition of small targets in event cameras. This invention adopts the definition in the image. In the following embodiments of this invention, the camera resolution is 1920*1080, and the number of pixels triggered by the target within 3ms is approximately 10-100, which meets the 2% requirement. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a spatiotemporal clustering method for small target detection in sparse event points, which addresses the shortcomings of existing technologies and solves the problems of difficulty in small target detection and target feature extraction in event point clouds.

[0013] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a spatiotemporal clustering method for small target detection based on sparse event points, comprising the following steps:

[0014] The raw data captured by the event camera is preprocessed to obtain preprocessed event data;

[0015] Iterate through all event points in the event data, assign a weight value to each event point, and sort all event points in descending order according to the weight values;

[0016] The average weight of the M% of event points ranked last is used as the clustering threshold. The event data is then clustered into nearest neighbors based on the clustering threshold to obtain preliminary detection results.

[0017] Point cloud filtering is applied to the preliminary detection results to obtain the final detection results.

[0018] This invention effectively filters out event point clutter triggered by thermal noise and interference event points generated by the movement of large objects in a static background, greatly improving the detection accuracy of small targets in event point clouds. This invention directly detects small targets in event point clouds without requiring a dataset. Compared to methods using morphology and edge features, this invention uses density features for detection, identifying the significant density difference between background noise event points and target event points. This allows for a clearer distinction between noise event points and target event points, resulting in better noise suppression and detection performance.

[0019] To reduce useless data, improve processing efficiency, and reduce storage resource requirements, this invention preprocesses the raw data captured by the event camera to obtain preprocessed event data. The specific implementation process includes:

[0020] Remove the Z-coordinates of four-dimensional event points from the original data. i Given the polarity p of the event point coordinates (x, y, t, p), we downsample the event point coordinates after removing the polarity to obtain the 3D spatiotemporal point cloud data [x, y, t]. T ×N, where the three-dimensional spatiotemporal point cloud data is the preprocessed event data; (x,y) is the two-dimensional position coordinate of the imaging pixel, t is the imaging time, and N is the number of event points in the original data.

[0021] In this invention, in order to further improve processing efficiency, the event data can be sliced ​​with a fixed step size to obtain multiple data segments, and a weight value can be assigned to each event point in each data segment; the event points of all data segments are arranged in descending order according to the weight value.

[0022] The weight W of any event point a a The calculation formula is: Where D max Let D be the Euclidean distance between event point a and the event point farthest from event point a. ai Let represent the Euclidean distance between event point a and the i-th event point, and m be the total number of input events. This invention assigns a weight value to each data point, thereby obtaining the global spatiotemporal features of the data, fully exploring the original spatiotemporal features of the event data, and further improving the accuracy of small target detection.

[0023] In this invention, event points with weight values ​​distributed below 50% are selected, and the average weight of the selected event points is calculated to obtain the adaptive clustering range l. Statistical analysis of the captured event point clouds shows that, under stationary shooting conditions, small targets account for approximately 30%-50% of the entire point cloud. Selecting 50% ensures that target event points are not lost during the weight analysis stage, while also effectively suppressing background noise and clutter.

[0024] The process of obtaining the preliminary test results includes:

[0025] Based on the clustering threshold, the event data is clustered using nearest neighbor clustering to obtain a clustering result [x,y,t] that includes the detection results. T ×M; M is the total number of event points included in the clustering result;

[0026] If the clustering result includes multiple categories, the two-dimensional spatial coordinate deviation between each category and the other categories is calculated. If the two-dimensional spatial coordinate deviation does not exceed the two-dimensional position coordinates in the clustering result, a uniform difference is applied at the break point to supplement the break trajectory and obtain a preliminary detection result.

[0027] When a small target moves too fast, the event points captured by the camera will be relatively sparse. When the target speed changes significantly, the density of event points will change significantly, which will cluster the same target into multiple categories. Uniform interpolation at the break point helps to deal with this situation and improve the detection accuracy of small targets.

[0028] The specific implementation process of point cloud filtering on the preliminary detection results includes: calculating the time span of each cluster category; if the time span of a certain cluster category does not exceed 10% of the point cloud time event dimension, then the cluster category is deleted. The target captured by the camera exists in the point cloud for a long time, while clutter interference occurs for a shorter period. However, clutter is more concentrated than random noise, thus there is a probability of false detection. This invention can filter out short-term clutter, capable of filtering out clutter interference that occurs for less than 1 / 10 of the time.

[0029] As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the method described above.

[0030] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon; when the computer program / instructions are executed by a processor, they implement the steps of the method described above.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention can directly extract small target features from event data for detection, and under static background, it can effectively filter out event point clutter triggered by thermal noise and interference event points generated by the movement of large objects, resulting in high detection accuracy. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the method in an embodiment of the present invention;

[0033] Figure 2 This is a flowchart illustrating the weight value calculation process in an embodiment of the present invention.

[0034] Figures 3(a) to 3(d) This is a visualization of the method in an embodiment of the present invention;

[0035] Figures 4(a) and 4(b) show the results of the image-based inter-frame difference method. Detailed Implementation

[0036] The method of the present invention is as follows: Figure 1 As shown, it includes the following steps:

[0037] Step 1 involves preprocessing the input data. The main task is to remove invalid portions of the raw data captured by the event camera and downsample it. Since polarity is not used in the data processing, the polarity portion of each event point is removed. This allows each event point to be represented as a three-dimensional vector Z. i =[x,y,t] T This reduces the amount of useless data in the methods of the embodiments of the present invention.

[0038] In this embodiment of the invention, the raw data is an "event point cloud containing small targets". The specific shooting method is as follows: the camera position is fixed, and moving small targets are photographed. The shooting scene can be indoors or in a stadium, etc. The subject can be a ping-pong ball, and the raw data includes random noise generated by the camera and clutter interference (such as pedestrians) that occurs during the shooting process.

[0039] Because event cameras have extremely high dynamic range and temporal resolution, the amount of event data they capture is relatively large. To improve the computing speed, this embodiment of the invention performs a downsampling operation on the raw event camera data. Alternatively, a downsampling operation can be performed on the preprocessed data after removing polarity to further improve data processing efficiency.

[0040] Step 2, Event Weight Analysis: This step involves iterating through all event points in the event data and assigning a weight value to each event point. This operation is to obtain the global spatiotemporal characteristics of the data.

[0041] Considering the large amount of event data, one implementation of this invention employs a fixed-interval step size to slice the data. Each data segment retains a certain number of event points (the number can range from 5,000 to 50,000 depending on the computing power; too few points will result in insufficient data to accurately extract small target event points, while too many points will lead to excessive time and cost). The above calculation method is performed within each data segment (i.e., assigning weight values ​​to event points in each data segment). The Euclidean distance between each point and all other event points is calculated. This operation can fully exploit the original spatiotemporal characteristics of the event data.

[0042] Step 3: Sort all events in descending order of their weights. Events with lower weights are more likely to be the target to be detected.

[0043] Step 4: After sorting and analyzing the weights obtained from the global event point weight analysis module, the adaptive clustering range is calculated. Considering that small targets account for approximately 50% of the global event data when the background is stable, the method in this embodiment uses the average Euclidean distance of the event points ranked in the bottom 50% as the threshold input for the point cloud spatiotemporal clustering module.

[0044] The global event point weight analysis module in this embodiment of the invention mainly implements the following functions:

[0045] Event point weight calculation involves calculating the weights of the preprocessed data and assigning a global weight value to each event point.

[0046] Weighted sorting and filtering are performed because clutter and other background interference do not persist for long. If an event point is the target to be detected, its Euclidean distance to other event points is relatively small, so its weight will be lower. Clutter and background interference usually manifest as outliers in the data. These event points have relatively large Euclidean distances from other points, resulting in a relatively large sum of calculated Euclidean distances, and correspondingly, their weights will also be larger. The event points are sorted according to the detection requirements to obtain candidate event points.

[0047] Clustering threshold calculation: Based on the Euclidean distance of candidate event points, calculate the range of adaptive spatiotemporal clustering, which serves as the input for the next module (point cloud spatiotemporal clustering module).

[0048] The point cloud spatiotemporal clustering module in this embodiment of the invention mainly implements the following functions:

[0049] Nearest neighbor clustering: Based on an adaptive threshold, the event data is clustered into nearest neighbors to obtain preliminary detection results.

[0050] Point cloud filtering merges the broken trajectories in the clustering results and removes short trajectories caused by clutter to obtain the final detection result.

[0051] Step 5: The nearest neighbor clustering method is used to obtain the best detection result. Since the nearest neighbor spatiotemporal clustering method requires knowledge of the type of some event data, but in actual detection, the type information of the data points is unknown, yet the target has spatiotemporally dense characteristics, this method first obtains a threshold (i.e., clustering range) input from the global event point weight analysis module. Event points that meet the threshold condition (assuming the calculated threshold is k, then event points that meet the threshold condition refer to points whose average distance to neighboring points is less than k in the nearest neighbor clustering algorithm) are considered as the target to be detected.

[0052] Step 6, point cloud filtering, mainly removes excessively short trajectories at the output end, filters out small sample categories generated in the clustering (most of which are interferences that have existed for a long time), and supplements broken trajectories. The final clustering result is the detection result.

[0053] In one embodiment, the data preprocessing step includes;

[0054] Input the four-dimensional event point coordinates Z i =(x,y,t,p) removes the polarity p and downsamples the data to obtain the 3D spatiotemporal point cloud data [x,y,t]. T ×N. N refers to the number of points, that is, the number of event points in the original data.

[0055] In one implementation of this invention, the method for event weight analysis includes:

[0056] Data slicing involves dividing the data into event point clouds of 10,000 event points (the number of event points can be set according to actual needs), and calculating the Euclidean distance D between any event point 'a' and the other i event points. ai The sum of the Euclidean distances between an event point and other events is defined as the event weight, and the weight calculation formula is as follows:

[0057]

[0058] Where D max D is the Euclidean distance from event a to the event farthest among all events. ai Let represent the Euclidean distance between event a and the i-th event, and m be the total number of input events.

[0059] In one implementation of this invention, when calculating event weights, the event points that have been assigned weights are arranged in descending order of weight size. Event points with weight values ​​distributed below 50% are selected, and their average weight is calculated, which is the adaptive clustering range l.

[0060] In one implementation of this invention, the overall process of weight analysis is as follows: Figure 2 As shown.

[0061] In one implementation of this invention, during point cloud spatiotemporal clustering calculation, the clustering threshold input is obtained from the global event point weight analysis module, such as... Figure 1 As shown, the specific steps of the clustering module operation at this time include:

[0062] Using a clustering range l, the nearest neighbor clustering algorithm is applied to all event points to obtain a clustering result [x,y,t] that includes the detection results. T ×M; The input number of points is 10000. After nearest neighbor clustering, a clustering result containing ≤10000 points will be obtained. M is the total number of points contained in the clustering result.

[0063] If the clustering result is multiple categories, calculate the two-dimensional spatial coordinate difference between the first and last two ends of each category (i.e., the first and last points that appear) and other categories. If the two-dimensional spatial coordinate deviation does not exceed the three-dimensional clustering threshold l (i.e., the clustering range), then perform uniform interpolation at the break point to supplement the break trajectory.

[0064] In this embodiment of the invention, calculating the two-dimensional spatial coordinate difference between each category and other categories refers to calculating the two-dimensional spatial coordinate difference between the first and last points (i.e., the first and last points of each category) and other categories. For example, if the first point of category 1 appears at time T = a, the last point of category 1 appears at time T = b, and the first point of category 2 (the time of its first point appearance) appears at time T = c, where c > b, then category 2 appears after category 1. Therefore, the two-dimensional spatial coordinate deviation is the difference in horizontal coordinates between the last point of category 1 and the first point of category 2. Similarly, if the last point of category 2 appears at time T = d, where d < a, then category 2 appears before category 1. The two-dimensional spatial coordinate deviation is the difference in horizontal coordinates between the last point of category 2 and the first point of category 1.

[0065] The clustering results may contain multiple categories, including "real targets," "long-standing clutter interference," and "dense noise points." Clutter interference and noise points exist independently and are therefore far apart, without any "breaks." Targets may trigger fewer event points in certain time domains due to factors such as non-uniform motion and camera acquisition errors. If a high threshold is set, these event points will be filtered out by the "nearest neighbor clustering" algorithm, resulting in breaks. Therefore, in this embodiment of the invention, the break points are the locations where the clustering algorithm has filtered out the relevant event points, causing the event point break.

[0066] Point cloud filtering is applied to the overall clustering results, and the time span of each category is calculated, considering only the time span of the first and last events in the clustering results (t). nIf the time dimension of the event point cloud does not exceed 10%, then the category is considered clutter interference and is removed.

[0067] The filtered result effectively suppresses short-term clutter and can fill in some broken trajectories. The final filtered result is the detection result.

[0068] The following are experiments in the embodiments of the present invention:

[0069] Shooting environment: Indoors, camera position fixed

[0070] Subject of the photo: Ping pong ball

[0071] Types of noise and clutter: camera thermal noise, pedestrians.

[0072] Figures 3(a) and 3(b) show the three-dimensional visualization effect of the method of the embodiment of the present invention. Figure 3(a) shows the original input event point cloud, with the target position highlighted by a black box. Figure 3(b) shows the final detection result. It is clear that the embodiment of the present invention effectively suppresses background noise. Figures 3(c) and 3(d) are two-dimensional projection effect diagrams, which project the results of Figures 3(a) and 3(b) in the (x,y) direction.

[0073] Figures 4(a) and 4(b) show the results of the image-based inter-frame difference method, with the detected moving targets within the black boxes.

[0074] Figures 4(a) and 4(b) show the detection results. Inter-frame differencing was performed using a grayscale image sequence with a frame rate of 30 to obtain the moving target between two frames. Figure 4(a) shows a relatively good detection result due to less clutter interference (the pedestrian did not move significantly), but it only obtains the target position in the image sequence, completely losing temporal information between the two frames. Figure 4(b) shows a significantly worse result, failing to completely eliminate clutter interference.

[0075] It is evident that the embodiments of the present invention reduce false detections and false alarms while retaining three-dimensional temporal information.

Claims

1. A spatiotemporal clustering method for small target detection based on sparse event points, characterized in that, Includes the following steps: The raw data captured by the event camera is preprocessed to obtain preprocessed event data; Iterate through all event points in the event data, assign a weight value to each event point, and sort all event points in descending order according to the weight values; The average weight of the M% of event points ranked last is used as the clustering threshold. The event data is then clustered into nearest neighbors based on the clustering threshold to obtain preliminary detection results. The process of obtaining the preliminary test results includes: The event data is clustered into nearest neighbor clusters based on the clustering threshold to obtain clustering results that include the detection results. M represents the total number of event points included in the clustering result. If the clustering result includes multiple categories, the two-dimensional spatial coordinate deviation between each category and the other categories is calculated. If the two-dimensional spatial coordinate deviation does not exceed the two-dimensional position coordinates in the clustering result, uniform interpolation is performed at the break point to supplement the break trajectory and obtain the preliminary detection result. Point cloud filtering is applied to the preliminary detection results to obtain the final detection results.

2. The spatiotemporal clustering small target detection method for sparse event points according to claim 1, characterized in that, The specific implementation process of preprocessing the raw data captured by the event camera to obtain preprocessed event data includes: Remove four-dimensional event point coordinates from the original data The polarity p is used to downsample the coordinates of event points after removing the polarity, resulting in three-dimensional spatiotemporal point cloud data. The three-dimensional spatiotemporal point cloud data is the preprocessed event data; These are the two-dimensional position coordinates of the imaging pixels. Where N is the imaging time, and N is the number of event points in the original data.

3. The spatiotemporal clustering small target detection method for sparse event points according to claim 1, characterized in that, The process of preprocessing the raw data captured by the event camera includes: performing a downsampling operation on the raw data.

4. The spatiotemporal clustering small target detection method for sparse event points according to claim 1, characterized in that, The event data is sliced ​​to obtain multiple data segments, and each event point in each data segment is assigned a weight value; the event points of all data segments are sorted in descending order according to the weight values.

5. The spatiotemporal clustering method for small target detection based on sparse event points according to claim 1 or 4, characterized in that, Any event point weight value The calculation formula is: ;in For event points and event points The Euclidean distance between the farthest event points. Representative event point With the The Euclidean distance between the event points, where m is the total number of input events.

6. The spatiotemporal clustering small target detection method for sparse event points according to claim 5, characterized in that, Select event points whose weight values ​​are distributed below 50%, and calculate the average weight of the selected event points to obtain the adaptive clustering range. .

7. The spatiotemporal clustering small target detection method for sparse event points according to claim 1, characterized in that, The specific implementation process of point cloud filtering on the preliminary detection results includes: calculating the time span of each cluster category; if the time span of a certain cluster category does not exceed 10% of the point cloud time dimension, then the cluster category is deleted.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory; characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon; characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images