A Method and System for Detecting Faint Targets in Space Based on Spatiotemporal Point Cloud Representation by Event Cameras

Abstract

The application belongs to the technical field of target detection, and discloses a space dim and weak target detection method and system based on an event camera space-time point cloud representation, which comprises the following steps: obtaining a target preliminary detection result at the current moment based on an event frame image at the current moment, and modeling a space-time structure of a historical target event point cloud to predict the coordinate position of the historical target in the event frame image at the current moment; further, matching the predicted target with the preliminary detection target, supplementing a missed detection target to the preliminary detection result based on a matching result, filtering out false alarms, and outputting a current detection result; meanwhile, expanding the point cloud of the historical target for target prediction at the next moment; and based on the comprehensive judgment of the above preliminary detection result and prediction result, the comprehensive detection performance of the algorithm can be improved, so that the dim and weak target in space can be accurately identified, the event point cloud of the historical target is continuously updated and enhanced, which is beneficial to improving the prediction accuracy at the next moment, and then improving the detection accuracy of the target.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and more specifically, relates to a method and system for detecting spatially faint targets based on spatiotemporal point cloud representation by an event camera. Background Technology

[0002] Optoelectronic imaging detection of space targets is a key technology for applications such as space debris and on-orbit spacecraft detection. Space targets move at high speeds, are small in size, and have low energy, making them easily obscured by the starry background under complex space lighting conditions, thus hindering detection. For example, space debris generated by spacecraft collisions tumbles at high speeds, is small, does not emit its own light, and has low energy. Under space background and lighting conditions such as stellar flares and solar illumination, it appears as a faint, low-signal-to-noise ratio target on the optoelectronic imaging surface. Traditional optoelectronic frame cameras have low temporal resolution and narrow dynamic range, making it difficult to stably detect and track high-speed tumbling space debris under complex lighting and backgrounds such as backlighting or near the Earth's edge. Event cameras, with their advantages of fast response, wide dynamic range, and low power consumption, are a novel optoelectronic imaging sensor for detecting faint space targets and have broad application prospects.

[0003] Patents such as "CN202310239102.8 - Detecting Space Targets from Event Camera Voxel Data Based on Convolutional Neural Networks"; Wang Li's team at the Beijing Institute of Control Engineering (China Space Science and Technology, Vol. 44, No. 3, 2024) "Detecting Space Targets from Event Camera Data Streams Based on Spike Neural Networks"; and Gregory Cohen's team at Western Sydney University (Sensor Journal, 2020, vol. 20 no. 24) construct two-dimensional event frame representations along the time projection of the event stream to detect space targets. However, these detection methods are suitable for high signal-to-noise ratio space targets but cannot handle low signal-to-noise ratio, faint targets. This is because the intermittent triggering of faint space target events reduces their number, causing them to be submerged in background noise events, leading to missed detections and high false alarm rates.

[0004] Therefore, there is an urgent need to invent a method for detecting dim targets in space that is applicable to complex lighting conditions and background interference. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and system for detecting dim targets in space based on spatiotemporal point cloud representation of event camera. Its purpose is to improve the detection accuracy of targets, so as to be applicable to the detection of dim targets in space with complex lighting and background interference.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for detecting spatially faint targets based on spatiotemporal point cloud representation by an event camera is provided, comprising:

[0007] Initial target detection: Acquire the event stream of the event camera and construct event frame images. Based on the event frame images at the current moment, obtain the initial target detection results at the current moment and determine the initial target at the current moment.

[0008] Target prediction: Obtain the historical targets detected in the previous moment. For each historical target, establish a three-dimensional spatiotemporal structure model of its three-dimensional target event point cloud in the historical event stream up to the previous moment, and predict its position in the event frame image at the current moment based on the model to obtain the target prediction result at the current moment and determine the predicted target at the current moment.

[0009] Target verification and false alarm elimination: The predicted target is matched with the initial detection target. All predicted targets A are traversed. If an initial detection target B with a distance less than the distance threshold can be matched, the initial detection target B is output as the current detection result, and the target event point cloud used to generate the predicted target A is marked as an augmentable target event point cloud. Otherwise, the predicted target A is output as the current detection result, and the target event point cloud used to generate the predicted target A is marked as a target event point cloud that can be completed. If there are still unmatched initial detection targets after the traversal, the initial detection target whose reliability meets the preset requirements is selected from the target events based on the spatiotemporal distribution characteristics and output as the current detection result.

[0010] Event point cloud update: Based on the newly received event stream, the 3D target event point cloud of each historical target up to the previous time step is expanded to the current time step. For target event point clouds that can be enhanced, if the number of target event point clouds after expansion is less than the threshold, target events are randomly added during the expanded time step. For target event point clouds that can be completed, target events are randomly added during the expanded time step. The updated 3D target event point cloud is used for target prediction at the next time step.

[0011] Optionally, in the initial target detection, obtaining the initial target detection result at the current time includes:

[0012] The event stream is sliced ​​at set time intervals, and the event stream slice at the current moment is projected along the time axis to construct the event frame image at the current moment;

[0013] Binarize the event frame image at the current moment to obtain the event binary image;

[0014] Perform a convolution operation on the event binary map at the current time to obtain a spatial density feature map, and solve the segmentation threshold based on the standard deviation and mean of the spatial density feature map at the current time.

[0015] The spatial density feature map at the current time is binary segmented using a segmentation threshold. Pixel regions greater than or equal to the segmentation threshold are taken as the segmentation results. Connected component labeling is performed on the segmentation results to obtain the initial target detection result at the current time.

[0016] Optionally, in the target prediction, the distribution of the 3D target event point cloud is set to follow a linear model; for each historical target, the operation of predicting its position in the event frame image at the current moment includes the following steps:

[0017] Extract the target events of the historical target from the historical event stream and unfold them along the timeline to obtain the three-dimensional target event point cloud of the historical target;

[0018] The centroid of the three-dimensional target event point cloud is determined as a point in the linear model, and the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the three-dimensional target event point cloud is determined as the direction vector of the linear model, thereby determining the linear model;

[0019] The position of the historical target in the event frame image at the current moment is obtained based on the linear model.

[0020] Optionally, extracting the target event of the historical target from the historical event stream up to the previous moment includes moving a time window of fixed length to the previous moment and extracting the target event of the historical target from the historical event stream located within the time window.

[0021] Optionally, in the target prediction, if the predicted position of a historical target at the current moment exceeds the image range, the predicted target will not be used.

[0022] Optionally, in the target verification and false alarm rejection process, the operation of analyzing the spatiotemporal distribution characteristics of each initially detected target that was not matched to determine its reliability includes:

[0023] First, extract the target events of the initial detection target from the event stream within a certain period up to the current time and expand them along the time axis to obtain the three-dimensional target event point cloud of the initial detection target;

[0024] The reliability of the target is then determined based on the distribution characteristics of the three-dimensional target event point cloud of the initial detection target: if the difference in timestamps between adjacent event points in the target event point cloud is less than the preset time interval, and the number of target events in the target event point cloud is greater than the preset number, then the initial detection target is considered to be a real target and is output as the detection result at the current moment; otherwise, the initial detection target is considered to be noise and is not output.

[0025] Optionally, in the event point cloud update, for each three-dimensional target event point cloud to be expanded, the Euclidean distance between each target event in the three-dimensional target event point cloud and the axis center vector of the three-dimensional target event point cloud is calculated and averaged to obtain the average distance. When randomly supplementing target events within the expanded time period, target events are randomly supplemented within a spatial range with the corresponding point cloud axis center vector as the central axis and the average distance as the radius.

[0026] According to a second aspect of the present invention, a spatially faint target detection system based on spatiotemporal point cloud representation by an event camera is provided, comprising:

[0027] The target initial detection module is used to acquire the event stream of the event camera and construct event frame images. Based on the event frame images at the current moment, it acquires the target initial detection results at the current moment and determines the target to be detected at the current moment.

[0028] The target prediction module is used to acquire the historical targets detected in the previous moment. For each historical target, a three-dimensional spatiotemporal structure model is established for its three-dimensional target event point cloud in the historical event stream up to the previous moment. Based on the model, its position in the event frame image at the current moment is predicted to obtain the target prediction result at the current moment and determine the predicted target at the current moment.

[0029] The target verification and false alarm elimination module is used to match the predicted target with the initial detection target. It iterates through all predicted targets A. If a matching initial detection target B with a distance less than a distance threshold can be found, the initial detection target B is output as the current detection result, and the target event point cloud used to generate the predicted target A is marked as an augmentable target event point cloud. Otherwise, the predicted target A is output as the current detection result, and the target event point cloud used to generate the predicted target A is marked as a target event point cloud that can be completed. If there are still unmatched initial detection targets after the iteration, the initial detection target whose reliability meets the preset requirements is selected from them based on the spatiotemporal distribution characteristics of the target events and output as the current detection result.

[0030] The event point cloud update module is used to expand the 3D target event point cloud of each historical target up to the previous moment to the current moment based on the newly received event stream. For target event point clouds that can be enhanced, if the number of target event point clouds after expansion is less than a threshold, target events are randomly added during the expanded time period. For target event point clouds that can be completed, target events are randomly added during the expanded time period. The updated 3D target event point cloud is used for target prediction in the next moment.

[0031] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0032] According to a fourth aspect of the invention, a computer program product is provided, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method described in any of the preceding claims.

[0033] In summary, compared with the prior art, the technical solutions conceived in this invention have the following main advantages:

[0034] This invention performs initial target detection based on event frame images, obtaining initial detection results. It then incorporates historical detection results to analyze target motion patterns and predict the current position of historical targets. The initial detection results and prediction results are matched to identify true targets, missed targets, and uncertain targets requiring further confirmation. True and missed targets are considered as currently output detection results, while uncertain targets are further filtered based on the spatiotemporal distribution characteristics of target events to identify true targets as currently output detection results. This comprehensive judgment based on the initial and prediction results improves target detection accuracy, enabling accurate identification of faint targets in space. Furthermore, during target verification, the 3D target event point cloud of historical targets can be marked, distinguishing between augmentable and incomplete point clouds. The point cloud is augmented or incomplete during event point cloud updates and used for target prediction in the next moment, further improving prediction accuracy and thus target detection accuracy. Attached Figure Description

[0035] Figure 1 This is a flowchart of the steps of a spatially weak target detection method according to an embodiment of the present invention.

[0036] Figure 2 This is a signal flow diagram of a spatial faint target detection method according to an embodiment of the present invention.

[0037] Figure 3 This is a visualization of the initial detection process in one embodiment of the present invention, wherein (a) is an event frame image obtained by projecting the original event stream along the time axis, (b) is a spatial density feature map constructed by convolution, and (c) is an initial detection result image after threshold segmentation.

[0038] Figure 4 This is a schematic diagram of establishing a three-dimensional spatiotemporal structure model based on the three-dimensional target event point cloud of historical targets and performing motion prediction in one embodiment of the present invention, wherein (a) is the three-dimensional target event point cloud of historical targets, and (b) is a visualization diagram of the structure model and target prediction.

[0039] Figure 5This is an intermediate result diagram of the three-dimensional target event point cloud before and after enhancement in one embodiment of the present invention, wherein (a) is the spatiotemporal point cloud distribution before enhancement, and (b) is the spatiotemporal point cloud distribution after enhancement by randomly generating event points.

[0040] Figure 6 These are the results of the detection and tracking by the method of the present invention, where (a) to (e) are the targets detected at different times.

[0041] Figure 7 The results are the ground truth annotations for the EBSSA dataset, where (a) to (e) represent the targets at different times. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Example 1

[0044] This invention provides a method for detecting spatially faint targets based on spatiotemporal point cloud representation from an event camera, such as... Figure 1 The diagram shown is a flowchart of the steps of a spatially weak target detection method according to an embodiment of the present invention. Figure 2 The diagram shown is a signal flow diagram of a spatially faint target detection method according to an embodiment of the present invention. The steps therein are described in detail below.

[0045] S1. Obtain the event stream of the event camera and construct the event frame image. Based on the event frame image at the current moment, obtain the initial detection result of the target at the current moment and determine the initial detection target at the current moment.

[0046] Specifically, an event camera is used to capture images of spatial targets, obtaining the raw event stream. This event stream is then sliced ​​over time to construct event frame images at different times. Conventional target detection methods are then used to detect each event frame image, yielding initial detection results for the corresponding spatial targets. However, since detecting faint targets in space is challenging, the accuracy of the initial detection results is not high. Therefore, subsequent steps are needed to improve the detection accuracy.

[0047] In one embodiment, step S1 can be divided into sub-steps S11 to S14.

[0048] S11. Slice the event stream at set time intervals, project the event stream slice at the current moment along the time axis, and construct the event frame image at the current moment.

[0049] Specifically, slices are made at fixed time intervals Δt, for example, every 100ms. Let the starting time be t0, and the slices be made at t0, t1, t2, ..., t... k-1 t k Perform slicing, t k The event stream cut at time t k-1 ~t k The event flow within the time period will be t k-1 ~t k Projecting the event stream within a time period along the time axis can construct t k An event frame image at a given moment.

[0050] The event flow within each time period is recorded as E={e n |e n =(p n ,t n ,x n ,y n )}, where e n The timestamp is t n The event, p n For event e n The polarity of x n ,y n For event e n The coordinates are given. Projecting the event stream E along the time axis yields the event frame image F at the corresponding moment. The pixel value F(i,j) at coordinates (i,j) of the event frame image F is:

[0051] ;

[0052] In the formula, .

[0053] S12. Perform binarization on the event frame image at the current moment to obtain the event binary image.

[0054] Specifically, let the event binary graph be denoted as event binary graph B, and let the value B(i,j) at coordinate (i,j) of event binary graph B be:

[0055] .

[0056] S13. Perform a convolution operation on the event binary map at the current time to obtain a spatial density feature map. Solve for the segmentation threshold based on the standard deviation and mean of the spatial density feature map at the current time.

[0057] For example, a 3×3 convolution kernel can be used to convolve the binary image B to obtain the spatial density feature map f. The standard deviation σ and mean μ of the feature map f can be calculated, and the segmentation threshold τ of the spatial density feature map f can be calculated using the following formula:

[0058] ;

[0059] In the formula, λ is a preset hyperparameter, which is generally between 5 and 10.

[0060] S14. Use the segmentation threshold to perform binary segmentation on the spatial density feature map at the current time, take the pixel region that is greater than or equal to the segmentation threshold as the segmentation result, and mark the connected components of the segmentation result to obtain the initial detection result of the target at the current time.

[0061] like Figure 3 The diagram shown is a visualization of the initial detection process in one embodiment of the present invention, wherein (a) is an event frame image obtained by projecting the original event stream along the time axis, (b) is a spatial density feature map constructed by convolution, and (c) is an initial detection result diagram after threshold segmentation.

[0062] S2. Obtain the historical targets detected in the previous moment. For each historical target, perform a three-dimensional spatiotemporal structure model of its three-dimensional target event point cloud in the historical event stream up to the previous moment, and predict its position in the event frame image at the current moment based on the model. Obtain the target prediction result at the current moment and determine the predicted target at the current moment.

[0063] Specifically, this step involves establishing an xyt three-dimensional structural model of the target event point cloud in the historical event stream based on the detection results of the previous moment. This invention assumes that the event point cloud, from the previous moment to the current moment, presents a linear tubular structure. Therefore, the target event can be modeled and its movement position predicted based on a linear model. The possible coordinate positions of each target from the previous moment are predicted for the current moment and used as the predicted target for the current moment. Specifically, a matching model can be selected based on the distribution trend of the target event point cloud in the historical event stream. The selected model is then fitted based on the target event point cloud in the historical event stream to determine the model parameters, thereby determining the geometric model of the historical target. After determining the geometric model of the historical target, the image position corresponding to the historical target at the current moment can be directly solved based on the model. Solving for the position of each historical target in the current event frame image yields the predicted position of the historical target at the current moment, which is the predicted target for the current moment.

[0064] In one embodiment, the spatiotemporal distribution of the target event point cloud can be assumed to follow a linear model. In this case, predicting the position of each historical target in the current event frame image includes the following steps.

[0065] S21. Extract the target events of the historical target from the historical event stream and expand them along the time axis to obtain the three-dimensional target event point cloud of the historical target.

[0066] Specifically, the target events in the corresponding region can be extracted from the historical event stream based on the detection box of the historical target and expanded along the time axis to obtain the three-dimensional target event point cloud of the historical target.

[0067] In one embodiment, extracting the target event of a historical target from a historical event stream includes moving a time window of fixed length to the previous moment and extracting the target event of the historical target from the historical event stream located within the current time window. For example, the current moment is t. k If the length of the time window is three times the time interval, then the time window is moved to t. k-1 From t k-4 ~ t k-1 Extract the target event of the historical target from the historical event stream during the period.

[0068] S22. Solve for the centroid of the 3D target event point cloud as a point in the linear model, and solve for the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the 3D target event point cloud as the direction vector of the linear model to determine the linear model.

[0069] Specifically, the linear model can be denoted as:

[0070] ;

[0071] In the formula, p0 is a point in the linear model. In this invention, the centroid of the three-dimensional target event point cloud is taken as p0, and the coordinates of p0 in three-dimensional spacetime are p0=(x0,y0,t0). The direction vector of the straight line model is obtained by solving for the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the three-dimensional target event point cloud. , recorded as s is the independent variable in the linear model, and r(s) is the dependent variable in the linear model.

[0072] S23. Based on the linear model, the position of the historical target in the event frame image at the current moment is obtained.

[0073] Assume the current time is t. k The location of the historical target in the event frame image at the current moment is (x k ,y k If ), then its coordinates in three-dimensional spacetime are (x k ,y k , t k At this point, the independent variable s is denoted as si. k Substituting into the linear model, we get:

[0074] ;

[0075] Solving the model, we get:

[0076] ;

[0077] ;

[0078] ;

[0079] Therefore, it is possible to predict the position (x) of the historical target in the current event frame image. k ,y k ).

[0080] like Figure 4 The diagram shown is a schematic diagram of establishing a three-dimensional spatiotemporal structure model based on the three-dimensional target event point cloud of historical targets and performing motion prediction in an embodiment of the present invention, wherein (a) is the three-dimensional target event point cloud of historical targets, and (b) is a visualization diagram of the structure model and target prediction.

[0081] Understandably, if the predicted location of a historical target at the current moment exceeds the image range, the predicted target will not be used, meaning the predicted target will not be included in the prediction result.

[0082] S3. Match the predicted target with the initial detection target. Traverse all predicted targets A. If a matching initial detection target B with a distance less than the distance threshold can be found, output the initial detection target B as the current detection result and mark the target event point cloud used to generate the predicted target A as an augmentable target event point cloud. Otherwise, output the predicted target A as the current detection result and mark the target event point cloud used to generate the predicted target A as a target event point cloud that can be completed. If there are still unmatched initial detection targets after the traversal, select the initial detection target whose reliability meets the preset requirements based on the spatiotemporal distribution characteristics of the target events and output it as the current detection result.

[0083] This step involves using historical detection information to correct and supplement the current preliminary detection results, achieving complementary fusion between preliminary detection and prediction, thereby improving the accuracy of target detection.

[0084] Specifically, matching can be performed by calculating the Euclidean distance between the initially detected target location and the predicted target location, with a preset distance threshold τ. distance For example, τ distance =10 pixels, iterate through each predicted target A in the prediction results, and find the matching initial detection target from the initial detection results. There are two matching results as follows:

[0085] Scenario 1: The initial detection results show a location less than τ away from the currently predicted target A. distanceIf the initial detection target B is not a false alarm, it is determined that the initial detection target B is a real target and will not participate in the subsequent false alarm filtering operation. It will be directly used as the output detection result, and the target event point cloud used to generate the predicted target A will be marked as an augmentable target event point cloud.

[0086] The second scenario: The current preliminary detection result does not contain a target A whose distance is less than τ. distance If the initial detection target is not found, it is determined that the predicted target A was missed in the initial detection result at the current moment, and the predicted target A is output as the current detection result. At the same time, the target event point cloud used to generate the predicted target A is marked as a target event point cloud that can be completed.

[0087] Understandably, if the 3D target event point cloud of a certain historical target is marked as a point cloud that can be completed at multiple consecutive time points, it means that the historical target is likely no longer present in the current detection space, so the historical target can also be deleted.

[0088] After the traversal is completed, there may still be unmatched targets in the initial detection results. In this case, the targets that are judged as false alarms are further removed by the spatiotemporal distribution characteristics of the target events, and the real targets are retained as the output detection results.

[0089] Specifically, for each initially detected target that was not matched, the following steps are taken to determine its reliability by analyzing its spatiotemporal distribution characteristics.

[0090] The first step is to extract the target events of the initial target from the event stream up to the current time and expand them along the time axis to obtain the three-dimensional target event point cloud of the initial target.

[0091] For example, first move a time window of fixed length to the current moment, extract the target events of the initial detection target from the event stream located in the current time window and expand them along the time axis to obtain the three-dimensional target event point cloud of the initial detection target.

[0092] For example, the current time is t k If the length of the time window is three times the time interval, then the time window is moved to t. k From t k-3 ~ t k The target events of the initial detection target are extracted from the event stream during the period and expanded along the time axis to obtain the three-dimensional target event point cloud of the initial detection target.

[0093] Specifically, the target events in the corresponding area can be extracted from the historical event stream based on the detection box of the initial target and expanded along the time axis to obtain the three-dimensional target event point cloud of the initial target.

[0094] The second step is to determine the reliability of the target event point cloud based on its distribution characteristics, and then determine whether it is a false alarm. If the difference between the timestamps of adjacent event points in the target event point cloud is less than the preset time interval (e.g., 5ms), and the number of target events in the target event point cloud is greater than the preset number (e.g., 100), then the initial detection target is considered to be a real target and is output as the detection result at the current time. Otherwise, the initial detection target is considered to be a false alarm (i.e., noise) and no output is made.

[0095] In the above embodiments, the differences in the distribution of targets and noise in three-dimensional spatiotemporal point clouds are utilized to filter out false alarms in high-fidelity initial detection results and reduce the false alarm rate.

[0096] S4. Based on the newly received event stream, expand the 3D target event point cloud of each historical target up to the previous moment to the current moment. For target event point clouds that can be enhanced, if the number of target event point clouds after expansion is less than the threshold, target events are randomly added during the expanded time period. For target event point clouds that can be completed, target events are randomly added during the expanded time period. The updated 3D target event point cloud is used for target prediction at the next moment.

[0097] In one embodiment, when randomly supplementing target events within the extended time period, the spatial range of the supplemented target events is further defined. Specifically, for each 3D target event point cloud to be expanded, the Euclidean distance between each target event in the 3D target event point cloud and the axis center vector of the 3D target event point cloud is calculated and averaged to obtain the average distance. When randomly supplementing target events within the extended time period, the target events are randomly supplemented within a spatial range with the corresponding point cloud axis center vector as the central axis and the average distance as the radius.

[0098] For example, for any 3D target event point cloud used in target prediction, the steps for updating the point elements are as follows:

[0099] S41. Calculate each event point p. ei To the target event point cloud axis vector Euclidean distance d i And calculate all event points p in the point cloud. ei to the axis vector average distance d m =mean(d i );

[0100] Euclidean distance d i The calculation formula is:

[0101] ;

[0102] S42. If the 3D target event point cloud is an augmentable 3D target event point cloud, then extend it along the time axis to the current time, and determine the sparsity of the target event point cloud within the interval from the previous time to the current time. If the number of target event point clouds is lower than a certain threshold, then use the axis-centered vector... With d as the axis m Using a radius, randomly generate event points to enhance the current target event point cloud;

[0103] S42. If the 3D target event point cloud is a complete 3D target event point cloud, then the axis-centered vector is used as the reference point. With d as the axis m Using a radius, randomly generate target event points to complete the target events within the interval from the previous time to the current time.

[0104] like Figure 5 The figure shown is an intermediate result diagram of the three-dimensional target event point cloud before and after enhancement in an embodiment of the present invention, wherein (a) is the spatiotemporal point cloud distribution before enhancement, and (b) is the spatiotemporal point cloud distribution after enhancement by randomly generating event points.

[0105] This step involves expanding and supplementing the 3D target event point cloud of historical targets. For point clouds that can be enhanced, enhancement is performed only when their density is below a set threshold, and the historical target event point cloud is expanded to the current moment to maintain the continuity of target events in the temporal dimension. For point clouds that can be completed, missing target events are completed in the spatiotemporal dimension to ensure the integrity and continuity of target events. The enhanced and completed point cloud is then used for target prediction in the next moment, which can improve prediction accuracy, thereby improving the accuracy of target detection in the next moment.

[0106] The target event point cloud enhancement and completion strategy aims to expand the three-dimensional target event (xyt) along the time axis over time for star point target detection at subsequent time steps. At the same time, target tracking is achieved during the prediction-detection-expansion process, thereby realizing the fusion of the three tasks of prediction-detection-tracking.

[0107] This invention performs initial target detection based on event frame images, obtaining initial detection results. It then incorporates historical detection results to analyze target motion patterns and predict the current position of historical targets. The initial detection results and prediction results are matched to identify true targets, missed targets, and uncertain targets requiring further confirmation. True and missed targets are considered as currently output detection results, while uncertain targets are further filtered based on the spatiotemporal distribution characteristics of target events to identify true targets as currently output detection results. Based on the comprehensive judgment of the initial and prediction results, the false alarm rate and missed detection rate can be reduced, thus accurately identifying faint targets in space. Furthermore, during target verification and false alarm elimination, the 3D target event point cloud of historical targets can be marked, distinguishing between augmentable and incomplete point clouds. The point cloud is augmented or incomplete during event point cloud updates and used for target prediction in the next moment, further improving prediction accuracy and consequently target detection accuracy.

[0108] This invention utilizes the prediction information of historical detected targets to fuse and enhance the current preliminary detection results, addressing the problem of missed target detection caused by intermittent triggering events of faint targets. For space debris that flies at high speed and tumbles with fluctuating reflectivity, the above prediction-detection fusion strategy can achieve high-precision and stable detection of debris targets.

[0109] Example 2

[0110] The present invention also relates to a spatial faint target detection system based on spatiotemporal point cloud representation of an event camera, which includes a target initial detection module, a target prediction module, a target verification module and an event point cloud update module.

[0111] The target initial detection module is used to acquire the event stream of the event camera and construct event frame images. Based on the event frame images at the current moment, it acquires the target initial detection results at the current moment and determines the target to be detected at the current moment.

[0112] The target prediction module is used to acquire the historical targets detected in the previous moment. For each historical target, a three-dimensional spatiotemporal structure model is established for its three-dimensional target event point cloud in the historical event stream up to the previous moment. Based on the model, its position in the event frame image at the current moment is predicted to obtain the target prediction result at the current moment and determine the predicted target at the current moment.

[0113] The target verification and false alarm elimination module is used to match the predicted target with the initial detection target. It iterates through all predicted targets A. If a matching initial detection target B with a distance less than the distance threshold can be found, the initial detection target B is output as the current detection result, and the target event point cloud used to generate the predicted target A is marked as an augmentable target event point cloud. Otherwise, the predicted target A is output as the current detection result, and the target event point cloud used to generate the predicted target A is marked as a target event point cloud that can be completed. If there are still unmatched initial detection targets after the iteration, the initial detection target whose reliability meets the preset requirements is selected from them based on the spatiotemporal distribution characteristics of the target events and output as the current detection result.

[0114] The event point cloud update module is used to expand the 3D target event point cloud of each historical target up to the previous moment to the current moment based on the newly received event stream. For target event point clouds that can be enhanced, if the number of target event point clouds after expansion is less than a threshold, target events are randomly added during the expanded time period. For target event point clouds that can be completed, target events are randomly added during the expanded time period. The updated 3D target event point cloud is used for target prediction in the next moment.

[0115] Understandably, the above modules can implement the corresponding steps in the spatially dim target detection method in Example 1. For details, please refer to the description in Example 1, which will not be repeated here.

[0116] The following will illustrate this with specific examples.

[0117] The EBSSA dataset is a real-world dataset of space objects such as planets and stars collected by Gregory Cohen's team at Western Sydney University using an event camera. It is manually labeled at 10ms intervals and used for space object detection and tracking tasks. The method of this invention was tested on this publicly available dataset to verify its effectiveness. Specific experimental results are as follows: Figure 6 and Figure 7 As shown, where Figure 6 The results of the detection and tracking by the method of the present invention are shown in (a) to (e), where (a) to (e) represent the targets detected at different times. Figure 7 The table shows the ground truth annotations for the EBSSA dataset, where (a) to (e) represent targets at different times. Experimental results demonstrate that the method of this invention can effectively detect moving targets in space, and has excellent detection and tracking capabilities for both slow-moving and fast-moving targets.

[0118] Example 3

[0119] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0120] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0121] Example 4

[0122] This invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this invention.

[0123] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" are intended to illustrate the present invention and are not intended to limit the present invention.

[0124] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A spatial dim and weak target detection method based on event camera spatiotemporal point cloud representation, characterized in that, Comprise: Target preliminary detection: acquire event stream of event camera and build event frame image, acquire target preliminary detection result of current moment based on event frame image of current moment, determine preliminary detection target of current moment; Target prediction: acquire historical target detected in last moment, for each historical target, build three-dimensional space-time structure model for three-dimensional target event point cloud of the historical target in historical event stream up to last moment, and predict its position in event frame image of current moment based on the model, obtain target prediction result of current moment, determine prediction target of current moment; Target verification and false alarm elimination: match the prediction target with the preliminary detection target, traverse all prediction targets A, if a preliminary detection target B with distance less than distance threshold can be matched, the preliminary detection target B is taken as current detection result output, and the target event point cloud used to generate the prediction target A is marked as enhanced target event point cloud; otherwise, the prediction target A is taken as current detection result output, and the target event point cloud used to generate the prediction target A is marked as complemented target event point cloud; if there are still unmatched preliminary detection targets after traversal, select the preliminary detection target with reliability meeting preset requirement as current detection result output through space-time distribution characteristics of target event; Event point cloud update: according to newly received event stream, expand three-dimensional target event point cloud of each historical target up to last moment to current moment, for enhanced target event point cloud, if the number of target event point cloud after expansion is less than threshold, randomly supplement target event in the expanded period, for complemented target event point cloud, also randomly supplement target event in the expanded period, use updated three-dimensional target event point cloud for target prediction of next moment.

2. The space dim target detection method of claim 1, wherein, In the target preliminary detection, acquiring target preliminary detection result of current moment comprises: Slice the event stream at a set time interval, project the event stream slice of current moment along time axis, build event frame image of current moment; Carry out binaryzation processing on event frame image of current moment, obtain event binary image; Carry out convolution operation on event binary image of current moment, obtain spatial density feature map, solve segmentation threshold according to standard deviation and mean value of spatial density feature map of current moment; Carry out binary segmentation on spatial density feature map of current moment by using segmentation threshold, take pixel region greater than or equal to the segmentation threshold as segmentation result, carry out connected domain labeling on the segmentation result, obtain target preliminary detection result of current moment.

3. The space dim target detection method of claim 1, wherein, In the target prediction, set that distribution of three-dimensional target event point cloud follows straight line model; for each historical target, the operation of predicting its position in event frame image of current moment comprises the following steps: Extract target event of the historical target from historical event stream and expand along time axis, obtain three-dimensional target event point cloud of the historical target; Solve the centroid of the three-dimensional target event point cloud as a point in the straight line model, solve the eigenvector corresponding to the maximum eigenvalue of the covariance matrix of the three-dimensional target event point cloud as the direction vector of the straight line model, determine the straight line model; Based on the straight line model, a position of the historical target in a current time event frame image is obtained.

4. The space dim target detection method of claim 3, wherein, Target events of the historical target are intercepted from a historical event stream up to a last time, including moving a time window with a fixed length to the last time, and intercepting the target events of the historical target from the historical event stream within the time window.

5. The method of claim 1, wherein, In the target prediction, if a predicted position of the historical target at the current time exceeds an image range, the predicted target is not adopted.

6. The space dim target detection method of claim 1, wherein, In the target verification and false alarm elimination, for each primary detection target that is not matched, an operation of analyzing a time and space distribution characteristic of the primary detection target to determine reliability of the primary detection target includes: first, target events of the primary detection target are intercepted from an event stream in a period of time up to the current time and are expanded along a time axis to obtain a three-dimensional target event point cloud of the primary detection target; then, the reliability of the primary detection target is determined according to a distribution characteristic of the three-dimensional target event point cloud of the primary detection target: if a time stamp difference between adjacent event points in the target event point cloud is less than a preset time interval, and a number of target events in the target event point cloud is greater than a preset number, the primary detection target is considered to be a real target and is output as a detection result at the current time; otherwise, the primary detection target is considered to be noise and is not output.

7. The method of claim 1, wherein the step of detecting the spatial dim target comprises the steps of: determining a first intensity of the first target; determining a second intensity of the second target; and determining a third intensity of the third target. In the event point cloud updating, for each three-dimensional target event point cloud to be expanded, an average distance is obtained by calculating and averaging a Euclidean distance between each target event in the three-dimensional target event point cloud and an axis vector of the three-dimensional target event point cloud, and target events are randomly supplemented in a space range with the corresponding point cloud axis vector as a central axis and with the average distance as a radius in an expanded period.

8. A spatial dim and weak target detection system based on event camera spatiotemporal point cloud representation, characterized in that, The method includes: a target primary detection module, configured to acquire an event stream of an event camera and construct an event frame image, acquire a target primary detection result at a current time based on the event frame image at the current time, and determine primary detection targets at the current time; a target prediction module, configured to acquire historical targets detected at a last time, for each historical target, establish a three-dimensional time and space structure model for a three-dimensional target event point cloud of the historical target in a historical event stream up to the last time, and predict a position of the historical target in an event frame image at the current time based on the model to obtain a target prediction result at the current time and determine predicted targets at the current time; a target verification and false alarm elimination module, configured to match the predicted targets and the primary detection targets, traverse all predicted targets A, if a primary detection target B with a distance less than a distance threshold can be matched to the predicted target A, output the primary detection target B as a current detection result, and mark a target event point cloud used to generate the predicted target A as an enhanced target event point cloud; otherwise, output the predicted target A as the current detection result, and mark a target event point cloud used to generate the predicted target A as a complemented target event point cloud; if there are still primary detection targets that are not matched after the traversal, select a primary detection target with reliability meeting a preset requirement from the primary detection targets as the current detection result output through a time and space distribution characteristic of a target event. An event point cloud updating module is configured to, according to a newly received event stream, expand a three-dimensional target event point cloud of each historical target up to a previous time instant to a current time instant, for an augmentable target event point cloud, if a quantity of the expanded target event point cloud is less than a threshold, randomly supplement target events in an expanded time period, and for a completable target event point cloud, also randomly supplement target events in the expanded time period, and use the updated three-dimensional target event point cloud for target prediction at a next time instant.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 7.

Images