An event camera-based target detection and tracking method, device and electronic equipment

Abstract

The application discloses a target detection tracking method and device based on an event camera and electronic equipment, and belongs to the technical field of target detection. The target detection tracking method comprises the following steps: matching and detecting a target preliminary detection result at a current moment with a target tracking result at a previous moment to determine whether a new category appears; and performing data association on multiple targets to realize tracking of the multiple targets. The new category can be accurately found out in a complex scene such as track intersection and target occlusion, and tracking can be performed. Further, the position of a target instance after target tracking is predicted, the prediction information of the target is matched with target detection information at a next moment, the target detection area is redrawn according to a matching result, and then the target area is reduced, the multiple target intersection time is reduced, and the target tracking robustness is improved.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and more specifically, relates to a target detection and tracking method, apparatus and electronic device based on an event camera. Background Technology

[0002] Space multi-target detection and tracking technology is crucial for key applications such as space debris monitoring and on-orbit spacecraft identification. On one hand, the complex lighting environment in space makes small space targets, such as space debris, easily obscured by the starry background, leading to missed detections and issues like discontinuities and drift in trajectory data, increasing the complexity of multi-target tracking. On the other hand, the flight direction of space debris is highly uncertain, its trajectory exhibits nonlinear characteristics, and phenomena such as trajectory intersections, target convergence, or short-term occlusion further exacerbate the difficulty of identifying different target trajectories, making stable and continuous target matching and association difficult. Traditional frame-based optical cameras, due to their low temporal resolution and sparse sampling of continuous space target motion, struggle to capture transient motion processes such as multi-target convergence. Simultaneously, their narrow dynamic range makes targets easily missed under complex lighting and background interference, rendering them unsuitable for space target detection and tracking tasks. In contrast, event cameras, with their advantages of fast response, wide dynamic range, and low power consumption, represent a novel optoelectronic imaging sensor for addressing space multi-target detection and tracking tasks, with broad application prospects.

[0003] Because traditional frame cameras have low temporal resolution, they cannot capture the transient motion process when multiple targets intersect, which makes it easy for problems such as tracking ID switching and trajectory breakage to occur when multiple targets intersect. Furthermore, existing detection and tracking methods based on traditional frame cameras are often suitable for tracking a small number of targets with simple motion trajectories and no intersection, but cannot handle multi-target tracking in complex scenarios such as trajectory intersection and target occlusion. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a target detection and tracking method, device and electronic device based on an event camera. Its purpose is to solve the technical problem that the existing detection and tracking methods cannot handle multi-target tracking in complex scenarios such as trajectory intersection and target occlusion.

[0005] To achieve the above objectives, according to one aspect of the present invention, a target detection and tracking method based on an event camera is provided, comprising:

[0006] S1: Slice the raw event stream captured by the event camera to obtain a subset of events at each moment;

[0007] S2: Obtain the target detection feature map corresponding to the event subset at each time step;

[0008] S3: Traverse the event subsets at each time step, perform preliminary detection on the current target detection feature map to obtain the target preliminary detection result at the current time step; if the current time step is the initial time step, proceed to S4 step; otherwise, proceed to S5 step.

[0009] S4: Initialize the global tracking cache using the initial target detection result at the initial time, and use the initial target detection result at the initial time as the target tracking result at the initial time;

[0010] S5: Match the current target initial detection result with the target tracking result of the previous time to determine whether a new category has appeared; if so, add the new category to the target category in the global tracking cache of the previous time, and update the target trajectory of the existing category in the global tracking cache of the previous time to obtain the target tracking result of the current time; otherwise, update the trajectory of the target of the existing category in the global tracking cache of the previous time to obtain the target tracking result of the current time; then proceed to S6;

[0011] S6: Perform position prediction for each target instance recorded in the global tracking cache at the current moment, determine the target region at the next moment based on the position prediction result, fuse and update the target region at the next moment with the target detection feature map corresponding to the event subset at the next moment, and then return to S3, until the event subsets at each moment have been traversed.

[0012] Further, S1 includes: slicing the raw event stream acquired by the event camera at set time intervals to obtain event data at each moment, and dividing the event data at the current moment into a positive polarity event subset and a negative polarity event subset according to polarity attributes.

[0013] Further, S2 includes:

[0014] S21: Extract features from the event frame images corresponding to the positive and negative polarity event subsets at the current moment to obtain positive polarity segmentation feature maps and negative polarity segmentation feature maps;

[0015] S22: The intersection of the positive polarity segmentation feature map and the negative polarity segmentation feature map is taken as the target detection feature map at the current moment.

[0016] Further, S21 includes:

[0017] The positive and negative event subsets at the current moment are projected along the time axis to construct corresponding positive and negative event images;

[0018] The positive and negative event images are normalized and filtered, and then an adaptive threshold segmentation operation is used to obtain the positive and negative segmentation feature maps respectively.

[0019] Further, the step of matching the initial target detection result at the current moment with the target tracking result at the previous moment to determine whether a new category has appeared includes:

[0020] The target category and its instances in the current target initial detection result are compared with the target category and its instances in the previous target tracking result to determine whether the new category has appeared.

[0021] Further, the step of matching the initial target detection result at the current moment with the target tracking result at the previous moment to determine whether a new category has appeared includes:

[0022] The target category and its instance in the current target initial detection result are matched with all target instances that have completed target tracking in the target tracking result of the previous time using Euclidean distance. If the Euclidean distance is lower than the first preset value, it is considered to have passed the detection.

[0023] For target categories that fail to be matched by Euclidean distance, a matching strategy based on linear equations is introduced to determine the degree of association between them and all target tracking instances in the target tracking results of the previous time step. If the degree of association is lower than a second preset value, they are regarded as the new category.

[0024] Further, the step of fusing and updating the target region at the next time step with the target detection feature map corresponding to the event subset at the next time step includes:

[0025] Define a zero-based feature map with the same size as the target detection feature map at the next time step, and determine whether the position prediction result is in the target detection segmentation feature map at the next time step;

[0026] If not, no action will be taken;

[0027] If so, determine whether there are multiple location prediction results corresponding to the same target area; if not, use the center coordinates of the target area corresponding to the location prediction result as the center, draw a circular area with a radius of a preset number of pixels on the all-zero feature map and fill it; if so, use each of the location prediction results as the center, draw a circular area with a radius of a preset number of pixels on the all-zero feature map and fill it.

[0028] To achieve the above objectives, according to one aspect of the present invention, a target detection and tracking device based on an event camera is provided, comprising:

[0029] The slicing module is used to slice the raw event stream captured by the event camera to obtain a subset of events at each moment;

[0030] The traversal module is used to obtain the target detection feature maps corresponding to the event subsets at each time step;

[0031] The initial detection module is used to traverse the event subsets at each time point, perform initial target detection on the current target detection feature map to obtain the initial target detection result at the current time point; if the current time point is the initial time point, then enter the initialization module; otherwise, enter the loop module.

[0032] An initialization module is used to initialize the global tracking cache using the initial target detection result at the initial time, and to use the initial target detection result at the initial time as the target tracking result at the initial time.

[0033] The loop module is used to match the target initial detection result at the current moment with the target tracking result at the previous moment to determine whether a new category has appeared. If a new category has appeared, it is added to the target category in the global tracking cache at the previous moment, and the target trajectory of the existing category in the global tracking cache at the previous moment is updated to obtain the target tracking result at the current moment. Otherwise, the trajectory of the target of the existing category in the global tracking cache at the previous moment is updated to obtain the target tracking result at the current moment. Then, the module continues to update.

[0034] The update module is used to predict the location of each target instance recorded in the global tracking cache at the current time, determine the target region at the next time based on the location prediction result, fuse the target region at the next time with the target detection feature map corresponding to the event subset at the next time and update it, and then return to the initial detection module until the event subsets at each time have been traversed.

[0035] To achieve the above objectives, according to one aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the event camera-based target detection and tracking method.

[0036] To achieve the above objectives, according to one aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the event-camera-based target detection and tracking method.

[0037] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0038] (1) This invention provides a target detection and tracking method based on an event camera. The method matches the initial target detection result at the current moment with the target tracking result at the previous moment to determine whether a new category has appeared. Data association is performed on multiple targets to achieve multi-target tracking. This method can accurately identify new categories and perform tracking in complex scenarios such as trajectory intersection and target occlusion. Furthermore, the method predicts the position of the target instance after target tracking and matches the predicted target information with the target detection information at the next moment. The target detection area is redrawn according to the matching result, thereby reducing the target area, reducing the multi-target intersection time, and improving the robustness of target tracking.

[0039] (2) This scheme proposes a point target detection method guided by event polarity. Event time images are constructed for positive and negative polarity event subsets respectively. The intensity of the target is reflected by the accumulation of timestamps. The target region is segmented by an adaptive threshold method. The intersection of the segmentation feature maps of positive and negative polarities is taken to obtain the final target detection result. For the first time in the traditional event target detection process, the polarity information of the event is introduced, which can effectively reduce the interference of event noise on point target detection and improve the target detection accuracy.

[0040] (3) This scheme introduces Euclidean distance matching mechanism and direct distance matching mechanism to realize multi-level data association between the detected target and the tracking trajectory, which can improve the tracking performance of the algorithm in complex scenarios such as target missed detection and short-term occlusion, and effectively alleviate problems such as ID duplicate allocation and trajectory breakage.

[0041] (4) This scheme determines whether there are multiple location prediction results corresponding to the same target area, and then redraws the corresponding target areas respectively, which can effectively avoid the situation of the target being followed incorrectly. Attached Figure Description

[0042] Figure 1 This is a flowchart of the target detection and tracking method based on an event camera provided in Embodiment 1 of the present invention.

[0043] Figure 2 This is a signal flow diagram of the target detection and tracking method based on an event camera provided in Embodiment 1 of the present invention.

[0044] Figure 3 This is a comparison image before and after the target feature map is redrawn using the prediction information of the target detection and tracking method based on the event camera in Embodiment 1 of the present invention.

[0045] Figure 4 This is an example of the visualization of the detection and tracking results of different numbers of targets using the target detection and tracking method based on an event camera in Embodiment 2 of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides a target detection and tracking method based on an event camera, including: S1-S6. (As...) Figure 2 The diagram shown is a signal flow diagram of the target detection and tracking method based on an event camera in Embodiment 1 of the present invention.

[0049] S1: Slice the raw event stream captured by the event camera to obtain a subset of events at each moment.

[0050] S2: Obtain the target detection feature map corresponding to the event subset at each time step.

[0051] S3: Traverse the event subsets at each time step, perform a preliminary detection on the current target detection feature map to obtain the target preliminary detection result at the current time step; if the current time step is the initial time step, proceed to S4 step, otherwise proceed to S5 step.

[0052] S4: Initialize the global tracking cache using the initial target detection result at the initial time, and use the initial target detection result at the initial time as the target tracking result at the initial time.

[0053] S5: Match the current target initial detection result with the target tracking result of the previous time to determine whether a new category has appeared; if so, add the new category to the target category in the global tracking cache of the previous time, and update the target trajectory of the existing category in the global tracking cache of the previous time to obtain the target tracking result of the current time; otherwise, update the target trajectory of the existing category in the global tracking cache of the previous time to obtain the target tracking result of the current time; then proceed to S6.

[0054] S6: Perform position prediction for each target instance recorded in the global tracking cache at the current moment, determine the target region at the next moment based on the position prediction result, fuse and update the target region at the next moment with the target detection feature map corresponding to the event subset at the next moment, and then return to S3, until the event subsets at each moment have been traversed.

[0055] As an optional implementation, S1 includes: slicing the raw event stream acquired by the event camera at set time intervals to obtain event data at each moment, and dividing the event data at the current moment into a positive polarity event subset and a negative polarity event subset according to polarity attributes.

[0056] The original event stream is sliced ​​at set time intervals, and the event data within the slice is divided into two sets of event subsets based on the polarity attribute of the event.

[0057] Specifically, slices are made at fixed time intervals Δt, for example, every 40ms. Let the starting time be t0, and the slices be made at t0, t1, t2, ..., t... k-1 t k Perform slicing, t k The event stream cut at time t k-1 ~t k The event flow within a time slice. Further, the event data within the slice is grouped by polarity, dividing it into positive and negative polarity event subsets. The event flow within each time slice is denoted as:

[0058] E = {e n |e n = (p n , t n , x n , y n )},

[0059] Among them, e n For timestamp t n Events are triggered at any time, p n For event e n polarity (p) n = 1 indicates a positive polarity event, p n = 0 indicates a negative polarity event), x n y n Event e n The horizontal and vertical coordinates are given. The positive and negative polarity event subsets are denoted as PE and NE, respectively, and can be further represented as:

[0060] E = {PE, NE}, PE = {e n |e n = (1, t n , x n , y n )},NE = {e n |e n = (0, t n , x n ,y n )}.

[0061] Then, positive and negative polarity event frame images are constructed for the positive and negative polarity event subsets according to the timestamp attribute. Threshold segmentation is performed based on the event frame images to achieve initial target detection. The intersection of the positive and negative polarity initial detection feature maps is taken to obtain the final target detection result, which is used for subsequent target tracking.

[0062] As an optional implementation, S2 includes: S21: extracting features from the event frame images corresponding to the positive polarity event subset and the negative polarity event subset at the current moment to obtain a positive polarity segmentation feature map and a negative polarity segmentation feature map; S22: taking the intersection of the positive polarity segmentation feature map and the negative polarity segmentation feature map as the target detection feature map at the current moment.

[0063] Specifically, time-series images are constructed for subsets of positive and negative polarity events to detect target regions. The two detection results are then fused to improve target detection accuracy. In this embodiment, target events often involve both positive and negative polarity events; that is, the target location contains both positive and negative polarity events. Conversely, noise events exhibit random polarity and do not have accompanying polarity characteristics. Based on this finding, a "divide and conquer" approach is adopted to detect target regions separately for positive and negative polarity events, and finally, the detection results are fused to achieve high-precision detection.

[0064] The final target feature map is obtained by intersecting the target segmentation feature maps generated from subsets of positive and negative polarity events. Finally, connected component labeling is used to obtain the target detection result at the current time. Specifically, the positive and negative polarity target segmentation feature maps are labeled as pf s and nf s Then the operation of taking the intersection of the two can be expressed as:

[0065] ,

[0066] In the formula, (i, j) are the pixel coordinates in the feature map, and f s The final target feature map is obtained by intersecting the segmentation feature maps of positive and negative polarity targets. Finally, connected component analysis is used to analyze the segmentation result f. s Morphological processing is performed to extract all candidate target regions. By calculating the geometric properties such as the bounding box, area, and pixel mask of each connected region, the target can be accurately located and its features described, providing structured input for subsequent tasks such as target tracking or behavior recognition.

[0067] As an optional implementation, S21 includes: projecting the positive and negative event subsets at the current moment along the time axis to construct positive and negative event images; normalizing and filtering the positive and negative event images, and then using an adaptive threshold segmentation operation to obtain the positive and negative segmentation feature maps.

[0068] Specifically, a time image is constructed by projecting a subset of events with positive and negative polarities along the time axis, and then normalized to the interval [0, 255]. Specifically, projecting event streams PE and NE along the time axis yields time images PF and NF at the corresponding times, respectively. The pixel value PF(i, j) at coordinate (i, j) in time image PF is:

[0069]

[0070] In the formula, The pixel value NF(i, j) at coordinate (i, j) in the time-matrix image NF is:

[0071] ,

[0072] In the formula, .

[0073] Since the timestamp t of an event is often quite large and may accumulate, a normalization operation is introduced to prevent the pixel values ​​in the PF / NF of the time image from being too large:

[0074] ,

[0075] ,

[0076] In the formula, PF norm and NF norm These are the normalized positive and negative polarity time images, respectively.

[0077] Median filtering is used to filter the normalized time image to remove random noise, and adaptive threshold segmentation is used to obtain the target segmentation feature map.

[0078] Specifically, due to the high sensitivity of event cameras, a large number of noisy events, especially thermal noise, inevitably exist in the event stream. This noise often exhibits scattered and randomly distributed characteristics, easily interfering with target detection results. Therefore, in the normalized temporal image PF... norm / NF normMedian filtering is introduced to effectively suppress isolated noise points through local neighborhood statistics while preserving the structural information of the target region, thereby improving the robustness of subsequent segmentation. The positive and negative polarity feature maps after median filtering are denoted as pf and nf, respectively. Then, the standard deviation σ and mean μ of the feature maps are calculated, and the segmentation threshold τ of the spatial density feature map f is calculated using the following formula.

[0079] τ = μ +σλ;

[0080] In the formula, λ is a preset hyperparameter, typically ranging from 5 to 10. The feature map pf / nf is segmented using a segmentation threshold to obtain the target segmentation feature map.

[0081] Regarding S4, in the initial stage of target tracking, the global tracking cache is initialized based on the target detection results; in the subsequent target tracking stages, the target detection results at the current moment are matched with the target tracking results at the previous moment at multiple levels to achieve multi-target data association.

[0082] Specifically, this step first initializes target category information based on the target detection results of each frame, assigning a unique tracking ID to each initial target. Next, an Euclidean distance matching algorithm is used to match the currently detected target with existing tracking results. This method is based on spatial similarity, determining whether a target in the current frame corresponds to a known target in the previous frame. For target detection results that fail Euclidean distance matching, this invention assumes that the trajectory of each spatial target is a straight line within a short time, and then introduces a straight-line distance determination mechanism for further processing. If a detection result cannot be matched by either Euclidean distance or straight-line distance, it is considered a newly emerging target and inserted into the cache with a new target category.

[0083] In Example 1, step S4 can be divided into sub-steps S41 to S44.

[0084] S41. After completing the target detection of the first frame and before entering the target tracking stage, initialize the global tracking information based on the target detection results of the first frame and record the detection results in the global tracking cache.

[0085] Specifically, firstly, a unique tracking identifier (tracking ID) is assigned to each target detected in the first frame. These IDs are used to distinguish different target identities and maintain consistency throughout the tracking process. Secondly, for each detected target, its corresponding bounding box information is stored, including the coordinates of the top left corner and width and height parameters. Thirdly, the center position of each target is calculated and recorded based on the bounding box information. The calculation method is as follows: the x-coordinate of the center point is cx = x + w / 2, and the y-coordinate is cy = y + h / 2, where (x, y) is the coordinate of the top left corner of the detection box, and (w, h) is the width and height of the detection box. Finally, a set of initial trajectory line parameters is assigned to each initialized target category. The parameter values ​​are set to [0, 0], which can represent the slope and intercept of the straight line model.

[0086] S42. Match the target detection results of each frame (starting from the second frame) with all class instances that have completed target tracking in the previous frame using Euclidean distance to achieve continuous temporal association and trajectory continuation of the target. This matching process is based on a minimum Euclidean distance strategy.

[0087] Specifically, for each target detection result in the current frame, the Euclidean distance between it and the latest detection coordinates of each target category instance in the global tracking cache is calculated sequentially. The Euclidean distance is calculated based on the target center point coordinates, in the form:

[0088]

[0089] In the formula, (x0, y0) are the center coordinates of the target detected in the current frame; (x1, y1) are the latest detected coordinates of each target category instance in the global tracking cache. If the distance d1 is less than the algorithm's preset matching threshold (e.g., 10 pixels), the two targets are considered to belong to the same category instance, and a successful match can be considered. At the same time, the detected target is added to the corresponding category instance in the global tracking cache and assigned the same tracking ID.

[0090] S43. For detected targets that fail to pass the Euclidean distance matching, this invention assumes that the trajectory of each spatial target is a straight line in a short time, and further introduces a matching strategy based on the straight line equation to enhance the fault tolerance of the target tracking algorithm in complex tracking scenarios, and realize multi-level data association between the detected target and the tracking trajectory.

[0091] Specifically, for all target detection results that fail to match Euclidean distance, the distance between each result and each trajectory line equation is calculated sequentially (note that this line equation is the trajectory line equation corresponding to all category instances in the global tracking cache that have not yet had new targets added). This distance is represented in the form of point-to-line geometric distance, and the detailed calculation formula is as follows:

[0092]

[0093] In the formula, d2 is the distance from point (x, y) to the line; a and b are the parameters of the line equation. If this distance is less than the algorithm's preset threshold (e.g., 5 pixels), the detection result is considered to be highly consistent with the trajectory trend of the current category instance, belonging to the same target category instance, and thus the line distance match is deemed successful. At the same time, the bounding box, center point coordinates, and other information of the currently detected target are added to the corresponding category instance in the global tracking cache, and a tracking ID for that category is assigned.

[0094] S44. For a current target that has not passed both Euclidean distance matching and linear distance matching, the algorithm treats it as a "newly appearing target" and assigns a new tracking ID to the target (the value of which is the current global maximum tracking ID plus 1). The corresponding box bounding box, center point coordinates, line initial parameters and other information are inserted into the global tracking cache as a new category instance.

[0095] In this step, the linear equation corresponding to each category instance is obtained by fitting the equation to several target locations (i.e., the coordinates of the target's center point) within that category instance using the least squares method. The linear equation is expressed by the following formula:

[0096] ,

[0097] In the formula, a and b are the parameters of the linear equation. For each newly added category instance, the parameters a and b are initially set to [0,0]. When a target of a certain category (i.e., the target corresponding to the same tracking ID) is detected and successfully associated at multiple consecutive time points, the number of its historical center points gradually accumulates. Once this number exceeds a certain threshold (set to 10 in this invention), it is considered that the target's trajectory information has sufficient temporal samples, at which point the least squares method can be used to perform linear fitting on the center coordinate distribution.

[0098] To maintain the timeliness and dynamism of the linear model, a fitting operation is automatically triggered whenever a new target is added to a category instance. The parameters a and b of the linear equation are recalculated using all historical center coordinates in the current category instance as input, and are synchronously recorded in the global tracking information cache corresponding to that target category.

[0099] Regarding S5, based on the target tracking results at the current moment, the data in the global tracking cache is updated, and resource reuse is achieved at the same time.

[0100] Specifically, after all target detection results in the current frame have undergone Euclidean distance matching and trajectory fitting matching, the algorithm ensures that each detected target is successfully associated with a tracking category and assigned a unique tracking ID. At this point, each detection result, along with its corresponding bounding box, center point coordinates, fitting parameters, and other information, will be completely inserted into the global tracking information cache of the corresponding category. Simultaneously, to ensure the temporal continuity and global data consistency of the entire multi-target tracking algorithm, a necessary maintenance and update operation must be performed on the existing global tracking information before processing the target data of the next frame.

[0101] As an optional implementation, the step of matching the target initial detection result at the current moment with the target tracking result at the previous moment to determine whether a new category has appeared includes: performing multi-level matching detection on the target category and its target instance in the target initial detection result at the current moment with the target category and its target instance in the target tracking result at the previous moment to determine whether the new category has appeared.

[0102] Further, the step of matching the current target preliminary detection result with the previous target tracking result to determine whether a new category has appeared includes: performing Euclidean distance matching between the target category and its instances in the current target preliminary detection result and all target instances that have completed target tracking in the previous target tracking result; if the Euclidean distance is lower than a first preset value, it is considered to have passed the detection. For target categories that fail the Euclidean distance matching, a matching strategy based on linear equations is introduced to determine the degree of association between them and all category instances that have completed target tracking in the previous target tracking result; if the degree of association is lower than a second preset value, it is considered to be the new category.

[0103] In Example 1, step S5 can be divided into sub-steps S51 to S53.

[0104] S51. Iterate through each target instance in the global tracking cache to check if any new targets have been inserted. For existing categories where no new targets have been inserted, fill them with "empty frame" markers.

[0105] Specifically, at the current detection time, some existing target categories may not match any new detected targets due to temporary occlusion, weak excitation events, or missed detections. To ensure that each target category has a consistent temporal resolution and a data structure of equal length in global temporal tracking, the algorithm inserts a placeholder for categories that have not been updated. This placeholder is in the form of all-zero data (i.e., box coordinates, center point, fitting parameters, etc., are all zero values). This operation is equivalent to explicitly declaring "not detected in this frame" for the target category at the current time step, providing a clear basis for subsequent determination of whether the target still exists or has left the field of view.

[0106] S52. Further determine the number of "empty frames" for each tracking category, and then clear the tracking cache that has not been inserted with new targets for a long time, giving the algorithm the ability to manage the life cycle of targets.

[0107] Specifically, as time goes on, some targets may have left the camera's field of view. The algorithm needs to have the ability to manage the target lifecycle and promptly clean up invalid category information to avoid memory resource redundancy and ID resource waste. To this end, this invention establishes a clearing mechanism: if a tracking category has 25 consecutive frames of all-zero marker information in its global tracking cache, meaning the target has not been detected for 25 consecutive frames, then the target is determined to have permanently disappeared. Once this condition is met, the algorithm will immediately clear the cached data for that category, releasing its memory space.

[0108] S53. Reclaim the target tracking ID of the cleared target category to save system resource overhead in tracking scenarios with a large number of targets.

[0109] Specifically, as a resource reuse strategy, to avoid excessive system maintenance burden caused by the continuous growth of tracking IDs, this invention designs an ID recycling mechanism: cleared category IDs are marked as idle. If a new target appears subsequently (i.e., one that neither meets distance matching nor trajectory fitting requirements), the algorithm will prioritize reusing these idle IDs, rather than simply adding one to the maximum ID. This design can effectively save system resource overhead and improve the overall operating efficiency of the algorithm when there are many targets or targets frequently enter and exit the scene.

[0110] S6. Predict the position of each target instance recorded in the global tracking cache, and redraw the target region of the target detection feature map at the next time step based on the predicted position to reduce the interference of intersection scenes during subsequent target tracking. Figure 3 This is a comparison image before and after the target feature map is redrawn using the prediction information of the target detection and tracking method based on the event camera in Embodiment 1 of the present invention.

[0111] Specifically, the global tracking cache stores the target bounding box, center point coordinates, and linear equation parameters [a, b] for each category at each time step during the tracking process (if a target in that category is not detected at a certain time, it is filled with all zeros). This tracking information contains the historical trajectory of the target movement for each category, which can provide prior information guidance for subsequent target detection. This invention predicts the future position of tracked target instances, and then redraws the target region in the next target detection stage, reducing the target area, decreasing the overlap time when multiple targets intersect, and improving the robustness of target tracking.

[0112] As an optional implementation, the step of fusing and updating the target region at the next time step with the target detection feature map corresponding to the event subset at the next time step includes:

[0113] Define a zero-based feature map with the same size as the target detection feature map at the next time step, and determine whether the position prediction result is in the target detection segmentation feature map at the next time step.

[0114] If not, no action will be taken.

[0115] If so, determine whether there are multiple location prediction results corresponding to the same target area; if not, use the center coordinates of the target area corresponding to the location prediction result as the center, draw a circular area with a radius of a preset number of pixels on the all-zero feature map and fill it; if so, use each of the location prediction results as the center, draw a circular area with a radius of a preset number of pixels on the all-zero feature map and fill it.

[0116] In Example 1, step S6 can be divided into sub-steps S61 to S62.

[0117] S61. First, obtain the center point coordinates of each target category in the latest two frames (assuming the current detection frame is frame k, and the latest two frames are frame (k-1) and frame k) from the global tracking cache. Then, use the following formula to predict the spatial position of the target in the next time step:

[0118] x kp = x k + σ x

[0119] y kp = y k + σ y

[0120] In the formula, (x kp , y kp σ represents the predicted position of the target at the next moment. x = x k -x k-1 σ represents the displacement difference in the x-coordinate. y =y k -y k-1 This represents the positional difference in the y-coordinate.

[0121] S62. After obtaining the predicted target location for each category, during the next target detection step, determine whether the predicted location is in the target detection segmentation feature map, and redraw the target region in the detection feature map based on the determination result. The target detection segmentation feature map comes from the target segmentation feature map after event polarity information fusion. Before redrawing the target, a zero-map with the same size as the feature map needs to be defined, and all redrawing operations are performed on this zero-map. The specific determination steps are as follows:

[0122] (1) Connected component labeling is performed on the segmentation feature map of the target detection, and the mask map of the target region is obtained. Different target regions in the mask map are labeled with different numbers, and the background region is labeled with 0.

[0123] (2) Determine whether the pixel value at the predicted position in the target mask image is 0. If it is 0, skip it directly without processing.

[0124] (3) If the pixel value at the predicted location in the target mask image is not 0, then the predicted location is the target area, indicating that the prediction result and the detection result are consistent, and the target area can be redrawn. At this time, it is necessary to further determine whether there is a numerical label for the target area in the local list, and then determine whether the target has been redrawn before. When the motion trajectories of multiple targets intersect, multiple targets overlap each other. At this time, the target detection can only detect one target, and the other overlapping targets are "missed". Multiple targets share one area and are labeled with one number. Therefore, special processing is required for this situation.

[0125] (4) If the pixel value at the predicted position in the target mask image is not 0, and there is no numerical label of the target area recorded in the local list, it means that the target is being redrawn for the first time. Then, a circle with a radius of 3 pixels is drawn on the zero map with the center coordinates of the target as the center as the new target, and the circle is filled with 255. At the same time, the numerical label of the target on the mask image and the predicted position are stored in the local list for subsequent determination of whether to redraw. If the pixel value at the predicted position in the target mask image is not 0, and there is a numerical label of the target area recorded in the local cache, it means that the target is not being redrawn for the first time and there is a situation of multiple targets intersecting. In this case, a circle with a radius of 3 pixels is drawn on the zero map with the predicted position as the center as the new target, and the circle is filled with 255.

[0126] (5) Finally, redraw the target that has not been redrawn. Draw a circle with a radius of 3 pixels on the zero map with the center coordinates of the target as the center and fill the circle with 255.

[0127] Beneficial effects: After redrawing the target area based on the predicted information, the area of ​​the new target is significantly smaller, which can effectively reduce the target overlap time and thus improve the robustness of multi-target tracking in intersection scenarios.

[0128] Example 2

[0129] This embodiment provides a target detection and tracking device based on an event camera, including: a slicing module, a traversal module, a preliminary detection module, an initialization module, a loop module, and an update module.

[0130] The slicing module is used to slice the raw event stream captured by the event camera to obtain a subset of events at each moment.

[0131] The traversal module is used to obtain the target detection feature maps corresponding to the event subsets at each time step;

[0132] The initial detection module is used to traverse the event subsets at each time point, perform initial target detection on the current target detection feature map to obtain the target initial detection result at the current time point; if the current time point is the initial time point, then enter the initialization module; otherwise, enter the loop module.

[0133] An initialization module is used to initialize the global tracking cache using the initial target detection result at the initial time, and to use the initial target detection result at the initial time as the target tracking result at the initial time.

[0134] The loop module is used to match the target initial detection result at the current moment with the target tracking result at the previous moment to determine whether a new category has appeared. If a new category has appeared, the new category is added to the target category in the global tracking cache at the previous moment, and the target trajectory of the existing category in the global tracking cache at the previous moment is updated to obtain the target tracking result at the current moment. Otherwise, the trajectory of the target of the existing category in the global tracking cache at the previous moment is updated to obtain the target tracking result at the current moment. Then, the module enters the update module.

[0135] The update module is used to predict the location of each target instance recorded in the global tracking cache at the current time, determine the target region at the next time based on the location prediction result, fuse the target region at the next time with the target detection feature map corresponding to the event subset at the next time and update it, and then return to the initial detection module until the event subsets at each time have been traversed.

[0136] This invention designs and implements a hardware-in-the-loop simulation experiment to acquire and analyze event data. Regarding the experimental platform setup, simulation software is first used on a host computer (control terminal) to set the motion trajectories and parameters of multiple point targets, including target brightness, speed, and path. These simulated motion processes are visualized using a "point target simulator," simulating the motion of point targets in the field of view in a real environment. The simulator's output is captured in real-time by an event camera. During the data acquisition phase, the Prophesee EVK4 event camera was selected. This device has high temporal resolution and good dynamic response capabilities, enabling it to accurately record every brightness change event generated by the point targets in the simulator. Event data is saved in the form of an event stream, where each event record includes the event's timestamp, trigger pixel coordinates, and polarity information.

[0137] To comprehensively verify the algorithm's performance under different target numbers, several representative data segments were carefully selected as test samples based on collected and organized hardware-in-the-loop simulation data. These data segments cover various typical scenarios with 1, 3, 8, and 10 targets, fully demonstrating the progressive experimental strategy from simple scenarios to complex high-density scenarios. Experimental results are as follows: Figure 4 As shown, this algorithm demonstrates excellent performance in both simple scenarios with a small number of targets and complex environments with dense targets. Especially in high-density scenarios with 10 targets simultaneously, the algorithm can still stably and accurately complete the detection and continuous tracking of all targets without any anomalies such as target loss, false detection, mistracking, or broken target trajectories.

[0138] Example 3

[0139] The present invention also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0140] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0141] Example 4

[0142] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0143] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0144] Example 5

[0145] This invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this invention.

[0146] The technical features of the embodiments described above can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "again" in this invention are intended to illustrate the invention and are not intended to limit the invention.

[0147] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An event camera based object detection and tracking method, characterized in that, Comprise: S1: slicing the original event stream collected by the event camera to obtain event subsets at each time; S2: obtaining the target detection feature map corresponding to the event subset at each time; S3: traversing the event subset at each time, performing preliminary detection on the current target detection feature map to obtain the target preliminary detection result at the current time; if the current time is the initial time, enter S4, otherwise enter S5; S4: initializing the global tracking cache using the target preliminary detection result at the initial time, and taking the target preliminary detection result at the initial time as the target tracking result at the initial time; S5: matching and detecting the target preliminary detection result at the current time and the target tracking result at the last time to determine whether a new class appears; If it appears, add the new class to the target class in the global tracking cache at the last time, update the target trajectory of the existing class in the global tracking cache at the last time to obtain the target tracking result at the current time; Otherwise, update the target trajectory of the existing class in the global tracking cache at the last time to obtain the target tracking result at the current time; Then enter S6; S6: position prediction is performed on each target instance recorded in the global tracking cache at the current time, and the next target region is determined according to the position prediction result, and the next target region is fused with the target detection feature map corresponding to the event subset at the next time and is updated, and then returns to S3 until the event subset at each time is traversed; The fusion and update of the next target region and the target detection feature map corresponding to the event subset at the next time include: defining a full-zero feature map with the same size as the target detection feature map at the next time, and judging whether the position prediction result is in the target detection segmentation feature map at the next time; if not, do not perform any processing; if yes, judge whether there are multiple position prediction results corresponding to the same target region; if not, take the center coordinates of the target region corresponding to the position prediction result as the center, draw a circular region with a radius of a preset number of pixels on the full-zero feature map and fill it; if yes, take each position prediction result as the center, draw a circular region with a radius of a preset number of pixels on the full-zero feature map and fill it.

2. The event camera based object detection and tracking method of claim 1, wherein, The S1 comprises: slicing the original event stream collected by the event camera at a set time interval to obtain event data at each time, and dividing the event data at the current time into positive polarity event subsets and negative polarity event subsets according to the polarity attribute.

3. The event camera based object detection and tracking method of claim 1, wherein, The S2 comprises: S21: feature extraction is performed on the event frame image corresponding to the positive polarity event subset and the negative polarity event subset at the current time to obtain a positive polarity segmentation feature map and a negative polarity segmentation feature map; S22: the intersection of the positive polarity segmentation feature map and the negative polarity segmentation feature map is taken as the target detection feature map at the current time.

4. The event camera based object detection and tracking method of claim 3, wherein, The S21 comprises: projecting the positive polarity event subset and the negative polarity event subset at the current time along the time axis to construct a positive polarity event image and a negative polarity event image; The positive polarity event image and the negative polarity event image are normalized and filtered, and then the positive polarity segmentation feature map and the negative polarity segmentation feature map are obtained through adaptive threshold segmentation.

5. The event camera based object detection and tracking method of claim 1, wherein, The matching detection of the target initial detection result of the current moment and the target tracking result of the previous moment to determine whether a new class appears comprises: The matching detection of the target class and the target instance in the target initial detection result of the current moment and the target class and the target instance in the target tracking result of the previous moment at multiple levels to determine whether the new class appears.

6. The event camera based object detection and tracking method of claim 1, wherein, The matching detection of the target initial detection result of the current moment and the target tracking result of the previous moment to determine whether a new class appears comprises: The Euclidean distance matching of the target class and the target instance in the target initial detection result of the current moment and all the target instances completing target tracking in the target tracking result of the previous moment is performed, and if the Euclidean distance is lower than a first preset value, the matching detection is considered to pass; For the target class that fails to pass the Euclidean distance matching, a matching strategy based on a straight line equation is introduced to determine the correlation degree of the target class and all the target instances completing target tracking in the target tracking result of the previous moment, and if the correlation degree is lower than a second preset value, the target class is considered to be the new class.

7. An event camera based object detection and tracking apparatus, characterized by, A method for performing the event camera-based target detection and tracking method of any one of claims 1-6 comprises: a slicing module configured to slice original event streams collected by an event camera to obtain event subsets of each moment; a traversal module configured to obtain target detection feature maps corresponding to the event subsets of each moment; an initial detection module configured to traverse the event subsets of each moment, perform target initial detection on a current target detection feature map, and obtain a target initial detection result of a current moment; if the current moment is an initial moment, the method enters an initialization module; otherwise, the method enters a loop module; the initialization module is configured to initialize a global tracking cache by using the target initial detection result of the initial moment, and use the target initial detection result of the initial moment as a target tracking result of the initial moment; the loop module is configured to perform matching detection of the target initial detection result of the current moment and a target tracking result of a previous moment to determine whether a new class appears; if the new class appears, the new class is added to target classes in the global tracking cache of the previous moment, target trajectories of the existing classes in the global tracking cache of the previous moment are updated, and a target tracking result of the current moment is obtained; otherwise, target trajectories of the existing classes in the global tracking cache of the previous moment are updated, a target tracking result of the current moment is obtained, and the method enters an updating module; the updating module is configured to perform position prediction on each target instance recorded in the global tracking cache of the current moment, determine a target region of a next moment according to a result of the position prediction, perform fusion and updating of the target region of the next moment and a target detection feature map corresponding to an event subset of the next moment, and return to the initial detection module until the event subsets of each moment are traversed. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the method of any one of claims 1-6. The processor executes the computer program to implement the steps of the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images