Event Camera-based Object Detection and Tracking Method, System, and Storage Medium

Abstract

The present invention provides a target detection and tracking method, system, and storage medium based on an event camera. The method includes: reading a DVS event sequence and performing background noise reduction processing; setting a time window; initializing Kalman filter parameters; converting the DVS event sequence within the time window into a normalized average time plane; performing target event denoising in the normalized average time plane; detecting the target position and simultaneously predicting the target position using a Kalman filter, but the initial position is determined by the detection; calculating the distance between the target position detected at the current moment and the target position at the previous moment, balancing the detected position with the position predicted by the Kalman filter based on the distance information, thereby determining the target position at the current moment; performing trajectory smoothing; moving to the next time window and repeating the above steps until the DVS event sequence ends. The present invention has the characteristics of low latency, avoiding noise interference, avoiding inaccurate position detection, and preventing target frame jitter.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to target detection and tracking based on computer vision. Background Art

[0002] While many detection and tracking solutions exist for current computer vision tasks, these solutions are insufficient for handling high-speed motion, dynamic changes, and fluctuating lighting conditions. Traditional cameras capture images at a fixed frame rate, which can cause motion blur when capturing fast-moving objects. Furthermore, in low light or when the lighting changes, the object's features change, making it difficult to distinguish from the background. Furthermore, if the background is multicolored or the target and background have similar colors, the target will be lost in the background, making it difficult to detect.

[0003] Dynamic vision sensors (DVS), also known as event-based cameras, use an event-driven approach to capture dynamic changes in a scene. Unlike traditional cameras, which capture complete images at a specified frame rate (such as 30 fps), event cameras, such as DVS, have no concept of frame rate. Each pixel responds asynchronously and independently to changes in brightness in the scene. The output of an event camera is a sequence of "events" or "pulses" at a variable rate. Each event represents a change in light brightness, and a pulse is generated when the light intensity changes by more than a certain threshold compared to the previous moment.

[0004] In videos shot by traditional cameras, it is difficult to detect moving targets in complex backgrounds. However, DVS cameras imitate the pulse events of biomembrane potential, and only moving targets trigger events, so they have the advantage of being able to quickly detect moving targets.

[0005] Prior art CN112927261A discloses a target tracking method that integrates position prediction and correlation filtering. The method includes: based on detection using a kernel correlation filter, a Kalman filter is introduced to correct the tracking results. At the start of tracking, the initial coefficients of the kernel correlation filter are calculated in the initial frame video image based on the target position and size in the label pre-given by the video sequence. In subsequent frames, the target position of the previous frame is taken as the center, and the target range is expanded by 2.5 times as the target search area. The kernel correlation filter response is calculated, and the position corresponding to the maximum response value is used as the initial target result. This result is then used as the observation value, and the tracking result is corrected using a Kalman filter and the kernel correlation filter coefficients are updated. This process is repeated until the tracking is completed. The target detection and tracking in the above-mentioned prior art is still based on traditional RGB cameras, which require the extraction of target image features. Therefore, the asynchronous and low-latency characteristics of DVS cannot be utilized.

[0006] Prior art CN112949512A discloses a dynamic gesture recognition method, gesture interaction method, and interaction system. The method includes: a dynamic vision sensor (DVS) adapted to trigger an event based on the relative motion of an object in the field of view and the dynamic vision sensor, and output an event data stream to a hand detection module; a hand detection module adapted to process the event data stream to determine the initial hand position; and a hand tracking module adapted to determine a series of state vectors indicating the hand motion state in the event data stream using a Kalman filter based on the initial hand position. The target detection and tracking in the above-mentioned prior art are susceptible to noise interference; and the predicted position of the Kalman filter in the above-mentioned prior art is prone to offset when the target stops or turns.

[0007] Therefore, existing technologies, such as traditional RGB video, require extracting target image features rather than directly utilizing event information. This makes it impossible to leverage the asynchronous and low-latency nature of DVS. Static backgrounds are also included in the calculation, resulting in high computational overhead and making it unsuitable for low-power computing scenarios. Existing DVS-based target detection and tracking technologies are susceptible to noise interference on target positions, resulting in inaccurate target bounding boxes during detection. Furthermore, the use of Kalman filtering in these existing technologies is prone to drift when the target stops or turns. These technologies use a template near the target position predicted by the Kalman filter to update the detector, which then uses the detected target position to further update the Kalman filter parameters. This can lead to error accumulation in complex scenes, ultimately leading to target loss. Existing technologies require the target position to be specified in the initial frame, which incurs labor costs. However, many scenarios lack the necessary human resources to constantly monitor the video to determine the target's initial position. In subsequent frames, existing technologies require searching within a window 2.5 times the size of the target bounding box detected in the previous frame, resulting in high computational overhead and target loss if the target strays from the search range. Summary of the Invention

[0008] To overcome the above-mentioned shortcomings of the prior art, the present invention provides a method, system, and storage medium for target detection and tracking based on an event camera. These methods have the advantages of low latency, noise interference avoidance, position inaccuracy avoidance, and target frame jitter prevention.

[0009] The present invention provides a target detection and tracking method based on an event camera, which is characterized by comprising:

[0010] Step 1: Read the DVS event sequence and perform background noise reduction; set the time window; initialize the parameters of the Kalman filter;

[0011] Step 2: Convert the DVS event sequence within the time window into a normalized average time plane;

[0012] Step 3: denoise the target events in the normalized average time plane; detect the target position and use Kalman filtering to predict the target position, but the initial position is determined by the detection;

[0013] Step 4: Calculate the distance between the target position detected at the current moment and the target position at the previous moment, and balance the detected position with the position predicted by the Kalman filter based on the distance information to determine the target position at the current moment.

[0014] Step 5: Smooth the trajectory; move to the next time window and repeat steps 2 to 5 until the DVS event sequence ends.

[0015] The present invention provides a target detection and tracking system based on an event camera, which is characterized by comprising:

[0016] The preprocessing module is used to read the DVS event sequence, perform background noise reduction, set the time window, and initialize the parameters of the Kalman filter;

[0017] Normalized average time surface module, used to convert the DVS event sequence within the time window into a normalized average time surface;

[0018] The target position prediction module is used to perform target event denoising and detect the target position in the normalized average time plane. Kalman filtering is also used to predict the target position, but the initial position is determined by the detection.

[0019] The target position update module is used to calculate the distance between the target position detected at the current moment and the target position at the previous moment, and balance the detected position with the position predicted by the Kalman filter based on the distance information to determine the target position at the current moment;

[0020] The trajectory smoothing module is used to smooth the trajectory; moving to the next time window, the above module continues processing until the DVS event sequence ends.

[0021] The present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned event camera-based target detection and tracking method.

[0022] Based on the above scheme, the present invention directly utilizes event information, avoiding the computation of unnecessary information in traditional images and offering low latency and asynchronous performance. Background and target event denoising are employed to prevent noise interference with target positions. Combining detection with Kalman filtering avoids inaccurate position detection and offsets in Kalman filter predictions. A trajectory smoothing strategy is employed to effectively prevent jitter in the target frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 Flowchart of the method of the present invention.

[0025] Figure 2 This is the background denoising effect diagram.

[0026] Figure 3 This is a visualization of the normalized average time surface.

[0027] Figure 4 This is the target detection effect diagram.

[0028] Figure 5 To balance the detection position and the position predicted by the Kalman filter, detection represents detection and KCF represents Kalman filter.

[0029] Figure 6 This is a trajectory smoothing effect diagram. In the figure, detection means detection and KCF means Kalman filter. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] Explanation of relevant terms:

[0032] Event camera: Also known as an event camera, it is a camera that uses an asynchronous sensor to sample light based on the dynamics of the scene. Standard cameras acquire complete images at a specified frame rate (such as 30fps), but event cameras have no concept of frame rate. Each pixel responds asynchronously and independently to changes in brightness in the scene. The output of an event camera is a sequence of "events" or "pulses" at a variable rate. Each event represents a change in light brightness, and a pulse is generated when the light intensity changes by more than a certain threshold compared to the light intensity at the previous moment. The event contains information such as location, positive or negative polarity (light becomes stronger or weaker), and the current time. It is based on the pulse mechanism in biological vision.

[0033] DVS: Dynamic Vision Sensor, also known as Neuromorphic Vision Sensor. This refers to a sensor that outputs positive and negative polarity events (light intensity changes or decreases).

[0034] Event: When the light intensity at each location in an event camera changes beyond a certain threshold, an "event" or "pulse" is generated. An event contains information about the location, time, and polarity (light intensity).

[0035] Time plane: This represents the location and time information of an event as a two-dimensional graph. The time information is used as the “pixel value” of each location.

[0036] Object detection: Find all objects of interest in the image, including two subtasks: object localization and object classification, and determine the category and location of the object at the same time.

[0037] Target tracking: It is divided into single target tracking and multi-target tracking. Single target tracking: Given a target, track the position of this target. Multi-target tracking: track the positions of multiple targets.

[0038] Membrane potential: Membrane potential generally refers to the potential difference between two solutions separated by a membrane. It generally refers to the electrical phenomena that accompany cellular life processes, and the potential difference that exists across the cell membrane. Membrane potential plays a crucial role in the communication between nerve cells.

[0039] Complex background: For example, a background with a color similar to the target, or with many obstructions, or with moving objects similar to the target.

[0040] Figure 1 is a flowchart of a method according to an embodiment of the present invention.

[0041] In one embodiment, the present invention provides a method for target detection and tracking based on an event camera, characterized by comprising:

[0042] Step 1: Read the DVS event sequence and perform background noise reduction; set the time window; initialize the parameters of the Kalman filter;

[0043] Step 2: Convert the DVS event sequence within the time window into a normalized average time plane;

[0044] Step 3: denoise the target events in the normalized average time plane; detect the target position and use Kalman filtering to predict the target position, but the initial position is determined by the detection;

[0045] Step 4: Calculate the distance between the target position detected at the current moment and the target position at the previous moment, and balance the detected position with the position predicted by the Kalman filter based on the distance information to determine the target position at the current moment.

[0046] Step 5: Smooth the trajectory; move to the next time window and repeat steps 2 to 5 until the DVS event sequence ends.

[0047] Optionally, step 1 includes:

[0048] Step 1a: specify a time interval and determine whether there is an event in the 8-neighborhood interval of each event location (x, y) within the time interval. If not, it is considered to be background noise and filtered out;

[0049] Step 1b, set the time window and initialize the Kalman filter parameters as follows;

[0050] Set the state transfer matrix A = [A s , 0 ； 0, A s ];

[0051] Among them, A s , 0 means in A s The following is parallel and A s All-zero matrices of the same size; 0, A s Indicates that in A s The previous parallel and A s All-zero matrices of the same size; A s , 0 ； 0, A s The semicolon in the string means going to the next line;

[0052] Set the prediction noise covariance matrix Q = [Q s , 0 ； 0, Q s ];

[0053] Among them, Q s , 0 means in Q s Followed by parallel and Q s All-zero matrices of the same size; 0, Q s Indicates that in Q s The front is parallel to Q s All-zero matrices of the same size; Q s , 0 ； 0, Q s The semicolon in the string means going to the next line;

[0054] Set the observation matrix H = [H s ,0;0,H s ];

[0055] Among them, H s , 0 means in H s The following is parallel and H s All-zero matrices of the same size; 0, H sIndicates that H s The front is parallel to H s All-zero matrices of the same size; H s , 0 ； 0,H s The semicolon in the string means going to the next line;

[0056] Set the observation noise covariance matrix R;

[0057] Set the state covariance matrix P;

[0058] Set the target's motion state to x = [p col , v col , a col , p row , v row , a row ] T , where (p row , p col ) is the center coordinate of the target, (v row , v col ) is the target speed, (a row , a col ) is the acceleration of the target.

[0059] Optionally, in step 1b,

[0060] The time window is 5ms;

[0061] Optionally, in step 1b,

[0062] A s =[1, 1, 0.5; 0, 1, 1; 0, 0, 1];

[0063] Optionally, in step 1b,

[0064] Q s =diag([15, 15, 10]), where diag is used to create a diagonal matrix;

[0065] Optionally, in step 1b,

[0066] H s =[1,0,0];

[0067] Optionally, in step 1b,

[0068] The observation noise covariance matrix is R = [25, 0; 0, 25];

[0069] Optionally, in step 1b,

[0070] The state covariance matrix is P = diag([10 5 , 10 5 , 105 , 10 5 , 10 5 , 10 5 ]);

[0071] Optionally, in step 1b,

[0072] The initial value of the target's velocity is (0, 0), and the initial value of the target's acceleration is (0, 0).

[0073] like Figure 2 As shown, this is the background denoising effect of step 1 of the present invention.

[0074] Optionally, step 2 includes:

[0075] Step 2a, calculate the average time plane; the time information of each event at position (i, j) in the time window is t, and the cumulative number of events is I i，j The average time surface is

[0076] Step 2b, calculate the normalized mean time surface; defined as Where (i, j)∈T represents the event position on the average time plane.

[0077] like Figure 3 As shown, it is a visualization diagram of the normalized average time surface of step 2 of the present invention.

[0078] Optionally, step 3 includes:

[0079] Usually, the temporal information of a moving target is greater than the background noise, so a threshold λ is set to filter out the noise.

[0080] Step 3a, target detection; set a threshold λ to filter out noise, the target is O = {(i, j) | N i，j >λ}; the target detection position at time k is z dk =mean(O), where mean is the mean operation;

[0081] Step 3b, target tracking; predict the state at time k in is the actual value of the state at time k-1, Represents the predicted value at time k; predicts the state covariance matrix at time k Among them, P k-1 is the state covariance matrix at time k-1.

[0082] Optionally, in step 3a, λ may be set to 0.2. To filter out noise, the values of the smaller 20% portion and the larger 20% portion of the target O may be removed.

[0083] like Figure 4 As shown, this is a target detection effect diagram of step 3 of the present invention.

[0084] Optionally, step 4 includes:

[0085] Step 4a, based on the target detection position z obtained in step 3a dk , calculate the Euclidean distance between the current k-time position and the k-1-time position Define the threshold thres. When d1≤thres, the detected target position is considered correct. The parameters used to update the Kalman filter are as follows:

[0086] Calculate the Kalman gain at time k

[0087] Update the state at time k

[0088] Update the state covariance matrix at time k

[0089] When d1>thres, calculate the predicted position of the Kalman filter at the current k moment

[0090] Step 4b: Calculate the Euclidean distance between the target position predicted by the Kalman filter at the current k moment and the target position at the k-1 moment

[0091] Step 4c: Count the number of events in the target box where the Kalman filter predicts the position and the target box where the detection position is located, and take the center point of the one with the larger number of events as the current target position z k .

[0092] Optionally, in step 4a, thres may be 30.

[0093] like Figure 5 As shown, the effect of balancing the detected position and the Kalman filter predicted position in step 4 of the present invention is shown.

[0094] Optionally, step 5 includes:

[0095] When the target motion slows down, the number of events will decrease, and the current predicted target position may deviate far from the target position at the previous moment, causing the target frame to jitter. To solve this problem, a trajectory smoothing strategy is adopted.

[0096] If d2>thres in step 4b, then the target position z at the current time k is k =ω1·z k +ω2·z k-1 , where ω1 and ω2 are weights.

[0097] Optionally, ω1=0.5, ω2=0.5.

[0098] like Figure 6 As shown, the trajectory smoothing effect of step 5 of the present invention is achieved.

[0099] In another embodiment, the present invention provides an event camera-based target detection and tracking system, characterized by comprising:

[0100] The preprocessing module is used to read the DVS event sequence, perform background noise reduction, set the time window, and initialize the parameters of the Kalman filter;

[0101] Normalized average time surface module, used to convert the DVS event sequence within the time window into a normalized average time surface;

[0102] The target position prediction module is used to perform target event denoising and detect the target position in the normalized average time plane. Kalman filtering is also used to predict the target position, but the initial position is determined by the detection.

[0103] The target position update module is used to calculate the distance between the target position detected at the current moment and the target position at the previous moment, and balance the detected position with the position predicted by the Kalman filter based on the distance information to determine the target position at the current moment;

[0104] The trajectory smoothing module is used to smooth the trajectory; moving to the next time window, the above module continues processing until the DVS event sequence ends.

[0105] In another embodiment, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores multiple programs; when the programs are running, the device where the computer-readable storage medium is located is controlled to load and execute the above-mentioned event camera-based target detection and tracking method.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection and tracking method based on event camera, characterized in that: include: Step 1: Read the DVS event sequence and perform background noise reduction processing; Set the time window; Initialize the parameters of the Kalman filter; Step 2: Convert the DVS event sequence within the time window into a normalized average time plane; Step 3: denoise the target event in the normalized average time plane; Detect the target position and use Kalman filtering to predict the target position, but the initial position is determined by the detection; Step 4: Calculate the distance between the target position detected at the current moment and the target position at the previous moment, and balance the detected target position with the target position predicted by the Kalman filter based on the distance information to determine the target position at the current moment. Step 5, perform trajectory smoothing; Move to the next time window and repeat steps 2 to 5 until the DVS event sequence ends; Wherein, step 1 includes: Step 1a: specify a time interval and determine whether there is an event in the 8-neighborhood interval of each event location within the time interval. If not, it is considered to be background noise and filtered out; Step 1b, set the time window at the specified time interval and initialize the parameters of the Kalman filter as follows; Set the state transfer matrix A; Set the prediction noise covariance matrix Q; Set the observation matrix H; Set the observation noise covariance matrix R; Set the state covariance matrix P; Set the target's motion state to x = [p col ,v col ,a col ,p row ,v row ,a row ] T , where (p row ,p col ) is the center coordinate of the target, (v row ,v col ) is the target speed, (a row ,a col ) is the acceleration of the target; Wherein, step 2 includes: Step 2a, calculate the average time plane; the time information of the event at position (i, j) in the time window is t, and the cumulative number of events is I i,j ; then the average time surface is Step 2b, calculate the normalized mean time surface; defined as where (i,j)∈T i,j represents the event position on the average time plane; Wherein, step 3 includes: Step 3a, target detection; set a threshold λ to filter out noise, the target is O = {(i, j) | N i,j >λ}; the target position detected at time k is z dk =mean(O), where mean is the mean operation; Step 3b, target tracking; predict the state at time k in is the actual value of the state at time k-1, Represents the predicted value at time k; predicts the state covariance matrix at time k Among them, P k-1 is the state covariance matrix at time k-1; Wherein, step 4 includes: Step 4a, based on the detected target position z obtained in step 3a dk , calculate the Euclidean distance between the target position detected at the current k moment and the target position at the k-1 moment Define the threshold thres. When d1≤thres, the detected target position is considered correct. The parameters used to update the Kalman filter are as follows: Calculate the Kalman gain at time k Update the state at time k Update the state covariance matrix at time k When d1>thres, calculate the target position predicted by the Kalman filter at the current k moment Step 4b: Calculate the Euclidean distance between the target position predicted by the Kalman filter at the current k moment and the target position at the k-1 moment Step 4c: Count the number of events in the target box where the target position predicted by the Kalman filter is located and the target box where the detected target position is located, and take the center point of the target box with the larger number of events as the current target position z k ; Wherein, step 5 includes: If d2>thres in step 4b, then the target position z at the current k moment is k =ω1·z k +ω2·z k-1 , where ω1 and ω2 are weights.

2. The method according to claim 1, characterized in that In step 3a, λ can be set to 0.

2. To filter out noise, the values of the smaller 20% portion and the larger 20% portion of the target O can be removed.

3. The method according to claim 2, characterized in that ω1=0.5,ω2=0.

5.

4. An event camera-based target detection and tracking system, characterized in that: include: The preprocessing module is used to read the DVS event sequence, perform background noise reduction, set the time window, and initialize the parameters of the Kalman filter. It specifically includes: Specify a time interval and determine whether there is an event in the 8-neighborhood interval of each event location within the time interval. If not, it is considered to be background noise and filtered out. The time window is set at a specified time interval and the parameters of the Kalman filter are initialized as follows; Set the state transfer matrix A Set the prediction noise covariance matrix Q; Set the observation matrix H; Set the observation noise covariance matrix R; Set the state covariance matrix P; Set the target's motion state to x = [p col ,v col ,a col ,p row ,v row ,a row ] T , where (p row ,p col ) is the center coordinate of the target, (v row ,v col ) is the target speed, (a row ,a col ) is the acceleration of the target; The normalized average time plane module is used to convert the DVS event sequence within the time window into a normalized average time plane. Specifically, it includes: Find the average time surface; the time information of the event at position (i, j) in the time window is t, and the cumulative number of events is I i,j ; then the average time surface is Compute the normalized mean time surface; defined as where (i,j)∈T i,j represents the event position on the average time plane; The target position prediction module is used to perform target event denoising and detect the target position in the normalized average time plane. It also uses Kalman filtering to predict the target position, but the initial position is determined by the detection. Specifically, it includes: Target detection: set a threshold λ to filter out noise, the target is O = {(i, j) | N i,j >λ}; the target position detected at time k is z dk =mean(O), where mean is the mean operation; Target tracking; predicting the state at time k in is the actual value of the state at time k-1, Represents the predicted value at time k; predicts the state covariance matrix at time k Among them, P k-1 is the state covariance matrix at time k-1; The target position update module is used to calculate the distance between the target position detected at the current moment and the target position at the previous moment, and balance the detected target position with the target position predicted by the Kalman filter based on the distance information to determine the target position at the current moment. Specifically, it includes: According to the detected target position z dk , calculate the Euclidean distance between the target position detected at the current k moment and the target position at the k-1 moment Define the threshold thres. When d1≤thres, the detected target position is considered correct. The parameters used to update the Kalman filter are as follows: Calculate the Kalman gain at time k Update the state at time k Update the state covariance matrix at time k When d1>thres, calculate the target position predicted by the Kalman filter at the current k moment Calculate the Euclidean distance between the target position predicted by the Kalman filter at the current k moment and the target position at the k-1 moment Count the number of events in the target box where the target position predicted by the Kalman filter is located and the target box where the detected target position is located, and take the center point of the target box with the larger number of events as the current target position z k ; The trajectory smoothing module is used to smooth the trajectory and move to the next time window to continue processing until the DVS event sequence ends. Specifically, it includes: If d2>thres, then the target position z at the current k moment k =ω1·z k +ω2·z k-1 , where ω1 and ω2 are weights.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 3.

Images