A method for estimating the start-up delay between event cameras

Abstract

This invention provides a method for estimating the start-up delay between event cameras, belonging to the field of computer vision. It includes the following steps: First, acquiring data from multiple event cameras; next, selecting corresponding data from multiple events according to a predefined method; then, preprocessing the selected corresponding data, followed by normalization and Gaussian filtering to obtain a distribution function; finally, obtaining a minimization objective function based on the distribution function, and calculating the minimization objective function using the negative value of the correlation to complete the start-up delay estimation. The event camera time synchronization method proposed in this invention can synchronize event cameras solely based on event streams to obtain the start-up delay between event cameras, simplifying the synchronization process, making it easy to promote, and applicable to synchronization between any event cameras.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and more specifically, relates to a method for estimating the start-up delay between event cameras. Background Technology

[0002] In recent years, a new type of sensor (event camera) inspired by the biological retina has been developed. It focuses on image changes and outputs timestamps, x and y coordinates, and the polarity of the coordinate point, ultimately obtaining an event stream containing these four sets of data.

[0003] Event cameras, also known as dynamic vision sensors, are sensors that have gained increasing influence in recent years. Inspired by the biological retina, they can detect changes in images. When a change in brightness exceeds a certain threshold, an "event" is generated. An event contains three elements: pixel location, generation time, and polarity (brightening / darkening). The chip outputs the event stream to the camera board via the AER bus, and the camera board transmits the data to a computer via USB 3.0. Event cameras feature extremely low latency, no motion blur, high dynamic range, and extremely low power consumption, making them widely used in target detection, autonomous driving, and other fields. Specific applications include data encoding and storage, super-resolution, image denoising, video reconstruction, target recognition, optical flow estimation, motor control, and SLAM. These scenarios often require multiple event cameras to form a binocular or multi-view system, necessitating multi-sensor synchronization, i.e., acquiring the startup delay between several event cameras. Most event cameras support hardware synchronization, such as two cameras simultaneously receiving a high-frequency electrical signal, but this requires additional synchronization circuitry. Furthermore, the synchronization principles differ between different event camera models, leading to increased hardware costs and complex acquisition systems.

[0004] Patent application CN110740227A, entitled "Camera Time Synchronization Device and Method Based on GNSS Timing and Image Display Information Encoding Method," describes a device that receives GNSS time as its internal time reference and calibrates its internal time in real time. It uses a capture display screen and LED light group encoding to implement GNSS timestamp marking for optical image sensors such as cameras. While this invention does not require any hardware connection or software programming to the camera, and does not necessitate additional trigger control circuitry or functions on the camera, event cameras do not support GNSS timing, thus making this method unsuitable for estimating startup delays.

[0005] Patent application CN115147753A proposes a "Multi-camera Time Synchronization Method Based on Human Pose." This method determines the time difference during video capture by using a matching matrix and matching vectors between frames to determine the time difference during capture, thus achieving video synchronization. While this method does not use any hardware, it has limitations regarding the subjects being captured, and the data modes of images from ordinary cameras and data streams from event cameras differ, making it unsuitable for direct application.

[0006] Patent application CN114463399A proposes a "spatiotemporal matching method between an event camera and a conventional optical camera." Based on scene structure similarity indices, it matches and searches the image frames and event stream data output by the event camera with the image frame data output by the conventional optical camera to obtain the optimal matching field of view and time difference, thereby achieving spatiotemporal matching between the event camera and the conventional optical camera. While this method further completes the spatiotemporal matching between the camera and the event camera, it still does not solve the problem of estimating the start-up delay between event cameras.

[0007] Therefore, it is necessary to propose a method for estimating the start-up delay between event cameras in order to address the above-mentioned shortcomings of the existing technology. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a method for estimating the start-up delay between event cameras. This method compares data captured by different event cameras over a period of time, finds the offset time that can synchronize the two, and completes the estimation of the start-up delay of data acquired by different event cameras.

[0009] To achieve the above objectives, the present invention provides a method for estimating the start-up delay between event cameras, comprising the following steps:

[0010] S1: Acquire data from multiple event cameras.

[0011] S2: Select corresponding data for multiple events according to the set method.

[0012] S3: First, preprocess the selected corresponding data, then perform normalization and Gaussian filtering to obtain the distribution function.

[0013] S4: Obtain the minimum objective function based on the distribution function, and calculate the minimum objective function by combining the negative value of the correlation, thereby completing the start-up delay estimation.

[0014] Furthermore, step S2 specifically involves:

[0015] S2: Extract data within a time period T after the i-th event camera starts as the corresponding data for calibration. The minimum time unit is defined as Δt. When the event camera records a time of nΔt, the number of events occurring at the i-th event camera within this time period is m. i (n), where i is a positive integer and n is also a positive integer. n is a dimensionless number, which can be interpreted as the integer part of the time t from the beginning to the present time divided by Δt.

[0016] Furthermore, step S3 specifically involves: determining the number of events m that occurred at the camera for the i-th event. i (n) First, normalize the data, then perform Gaussian filtering to obtain the distribution function f of the number of events for different event cameras. i (n).

[0017] Furthermore, step S4 specifically involves setting the startup delay of the i-th event camera relative to the 1st event camera to be... The smallest unit of time, where i ≠ 1, and f in the following text i (x) is the distribution function f of the number of events of the event camera. i (n), the objective function to be minimized is F(f1(x), f i (x+δn i )), δn obtained by minimizing the objective function i This is the estimated startup delay between event cameras.

[0018] Furthermore, each time data is collected from multiple event cameras, the startup delay between the event cameras during that collection process needs to be calculated.

[0019] Furthermore, step S2 specifically involves:

[0020] Let e ​​be an event generated by the event camera. j The timestamp of the event is A timestamp is the time assigned by the event camera when the event occurred, used to represent the time interval between the occurrence of the event and the start time of the event camera.

[0021] Based on the timestamps of the events, data collected by different event cameras is extracted. Data of duration T is extracted after each event camera starts. The smallest time unit Δt represents the latency accuracy required by the actual scenario. Let the time duration T contain N smallest time units, T = Δt * N. Let E be the set of all events generated by the i-th event camera within a certain unit time ((n-1)Δt, nΔt). i (n), n is an integer greater than 0 and less than N. The total number of events that occur within the aforementioned time period is denoted as m. i(n), m i (n)=num(E i (n)), where num represents the number of elements in the set, and the total number of events generated by the camera during the i-th event in time T is denoted as .

[0022] Furthermore, step S3 specifically involves determining the number of events m for the camera at the i-th event. i (n) Normalization processing is performed to confine the data to (0,1) while preserving the original relationships between the data. The data from the camera of the i-th event is normalized to... After normalizing the data, Gaussian filtering is applied to each data point. The distribution function of the camera data for the i-th event after filtering is as follows:

[0023] Furthermore, with different event cameras pointing in the same direction, their centers on the same horizontal plane, and the cameras fixed, the simultaneously recorded data, after normalization and time synchronization, yields the distribution function f of the number of events for different event cameras. i (n) similar.

[0024] Furthermore, in step S4, the objective function obtained from the distribution function is F(f1(x), f i (x+δn i The objective function is the sum of squared errors. Bhattacharyy distance Or the negative value of the correlation

[0025] Let the startup delay be... but

[0026] Furthermore, The specific calculation method is as follows:

[0027] Define the search range for startup time delay, and set the startup time delay to δn. min *Δt to δn max Between *Δt, δn min ,δn max If is a positive integer, then

[0028] Based on the characteristics of the objective function, F(f1(x),f i (x+δn i The calculation result of )) is between 0 and 1, let δn i When taking different values, F(f1(x),f i (x+δn i The minimum value of )) is h, initialize h = 1.0

[0029] In δn i -1<δn max The following two steps are executed in a loop:

[0030] (1) If Then put Assign the value of δn to h, and then assign δn to h. i The value assigned to

[0031] (2) Put δn i The value of +1 is assigned to δn i ,

[0032] Until iteration reaches δn i -1>δn max Stop the loop.

[0033] After the loop completes This is an estimate of the startup delay between event cameras.

[0034] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0035] The event camera time synchronization method proposed in this invention can synchronize the time of event cameras by relying solely on event streams, obtain the start-up delay between event cameras, make the synchronization process simpler, easier to promote and applicable to the synchronization between any event cameras. Attached Figure Description

[0036] Figure 1 This invention provides a method for estimating the start-up delay between event cameras.

[0037] Figures 2(a) and 2(b) are normalized data images of the CeleX5 event camera and the DAVIS346 event camera in the embodiment of the present invention;

[0038] Figures 3(a) and 3(b) are the normalized data images in Figures 2(a) and 2(b) of the embodiment of the present invention, and then filtered.

[0039] Figure 4 As described in the embodiments of the present invention The calculation flowchart;

[0040] Figures 5(a) and 5(b) are the results of the change in data volume over time after delay calibration of the CeleX5 event camera and the DAVIS346 event camera in the embodiments of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] This invention proposes a method for estimating the start delay between event cameras. It can obtain the start delay between event cameras by relying solely on the event stream for time synchronization of event cameras, making the synchronization process simpler, easier to promote, and applicable to the synchronization between any event cameras.

[0043] Figure 1 This invention provides a method for estimating the start-up delay between event cameras. As shown in the figure, the core steps of this invention include the following:

[0044] S1: Acquire data collected by the event camera.

[0045] S2: Select the corresponding data according to the set method, preprocess the selected data, and then perform normalization and Gaussian filtering.

[0046] S3: Calculate and estimate the startup delay based on the negative value of the correlation.

[0047] In one specific embodiment, the method of the present invention includes the following detailed steps and processes:

[0048] S1: Acquire data from the event cameras. Select the corresponding data according to the set method. Specifically, use a CeleX5 event camera and a DAVIS346 event camera, fix the two cameras together with the same lens orientation, lens centers on the same horizontal plane, and a 20cm distance between them. Connect the cameras to the computer via USB 3.0 and start recording. Shake the two fixed cameras randomly for T seconds (T=8 seconds), then record the data. Export the raw data recorded by the two cameras as a .csv file. Process the data using MATLAB. Take the smallest time unit as Δt=1ms and extract the data within time T (T=8s). Let the CeleX5 be the first event camera and the DAVIS346 be the second event camera. Let the number of events for the first event camera be m1(n) and the number of events for the second event camera be m2(n).

[0049] S2: Preprocess the selected data, then perform normalization and Gaussian filtering. Specifically, normalize m1(n) and m2(n) in S100. The first event camera data is normalized to... The second event camera data is normalized to In this example, the normalized data images of the CeleX5 event camera and the DAVIS346 event camera are shown in Figures 2(a) and 2(b). Gaussian filtering was then applied to the normalized data. The Gaussian filtered data from the first event camera is shown below. The second event camera data, after Gaussian filtering, is The Gaussian filter window size is set to 100 minimum time units, i.e., 0.1s. The filtered data images in this embodiment are shown in Figure 3(a) and Figure 3(b).

[0050] S3: Calculate and estimate the startup delay based on the negative value of the correlation. Specifically, assume the startup delay of the second event camera relative to the first event camera is n. i If there are a minimum time unit, then minimize the objective function F(f1(n),f i (n+δn i The obtained δn i This refers to the startup time delay. The objective function is F(f1(n),f2(n+δn)). i The sum of squares can be the error: or Bhattacharyy distance Or the negative value of the correlation Startup delay In this example, the negative value of the correlation is used: As a basis for judging relevance, The calculation flowchart is as follows Figure 4 As shown, summarize Figure 4 The core steps are as follows, and the specific calculation method is as follows:

[0051] Step 1: Calculate and accumulate the probability density of CeleX5 event cameras. Corresponding time scale, i.e. Find the cumulative probability density of DAVIS346 events to camera. Corresponding time scale Find n 1 With n 2 The difference n 0 That is, n 0 =|n 1 -n 2 |. It is believed The range of values ​​for δn is min to δn max (δn min ,δn max (where δn is a positive integer) min =0.8n 0 ,δn max =1.2n0 .

[0052] Step 2, make h = 1, δn i =δn min If δn i -1<δn max Then proceed to steps three and four, and repeat steps three and four in sequence until δn. i -1>δn max ,

[0053] Step 3, if make make

[0054] Step 4, δn i ←δn i +1,

[0055] Step 5: Iterate until δn i -1>δn max ,at this time This is the desired startup time delay.

[0056] In this example, the timestamp of the second event camera data is added. The startup delay between event cameras can then be estimated. This method shows that the startup delay between cameras in this example is 2192ms. The changes in the amount of event camera data over time after delay calibration are shown in Figures 5(a) and 5(b). As can be seen from Figures 5(a) and 5(b), the event cameras have achieved time synchronization after the above processing.

[0057] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for estimating the start-up delay between event cameras, characterized in that, It includes the following steps: S1: Acquire events captured by multiple event cameras. S2: Select the number of events occurring in multiple event cameras according to the set method. Step S2 is as follows: Extract the first The duration after the camera starts up. The data within is used as the corresponding data for calibration, and the minimum time unit is specified as [missing information]. When the event camera itself records the time is At that time, the first The function for the number of events occurring in a camera is: ,in, It is a positive integer. It is also a positive integer. S3: First, preprocess the selected corresponding data, then perform normalization and Gaussian filtering to obtain the distribution function. Specifically, step S3 involves: processing the first... The function for counting the number of events that occur in a camera. First, normalization is performed, then Gaussian filtering is applied to obtain the distribution function of the number of events for different event cameras. , S4: Obtain the minimum objective function based on the distribution function, and calculate the minimum objective function by combining it with the negative value of the correlation, thus completing the start-up delay estimation. Step S4 specifically involves assuming the first... The startup delay of each event camera relative to the first event camera is... The smallest unit of time, Minimize the objective function as The result obtained by minimizing the objective function For the estimated event camera startup delay, where, That is, the first Distribution function of the number of events of a camera. .

2. The method for estimating the start-up delay between event cameras as described in claim 1, characterized in that, Each time data is collected from multiple event cameras, the startup delay between the event cameras during that collection process needs to be calculated.

3. The method for estimating the start-up delay between event cameras as described in claim 2, characterized in that, Step S2 is as follows: Let an event generated by the event camera be denoted as . The timestamp of the event is The timestamp is the time assigned by the event camera when the event occurred, used to represent the time interval between the occurrence of the event and the start time of the event camera. Based on the event timestamps, data collected by cameras for different events is extracted, with the extraction time being the duration after each event camera starts. Data, smallest unit of time To meet the latency accuracy requirements of real-world scenarios, let the time length be... Include The smallest unit of time, Record a unit of time (( , Inner The set of all events generated by the event camera is , , greater than 0 and less than The integer, representing the total number of events occurring within the aforementioned time period, is a function of the number of events, denoted as . , ,in, The number of elements in a set is denoted by . Within a time period The total number of events generated by the event camera is .

4. The method for estimating the start-up delay between event cameras as described in claim 3, characterized in that, Step S3 specifically involves, for the first Event count function for each event camera Normalization process limits the data to... And retain the original relationships between the data, and the first The data from the event camera is normalized to The normalized data are then subjected to Gaussian filtering. The distribution function of the event camera data after filtering is: , For a standard deviation of Gaussian filter.

5. The method for estimating the start-up delay between event cameras as described in claim 1, characterized in that, Different event cameras have the same lens orientation, with their centers on the same horizontal plane. The cameras are fixed, and the data recorded simultaneously is normalized and time-synchronized to obtain a distribution function of the number of events from different event cameras. resemblance.

6. The method for estimating the start-up delay between event cameras as described in any one of claims 1-5, characterized in that, In step S4, the objective function obtained from the distribution function is: The objective function is the sum of squared errors. Bhattacharyy distance Or the negative value of the correlation , Let the startup delay be... ,but .

7. The method for estimating the start-up delay between event cameras as described in claim 6, characterized in that, calculate , The specific calculation method is as follows: Specify the search range for startup time delay, and set the startup time delay within... arrive between, It is a positive integer. , Based on the characteristics of the objective function, The calculation result is between 0 and 1. Let's assume... When taking different values, The minimum value is ,initialization , exist The following two steps are executed in a loop: (1) If Then (2) The value assigned to , Until iteration to Stop the loop. After the loop is completed This is an estimate of the startup delay between event cameras.

Images