A calibration device and method for a large field of view event imaging system

Abstract

The application provides a kind of large field of view event imaging system calibration device and method, solve the technical problems that existing calibration device and method are not suitable for calibrating event camera in large field of view, or need to arrange control points and distribution is limited, or operation is cumbersome, consumes manpower and time.The calibration device includes a drone, a satellite navigation positioning time service device arranged on the drone, a light source arranged below the drone, a base and a plurality of event cameras mounted on the base, the plurality of event cameras are directed to a predetermined area in a large field of view;The drone and the event camera are in communication with the ground station;The drone is used to simulate the specific position of the predetermined area;The satellite navigation positioning time service device receives the signal emitted by the navigation positioning satellite, is used to record the three-dimensional coordinates and the first time stamp in the world coordinate system when the light source is bright and dark, and is transmitted to the ground station in real time through the drone;The light source is used to form the event stream collected by the event camera.

Description

Technical Field

[0001] This invention relates to a calibration device and method, specifically to a calibration device and method for a large field-of-view event imaging system. Background Technology

[0002] Currently, inspired by insect compound eyes, using multiple cameras to form a bionic compound eye has become an important means of obtaining the three-dimensional coordinates of targets in real time with high precision within a large field of view. Event cameras, with their advantages of extremely fast response speed, reduced invalid information, and high dynamic range, have become one of the best choices for constructing bionic compound eyes. However, before measurement, the event camera needs to be calibrated. The relationship between the three-dimensional position of a point in space and its corresponding point in the image is determined by the geometric model of the event camera imaging. The parameters in these geometric models are the event camera parameters. The accuracy of the event camera calibration directly affects the accuracy of the bionic compound eye composed of event cameras in measuring the three-dimensional coordinates of the target.

[0003] Traditional calibration methods typically employ Zhang Zhengyou's calibration method based on a checkerboard target. This method is simple to operate and low-cost, but it is only suitable for calibrating event cameras in close-range, small field-of-view situations, and cannot calibrate event cameras in large field-of-view situations exceeding 10 meters. Alternatively, multiple control points can be deployed within the field of view of a regular camera, and the 3D coordinates of each control point can be measured with high precision to calibrate the regular camera. However, this method is cumbersome, time-consuming, and manpower-intensive, and the control points can only be placed on the ground, limiting their distribution. Another method utilizes a drone equipped with a flashing bulb to fly within the field of view, using events collected by the regular camera and the 3D coordinates collected by the drone to construct feature points for calibration. However, this method does not synchronize the time of the regular camera and the drone, resulting in cumbersome operation and significant time and manpower consumption. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problems that existing calibration devices and methods are not suitable for calibrating event cameras in a large field of view, or require the arrangement of control points with limited distribution, or are cumbersome to operate and consume manpower and time, and to provide a calibration device and method for a large field of view event imaging system.

[0005] The concept of this invention is:

[0006] By using a drone carrying a flashing light source, and combining it with the three-dimensional coordinate information of the drone collected by a satellite navigation, positioning and timing device to form feature points, and by using a high-precision satellite navigation, positioning and timing device and the PTP protocol to achieve time synchronization between the drone and the event camera, the event camera can be quickly calibrated.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A calibration device for a large field-of-view event imaging system, characterized by:

[0009] It includes a drone, a satellite navigation, positioning and timing device mounted on the drone, a light source mounted below the drone, a base and multiple event cameras mounted on the base, with the multiple event cameras facing a predetermined area in a large field of view;

[0010] Both the UAV and the event camera communicate with the ground station in the large field-of-view event imaging system to be calibrated.

[0011] The drone is used to simulate the specific location of a predetermined area;

[0012] The satellite navigation, positioning, and timing device receives signals from navigation and positioning satellites to record the three-dimensional coordinates and first timestamp of the light source in the world coordinate system when it turns on and off, and transmits them to the ground station in real time via the UAV.

[0013] The light source is used to generate the event stream captured by the event camera.

[0014] Furthermore, the event camera records a second timestamp of the light source on / off event via the PTP protocol;

[0015] The drone and the event camera are synchronized via a satellite navigation positioning and timing device and the PTP protocol.

[0016] Furthermore, the base is a hemispherical structure, and multiple mounting holes are provided on the base in the style of a biomimetic compound eye, with multiple event cameras correspondingly installed in the mounting holes.

[0017] Furthermore, the number of mounting holes is 18;

[0018] The PTP protocol is the IEEE 1588 standard.

[0019] Furthermore, the satellite navigation, positioning, and timing device is a GPS timing device or a BeiDou timing device;

[0020] The light source is an LED light source.

[0021] Meanwhile, the present invention also provides a calibration method for a large field-of-view event imaging system, which uses the above-mentioned calibration device for a large field-of-view event imaging system, and is characterized by including the following steps:

[0022] 1) Mount multiple event cameras on a base, facing the predetermined area in a large field of view;

[0023] 2) Control the UAV to hover at different positions in a predetermined area within a wide field of view, and control the light source to turn on and off once each time it hovers, forming an event stream; at the same time, use a satellite navigation, positioning and timing device to record the three-dimensional coordinates and first timestamp of the light source in the world coordinate system in real time when it turns on and off, and use an event camera to collect the event stream, which includes light source on and off events and background events;

[0024] 3) Filter background events in the event stream;

[0025] 4) Extract the image coordinates and second timestamp of the light source illumination / disillusionment event in the camera image coordinate system for each event;

[0026] 5) Extract the three-dimensional coordinates and first timestamp of the light source in the world coordinate system when it is on and off, and transform the three-dimensional coordinates of the light source in the world coordinate system to the preset coordinate system to obtain the three-dimensional coordinates of the light source in the preset coordinate system;

[0027] 6) Based on the first timestamp in step 5) and the second timestamp in step 4), the image coordinates of the light source on / off event in each event camera image coordinate system and the three-dimensional coordinates in the preset coordinate system are used to form feature points. Each event camera is calibrated using the feature points to form a connection between the preset coordinate system and the coordinate system of each event camera.

[0028] 7) By utilizing the connection between the preset coordinate system and the coordinate system of each event camera, establish the coordinate transformation relationship between different event cameras and complete the calibration of the large field-of-view event imaging system.

[0029] Further, step 5) specifically refers to:

[0030] Extract the three-dimensional coordinates and first timestamp of the light source in the world coordinate system when it is on and off, and convert the three-dimensional coordinates of the light source in the world coordinate system to the three-dimensional coordinates in the geodetic coordinate system. Then, based on the origin of the preset coordinate system, convert the three-dimensional coordinates in the geodetic coordinate system to the preset coordinate system to obtain the three-dimensional coordinates of the light source in the preset coordinate system.

[0031] Furthermore, in step 6), each event camera is calibrated using feature points according to Zhang Zhengyou's calibration method.

[0032] Further, step 7) specifically involves:

[0033] By using the rotation matrix and translation vector between the preset coordinate system and the coordinate system of each event camera, the coordinate transformation relationship between different event cameras is established, and the calibration of the large field-of-view event imaging system is completed.

[0034] Furthermore, step 3) specifically involves:

[0035] Background events in the event stream are filtered using a Gaussian filter.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] 1) The calibration device of the large field-of-view event imaging system of the present invention utilizes a drone carrying a flashing light source to hover at multiple locations to construct feature points, eliminating the need to arrange control points and avoiding the problem of limited control point layout in a large field of view.

[0038] 2) The calibration device of the large field-of-view event imaging system of the present invention utilizes a high-precision satellite navigation, positioning and timing device carried by a UAV, which can transmit the three-dimensional coordinate information of the light source in real time without having to download the coordinate information after recovering the UAV, making it more convenient.

[0039] 3) The calibration device of the large field-of-view event imaging system of the present invention utilizes a satellite navigation positioning and timing device and the PTP protocol to achieve high-precision synchronization between the UAV and the event camera. Therefore, the required information can be accurately selected based on the first and second timestamps, reducing the amount of data.

[0040] 4) The calibration method of the large field-of-view event imaging system of the present invention is simple to operate and reduces manpower and time costs. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the calibration device structure for a large field-of-view event imaging system according to the present invention;

[0042] Figure 2 This is a schematic diagram of the network topology used in an embodiment of the present invention;

[0043] Figure 3 This is a flowchart of the calibration method for the large field-of-view event imaging system of the present invention;

[0044] Figure 4 This is a schematic diagram of light source translation in an embodiment of the calibration method for the large field-of-view event imaging system of the present invention.

[0045] Explanation of reference numerals in the attached diagram: 1-UAV, 2-Satellite navigation, positioning and timing device, 3-Light source, 4-Base, 5-Event camera, 6-Navigation and positioning satellite, 7-Master clock. Detailed Implementation

[0046] like Figures 1-2As shown, a calibration device for a large field-of-view event imaging system includes a drone 1, a satellite navigation, positioning, and timing device 2 mounted on the drone 1, a light source 3 positioned below the drone 1, a hemispherical base 4, and 18 event cameras 5. In this embodiment, the satellite navigation, positioning, and timing device 2 is a GPS timing device. Specifically, the base 4 has 18 mounting holes arranged in a biomimetic compound eye pattern, and the 18 event cameras 5 are correspondingly installed in the mounting holes, facing a predetermined area in the large field of view. Both the drone 1 and the event cameras 5 communicate with the ground station in the large field-of-view event imaging system to be calibrated. The drone 1 is used to simulate the specific location of the predetermined area. The GPS timing device receives signals from the navigation and positioning satellite 6 to record the three-dimensional coordinates of the light source 3 in the world coordinate system when it turns on and off, as well as the first timestamp of the light turning on and off event. This data is then transmitted in real time to the ground station through the communication module of the drone 1. The light source 3 is used to form the event stream collected by the event cameras 5. The event cameras 5 record the second timestamp of the light source turning on and off event via the PTP protocol, which is the IEEE 1588 standard. The three-dimensional coordinates recorded by the GPS timing device and the light source illumination events recorded by the event camera 5 are synchronized using the first and second timestamps.

[0047] In this embodiment, there are 18 event cameras 5. The attributes of the event e(x,y,t,p) output by the event camera 5 include: x represents the pixel position in the x-axis image coordinate, y represents the pixel position in the y-axis image coordinate, t is time, and p is the polarity, which is related to the brightness change. The polarity includes positive polarity and negative polarity. If the brightness of the current pixel position is greater than the brightness of the previous time, a positive polarity event is generated, P=1; if the brightness of the current pixel position is less than or equal to the brightness of the previous time, a negative polarity event is generated, P=0.

[0048] Since all 18 event cameras 5 are mounted on the hemispherical base 4, it is unnecessary to equip each event camera 5 with a high-precision GPS timing device in this structure. In this case, the PTP protocol, i.e., the IEEE 1588 standard, can be used to achieve high-precision synchronization of these event cameras 5. The timing principle of the PTP protocol is as follows: within the same local area network, the master clock 7 periodically sends time synchronization messages, and the slave clocks receive these synchronization messages. Simultaneously, they randomly send delay request messages to the master clock 7 and then adjust their own clock deviations through a synchronization algorithm. The PTP protocol layer assembles the synchronization data stream in the system where the master clock resides, and then it passes through the transport layer, network layer, and data link layer. Network multicast is responsible for sending the data stream to the switch, which forwards the data packets to the same multicast group. The slave clocks in the same multicast group receive the synchronization messages and transmit them from the data link layer to the PTP protocol layer for unpacking. At the same time, the delay request messages sent by the slave clocks are assembled by the slave clock protocol layer and then transmitted back to the master clock 7 through the network layer. The principle of back-and-forth transmission is similar. After repeated calculations to obtain a relatively ideal deviation value, the phase difference and frequency difference between the slave clock and the master clock 7 are calculated by calculating the deviation ratio between the slave clock and the master clock 7. The obtained phase difference and frequency difference are then compensated to the slave clock, thereby achieving consistency between the slave clock and the master-slave clock.

[0049] The calibration device for the large field-of-view event imaging system of the present invention first utilizes a drone 1 equipped with a GPS timing device (in other embodiments, a Beidou timing device can be used) and a remotely controllable light source 3 to form a feature point construction device. Then, the drone 1 is controlled to hover at different positions, and feature points are obtained by combining the image coordinates of the light source on / off events collected by the event camera 5 and the three-dimensional coordinates of the light source 3 provided by the GPS timing device. The feature points are used to complete the calibration of the event camera 5. Finally, the coordinate transformation relationship between different event cameras 5 is realized by utilizing the connection between the preset coordinate system and the coordinate system of each event camera, thus completing the calibration of the large field-of-view event imaging system.

[0050] like Figures 3 to 4 As shown, a calibration method for a large field-of-view event imaging system, based on the aforementioned calibration device for a large field-of-view event imaging system, includes the following steps:

[0051] 1) Install 18 event cameras 5 in the mounting holes of the base 4, facing the predetermined area in the large field of view;

[0052] In a wide field of view, the distance between the predetermined area and the event camera 5 is 500m-1000m. At such a long distance, the required feature points can be calibrated using a UAV 1 equipped with a GPS timing device.

[0053] 2) Control the UAV 1 to hover at different positions in the predetermined area in the wide field of view. Each time it hovers, control the light source 3 to turn on and off once to form an event stream. At the same time, use the GPS timing device to record the three-dimensional coordinates and first timestamp of the light source 3 in the world coordinate system when it turns on and off in real time. Use the event camera 5 to collect the event stream, which includes the light source turning on and off events and background events.

[0054] 3) Filter background events in the event stream using Gaussian filtering, specifically:

[0055] The probability that an event is a true event needs to be calculated using a stream of events. For the current event to be processed, e0(x0,y0,t0,p0), a two-dimensional Gaussian kernel G(d,t) is used:

[0056]

[0057]

[0058] Δt i =|t i -t0|

[0059] In the formula: i takes values ​​from 1 to N, Δd i For the current event e0 and the i-th event e i Spatial distance, Δt i For the current event e0 and the i-th event e i The time distance, σ1 is Δd1……Δd N The standard deviation, σ², is Δt₁...Δt. N Standard deviation, x i Represents the i-th event e i The pixel position of the x-axis image coordinate, y i Represents the i-th event e i The pixel positions of the y-axis image coordinates, where x0 represents the pixel position of the current event e0 on the x-axis image coordinates, and y0 represents the pixel position of the current event e0 on the y-axis image coordinates. i Represents the i-th event e i The time, t0 represents the time of the current event e0.

[0060] Convolve the event stream using a two-dimensional Gaussian kernel to obtain the true event probability of the current event e0.

[0061]

[0062]

[0063] In the formula: E stream (d,t) represents the event stream. This represents the convolution operation.

[0064]

[0065] In the formula: λ is the set threshold, when the probability of a real event... When the probability is less than the threshold λ, the current event e0 is considered a background event and is filtered out; when the probability of a true event is less than the threshold λ, the current event e0 is considered a background event and is filtered out. When the current event e0 is greater than or equal to the threshold λ, it is considered a real event, i.e., the light source turning on or off event, and is retained.

[0066] 4) Extract the image coordinates and second timestamp of the light source illumination / disillusionment event in the camera image coordinate system for each event;

[0067] 5) Extract the 3D coordinates and first timestamp of light source 3 in the world coordinate system when it is on and off, and convert the 3D coordinates of light source 3 in the world coordinate system to 3D coordinates in the geodetic coordinate system. Then, based on the origin of the preset coordinate system, convert the 3D coordinates in the geodetic coordinate system back to the preset coordinate system to obtain the 3D coordinates of light source 3 in the preset coordinate system. The specific conversion process is as follows:

[0068] Assume that the coordinates of light source 3 in the world coordinate system (i.e., WGS-84) are P. G (B,L,H), with coordinates P in the geodetic coordinate system. E (X E ,Y E Z E ), with known coordinates P in the geodetic coordinate system O (X O ,Y O Z O As the origin of the preset coordinate system, the coordinates of light source 3 in the preset coordinate system are P. S (X S ,Y S Z S The coordinate transformation is performed using the following formula:

[0069]

[0070] Where B, L, and H represent the longitude, latitude, and altitude of light source 3 in the world coordinate system, respectively, and X E ,Y E Z E Let N represent the x, y, and z coordinates of light source 3 in the geodetic coordinate system, respectively. Let N represent the radius of curvature of the Earth's ellipsoid, and let E represent the first eccentricity of the Earth's ellipsoid.

[0071] Assuming a and b are the lengths of the Earth's semi-major and semi-minor axes, respectively, and χ is the Earth's ellipsoidal flattening rate, we can obtain:

[0072]

[0073] In turn, we can obtain:

[0074]

[0075] After the above steps, the coordinates P of light source 3 in the world coordinate system can be determined. G (B,L,H) converted to coordinates P in the geodetic coordinate system E (X E ,Y E Z E Then, convert it to coordinate P in the preset coordinate system using the following formula. S (X S ,Y S Z S ):

[0076] P S (X S ,Y S Z S ) = P E (X E ,Y E Z E )-P O (X O ,Y O Z O )

[0077] 6) Based on the first timestamp in step 5) and the second timestamp in step 4), construct feature points from the image coordinates of the light source on / off events in each event camera's image coordinate system and their three-dimensional coordinates in the preset coordinate system. Use these feature points to calibrate each event camera 5 according to Zhang Zhengyou's calibration method, thus establishing a connection between the preset coordinate system and each event camera's coordinate system, as shown in the following formula:

[0078] P i =R i P w +T i

[0079] In the formula, P w Let P be the coordinates of point P in the preset coordinate system. i Let R be the coordinates of point P in the camera coordinate system at the i-th event. i and T i Let be the rotation matrix and translation vector from the preset coordinate system to the coordinate system of the i-th event camera.

[0080] 7) Using the rotation matrix and translation vector between the preset coordinate system and the coordinate system of each event camera, establish the coordinate transformation relationship between different event cameras 5, and complete the calibration of the large field-of-view event imaging system, as follows:

[0081] like Figure 4 As shown, based on the known translation matrix T between the two hovering positions of UAV 1, P in step 5) S and P in step 6) i =R i P w +T i Establish the relationship between the preset coordinate system and the coordinate system of the camera for the m-th event:

[0082] P m =R m P s +T m

[0083] Establish the relationship between the preset coordinate system and the coordinate system of the nth event camera:

[0084] P n =R n (P s +T)+T n

[0085] In the formula: R m T m Let R be the rotation matrix and translation vector between the camera coordinate system and the preset coordinate system for the m-th event. n T n These are the rotation matrix and translation vector between the camera coordinate system and the preset coordinate system for the nth event, respectively, and P in the preset coordinate system. S After translation, it becomes P. S +T, P m For P S The coordinates of P in the camera coordinate system for the m-th event. n For P S +T is the coordinate of the camera in the nth event coordinate system. It should be noted that in this embodiment, light source 3 is an LED light source.

[0086] As can be seen from the above, using feature points in a preset coordinate system and their translated feature points as a medium, the coordinate transformation relationship between different event cameras 5 in a compound eye can be established. The calibration method in this embodiment utilizes feature points on the mobile drone 1 to cover the field of view of two adjacent event cameras 5. Since the translation matrix T is known, the rotation matrix between the two event cameras 5 can be calculated, thereby establishing the coordinate transformation relationship between adjacent event cameras 5. This embodiment uses the drone 1 to generate feature points and performs calibration by real-time identification of the flashing LED light source carried by the drone 1 within a large field of view, making it suitable for calibrating event cameras 5 within a large field of view.

Claims

1. A calibration device for a large field-of-view event imaging system, characterized in that: Includes a drone (1), a satellite navigation, positioning and timing device (2) mounted on the drone (1), a light source (3) mounted below the drone (1), a base (4) and multiple event cameras (5) mounted on the base (4), the multiple event cameras (5) facing a predetermined area in a large field of view; Both the UAV (1) and the event camera (5) communicate with the ground station in the large field-of-view event imaging system to be calibrated; The drone (1) is used to simulate the specific location of a predetermined area; The satellite navigation positioning and timing device (2) receives signals from the navigation and positioning satellite (6) to record the three-dimensional coordinates and first timestamp of the light source (3) in the world coordinate system when it is lit and turned off, and transmits them to the ground station in real time through the UAV (1); The light source (3) is used to form the event stream acquired by the event camera (5); The event camera (5) records the second timestamp of the light source on / off event via the PTP protocol; The UAV (1) and the event camera (5) are synchronized through the satellite navigation positioning and timing device (2) and the PTP protocol.

2. The calibration device for the large field-of-view event imaging system according to claim 1, characterized in that: The base (4) is a hemispherical structure. Multiple mounting holes are provided on the base (4) in the style of bionic compound eyes, and multiple event cameras (5) are installed in the mounting holes accordingly.

3. The calibration device for the large field-of-view event imaging system according to claim 2, characterized in that: The number of mounting holes is 18; The PTP protocol is the IEEE 1588 standard.

4. The calibration device for the large field-of-view event imaging system according to claim 1, characterized in that: The satellite navigation, positioning, and timing device (2) is a GPS timing device or a Beidou timing device; The light source (3) is an LED light source.

5. A calibration method for a large field-of-view event imaging system, employing the calibration device for a large field-of-view event imaging system as described in any one of claims 1 to 4, characterized in that, Includes the following steps: 1) Mount multiple event cameras (5) on the base (4) and face the predetermined area in the large field of view; 2) Control the UAV (1) to hover at different positions in a predetermined area in the wide field of view, and control the light source (3) to turn on and off once each time it hovers, forming an event stream; at the same time, use the satellite navigation positioning and timing device (2) to record the three-dimensional coordinates and first timestamp of the light source (3) in the world coordinate system in real time when it turns on and off, and use the event camera (5) to collect the event stream, which includes the light source turning on and off events and background events; 3) Filter background events in the event stream; 4) Extract the image coordinates and second timestamp of the light source illumination / discontinuation event in the camera image coordinate system for each event; 5) Extract the three-dimensional coordinates and first timestamp of the light source (3) in the world coordinate system when it is on and off, and transform the three-dimensional coordinates of the light source (3) in the world coordinate system to the preset coordinate system to obtain the three-dimensional coordinates of the light source (3) in the preset coordinate system; 6) Based on the first timestamp in step 5) and the second timestamp in step 4), the image coordinates of the light source on / off event in each event camera image coordinate system and the three-dimensional coordinates in the preset coordinate system are used to form feature points. Each event camera is calibrated using the feature points (5) to form a connection between the preset coordinate system and the coordinate system of each event camera. 7) By utilizing the connection between the preset coordinate system and the coordinate system of each event camera, establish the coordinate transformation relationship between different event cameras (5) and complete the calibration of the large field-of-view event imaging system.

6. The calibration method for the large field-of-view event imaging system according to claim 5, characterized in that, Step 5) specifically involves: Extract the three-dimensional coordinates and first timestamp of the light source (3) in the world coordinate system when it is on and off, and convert the three-dimensional coordinates of the light source (3) in the world coordinate system into the three-dimensional coordinates in the geodetic coordinate system. Then, according to the origin of the preset coordinate system, convert the three-dimensional coordinates in the geodetic coordinate system into the preset coordinate system to obtain the three-dimensional coordinates of the light source (3) in the preset coordinate system.

7. The calibration method for the large field-of-view event imaging system according to claim 6, characterized in that: In step 6), each event camera is calibrated using feature points according to Zhang Zhengyou's calibration method (5).

8. The calibration method for the large field-of-view event imaging system according to claim 7, characterized in that, Step 7) specifically refers to: By using the rotation matrix and translation vector between the preset coordinate system and the coordinate system of each event camera, the coordinate transformation relationship between different event cameras (5) is established, and the calibration of the large field-of-view event imaging system is completed.

9. The calibration method for the large field-of-view event imaging system according to claim 8, characterized in that, Step 3) specifically involves: Background events in the event stream are filtered using a Gaussian filter.

Images