Asynchronous-frame sensor joint calibration and data generation device and method

Abstract

This invention relates to an asynchronous-frame sensor joint calibration and data generation device and method, belonging to the field of high-speed dynamic machine vision perception and multimodal sensor joint calibration technology. It addresses the technical problem in existing technologies where asynchronous event vision sensors rely on changing "dynamic" excitation to output asynchronous event streams, while traditional visible light cameras require the calibration target to remain "static" to obtain clear, blur-free images. The device includes a display unit, an optical conversion unit, a beam splitting and synchronous data acquisition unit, and a data processing unit. This invention effectively resolves the contradiction between "dynamic" and "static" requirements in multi-sensor joint acquisition, utilizing the macroscopic static and microscopic dynamic characteristics of a pixelated light source array to achieve high-quality synchronous data acquisition and joint calibration of asynchronous event vision sensors and traditional frame-scanning visible light cameras in static environments.

Description

Technical Field

[0001] This invention relates to the field of high-speed dynamic machine vision perception and multimodal sensor joint calibration technology, and particularly to an asynchronous-frame sensor joint calibration and data generation device and method based on a static optical reference. Background Technology

[0002] With the increasing application of high-speed dynamic vision perception technology in fields such as autonomous driving and drones, traditional single vision sensors are no longer able to meet the application requirements in complex scenarios. Hybrid vision systems that integrate asynchronous event sensors and traditional frame-type visible light cameras have shown great potential because they can balance high dynamic range, microsecond-level latency and rich texture information.

[0003] However, the primary prerequisite for the effective fusion of information from two types of sensors is high-precision joint calibration, and existing calibration methods face fundamental challenges in this regard. The core contradiction lies in the fact that asynchronous event vision sensors rely on changing "dynamic" stimuli to output asynchronous event streams, while traditional visible light cameras require the calibration target to remain "static" to obtain clear, unblurred images. This fundamental contradiction makes developing a common calibration benchmark capable of simultaneously and efficiently stimulating event streams and providing high-quality static images a pressing problem for the industry. Furthermore, the current technological context severely lacks multimodal standard datasets that provide accurate geometric truth and standard platforms that can reproducibly evaluate various fusion algorithms, further hindering the development of related fields.

[0004] Therefore, there is an urgent need for a device that can provide a high-precision static common reference without mechanical movement. It can provide a common, high-precision calibration reference for two types of sensors in a static environment without mechanical movement, and further provide key data support for the research and evaluation of fusion algorithms. Summary of the Invention

[0005] This invention addresses the technical problem that existing asynchronous event vision sensors rely on changing "dynamic" excitation to output asynchronous event streams, while traditional visible light cameras require calibrated targets to remain "static" to obtain clear, unblurred images. It provides an asynchronous-frame sensor joint calibration and data generation device and method based on a static optical reference.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] An asynchronous-frame sensor joint calibration and data generation device, which is an asynchronous-frame sensor joint calibration and data generation device based on a static optical reference, includes: a display unit, an optical conversion unit, a beam splitting and synchronous data acquisition unit, and a data processing unit;

[0008] The display unit is used to display static calibration patterns under the control of the data processing unit;

[0009] The object-side focal plane of the optical conversion unit coincides with the display surface of the display unit, and is used to convert the divergent light emitted from each target point on the display unit into parallel light for emission.

[0010] The beam splitting and synchronous data acquisition unit includes: a beam splitter, a rigid mounting base, an attitude rotation mechanism, an asynchronous event vision sensor, and a frame-scanning visible light camera; the beam splitter, the asynchronous event vision sensor, and the frame-scanning visible light camera are respectively mounted on the rigid mounting base; the attitude rotation mechanism is connected to the rigid mounting base and is used to drive the rigid mounting base to rotate, thereby changing its spatial attitude relative to the parallel beam emitted by the optical conversion unit.

[0011] Beam splitters are used to receive parallel light emitted from optical conversion units;

[0012] An asynchronous event vision sensor and a frame-scanning visible light camera are used to receive parallel light after it has been split by a beam splitter.

[0013] The data processing unit is connected to the asynchronous event vision sensor and the frame-scanning visible light camera respectively. It is used to receive and process the event stream output by the asynchronous event vision sensor and the image acquired by the frame-scanning visible light camera, and solve for the internal parameters and relative pose parameters of the asynchronous event vision sensor and the frame-scanning visible light camera respectively.

[0014] In the above technical solution, a spectral premodulation filter element is provided in the optical path between the asynchronous event vision sensor and the beam splitter, and in the optical path between the frame-scanning visible light camera and the beam splitter. The spectral premodulation filter element is used to suppress ambient stray light and adjust the intensity of incident light.

[0015] In the above technical solution, the spectral premodulation filtering element is a filter.

[0016] In the above technical solution, the data processing unit is a computer.

[0017] In the above technical solution, the display unit is a pixelated programmable light source array.

[0018] In the above technical solution, the pixelated programmable light source array is a Micro-OLED microdisplay screen with a pixel size of 7.2µm.

[0019] In the above technical solution, the optical conversion unit is a collimating optical system.

[0020] In the above technical solution, the collimating optical system is a five-element lens group with a focal length of 60mm and a field of view of 17.5°.

[0021] An asynchronous-frame sensor joint calibration and data generation method, applicable to the aforementioned device, includes the following steps:

[0022] First, the device is calibrated; the data processing unit controls the display unit to continuously display the static calibration pattern; after the display unit enters a stable periodic brightness refresh state, the data processing unit simultaneously sends a rising edge pulse to the asynchronous event vision sensor and the frame scanning visible light camera as a synchronization trigger signal, and synchronously acquires event stream-static image data pairs;

[0023] Subsequently, the spatial attitude of the device is changed by the attitude rotation mechanism, and the above synchronous acquisition process is repeated to obtain multiple sets of event stream-static image data pairs from different perspectives.

[0024] After acquisition, the data processing unit extracts the sub-pixel coordinates of the calibration target from the event stream data and the static image, respectively. After associating these two sets of coordinates with the known object world coordinates, they are substituted into a unified camera imaging model. The overall reprojection error is minimized through a nonlinear optimization algorithm, thereby simultaneously solving for the internal parameters of the asynchronous event vision sensor, the internal parameters of the frame-scanning visible light camera, and the relative pose parameters between the asynchronous event vision sensor and the frame-scanning visible light camera.

[0025] The present invention has the following beneficial effects:

[0026] The asynchronous-frame sensor joint calibration and data generation device and method based on static optical reference of the present invention effectively solves the contradiction between "dynamic and static" requirements when multiple sensors are jointly acquired. By utilizing the macroscopic static and microscopic dynamic characteristics of pixelated light source array, it realizes high-quality synchronous data acquisition and joint calibration of asynchronous event vision sensor and traditional frame scanning visible light camera in static environment.

[0027] The asynchronous-frame sensor joint calibration and data generation device and method based on static optical reference of the present invention transcends the limitation of the single function of traditional calibration devices. It can construct an "event stream-image" paired dataset containing accurate intrinsic and extrinsic parameter values ​​by changing the relative pose of the two cameras and collecting data synchronously, thus providing a support platform for the reference data of multimodal fusion algorithms.

[0028] The asynchronous-frame sensor joint calibration and data generation device and method based on static optical reference of the present invention has important guiding significance for improving the perception reliability of hybrid vision systems in fields such as autonomous driving and intelligent robots. Attached Figure Description

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0030] Figure 1 This is a structural diagram of the asynchronous-frame sensor joint calibration and data generation device based on a static optical reference according to the present invention.

[0031] Figure 2 This is a flowchart of the asynchronous-frame sensor joint calibration and data generation method based on a static optical reference according to the present invention.

[0032] Figure 3 This is a logic diagram for the synchronous timing information acquisition of the present invention.

[0033] The reference numerals in the figure are:

[0034] 1-Pixelated programmable light source array; 2-Collimating optical system; 3-Beam splitter; 4-Rigid mounting base; 5-Attitude rotation mechanism; 6-Asynchronous event vision sensor; 7-Frame scanning visible light camera; 8-Spectral premodulation filter element. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings.

[0036] like Figure 1 As shown, the asynchronous-frame sensor joint calibration and data generation device based on a static optical reference of the present invention includes: a display unit, wherein the display unit is a pixelated programmable light source array 1, and the pixelated programmable light source array 1 is a Micro-OLED microdisplay screen; an optical conversion unit, wherein the optical conversion unit is a collimating optical system 2, and the collimating optical system 2 is a five-element collimating projection optical system; a beam splitting and synchronous data acquisition unit, wherein the beam splitting and synchronous data acquisition unit includes: a beam splitter 3, a rigid mounting base 4, an attitude rotation mechanism 5, an asynchronous event visual sensor 6, a frame scanning visible light camera 7, and a spectral premodulation filter element 8, wherein the spectral premodulation filter element 8 is a filter; and a data processing unit, wherein the data processing unit is a computer.

[0037] Pixelated programmable light source array 1, which is placed at the very front of the optical path ( Figure 1 The rightmost position in the diagram is used to display the static calibration pattern under the control of the data processing unit.

[0038] The object-side focal plane of the collimating optical system 2 coincides with the display surface of the pixelated programmable light source array 1, and is used to convert the divergent light emitted from each target point on the pixelated programmable light source array 1 into parallel light.

[0039] The beam splitter 3, the asynchronous event vision sensor 6, and the frame-scanning visible light camera 7 are respectively mounted on the rigid mounting base 4; the attitude rotation mechanism 5 is connected to the rigid mounting base 4 and is used to drive the rigid mounting base 4 to rotate, thereby changing its spatial attitude relative to the parallel beam emitted by the collimating optical system 2.

[0040] The beam splitter 3 is used to split the incident parallel light into two beams of similar energy, which are then guided to the asynchronous event vision sensor 6 and the frame-scanning visible light camera 7, respectively.

[0041] In this embodiment, the pixelated programmable light source array 1 uses a Micro-OLED microdisplay screen with a pixel size of 7.2µm to display a computer-generated 3×7 asymmetric circular grid calibration pattern. The collimating optical system 2 is a five-element lens group with a focal length of 60mm and a field of view of 17.5°. Its object-side focal plane is precisely aligned with the display surface of the Micro-OLED microdisplay screen. It is used to convert the divergent light emitted from each target point on the Micro-OLED microdisplay screen into highly collimated parallel light to establish a static optical reference without perspective distortion.

[0042] The asynchronous event vision sensor 6 (in this embodiment, an event camera with a resolution of 640×480) and the frame-scanning visible light camera 7 (in this embodiment, a high-resolution global shutter camera) are placed together with the beam splitter 3 on a rigid mounting base 4, forming a unit that can be precisely rotated around a single axis, thereby changing the spatial orientation of the two cameras (asynchronous event vision sensor 6 and frame-scanning visible light camera 7) relative to the incident parallel light reference. Filters serving as spectral pre-modulation filtering elements 8 are respectively provided before the entrances of the two beam splitting paths (the optical path between the asynchronous event vision sensor 6 and the beam splitter 3, and the optical path between the frame-scanning visible light camera 7 and the beam splitter 3) to suppress ambient stray light and precisely adjust the incident light intensity.

[0043] like Figure 2 As shown ( Figure 2 (Only a process outline is shown). The asynchronous-frame sensor joint calibration and data generation method based on a static optical reference of the present invention is applicable to the above-mentioned asynchronous-frame sensor joint calibration and data generation device based on a static optical reference. The process is as follows:

[0044] First, the system (i.e., the device of this invention) is calibrated. The computer controls the Micro-OLED microdisplay screen, which serves as a pixelated programmable light source array 1, to continuously display a static calibration pattern. After the Micro-OLED microdisplay screen enters a stable periodic brightness refresh state (refresh rate of 60Hz), the light enters the collimating optical system 2 and emits highly collimated parallel light. After being split by a semi-transparent mirror (beam splitter 3), the computer simultaneously sends a rising edge pulse to the asynchronous event visual sensor 6 and the frame-scanning visible light camera 7 as a synchronization trigger signal, and synchronously acquires event stream-static image data pairs.

[0045] The core logic of synchronous acquisition lies in resolving the contradiction between the need for "dynamic change" in the asynchronous event vision sensor 6 (acting as an event camera) and the need for "static clarity" in the frame-scanning visible light camera 7. The specific timing is as follows: Figure 3 As shown. Upon receiving the synchronization signal, the asynchronous event vision sensor 6 aligns its internal clock with the trigger time and begins continuously recording the event stream triggered by the micro-refreshes of the screen pixels. Upon receiving the synchronization signal, the frame-scanning visible light camera 7 immediately initiates a single-frame exposure with an exposure time T set to 100ms. This exposure duration covers approximately six refresh cycles of the screen (which serves as the Micro-OLED microdisplay screen of the pixelated programmable light source array 1). Its key functions are: firstly, to provide the event camera with continuous and sufficient dynamic excitation to generate a high signal-to-noise ratio event stream; and secondly, to average the multiple micro-brightness fluctuations of the screen (which serves as the Micro-OLED microdisplay screen of the pixelated programmable light source array 1) through the integration process of the frame-scanning visible light camera 7, ultimately capturing a high-quality static calibration image without flicker or stripes.

[0046] Subsequently, by changing the overall spatial attitude of the device through the attitude rotation mechanism 5, the above synchronous acquisition process is repeated to obtain multiple sets of "event stream-static image" data pairs from different perspectives.

[0047] After data acquisition, the computer extracts the sub-pixel coordinates of the calibration target points from the event stream data (extracted through spatiotemporal clustering and model fitting algorithms) and the static images (extracted through traditional image processing algorithms). These two sets of coordinates are then correlated with the known object-world coordinates and substituted into a unified camera imaging model. A nonlinear optimization algorithm is used to minimize the overall reprojection error, thereby simultaneously solving for the intrinsic parameters (focal length, principal point, distortion) of the asynchronous event vision sensor 6, the intrinsic parameters (focal length, principal point, distortion) of the traditional frame-scanning visible light camera 7, and the relative pose between the two.

[0048] Building upon the high-precision calibration of the system itself, the asynchronous-frame sensor joint calibration and data generation device based on a static optical benchmark of this invention can be further used to generate standard datasets. The computer, based on the precisely solved parameters, controls the measuring head to rotate to a series of preset orientations and acquires data using the same synchronous timing sequence, automatically packaging and generating a paired benchmark dataset of "event stream-image" with complete geometric truth values, serving the training and evaluation of subsequent fusion algorithms.

[0049] To verify the performance of the calibration system of the asynchronous-frame sensor joint calibration and data generation device based on static optical reference of the present invention in a real-world scenario, it can be switched to performance verification mode: the Micro-OLED microdisplay screen is replaced with a high dynamic range projector, projecting a simulated extreme scene pattern (such as a tunnel exit under strong light) onto the object-side focal plane of the collimating optical system 2. Based on the calibrated parameters, the system controls the rotation of the measuring head and acquires data using the aforementioned synchronous timing sequence, thereby obtaining multimodal data pairs capable of handling extreme lighting conditions under known precise geometric relationships. Inputting these data pairs into the fusion algorithm to be evaluated allows for quantitative analysis of the algorithm's robustness and accuracy under known conditions.

[0050] This embodiment fully demonstrates that the asynchronous-frame sensor joint calibration and data generation device based on static optical reference of the present invention has a clear structure and a well-defined process. It successfully integrates static display, synchronous acquisition and joint optimization into one, which not only realizes the precise calibration of multimodal vision sensors, but also provides a complete process from sensor calibration to dataset generation. It provides an effective new solution to solve the bottleneck problem of high dynamic vision perception technology.

[0051] The asynchronous-frame sensor joint calibration and data generation device and method based on static optical reference of the present invention effectively solves the contradiction between "dynamic and static" requirements when multiple sensors are jointly acquired. By utilizing the macroscopic static and microscopic dynamic characteristics of pixelated light source array, it realizes high-quality synchronous data acquisition and joint calibration of asynchronous event vision sensor and traditional frame scanning visible light camera in static environment.

[0052] The asynchronous-frame sensor joint calibration and data generation device and method based on static optical reference of the present invention transcends the limitation of the single function of traditional calibration devices. It can construct an "event stream-image" paired dataset containing accurate intrinsic and extrinsic parameter values ​​by changing the relative pose of the two cameras and collecting data synchronously, thus providing a support platform for the reference data of multimodal fusion algorithms.

[0053] The asynchronous-frame sensor joint calibration and data generation device and method based on static optical reference of the present invention has important guiding significance for improving the perception reliability of hybrid vision systems in fields such as autonomous driving and intelligent robots.

[0054] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An asynchronous-frame sensor joint calibration and data generation device, characterized in that, It is an asynchronous-frame sensor joint calibration and data generation device based on static optical reference, including: display unit, optical conversion unit, spectral and synchronous data acquisition unit and data processing unit; The display unit is used to display static calibration patterns under the control of the data processing unit; The object-side focal plane of the optical conversion unit coincides with the display surface of the display unit, and is used to convert the divergent light emitted from each target point on the display unit into parallel light for emission. The beam splitting and synchronous data acquisition unit includes: a beam splitter (3), a rigid mounting base (4), an attitude rotation mechanism (5), an asynchronous event vision sensor (6), and a frame-scanning visible light camera (7); the beam splitter (3), the asynchronous event vision sensor (6), and the frame-scanning visible light camera (7) are respectively mounted on the rigid mounting base (4); the attitude rotation mechanism (5) is connected to the rigid mounting base (4) and is used to drive the rigid mounting base (4) to rotate and change its spatial attitude relative to the parallel beam emitted by the optical conversion unit; The beam splitter (3) is used to receive parallel light emitted by the optical conversion unit; The asynchronous event vision sensor (6) and the frame-scanning visible light camera (7) are used to receive parallel light after it has been split by the beam splitter (3); The data processing unit is connected to the asynchronous event vision sensor (6) and the frame scanning visible light camera (7) respectively. It is used to receive and process the event stream output by the asynchronous event vision sensor (6) and the image acquired by the frame scanning visible light camera (7), and solve the internal parameters and relative pose parameters of the asynchronous event vision sensor (6) and the frame scanning visible light camera (7) respectively.

2. The asynchronous-frame sensor joint calibration and data generation device according to claim 1, characterized in that, In the optical path between the asynchronous event vision sensor (6) and the beam splitter (3), and in the optical path between the frame-scanning visible light camera (7) and the beam splitter (3), a spectral pre-modulation filter element (8) is provided respectively; the spectral pre-modulation filter element (8) is used to suppress ambient stray light and adjust the incident light intensity.

3. The asynchronous-frame sensor joint calibration and data generation device according to claim 2, characterized in that, The spectral pre-modulation filter element (8) is a filter.

4. The asynchronous-frame sensor joint calibration and data generation device according to claim 1, characterized in that, The data processing unit is a computer.

5. The asynchronous-frame sensor joint calibration and data generation device according to claim 1, characterized in that, The display unit is a pixelated programmable light source array (1).

6. The asynchronous-frame sensor joint calibration and data generation device according to claim 5, characterized in that, The pixelated programmable light source array (1) is a Micro-OLED microdisplay screen with a pixel size of 7.2µm.

7. The asynchronous-frame sensor joint calibration and data generation device according to claim 1, characterized in that, The optical conversion unit is a collimating optical system (2).

8. The asynchronous-frame sensor joint calibration and data generation device according to claim 7, characterized in that, The collimating optical system (2) is a five-element lens group with a focal length of 60 mm and a field of view of 17.5°.

9. An asynchronous-frame sensor joint calibration and data generation method, applicable to the device described in claim 1, characterized in that, Includes the following steps: First, the device is calibrated; the data processing unit controls the display unit to continuously display the static calibration pattern. After the display unit enters a stable periodic brightness refresh state, the data processing unit simultaneously sends a rising edge pulse to the asynchronous event visual sensor (6) and the frame scanning visible light camera (7) as a synchronization trigger signal to synchronously acquire event stream-static image data pairs. Subsequently, the spatial attitude of the device as a whole is changed by the attitude rotation mechanism (5), and the above synchronous acquisition process is repeated to obtain multiple sets of event stream-static image data pairs from different perspectives. After the acquisition is completed, the data processing unit extracts the sub-pixel coordinates of the calibration target from the event stream data and the static image respectively. After associating these two sets of coordinates with the known object world coordinates, they are substituted into a unified camera imaging model. The overall reprojection error is minimized through a nonlinear optimization algorithm, thereby simultaneously solving the internal parameters of the asynchronous event vision sensor (6), the internal parameters of the frame-scanning visible light camera (7), and the relative pose parameters between the asynchronous event vision sensor (6) and the frame-scanning visible light camera (7).

Images