Method and apparatus for hardware synchronization and calibration of an event camera with an infrared depth camera

Abstract

This invention discloses a hardware synchronization and calibration method and apparatus for an event camera and an infrared depth camera. It uses an external hardware-triggered reference pulse synchronization signal to achieve spatiotemporal alignment between multimodal cameras, thereby obtaining complete multimodal information and their spatial positional correspondences in each acquisition and scanning cycle, thus achieving stable optimization and reconstruction. This invention utilizes the external signal triggering mode of the event camera to embed precise timestamp markers in the event stream, performing hardware-level synchronization with the infrared depth camera during the exposure cycle to achieve deterministic timestamp alignment. This invention detects and corrects timestamp information that deviates continuously during the camera acquisition process, employing a flashing checkerboard calibration method to achieve simultaneous calibration of the event camera and the infrared depth camera. Finally, the effectiveness of the synchronization and calibration between multimodal cameras is verified through projection and back-projection methods, with the error of the joint calibration of intrinsic and extrinsic parameters controlled within 0.3 pixels.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and 3D vision, and in particular to a hardware synchronization and calibration method and apparatus for an event camera and an infrared depth camera (RGB-D camera). Background Technology

[0002] Multimodal sensor fusion technology can combine sensing data with different physical characteristics, and has wide applications in fields such as 3D spatial reconstruction, autonomous driving, and high-dynamic obstacle avoidance. Event cameras, as a breakthrough sensor in this field, achieve microsecond-level temporal resolution and extremely high dynamic range by asynchronously sensing event streams generated by changes in brightness. However, event cameras lack dense depth and color information, prompting researchers to combine them with traditional infrared depth cameras (RGB-D cameras) to achieve high-precision perception of complex scenes.

[0003] Existing multi-sensor alignment methods largely rely on system-level timestamp alignment, fundamentally due to the inherent mismatch between asynchronous event streams and synchronous image frames: event cameras passively trigger data based on pixel-level brightness changes, while RGB-D cameras actively sample according to a fixed exposure period. In real-world high-speed motion scenarios, system scheduling delays and communication bandwidth fluctuations typically lead to millisecond-level synchronization deviations. This not only limits the system's ability to reconstruct high-frequency moving objects but also easily generates significant reprojection artifacts during the multimodal fusion stage, further affecting the accuracy of 3D reconstruction and incurring significant computational alignment overhead. Therefore, achieving hardware-level synchronization between event cameras and RGB-D cameras, and constructing a high-precision calibration system compatible with asynchronous pulses and synchronous depth information, is an important research direction. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a hardware synchronization and calibration method for an event camera and an infrared depth camera. This method achieves reliable synchronization between the event camera and the infrared depth camera through a hardware-level reference synchronization control signal and achieves accurate calibration of the multimodal camera system through a flashing checkerboard calibration method, thereby enabling long-term stable data acquisition.

[0005] The objective of this invention is achieved through the following technical solution: a hardware synchronization and calibration method for an event camera and an infrared depth camera, the method comprising the following steps: (1) Multimodal camera synchronization at the hardware level: Based on a unified synchronization control signal, the timestamps of the event camera and the infrared depth camera are synchronized; (2) Calibration method of blinking checkerboard: The calibration method of blinking checkerboard is used to simultaneously realize the calibration operation of event camera and depth infrared camera; (3) Calibration and verification of projection and back projection: Based on the intrinsic parameters obtained from the calibration of the infrared depth camera, the pixels are projected in parallel into the spatial coordinates. Then, based on the relative position and attitude of the event camera obtained from the calibration, the spatial coordinate point cloud is projected onto the plane of the event camera. The calibration error is measured based on the error of the corresponding pixel coordinate position.

[0006] Furthermore, the specific steps for synchronizing the multimodal camera group at the hardware level are as follows: (1.1) Using the STM32F103 embedded tool as an external trigger source, the TIM basic timer is used at the hardware circuit level to generate the corresponding synchronization pulse signal. The same pulse synchronization signal is output from different channels of the same TIM and connected to the event camera and infrared depth camera to achieve timestamp alignment. (1.2) When the system device is powered on, extract the images of the first few frames acquired by the event camera and the infrared depth camera respectively to evaluate the power-on time deviation of the multimodal device. During the shooting and acquisition process, continuously adjust the deviation values ​​of the timestamps recorded by the event camera and the infrared depth camera according to the first-order information of time (i.e. the rate of change of deviation with time). The deviation is continuously corrected by the relative timestamp to avoid the situation where the error gradually increases during the operation of the device. (1.3) Due to the inconsistency in crystal frequency and precision between the event camera and the infrared depth camera, coupled with hardware and numerical precision rounding errors, higher precision values ​​will be discarded. This will cause serious frame errors during long-term operation. When correcting the deviation, each newly generated dynamic error needs to be added to the cumulative information as the deviation change rate to correct the total deviation.

[0007] Furthermore, the specific steps for calibrating the blinking checkerboard grid are as follows: (2.1) Use the flashing mode to capture the checkerboard image, so that the event camera can trigger the event of the event camera in the flashing mode in a static position, capture the event information, accumulate the events and remove dynamic points to obtain a stable event image with the corresponding acquisition frequency, and use the corner detection algorithm to find the checkerboard edge and the corresponding corner information for calibration; (2.2) The infrared depth camera acquires checkerboard images within its inherent acquisition cycle, and uses a corner detection algorithm to find the checkerboard edges and corresponding corner information for calibration; (2.3) Collect data at different distances, up and down, left and right on the two pixel planes. After each collection starts at a new position, keep the checkerboard stationary and record the stationary and flashing modes respectively. This allows both the event camera and the infrared depth camera to collect the checkerboard data at that position. The stability of the calibration operation is improved by using checkerboard image information with multiple segments of annotation. (2.4) The multimodal camera system is calibrated by combining intrinsic and extrinsic parameters in MATLAB software. The calibration operation of the system can control the error within 0.3 pixels. The calibration error of the multimodal camera system is checked by projection and reprojection methods.

[0008] Furthermore, the specific steps for the calibration and verification methods of projection and back projection are as follows: (3.1) Based on the intrinsic parameters obtained from the calibration of the infrared depth camera (RGB-D camera) and the standard model of pinhole imaging, calculate the camera coordinate information of each pixel in the RGB-D image at the corresponding timestamp, and use Nvidia's CUDA (Unified Computing Architecture) to efficiently project the corresponding pixels into the spatial coordinates in parallel. (3.2) Based on the position and attitude information of the event camera relative to the infrared depth camera obtained from the calibration information, the infrared depth camera is projected onto the point cloud information of the spatial coordinates through the standard model of pinhole imaging. The point cloud information is then projected back onto the imaging plane of the event camera using CUDA (Unified Computing Architecture) of the Nvidia graphics card. The pixel coordinate information of each spatial point cloud information corresponding to the event camera plane is calculated. (3.3) Under the same period, based on the image information accumulated by the event camera, compare it with the image information obtained by the back projection operation, measure the error of the corresponding pixel position, and thus determine the error of the calibration operation.

[0009] Furthermore, in the synchronization schemes for event cameras and infrared depth cameras, software synchronization schemes are basically based on multi-threading and multi-queue implementation. However, for high-frequency devices such as event cameras, data accumulation can easily lead to queue overflow and prevent effective synchronization. Hardware synchronization schemes include methods where the camera actively emits signals to achieve synchronization. However, not all devices support actively emitting signals as a driving source. Therefore, using an external circuit as the synchronization signal source (External Trigger) for the multimodal camera system can achieve stable timestamp alignment and precise control of the triggering cycle.

[0010] Furthermore, in the process of synchronizing multimodal camera groups, in addition to synchronization at the hardware level, a mechanism is also needed to find relevant frames between software. The two devices will have a basic deviation when they are powered on, and the deviation needs to be continuously corrected by the relative timestamp. In addition, the random error of the hardware timestamp of each frame needs to be continuously adjusted. The dynamic random deviation is much smaller than the trigger frame rate interval. Each newly generated dynamic error is added to the cumulative information, so as to correct the total deviation as the deviation change rate. The calculation method of the total deviation is obtained as follows: Total deviation = Basic deviation (power-on time deviation) + Dynamic (random) deviation * Number of acquired frames.

[0011] Furthermore, during the operation of the event camera and infrared depth camera system, the infrared depth camera (RGB-D camera) actively emits infrared light. The depth is calculated based on the reflection of infrared light at the corresponding location. The actively emitted infrared light has a wavelength of around 800nm, which can trigger the event camera with a receiving wavelength between 300nm and 1100nm. To eliminate the interference caused by the active infrared light, a visible light filter with a wavelength of 400nm to 700nm is used to filter the event camera. At the same time, it is checked and confirmed that adding the filter will not cause distortion of the images acquired by the event camera.

[0012] In a second aspect, the present invention provides a hardware synchronization and calibration device for an event camera and an infrared depth camera, comprising an event camera and an infrared depth camera, and a hardware circuit module for stably triggering an external synchronization signal. The event camera, the infrared depth camera, and the hardware circuit module are characterized in that, when the camera module and the hardware circuit module perform the synchronization operation, they implement a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-7.

[0013] Thirdly, the present invention provides a hardware synchronization and calibration device for an event camera and an infrared depth camera, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, when the processor executes the executable code, it implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-7.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a program stored thereon, characterized in that, when the program is executed by a processor, it implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-7.

[0015] Fifthly, the present invention provides a computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-7.

[0016] The beneficial effects of this invention are as follows: This invention proposes a hardware synchronization and calibration method for event cameras and infrared depth cameras, which effectively realizes hardware synchronization between event cameras and infrared depth cameras, enables the system to operate stably for a long time, keeps the matching error within ±1 error, and realizes accurate calibration of event cameras and infrared depth cameras simultaneously. The calibration effect after projection and reprojection error detection meets the accuracy requirements of three-dimensional reconstruction.

[0017] In summary, this invention provides a hardware synchronization and calibration method and apparatus for an event camera and an infrared depth camera. It proposes a hardware synchronization and calibration method for event cameras and infrared depth cameras, enabling the system to operate stably for extended periods within a ±1 error range and achieving simultaneous calibration of the event camera and the infrared depth camera. This method is applicable to the synchronization and calibration tasks of event cameras and infrared depth cameras and has further research value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below: Figure 1 A schematic diagram of the synchronization signal output for an infrared depth camera.

[0019] Figure 2 A schematic diagram of outputting a synchronization signal to the event camera.

[0020] Figure 3 A schematic diagram showing the output of a synchronization signal to an external hardware circuit.

[0021] Figure 4 This diagram illustrates the synchronized operation of an event camera and an infrared depth camera.

[0022] Figure 5 Point cloud fusion image after projection of images from event camera and infrared depth camera Figure 6 A schematic diagram illustrating the synchronization of an event camera and an infrared depth camera; Figure 7 A schematic diagram illustrating the simultaneous calibration of an event camera and an infrared depth camera; Figure 8 This is a schematic diagram for the calibration of projection and back projection; Figure 9 This is a structural diagram of a hardware synchronization and calibration device for an event camera and an infrared depth camera. Detailed Implementation

[0023] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This demonstrates a framework for using an infrared depth camera as a signal source to output a synchronization signal to an event camera for synchronization.

[0025] Figure 2 This demonstrates a framework for synchronizing an event camera as a signal source by outputting a synchronization signal to an infrared depth camera.

[0026] Figure 3 The demonstration showed that an external hardware circuit acts as a signal source to emit a synchronization control signal, enabling long-term stable synchronization of the timestamps of the infrared depth camera and the event camera.

[0027] Figure 4 The timeline of the event camera and infrared depth camera working together after synchronization is demonstrated. The basic deviation needs to be estimated at system startup, and the random error of the hardware timestamp for each frame needs to be adjusted at each corresponding clock cycle frequency.

[0028] Figure 5 This paper presents a fused image of a depth point cloud projected from an RGB image and aligned with an event camera. The background texture represents the mode of the RGB color point cloud projected onto the event pixels. Missing parts are caused by parallax, the FOV of the depth camera, and noise from the depth camera itself. The red and blue edges represent the cumulative effect of events triggered by the event camera, with red indicating positively polarized events and blue indicating negatively polarized events. The two camera modalities are aligned using hardware synchronization and the aforementioned flicker calibration module, providing excellent multimodal support for subsequent downstream tasks such as 3D reconstruction and high-speed object detection.

[0029] This invention provides a hardware synchronization and calibration method for an event camera and an infrared depth camera. While software synchronization schemes for event cameras and infrared depth cameras are generally based on multi-threading and multi-queue implementations, data accumulation in high-frequency devices like event cameras can easily lead to queue overflow and prevent effective synchronization. Hardware synchronization schemes include methods where the camera actively emits signals for synchronization, but not all devices support actively emitting signals as a driving source. Therefore, using an external circuit as the synchronization signal source (External Trigger) for the multimodal camera system can achieve stable timestamp alignment and precise control of the triggering period. The key steps of this invention are further described in detail below: (1) Hardware-level synchronization of event cameras and infrared depth cameras: such as Figure 6 As shown, the core of this invention lies in proposing hardware-level synchronization of an event camera and an infrared depth camera. An external synchronization signal trigger is designed to control the synchronization of the acquisition cycles of the event camera and the infrared depth camera. Using a unified synchronization control signal generated by an STM32 circuit trigger circuit as a reference, the timestamps of the event camera and the infrared depth camera are synchronized. For each arriving synchronization signal, the system continuously corrects the deviation based on the rate of change of the deviation over time, achieving stable acquisition over a long period and keeping the matching error within ±1%. The specific implementation steps are as follows: (1.1) This invention uses a track bracket to fix the two devices on the same plane and uses a hardware-triggered synchronization signal to achieve timestamp synchronization between the event camera and the infrared depth camera. The STM32F103 embedded tool is used as an external trigger source. At the hardware circuit level, the TIM basic timer is used to generate the corresponding synchronization pulse signal. Different channels of the same TIM output the same pulse synchronization signal to different devices to achieve timestamp alignment. (1.2) When the system device is powered on, the first few frames of the event camera and the infrared depth camera are extracted to evaluate the power-on time deviation of the multimodal device. During the shooting and acquisition process, the deviation values ​​of the timestamps recorded by the two devices are continuously adjusted according to the first-order information of time (i.e., the rate of change of deviation with time). The deviation is continuously corrected by the relative timestamps to avoid the situation where the error gradually increases during the operation of the device. (1.3) Due to the inconsistency in crystal frequency and accuracy between actual devices, coupled with hardware and numerical accuracy rounding errors, higher precision values ​​may be discarded. This can cause serious frame errors during long-term operation. When correcting the deviation, each newly generated dynamic error needs to be added to the cumulative information as the deviation change rate to correct the total deviation.

[0030] (2) Simultaneous calibration of event camera and infrared depth camera: such as Figure 7 As shown, the core of this invention lies in proposing a method for calibrating a blinking checkerboard pattern. The method involves capturing images of a blinking checkerboard pattern, using a corner detection algorithm to find and align the checkerboard edges and corners, and stably triggering the event camera's Event function to acquire corner information in a static position. Data is collected from both near and far, up, down, left and right sides of two pixel planes. Each time data collection begins at a new position, the checkerboard is kept still, and both static and blinking modes are recorded, ensuring that both devices acquire checkerboard data at that position. This simultaneously calibrates the event camera and the infrared depth camera. The specific implementation steps are as follows: (2.1) Use the flashing mode to capture the checkerboard image, so that the event camera can trigger the event of the event camera in the flashing mode in a static position, capture the event information, accumulate the events and remove dynamic points to obtain a stable event image with the corresponding acquisition frequency, and use the corner detection algorithm to find the checkerboard edge and the corresponding corner information for calibration; (2.2) The RGB camera acquires checkerboard images within its inherent acquisition cycle, and uses a corner detection algorithm to find the checkerboard edges and corresponding corner information for calibration; (2.3) Collect data at different distances, up and down, left and right on the two pixel planes. After each collection starts at a new position, keep the checkerboard stationary and record the stationary and flashing modes respectively, so that both devices can collect the checkerboard data at that position. Improve the stability of the calibration operation by using checkerboard image information with multiple segments of annotation. (2.4) The multimodal camera system is calibrated by combining intrinsic and extrinsic parameters in MATLAB software. The calibration operation of the system can control the error within 0.3 pixels. The calibration error of the multimodal camera system is checked by projection and reprojection methods.

[0031] (3) Calibration and verification of projection and back projection: such as Figure 8 As shown, the core of this invention lies in proposing a verification method for the simultaneous calibration effect of an event camera and an infrared depth camera. Based on the intrinsic parameters obtained from the infrared depth camera calibration, pixels are projected in parallel onto spatial coordinates. Then, based on the relative position and pose of the event camera obtained from the calibration, the point cloud is projected onto the plane of the event camera. The calibration error is measured based on the error in the corresponding pixel positions. The specific implementation steps are as follows: (3.1) Based on the intrinsic parameters obtained from the calibration of the infrared depth camera (RGB-D camera) and the standard model of pinhole imaging, calculate the camera coordinate information of each pixel in the RGB-D image at the corresponding timestamp, and use Nvidia's CUDA (Unified Computing Architecture) to efficiently project the corresponding pixels into the spatial coordinates in parallel. (3.2) Based on the position and attitude information of the event camera relative to the infrared depth camera obtained from the calibration information, the infrared depth camera is projected onto the point cloud information of the spatial coordinates through the standard model of pinhole imaging. The point cloud information is then projected back onto the imaging plane of the event camera using CUDA (Unified Computing Architecture) of the Nvidia graphics card. The pixel coordinate information of each spatial point cloud information corresponding to the event camera plane is calculated. (3.3) Within the same period, based on the image information accumulated by the event camera, compare it with the image information obtained by the back projection operation, measure the error of the corresponding pixel position, and thus determine the error of the calibration operation. The error of the corresponding pixel position is specifically calculated based on the coordinates of the feature points in the event camera image that match the infrared depth camera, and the coordinates of the feature points whose spatial point cloud is back-projected onto the event camera imaging plane. The distance error between the two coordinates is used as a standard to measure the calibration effect.

[0032] In the process of synchronizing a multimodal camera group, in addition to synchronization at the hardware level, a mechanism is also needed to find relevant frames in the software. The two devices will have a basic deviation when they are powered on, and the deviation needs to be continuously corrected by the relative timestamp. In addition, the random error of the hardware timestamp of each frame needs to be constantly adjusted. The dynamic random deviation is much smaller than the trigger frame rate interval. Each newly generated dynamic error is added to the cumulative information, so as to correct the total deviation as the deviation change rate. The calculation method of the total deviation is as follows: Total deviation = Basic deviation (power-on time deviation) + Dynamic (random) deviation × Number of acquisition frames.

[0033] During the operation of the event camera and infrared depth camera system, the infrared depth camera (RGB-D camera) actively emits infrared light. The depth is calculated based on the reflection of infrared light at the corresponding location. The actively emitted infrared light has a wavelength of around 800nm, which can trigger the event camera with a receiving wavelength between 300nm and 1100nm. To eliminate the interference caused by the active infrared light, a visible light filter with a wavelength of 400nm to 700nm is used to filter the event camera. At the same time, it is checked and confirmed that adding the filter will not cause distortion of the images acquired by the event camera.

[0034] Corresponding to the aforementioned embodiment of a hardware synchronization and calibration method for an event camera and an infrared depth camera, the present invention also provides an embodiment of a hardware synchronization and calibration device for an event camera and an infrared depth camera.

[0035] See Figure 9 The present invention provides a hardware synchronization and calibration device for an event camera and an infrared depth camera, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in the above embodiment.

[0036] The embodiment of the hardware synchronization and calibration method device for an event camera and an infrared depth camera provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 9 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which includes a hardware synchronization and calibration device for an event camera and an infrared depth camera provided by the present invention. Except for... Figure 9 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0037] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0038] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0039] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in the above embodiments.

[0040] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0041] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned hardware synchronization and calibration method for an event camera and an infrared depth camera.

[0042] In summary, this invention provides a hardware synchronization and calibration method and apparatus for an event camera and an infrared depth camera. The above embodiments are used to explain and illustrate this invention, not to limit it. Any modifications and alterations made to this invention within the spirit and scope of the claims fall within the protection scope of this invention.

Claims

1. A hardware synchronization and calibration method for an event camera and an infrared depth camera, characterized in that, The method includes the following steps: (1) Multimodal camera synchronization at the hardware level: Based on a unified synchronization control signal, the timestamps of the event camera and the infrared depth camera are synchronized; (2) Calibration method of blinking checkerboard: The calibration method of blinking checkerboard is used to simultaneously realize the calibration operation of event camera and depth infrared camera; (3) Calibration and verification of projection and back projection: Based on the intrinsic parameters obtained from the calibration of the infrared depth camera, the pixels are projected in parallel into the spatial coordinates. Then, based on the relative position and attitude of the event camera obtained from the calibration, the spatial coordinate point cloud is projected onto the plane of the event camera. The calibration error is measured based on the error of the corresponding pixel coordinate position.

2. The hardware synchronization and calibration method for an event camera and an infrared depth camera according to claim 1, characterized in that: The timestamps of the event camera and the infrared depth camera are aligned by a synchronization control signal triggered by an external hardware circuit. For each synchronization signal, the deviation is continuously corrected according to the rate of change of the deviation over time, so as to achieve stable image data acquisition. The matching error of the corresponding pixel coordinates of the two cameras obtained by projection and back projection operations is always controlled within the range of plus or minus one.

3. The hardware synchronization and calibration method for an event camera and an infrared depth camera according to claim 1, characterized in that: In step (1), the specific steps for multimodal camera synchronization at the hardware level are as follows: (1.1) Using the STM32F103 embedded tool as an external trigger source, the TIM basic timer is used at the hardware circuit level to generate the corresponding synchronization pulse signal. The same pulse synchronization signal is output from different channels of the same TIM and connected to the event camera and infrared depth camera to achieve timestamp alignment. (1.2) Extract the images of the first few frames acquired when the event camera and the infrared depth camera are powered on to evaluate the power-on time deviation. During the shooting and acquisition process, based on the first-order information of time, that is, the rate of change of deviation with time, continuously adjust the deviation values ​​of the timestamps recorded by the event camera and the infrared depth camera. The deviation is continuously corrected by the relative timestamps to avoid the situation where the error gradually increases during the operation of the equipment. (1.3) Due to the inconsistency in crystal frequency and precision between the event camera and the infrared depth camera, coupled with hardware and numerical precision rounding errors, higher precision values ​​will be discarded. This will cause serious frame errors during long-term operation. When correcting the deviation, each newly generated dynamic error needs to be added to the cumulative information as the deviation change rate to correct the total deviation.

4. The hardware synchronization and calibration method for an event camera and an infrared depth camera according to claim 1, characterized in that: In step (2), the specific steps for calibrating the blinking checkerboard grid are as follows: (2.1) Use the flashing mode to capture the checkerboard image, so that the event camera can trigger the event of the event camera in the flashing mode in a static position, capture the event information, accumulate the events and remove dynamic points to obtain a stable event image with the corresponding acquisition frequency, and use the corner detection algorithm to find the checkerboard edge and the corresponding corner information for calibration; (2.2) The infrared depth camera acquires checkerboard images within its inherent acquisition cycle, and uses a corner detection algorithm to find the checkerboard edges and corresponding corner information for calibration; (2.3) Collect data at different distances, up and down, left and right on the two pixel planes. After each collection starts at a new position, keep the checkerboard stationary and record the stationary and flashing modes respectively. This allows both the event camera and the infrared depth camera to collect the checkerboard data at that position. The stability of the calibration operation is improved by using checkerboard image information with multiple segments of annotation. (2.4) Perform joint calibration of intrinsic and extrinsic parameters of the event camera and infrared depth camera in MATLAB software, and check the calibration error of the event camera and infrared depth camera by projection and reprojection methods.

5. The hardware synchronization and calibration method for an event camera and an infrared depth camera according to claim 1, characterized in that: In step (3), the specific steps for calibration and verification of projection and back projection are as follows: (3.1) Based on the intrinsic parameters obtained from the infrared depth camera calibration and the standard model of pinhole imaging, calculate the camera coordinate information of each pixel in the RGB-D image at the corresponding timestamp, and use the CUDA of the graphics card to efficiently project the corresponding pixels into the spatial coordinates in parallel. (3.2) Based on the position and attitude information of the event camera relative to the infrared depth camera obtained from the calibration information, the infrared depth camera is projected onto the point cloud information of the spatial coordinates through the standard model of pinhole imaging. The CUDA of the graphics card is used to project it back onto the imaging plane of the event camera, and the pixel coordinate information of each spatial point cloud information corresponding to the event camera plane is calculated. (3.3) Under the same period, based on the image information accumulated by the event camera, compare it with the image information obtained by the back projection operation, measure the error of the corresponding pixel position, and thus determine the error of the calibration operation.

6. A hardware synchronization and calibration device for an event camera and an infrared depth camera, comprising an event camera and an infrared depth camera, and a hardware circuit module for stably triggering an external synchronization signal, wherein the event camera, the infrared depth camera, and the hardware circuit module are characterized in that, When the camera module and hardware circuit module perform the synchronization operation, they implement a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-5.

7. A hardware synchronization and calibration device for an event camera and an infrared depth camera, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-5.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a hardware synchronization and calibration method for an event camera and an infrared depth camera as described in any one of claims 1-5.

Images