A sensor acquisition time synchronization method and system based on ZYNQ

Abstract

This invention provides a sensor acquisition time synchronization method and system based on ZYNQ. A unified clock source is set in the FPGA to ensure that time differences do not accumulate and drift. Hardware triggering is performed using an IMU signal as a reference. Based on the trigger time combined with factors such as image exposure time and transmission delay, precise camera triggering is achieved, ensuring that the IMU data transmitted to memory and the camera image originate from the same real-time point. Sensor data is calibrated using a software algorithm, and the result is fed back to the sensor trigger signal offset processing module, forming a complete calibration loop. The introduction of closed-loop feedback achieves automated and adaptive high-precision synchronization. Because an IMU signal is used as the trigger for camera image acquisition, the image frame rate is a multiple of the IMU frequency. During actual operation, the offset processing module parameters can be modified to adjust the frame rate according to the required frame rate and the existing IMU frequency.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and positioning technology for intelligent robots or unmanned aerial vehicles, specifically a sensor acquisition time synchronization method and system based on ZYNQ. Background Technology

[0002] In autonomous and intelligent mobile robot systems, localization techniques for autonomous navigation in unknown environments mainly fall into two categories: utilizing global positioning systems such as GPS and motion capture to directly obtain global positioning information; and using the robot's own sensors to observe environmental and motion information to estimate robot motion. However, GPS is limited by signal transmission and cannot effectively locate robots in open outdoor environments due to obstacles such as building walls and forests. Conversely, the VIO algorithm analyzes measurement data from sensors equipped in unmanned systems to achieve self-localization in complex and unplanned environments. Furthermore, single-sensor VIO systems are often unsuitable for the working scenarios of modern autonomous systems, such as changes in lighting, inclement weather, high-speed movement, scene feature degradation, and dynamic environments. Therefore, unmanned systems are typically equipped with multiple sensors capable of providing local measurements, forming a multi-sensor fusion SLAM system. Commonly used sensors include those with external environment perception capabilities, such as industrial cameras; and sensors that perceive their own motion information, such as gyroscopes and accelerometers. On one hand, cameras and IMUs are complementary; the combination of these two sensors can significantly improve the accuracy and robustness of VIO. The camera provides clear visual measurement information, which can be used to identify and track feature points in a scene, thus providing information about relative motion. The IMU provides high-frequency body acceleration and angular velocity measurement data, which can be used to infer attitude changes and velocity information, and is unaffected by external lighting and weather. The two sensors have different characteristics and applicable conditions, effectively compensating for the shortcomings of a single sensor and maintaining the accuracy and robustness of the VIO system under complex conditions. However, due to various reasons such as data transmission, sensor latency, and operating system overhead, there is a delay in the arrival of sensor data at the receiving end. This results in a time delay in the actual sampling and recording by the sensors. When the delays of the two sensors differ, a relative time offset occurs. Existing research shows that this affects the positioning accuracy of VIO and may even cause the system to malfunction.

[0003] To enable multiple sensors to work collaboratively, it is essential to ensure their time is consistent. A common method is to provide all sensors with the same reference time through a unified clock source. Each sensor then calibrates its own clock time based on this reference time, achieving time synchronization in hardware. Furthermore, it is crucial to ensure that all sensors share a common time reference and clock source to prevent clock drift between different sensors, which could cause timestamps to deviate after a period of operation, even if the timestamps are initially aligned. In the field of autonomous driving, sensors are equipped with GNSS receivers. The navigation satellites they connect to have built-in high-precision atomic clocks. By interpreting the navigation satellite signals, the GNSS receivers can obtain ultra-high-precision clock signals. When the device receives PPS pulse signals and GPRM data messages, it extracts UTC time information and synchronizes it to the sensor clocks, ensuring complete time synchronization among all sensors. Additionally, PTP and NTP time synchronization schemes can be incorporated to achieve sub-microsecond clock synchronization between multiple slave nodes (various sensors) and the master node (host) in an Ethernet network. For example, the invention patent with publication number CN110906923A, "Vehicle-mounted Multi-sensor Tightly Coupled Fusion Positioning Method, System, Storage Medium, and Vehicle," acquires data from LiDAR, GPS, IMU, and chassis, and performs time and spatial synchronization tight coupling processing on the LiDAR, IMU, GPS, and chassis data, and calculates the vehicle's position and attitude. However, these GNSS-based timing methods heavily rely on GPS signals, resulting in poor performance in enclosed environments. Currently, some methods have emerged that do not rely on GPS signals and reduce the synchronization deviation between the IMU and image timestamps in visual odometry systems. For instance, the invention patent with publication number CN111405139A, "Time Synchronization Method, System, Visual Odometry System, and Storage Medium," sets the interruption period of the inertial measurement module to T and the image trigger period to nT. At time m*nT, the inertial measurement module sends a trigger signal to the image acquisition module. After receiving the trigger signal, the image acquisition module controls the image imaging time, making the image imaging time (m+1)*nT. By flexibly controlling the start and end of image exposure, the deviation between the image timestamp and the IMU is eliminated. This method uses multiple MCUs and ISPs for sensor signal triggering and data acquisition, which increases the on-chip device area and power consumption. Summary of the Invention

[0004] The technical problem to be solved by this invention is how to reduce the offset between the IMU and the camera timestamp without relying on GPS.

[0005] The present invention solves the above-mentioned technical problems through the following technical means:

[0006] A sensor acquisition time synchronization method based on ZYNQ, applied to a system where the camera has a fixed exposure, includes the following steps:

[0007] S1. Initialization: Initialize the parameters of the offset processing module; specifically, the camera trigger period is calculated as the IMU trigger period multiplied by a multiple n. During initialization, the camera trigger frequency is determined by the IMU frequency and the period multiple n, and the initial time offset is determined by the exposure parameters.

[0008] S2. Parameter Configuration: Set the IMU and camera to share the clock source provided by the FPGA; the IMU continuously acquires data at a fixed frequency; whenever n IMU data acquisitions are completed, the IMU outputs a hardware trigger pulse to trigger the camera exposure; at the same time, the trigger offset time is adjusted in the FPGA based on half of the initial time offset in step S1 to ensure that the midpoint of the exposure time of each frame of the image corresponds precisely with the nth IMU data.

[0009] S3. Calibration: The relative positional relationship between the camera and the IMU is obtained through calibration;

[0010] S4. Delay Calculation: Set the initial value of the time delay to zero, use polynomial or B-spline basis functions to represent the time-varying state, and calculate the time delay between the IMU and the camera;

[0011] S5. Parameter update; set half of the time delay calculated in step S4 as the offset parameter, modify the offset parameter of the offset processing module, recalculate the time offset, if the time offset gradually decreases and is less than the threshold, fix the offset parameter of the offset processing module; then set the IMU and image timestamp to the same value to complete the sensor synchronization setting; if the time offset is greater than the threshold, update the offset parameter, return to step S4, and re-estimate the time delay between the camera and IMU.

[0012] Furthermore, in step S2, the INT pin of the IMU is connected to the ZYNQ logic terminal trigger input interface.

[0013] Furthermore, the calibration method in step S3 involves wobbling the sensor in front of the checkerboard pattern while fully stimulating all rotational degrees of freedom to obtain the displacement of the comparison camera and IMU.

[0014] Furthermore, in step S4, the time delay calculation method is as follows: First, the camera pose is roughly calculated using the calibration board PnP algorithm, and the initial IMU pose is guessed using extrinsic parameters; the initial IMU pose is encoded as a sixth-order B-spline, the acceleration is encoded as a cubic polynomial, and the deviation is represented by a cubic B-spline; the information matrix constructed is iteratively calculated using the LM algorithm, and finally, the time delay between the IMU and the camera is obtained by iteratively solving using the CHOLMOD sparse matrix library.

[0015] Furthermore, the FPGA includes SPI, DMA, and VDMA IP cores for sending IMU and camera data to memory.

[0016] This invention also provides a sensor acquisition time synchronization system based on ZYNQ, applied to systems where the camera has a fixed exposure; comprising:

[0017] Initialization module: Initializes the parameters of the offset processing module; specifically, the camera trigger period is calculated as the IMU trigger period multiplied by a multiple n. During initialization, the camera trigger frequency is determined by the IMU frequency and the period multiple n, and the initial time offset is determined by the exposure parameters.

[0018] Parameter configuration module: Sets the IMU and camera to share the clock source provided by the FPGA; the IMU continuously acquires data at a fixed frequency; whenever n IMU data acquisitions are completed, the IMU outputs a hardware trigger pulse to trigger the camera exposure; at the same time, the trigger offset time is adjusted in the FPGA based on half of the initial time offset to ensure that the midpoint of the exposure time of each frame of the image corresponds precisely with the nth IMU data.

[0019] Calibration module: Obtains the relative positional relationship between the camera and the IMU through calibration;

[0020] Delay calculation module: Sets the initial value of the time delay to zero and calculates the time delay between the IMU and the camera;

[0021] The parameter update module sets half of the time delay as the offset parameter and modifies the offset parameter of the offset processing module; it recalculates the time offset, and if the time offset gradually decreases and is less than the threshold, it fixes the offset parameter of the offset processing module; then it sets the IMU and image timestamp to the same value, and if the time offset is greater than the threshold, it completes the sensor synchronization setting, updates the offset parameter, and re-estimates the time delay between the camera and IMU.

[0022] Furthermore, in the parameter configuration module, the INT pin of the IMU is connected to the ZYNQ logic terminal trigger input interface.

[0023] Furthermore, the calibration method in the calibration module involves shaking the sensor in front of a checkerboard pattern while fully stimulating all rotational degrees of freedom to obtain the displacement of the comparison camera and IMU.

[0024] Furthermore, in the delay calculation module, the time delay calculation method is as follows: First, the camera pose is roughly calculated using the calibration board PnP algorithm, and the initial IMU pose is guessed using extrinsic parameters; the initial IMU pose is encoded as a sixth-order B-spline, the acceleration is encoded as a cubic polynomial, and the deviation is represented by a cubic B-spline; the information matrix constructed is iteratively calculated using the LM algorithm, and finally, the time delay between the IMU and the camera is obtained by iteratively solving using the CHOLMOD sparse matrix library.

[0025] Furthermore, the FPGA includes SPI, DMA, and VDMA IP cores for sending IMU and camera data to memory.

[0026] The advantages of this invention are:

[0027] This invention generates a camera image acquisition trigger signal based on the IMU trigger signal, using parallel computation on an FPGA. The timing of the camera trigger signal can be adaptively adjusted via the CPU. A feedback channel is added to calculate the sensor delay, and parameters are continuously adjusted during calibration. Once the delay falls below a threshold, the parameters are fixed to achieve the desired effect.

[0028] Specifically, this invention calculates the camera trigger time based on the ratio of the IMU acquisition frequency to the camera frame rate, and fully considers the exposure delay. Half of the time delay between the IMU and the camera is set as the offset parameter. The IMU acquisition time is at the midpoint of the camera image exposure time. At this time, the sensor data fusion effect is better, which is conducive to improving the algorithm accuracy.

[0029] In the FPGA, the trigger offset time is adjusted based on the current IMU output trigger signal. The trigger signal time for the next frame is then calculated to ensure that the physical timing of the next frame's IMU and camera data acquisition is as similar as possible. Specifically, step S2 initially obtains the next frame's camera trigger time based on the IMU frequency and period multiple n, resulting in the overall camera trigger time axis. Step S4 estimates the time delay, takes half of that value, and sends it to the FPGA to adjust the trigger time from step S2. This is equivalent to shifting the trigger time axis to the left or right by a certain distance, thus ensuring that the physical timing of the next frame's IMU and camera data acquisition is as similar as possible.

[0030] This invention uses IMU signals as the trigger for camera image acquisition, completely eliminating the limitations of GPS and demonstrating strong adaptability to enclosed or underground environments. The image frame rate is a multiple of the IMU frequency; during actual operation, the offset processing module parameters can be modified to adjust the frame rate based on the required frame rate and the existing IMU frequency. Simultaneously, we use the IMU signal as the trigger source for camera acquisition, with the trigger module located at the PL terminal of the ZYNQ chip and the calculation module at the PS terminal. Compared to solutions using only a single microcontroller to generate the trigger signal, this SOC-based system architecture not only achieves hardware synchronization but also significantly reduces on-chip resource consumption and system power consumption. Attached Figure Description

[0031] Figure 1 This is a time synchronization architecture diagram of the ZYNQ heterogeneous chip sensor in an embodiment of the present invention;

[0032] Figure 2 This is a flowchart illustrating the time synchronization process of the ZYNQ heterogeneous chip sensor in an embodiment of the present invention.

[0033] Figure 3 This is a timing diagram of the ZYNQ heterogeneous chip sensor synchronization signal processing module in an embodiment of the present invention;

[0034] Figure 4 This illustrates the ideal sensor synchronization effect of the ZYNQ heterogeneous chip sensor in this embodiment of the invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] The ZYNQ heterogeneous chip sensor time synchronization architecture is shown in Figure 1. This calibration scheme sets up the IMU and camera signal triggers on the FPGA, selecting the commonly used BMI088 sensor as the test IMU. First, each module is initialized, and then a unified clock source is set in the FPGA to ensure that time differences do not accumulate and drift. Furthermore, since the acquisition cycles of different sensors are independent, it is impossible to guarantee that data is acquired at the same time. Therefore, a triggering mechanism is designed to trigger imaging from different sensors, and a camera that supports external hardware triggering and has a fixed exposure time is selected. By using the IMU signal as a reference for hardware triggering, and considering factors such as the image exposure time and transmission delay, precise camera triggering is performed, ensuring that the IMU data transmitted to memory and the camera image originate from the same real-time moment. At the CPU end, the received sensor data is calibrated using a software algorithm, and the result is fed back to the sensor trigger signal offset processing module via the AXI_GP bus, forming a complete calibration loop. This embodiment introduces closed-loop feedback, achieving automated and adaptive high-precision synchronization. Because the IMU signal is used as the trigger for the camera to acquire images, the image frame rate is a multiple of the IMU frequency. In actual operation, the parameters of the offset processing module can be modified according to the required frame rate and the existing IMU frequency to adjust the frame rate.

[0037] The ZYNQ heterogeneous chip sensor time synchronization process is shown in Figure 2. This process consists of the following steps:

[0038] Step 1: Perform initialization

[0039] First, the parameters of the sensor trigger signal offset processing module are initialized. The principle of this module is as follows: Figure 3As shown, the IMU sensor BMI088 outputs a trigger signal, indicating that the accelerometer and gyroscope data acquisition is complete. The camera trigger time is obtained by calculating the time offset after considering the camera exposure delay and communication delay. Since the IMU trigger signal is periodic, the calculated camera trigger period is a multiple of the IMU trigger period n. During initialization, the camera trigger frequency can be determined by the period multiple n and the IMU frequency, and the initial time offset can be determined by the exposure parameters.

[0040] In this embodiment, the camera uses fixed exposure. If image acquisition begins at the shutter trigger moment t1 and image readout is completed at the exposure end moment t2, then the time elapsed from the shutter trigger moment t1 to the exposure end moment t2 is the exposure delay plus the readout delay. If the IMU frequency is 100Hz and the camera frame rate is set to 25Hz, then the IMU triggers image acquisition every four times. At moment 0, the fourth moment, and so on, the camera trigger time is calculated using the exposure time, and the expected result is as follows. Figure 4 The effect is that the IMU acquisition time coincides with the midpoint of the camera image exposure time, resulting in better sensor data fusion and improved algorithm accuracy. The FPGA contains a system clock counter logic that timestamps the trigger signals from the IMU and camera, then transfers the timestamps to memory.

[0041] Step 2: Configure IMU, camera, and FPGA

[0042] The IMU and camera share a 100MHz clock source provided by the FPGA to avoid timestamp deviations caused by clock drift. The IMU is configured to continuously acquire accelerometer and gyroscope data at a fixed frequency of 100Hz, outputting a hardware trigger pulse each time data is ready. Its general-purpose INT pin is connected to the ZYNQ logic trigger input interface. After every four IMU data acquisitions, the IMU outputs a rising or falling edge pulse to trigger camera exposure. The camera is set to external trigger mode (rising edge trigger) with a fixed exposure mode. Simultaneously, the trigger offset time is adjusted in the FPGA to precisely control the exposure timing, ensuring that the midpoint of the exposure time for each frame corresponds accurately to the IMU data from a specific number of acquisitions. The FPGA is equipped with SPI, DMA, and VDMA IP cores to ensure smooth data transfer to DDR3 memory. The DMA in the CPU is set to interrupt mode with multi-level caching.

[0043] Step 3: Calibration Startup

[0044] The calibration process can be achieved by using a set of checkerboard images, by waving the device in front of the checkerboard while fully stimulating all rotational degrees of freedom, in order to clearly compare the relative positional relationship between the camera and the IMU.

[0045] Step 4: Delayed Calculation

[0046] The delay calculation module uses the unified spatiotemporal calibration framework for multi-sensor systems proposed by Furgale et al., which can be used to determine the transformation and time offset between the camera and the IMU. Through state parameterization, time-varying state quantities are represented as linear combinations of polynomial or B-spline basis functions. The continuous-time state is indirectly described by estimating the coefficient vector using a weighted sum of basis functions, and the problem of estimating the time offset is handled within the rigorous theoretical framework of maximum likelihood estimation.

[0047] In the camera and IMU time offset estimation in this module, B-spline functions are used to represent the time-varying state, and the initial guess of the time offset is set to zero. First, the camera pose is roughly calculated using the calibration board PnP algorithm, and the initial IMU pose is guessed using extrinsic parameters. The IMU pose is encoded as a sixth-order B-spline, the acceleration is encoded as a cubic polynomial, and the bias is represented by a cubic B-spline. The equations solved in each iteration of the LM algorithm are relatively large; the compact support of the B-spline basis functions can be used to reduce the solution complexity. In each iteration of the LM algorithm, the constructed information matrix exhibits a block hexagonal structure, resulting in non-zero blocks concentrated on the main diagonal and its five adjacent sub-diagonals. Therefore, the CHOLMOD sparse matrix library is used to leverage its sparsity for iterative solution to obtain the time delay between the IMU and the camera.

[0048] Step 5: Update parameters

[0049] Half of the time delay estimated in step four is set as the offset parameter. The offset processing module is modified via the AXI_GP bus. By shaking the device in front of the checkerboard pattern, the time offset is recalculated. If the time offset gradually decreases and falls below a threshold, the offset parameter of the offset processing module is fixed. At this point, the IMU trigger time is aligned with the camera exposure center time, and subsequent IMU and image timestamps are set to the same value, completing the sensor synchronization setup and allowing subsequent VIO algorithms to proceed. If the time offset exceeds the threshold, the offset parameter is updated, and the process returns to step four to re-estimate the camera and IMU time delay. This process is repeated until the time offset falls below the threshold.

[0050] In this embodiment, the camera image acquisition trigger signal is generated through parallel computation on the FPGA based on the IMU trigger signal, and the timing of the camera trigger signal can be adaptively adjusted by the CPU. A feedback channel is added to calculate the sensor delay, and the parameters are continuously adjusted during the calibration process. Once the delay is less than a threshold, the parameters are fixed to achieve the desired effect.

[0051] Specifically, in this embodiment, the camera trigger time is calculated based on the ratio of the IMU acquisition frequency to the camera frame rate, and the exposure delay is fully considered. Half of the time delay between the IMU and the camera is set as the offset parameter. The IMU acquisition time is at the midpoint of the camera image exposure time. At this time, the sensor data fusion effect is better, which is conducive to improving the algorithm accuracy.

[0052] In the FPGA, the trigger offset time is adjusted based on the current IMU output trigger signal. The trigger signal time for the next frame is then calculated to ensure that the physical timing of the next frame's IMU and camera data acquisition is as similar as possible. Specifically, step two uses the IMU frequency and period multiple n to initially determine the next frame's camera trigger time, resulting in the overall camera trigger time axis. Step four estimates the time delay, takes half of this value, and sends it to the FPGA to adjust the trigger time from step two. This is equivalent to shifting the trigger time axis to the left or right by a certain distance, thus ensuring that the physical timing of the next frame's IMU and camera data acquisition is as similar as possible.

[0053] This embodiment uses IMU signals as the trigger for camera image acquisition, completely eliminating the limitations of GPS and demonstrating strong adaptability to enclosed or underground environments. The image frame rate is a multiple of the IMU frequency. During actual operation, the offset processing module parameters can be modified to adjust the frame rate based on the required frame rate and the existing IMU frequency. Simultaneously, we use the IMU signal as the trigger source for camera acquisition; the trigger module is located at the PL terminal of the ZYNQ chip, and the calculation module is located at the PS terminal. Compared to schemes that use only a single microcontroller to generate the trigger signal, this SOC-based system architecture not only achieves hardware synchronization but also significantly reduces on-chip resource consumption and system power consumption.

[0054] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ZYNQ-based sensor acquisition time synchronization method applied to a system in which a camera is fixedly exposed, characterized in that, Comprise: S1. initialization: offset processing module parameter initialization; specifically, the camera trigger period obtained by calculation is the IMU trigger period multiplied by the multiple n, the camera trigger frequency is determined by the IMU frequency and the period multiple n during initialization, and the initial time offset is determined by the exposure parameter; S2. Parameter configuration: set the clock source shared by IMU and camera provided by FPGA; IMU continuously collects data at a fixed frequency; Every time n times of IMU data collection is completed, IMU outputs a hardware trigger pulse to trigger camera exposure; at the same time, the trigger offset time is adjusted in FPGA based on half of the initial time offset in S1; S3. Calibration: obtain the relative position relationship between camera and IMU through calibration; S4. Delay calculation: set the initial value of time delay to zero, and calculate the time delay of IMU and camera; S5. Parameter update; set half of the time delay as the offset parameter, modify the offset parameter of the offset processing module, recalculate the time offset, and if the time offset gradually decreases and is less than the threshold, fix the offset parameter of the offset processing module; then set the IMU and image time stamp to the same value, complete the sensor synchronization setting; if the time offset is greater than the threshold, update the offset parameter and return to S4 to reestimate the time delay of camera and IMU.

2. The ZYNQ-based sensor acquisition time synchronization method according to claim 1, wherein, In the step S2, the INT pin of the IMU is connected to the ZYNQ logic end trigger input interface.

3. The ZYNQ-based sensor acquisition time synchronization method of claim 1, wherein, The calibration method in step S3 is to shake the sensor in front of the chessboard, and fully excite all rotational degrees of freedom to obtain the displacement of the contrast camera and IMU.

4. The ZYNQ-based sensor acquisition time synchronization method of claim 1, wherein, In step S4, the time delay calculation method is: first, the camera pose is roughly calculated by PnP algorithm, and the initial IMU pose is guessed by external parameter; the initial IMU pose is encoded as six order B spline, the acceleration is encoded as cubic polynomial, and the deviation is represented by cubic B spline; the information matrix constructed is calculated by LM algorithm, and finally the CHOLMOD sparse matrix library is used for iterative solution to obtain the time delay of IMU and camera.

5. The ZYNQ-based sensor acquisition time synchronization method according to any one of claims 1 to 4, characterized in that, The FPGA includes SPI, DMA, VDMA IP core, which is used to send IMU and camera data to the memory.

6. A ZYNQ-based sensor acquisition time synchronization system applied to a system in which a camera is fixedly exposed; characterized in that, Comprise the following steps: Initialization module: offset processing module parameter initialization; specifically, the camera trigger period obtained by calculation is the IMU trigger period multiplied by the multiple n, the camera trigger frequency is determined by the IMU frequency and the period multiple n during initialization, and the initial time offset is determined by the exposure parameter; Parameter configuration module: set the clock source shared by IMU and camera provided by FPGA; IMU continuously collects data at a fixed frequency; Every time n times of IMU data collection is completed, IMU outputs a hardware trigger pulse to trigger camera exposure; at the same time, the trigger offset time is adjusted in FPGA based on half of the initial time offset; Calibration module: obtain the relative position relationship between camera and IMU through calibration; Delay calculation module: set the initial value of time delay to zero, and calculate the time delay of IMU and camera; The parameter updating module sets half of the time delay as an offset parameter, modifies the offset parameter of the offset processing module, recalculates the time offset, fixes the offset parameter of the offset processing module if the time offset gradually decreases and is less than a threshold value, sets the IMU and the image timestamp as the same value, and updates the offset parameter if the time offset is greater than the threshold value, reestimates the time delay of the camera and the IMU, and completes the sensor synchronization setting.

7. The ZYNQ-based sensor acquisition time synchronization system of claim 6, wherein, In the parameter configuration module, the INT pin of the IMU is connected to a ZYNQ logic end trigger input interface.

8. The ZYNQ-based sensor acquisition time synchronization system of claim 6, wherein, In the calibration module, the calibration method is to shake the sensor in front of the chessboard, and fully excite all rotational degrees of freedom to obtain the displacement of the contrast camera and the IMU.

9. The ZYNQ based sensor acquisition time synchronization system of claim 6, wherein, In the delay calculation module, the time delay calculation method is as follows: firstly, the pose of the camera is roughly calculated through the PnP algorithm of the calibration board, and the initial pose of the IMU is guessed through the external parameter; the initial pose of the IMU is encoded as a six-order B-spline, the acceleration is encoded as a cubic polynomial, and the deviation is represented by a cubic B-spline; the information matrix constructed is iteratively calculated by using the LM algorithm, and finally, the CHOLMOD sparse matrix library is used for iterative solution to obtain the time delay of the IMU and the camera.

10. The ZYNQ-based sensor acquisition time synchronization system according to any one of claims 6 to 9, characterized in that, The FPGA includes SPI, DMA and VDMA IP cores, and is used for sending the IMU and camera data to the memory.

Images