A method and device for time synchronization correction of a monocular camera using an IMU-assisted method for unmanned aerial vehicles (UAVs)

Abstract

The present application discloses a method and device for time synchronization of a monocular camera assisted by an IMU for unmanned aerial vehicle (UAV), which relates to the technical field of time synchronization. The method comprises the following steps: calibrating an inertial measurement unit (IMU) and a monocular camera, using a time signal generated by a clock source as a data acquisition synchronization point, performing time synchronization on the inertial measurement unit (IMU) and the monocular camera, placing a target in front of the monocular camera or making the monocular camera point to the target, photographing the target to obtain N frames of target images and corresponding IMU data, using an interpolation algorithm to align each frame of the image and the corresponding IMU data according to the timestamp, generating a data set with the same timestamp, finally analyzing the target image in the data set, tracking the movement speed of feature points in the pixel plane, calculating the UAV speed according to the IMU data, estimating and correcting the time deviation of the monocular camera based on the difference between the movement speed and the UAV speed, implementing dynamic compensation, and using an arbitrary target to calculate and compensate for the synchronization time error between the IMU and the monocular camera.

Description

Technical Field

[0001] The present application relates to the field of time synchronization technology, and in particular to a method and device for time synchronization of an unmanned aerial vehicle (UAV) IMU-assisted correction of a monocular camera. Background Art

[0002] The monocular camera technology based on IMU (Inertial Measurement Unit) is developing rapidly. Compared with mechanical monocular cameras, IMU monocular cameras have the characteristics of small size, low cost and high speed. They are widely used in various unmanned devices and intelligent equipment fields, such as unmanned driving, intelligent security, behavior prediction and other fields.

[0003] The IMU monocular camera generally contains a monocular camera, an IMU, a central control unit, a detector, a signal processing module and other devices. When the device is working, the central control unit will send a synchronous clock signal to trigger the IMU and the monocular camera, so that the monocular camera can take pictures and the IMU can start to vibrate. However, since the time required for the two to respond after receiving the trigger signal is different, there is a certain synchronization time error between the two. This time error will cause errors in the pose solution of the generated monocular camera image.

[0004] There was no effective automatic calculation method for synchronization time error before. Generally, time alignment between the two is achieved through manual recognition, that is, manually moving the collected signals one by one and observing whether the imaging results are consistent with the shooting scene. This time synchronization method is time-consuming and labor-intensive, and lacks corresponding evaluation standards, and cannot guarantee the accuracy of time synchronization. At the same time, when the shooting method or shooting angle of the monocular camera changes, the time error often changes significantly, and time synchronization needs to be re-performed.

[0005] In the prior art, most IMU monocular camera devices use the Visual-Inertial Navigation method to achieve shooting. The IMU (Inertial Measurement Unit) monocular camera device combines IMU technology with a monocular vision system, and achieves high-precision autonomous positioning and navigation capabilities by introducing an inertial measurement unit (IMU). This method can not only effectively reduce the uncertainty of a single sensor, but also remain stable when dealing with fast motion or complex environments, and is therefore widely used in fields such as drones, robot navigation, and autonomous driving. However, due to the inconsistent response time to the trigger signal and a certain delay in data output, a certain system delay time error will be generated. Strict calculations are required during the shooting process, and this part of the system delay time error must be dynamically compensated in the subsequent process. This error can reach several nanoseconds or even tens of nanoseconds, which seriously affects the visual navigation accuracy of the monocular camera.

[0006] Therefore, although some methods for calibrating the time error of monocular camera systems already exist, there are still many problems. For example, when calibrating the synchronization time error between IMU and monocular camera, it is heavily dependent on manual operation and lacks quantitative evaluation indicators. After changing the shooting method and shooting angle, it is necessary to repeat the complex calculation process, which is inefficient. When calculating the delay time error of the monocular camera system, there are high requirements for the time device, test environment and operation process, which greatly affects the efficiency of system control. Summary of the invention

[0007] The purpose of this application is to provide a method and device for time synchronization of a UAV IMU-assisted monocular camera in order to overcome the existing technical defects, correct the timestamp of the camera data stream according to the time offset, and gradually iterate and optimize until the time offset converges to zero, thereby completing time synchronization.

[0008] The purpose of this application is achieved through the following technical solutions:

[0009] In the first aspect, the present application proposes a time synchronization method for a drone IMU-assisted correction monocular camera, the method being applied to a time synchronization device, the method comprising:

[0010] Pre-set the inertial measurement unit model and calibrate the inertial measurement unit and monocular camera;

[0011] The time signal generated by the clock source is used as the data acquisition synchronization point to synchronize the time of the inertial measurement unit and the monocular camera;

[0012] Place the target in front of the monocular camera or point the monocular camera at the target;

[0013] The monocular camera and inertial measurement unit after running time synchronization shoot the target to obtain N frames of target images and corresponding IMU data;

[0014] Use the interpolation algorithm to align each frame image and the corresponding IMU data according to the timestamp to generate a data set with the same timestamp;

[0015] The target images in the data set are analyzed, the movement speed of the feature points in the pixel plane is tracked, the UAV speed is calculated based on the IMU data, the time deviation of the monocular camera is estimated and corrected based on the difference between the movement speed and the UAV speed, and dynamic compensation is implemented.

[0016] In one possible implementation, the model of the inertial measurement unit is:

[0017] ;

[0018] ;

[0019] in represents the estimated angular velocity, represents the estimated acceleration, represents the angular velocity of the body at time k, represents the acceleration of the body at time k, and is the time-varying bias, Internal errors of the sensor caused by various physical factors, and All are measurement noise.

[0020] In a possible implementation, the step of using a time signal generated by a clock source as a data acquisition synchronization point to synchronize the inertial measurement unit and the monocular camera includes:

[0021] Extract the time accuracy of the clock source to calibrate the inertial measurement unit and monocular camera;

[0022] The time signal generated by the clock source is sent to the inertial measurement unit and the monocular camera for decoding to generate a time stamp;

[0023] The inertial measurement unit and the monocular camera are controlled to achieve time synchronization based on the time-stamped synchronization point.

[0024] In a possible implementation, the method further includes:

[0025] In the case of signal transmission delay, the linear time-invariant system model is used to adjust the filter weights according to the transfer function so that the sum of the squares of the errors between the output signal and the expected signal is minimized.

[0026] In a possible implementation, the steps before using an interpolation algorithm to align each frame of image and corresponding IMU data according to timestamps to generate a data set with the same timestamp include:

[0027] Perform data cleaning on N frames of target images and corresponding IMU data to remove outliers and noise;

[0028] The timestamps of N frames of target images and the corresponding IMU data are unified to the same time base, and the missing data points are interpolated using adjacent data points.

[0029] In a possible implementation, the step of using an interpolation algorithm to align each frame of image and corresponding IMU data according to timestamps to generate a data set with the same timestamp includes:

[0030] Use interpolation algorithms to align each frame of image and the corresponding IMU data with a uniform time grid;

[0031] Generates datasets with the same timestamp and ensures that the datasets are consistent in time.

[0032] In a possible implementation, the interpolation algorithm uses quadratic polynomial interpolation.

[0033] In one possible implementation, the steps of analyzing the target image in the data set, tracking the movement speed of the feature points in the pixel plane, calculating the speed of the drone according to the IMU data, estimating and correcting the time deviation of the monocular camera based on the difference between the movement speed and the drone speed, and implementing the dynamic compensation include:

[0034] The ORB algorithm is used to detect and extract feature points from each frame of the image in the data set.

[0035] Based on feature points, the FLANN feature matching algorithm is used to quickly match the same or corresponding feature points between consecutive frames and construct the motion trajectory of the feature points;

[0036] In the motion trajectory, the pixel coordinate difference of the matched feature point between two consecutive frames is calculated, and then divided by the corresponding time interval to obtain the motion speed of the feature point in the pixel plane;

[0037] Compare the difference between the moving speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit to estimate the time deviation;

[0038] Use the time offset to correct the timestamps of the camera frames and / or IMU data, gradually correcting the time synchronization;

[0039] The motion speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit are iterated, and the time deviation is updated in each iteration. When the difference reaches the minimum value, the dynamic compensation is completed.

[0040] In the second aspect, the present application proposes a time synchronization device for a drone IMU-assisted correction monocular camera, the device comprising:

[0041] The calibration module is used to pre-set the model of the inertial measurement unit and calibrate the inertial measurement unit and the monocular camera;

[0042] The time synchronization module is used to use the time signal generated by the clock source as the data acquisition synchronization point to synchronize the time of the inertial measurement unit and the monocular camera;

[0043] A pointing module is used to place a target in front of the monocular camera or to make the monocular camera point to the target;

[0044] The shooting module is used to shoot the target with the monocular camera and inertial measurement unit after running time synchronization to obtain N frames of target images and corresponding IMU data;

[0045] The alignment module is used to align each frame of image and the corresponding IMU data according to the timestamp using an interpolation algorithm to generate a data set with the same timestamp;

[0046] The correction module is used to analyze the target image in the data set, track the movement speed of the feature points in the pixel plane, calculate the drone speed based on the IMU data, estimate and correct the time deviation of the monocular camera based on the difference between the movement speed and the drone speed, and implement dynamic compensation.

[0047] The above main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are many combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0048] The present application discloses a method and device for time synchronization of a drone IMU-assisted monocular camera correction. The method comprises the following steps: firstly calibrating the inertial measurement unit and the monocular camera, using the time signal generated by the clock source as the data acquisition synchronization point, time synchronization of the inertial measurement unit and the monocular camera, placing a target in front of the monocular camera or making the monocular camera point to the target, and shooting the target with the time-synchronized monocular camera and the inertial measurement unit to obtain N frames of target images and corresponding IMU data. An interpolation algorithm is used to align each frame of the image and the corresponding IMU data according to the timestamp to generate a data set with the same timestamp. Finally, the target image in the data set is analyzed, the movement speed of the feature point in the pixel plane is tracked, the drone speed is calculated according to the IMU data, and the time deviation of the monocular camera is estimated and corrected based on the difference between the movement speed and the drone speed, and dynamic compensation is implemented. The synchronization time error between the IMU and the monocular camera is calculated and compensated using an arbitrary target. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A flow chart of a method for time synchronization of a UAV IMU-assisted monocular camera correction proposed in an embodiment of the present application is shown.

[0051] Figure 2This is a flow chart of maintaining system time synchronization by using the IMU-assisted calibration of a monocular camera for a drone in the first specific implementation mode.

[0052] Figure 3 The device architecture diagram of the drone IMU-assisted correction monocular camera to maintain system time synchronization proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0054] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0055] In the prior art, there are many problems with the existing methods for calibrating the time error of some monocular camera systems. For example, when calibrating the synchronization time error between the IMU and the monocular camera, it is heavily dependent on manual operation and lacks quantitative evaluation indicators. After changing the shooting method and shooting angle, it is necessary to repeat the complex calculation process, which is inefficient. When calculating the delay time error of the monocular camera system, there are high requirements for the time device, test environment and operation process, which greatly affects the efficiency of system control.

[0056] Therefore, in order to solve the above-mentioned technical problems, the embodiments of the present application propose a time synchronization method and device for a drone IMU-assisted correction of a monocular camera, which can solve the problems of large fixed errors in system pose solution caused by system time errors such as the synchronization time error between the IMU and the monocular camera, the time delay error caused by system transmission between subsystems, and the low efficiency of the existing system time error calculation method. The method is described in detail below.

[0057] Please refer to Figure 1 , Figure 1 A flowchart of a method for time synchronization of a UAV IMU-assisted monocular camera correction proposed in an embodiment of the present application is shown. The method is applied to a time synchronization device and includes:

[0058] Step S100: pre-set the model of the inertial measurement unit, and calibrate the inertial measurement unit and the monocular camera.

[0059] The model of the inertial measurement unit is:

[0060] ;

[0061] ;

[0062] in represents the estimated angular velocity, represents the estimated acceleration, represents the angular velocity of the body at time k, represents the acceleration of the body at time k, and is the time-varying bias, Internal errors of the sensor caused by various physical factors, and All are measurement noise.

[0063] The IMU bias is modeled as a random walk, whose time derivative follows a Gaussian distribution:

[0064] ,in represents the time derivative of the angular velocity zero bias, represents the random noise of the angular velocity deviation, represents the derivative of the acceleration zero bias with respect to time, Random noise representing the deviation in acceleration.

[0065] The visual autonomous positioning method usually uses vision and IMU fusion to form a visual inertial odometer. The camera captures images at a frequency of about 20Hz, while the IMU measurement frequency is usually above 100Hz. It can be seen that there is a large amount of IMU measurement data between two adjacent image frames. According to the kinematic model, the IMU data between the kth frame and the k+1th frame of the camera can be integrated to obtain the position, velocity and attitude of the drone in the inertial system at time k+1:

[0066] ;

[0067] in ;

[0068] represents the position of the UAV in the world coordinate system (W) at time k+1, represents the position of the UAV in the world coordinate system (W) at time k, represents the time interval, that is, the time difference from time k to k+1, represents the rotation matrix, which transforms the local coordinate system measured by the IMU into the world coordinate system (W), represents the estimated value of acceleration, represents the zero bias of the IMU accelerometer, represents the noise of the IMU accelerometer, is the representation of gravitational acceleration in the world coordinate system (W); represents the speed of the drone in the world coordinate system (W) at time k+1, represents the speed of the drone in the world coordinate system (W) at time k, represents the attitude of the drone in the world coordinate system (W) at time k+1, usually expressed as a quaternion, represents the posture of the UAV in the world coordinate system (W) at time k, represents the estimated value of the angular velocity, Indicates the zero bias of the IMU gyroscope, represents the noise of the IMU gyroscope, Represents the attitude quaternion at time t; represents the angular velocity vector, T represents the transpose operation, represents antisymmetric, represents the angular velocity vector on the x-axis, represents the angular velocity vector on the y-axis, Represents the angular velocity vector on the z-axis.

[0069] It can be seen that the state update at time k+1 needs to rely on the state at time k. For optimization-based algorithms, once the initial state changes, the IMU measurements between the adjusted states need to be re-propagated. This update strategy consumes huge computing resources. In order to avoid repeated propagation, the IMU readings are usually pre-processed in the optimization-based sensor fusion algorithm to form a pre-integration term.

[0070] The reference coordinate system is changed from inertial system Adjust to Frame time machine system , by multiplying both sides It can be deduced that:

[0071] ;

[0072] here

[0073] ;

[0074] middle and It is called the pre-integration term. The pre-integration term is only related to the IMU measurement value and the zero bias and is completely independent of the system state. When the IMU zero bias is used to estimate the system state, since the zero bias does not change much in the short term, the pre-integration term can be approximated by a first-order Taylor expansion relative to the zero bias. The initial moment of the pre-integration ,β is the zero vector,γ is the unit quaternion. Due to the IMU noise term and is an unknown term, so it is set to zero. Then the observed value of the IMU pre-integration term is , and The discrete form of can be expressed as:

[0075] ;

[0076] represents the pre-integrated acceleration estimate of the k-th frame image at time step i+1, where i is the Frame image to The time period between frames The discrete time points of the IMU measurements, represents the pre-integrated acceleration estimate of the k-th frame image at time step i, is the time interval between two measurements, Representation about quaternion The rotation matrix. Quaternions are an extension of complex numbers and are used to represent rotations in three-dimensional space. The measured acceleration at time step i, The zero bias of the accelerometer, that is, the value that should be output when there is no acceleration, It represents a part of the second-order integration of acceleration taking into account the effects of rotation and zero bias. represents the pre-integrated rotation estimate of the k-th frame image at time step i+1, represents the pre-integrated rotation estimate of the k-th frame image at time step i, represents the quaternion estimate at time step i+1, represents the quaternion estimate at time step i, Represents quaternion multiplication, used to combine two consecutive rotations, Represents an update reduction matrix for computing quaternions, where is the angular velocity measurement at time step i, It represents the zero bias of the gyroscope.

[0077] Step S200: Using the time signal generated by the clock source as a data acquisition synchronization point to synchronize the time of the inertial measurement unit and the monocular camera.

[0078] The clock source generates an accurate time signal that can be received by both the camera and the IMU as a synchronization point for data acquisition, ensuring that the timestamps of the IMU and the camera come from the same clock source. The time of the two can also be synchronized through software, laying the foundation for subsequent data alignment.

[0079] Step S200 includes:

[0080] Extract the time accuracy of the clock source to calibrate the inertial measurement unit and monocular camera;

[0081] The time signal generated by the clock source is sent to the inertial measurement unit and the monocular camera for decoding to generate a time stamp;

[0082] The inertial measurement unit and the monocular camera are controlled to achieve time synchronization based on the time-stamped synchronization point.

[0083] Confirm the time accuracy of the clock source. Generally, an atomic clock or GPS is used as a reference source to provide a nanosecond or higher precision time base for the IMU and monocular camera. The clock synchronization module integrated or external to the IMU and monocular camera effectively receives and processes these time signals and accurately interprets the timestamp information. A unified time tag is embedded in the data packet to ensure the spatiotemporal correlation of each data. Finally, the IMU and camera are controlled to start data acquisition in a strictly synchronized state. They work at different frequencies, but start and stop data recording according to the same time base. Based on the shared time base, the start and stop instructions of different devices are coordinated to ensure data synchronization even if they work in different cycles.

[0084] The method further includes:

[0085] In the case of signal transmission delay, the linear time-invariant system model is used to adjust the filter weights according to the transfer function so that the sum of the squares of the errors between the output signal and the expected signal is minimized.

[0086] Signal processing techniques, such as least mean square error (LMS) adaptive filtering or other time delay estimation methods, are used to compensate for small delays in signal transmission and improve synchronization accuracy.

[0087] There is a slight delay in signal transmission. Establishing a linear time-invariant system model can accurately describe the entire process of signal from input to output. Determine the system's transfer function, the minimum mean square error criterion, and recursively adjust the filter weights to minimize the sum of square errors between the output signal and the expected signal to compensate for these delays and ensure that the time stamp accurately reflects the actual data acquisition moment.

[0088] Step S300: placing a target in front of the monocular camera or making the monocular camera point to the target.

[0089] For subsequent feature point tracking and data processing, objects with obvious features need to be placed within the camera's field of view.

[0090] Step S400: The monocular camera and the inertial measurement unit after running time synchronization shoot the target to obtain N frames of target images and corresponding IMU data.

[0091] The motion data of IMU and the image sequence of camera are collected synchronously, which will be the basis for subsequent processing.

[0092] The steps before step S500 include:

[0093] Perform data cleaning on N frames of target images and corresponding IMU data to remove outliers and noise;

[0094] The timestamps of N frames of target images and the corresponding IMU data are unified to the same time base, and the missing data points are interpolated using adjacent data points.

[0095] The collected N frames of target images and corresponding IMU data are cleaned to remove outliers and noise. Then, to ensure that the timestamps of all data sources are unified to the same time base, the timestamps are converted to a uniform unit (milliseconds) and any time deviation is eliminated. For missing data points, adjacent data points are used for interpolation to fill the gaps.

[0096] Step S500: Use an interpolation algorithm to align each frame of image and the corresponding IMU data according to the timestamp to generate a data set with the same timestamp.

[0097] Interpolation and other algorithms are used to handle the data timestamp mismatch caused by different hardware sampling rates and generate synchronized data sets.

[0098] Step S500 includes:

[0099] Use interpolation algorithms to align each frame of image and the corresponding IMU data with a uniform time grid;

[0100] Generates datasets with the same timestamp and ensures that the datasets are consistent in time.

[0101] The interpolation algorithm uses quadratic polynomial interpolation. For data points with asynchronous timestamps, their values ​​are adjusted through the interpolation algorithm to align them with a unified time grid. Time alignment usually involves time interpolation or time resampling, so that all data points can correspond to a unified time series. Check whether the generated data set is consistent in time, ensure that the interpolation process does not introduce additional errors, verify the time consistency and integrity of the interpolated data, and ensure that the processing process fills the gaps without introducing improper distortions.

[0102] Step S600, analyze the target image in the data set, track the movement speed of the feature points in the pixel plane, calculate the drone speed based on the IMU data, estimate and correct the time deviation of the monocular camera based on the difference between the movement speed and the drone speed, and implement dynamic compensation.

[0103] Based on visual information (the velocity of feature points in the pixel plane) and IMU data (the actual linear acceleration and angular velocity of the device), the system's time deviation is calculated and dynamic compensation is performed accordingly to improve data consistency and accuracy.

[0104] Step S600 includes:

[0105] The ORB algorithm is used to detect and extract feature points from each frame of the image in the data set.

[0106] Based on feature points, the FLANN feature matching algorithm is used to quickly match the same or corresponding feature points between consecutive frames and construct the motion trajectory of the feature points;

[0107] In the motion trajectory, the pixel coordinate difference of the matched feature point between two consecutive frames is calculated, and then divided by the corresponding time interval to obtain the motion speed of the feature point in the pixel plane;

[0108] Compare the difference between the moving speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit to estimate the time deviation;

[0109] Use the time offset to correct the timestamps of the camera frames and / or IMU data, gradually correcting the time synchronization;

[0110] The motion speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit are iterated, and the time deviation is updated in each iteration. When the difference reaches the minimum value, the dynamic compensation is completed.

[0111] Use ORB (Oriented FAST and Rotated BRIEF) to extract significant feature points in each frame of the image. Use the FLANN feature matching algorithm to match the same feature points between consecutive frames to determine their motion trajectories. Compare each matched feature point and calculate its pixel coordinate change between two consecutive frames. Divide the change in pixel coordinates by the time interval between the two frames to obtain the instantaneous speed of the feature point in the pixel plane. Check whether there is a difference in speed to ensure that the difference between visual and IMU speeds is minimized. The time deviation can be solved in reverse.

[0112] The estimated time bias is then applied to the timestamp of the camera frame or IMU data to correct the time synchronization error. This process requires multiple iterations, with the time bias estimate updated each time until the difference between the visual and IMU velocities is minimized.

[0113] As a feature extraction method, ORB features are suitable for quickly and accurately identifying and describing key points in images. FLANN matching ensures the correct correspondence of feature points across frames, which is beneficial for scenarios with high real-time requirements. The optical flow (i.e., the rate of change of pixel coordinates) of the feature points is calculated to obtain the speed, and the consistency between the speed measured by vision and the speed data provided by the IMU is observed and optimized. The time deviation is iteratively solved, and then the corrected time deviation is applied. This process is iterated until convergence, indicating that the speed difference between the visual and IMU data is minimized.

[0114] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0115] First, the synchronization time error caused by the different response times between the IMU and the monocular camera is solved. This error will lead to the instability of the pose solution and thus affect the control of visual navigation.

[0116] Second, calculating the synchronization time error between the IMU and the monocular camera, as well as the system delay time error between the reference target and the measurement target, can effectively solve the positioning error problem caused by the system time error, and can also effectively solve the problem of system posture solution error.

[0117] Third, the synchronization time error between the IMU and the monocular camera is calculated and compensated by using any target. Since the measurement target needs to pass through the system delay time error caused by the response time delay of other systems and detectors, the present invention does not require any manual measurement process and can automatically solve the time error in the target system, thereby improving the accuracy of the monocular camera visual navigation. Time synchronization work was previously completed through manual measurement. Compared with this time synchronization situation, the time synchronization method and device of the present invention can greatly improve the time synchronization accuracy and thus optimize the accuracy and reliability of visual navigation.

[0118] The following is a possible implementation of a time synchronization device for a drone IMU-assisted monocular camera correction, which is used to execute the various execution steps and corresponding technical effects of the time synchronization method shown in the above embodiments and possible implementations. The device includes:

[0119] The calibration module is used to pre-set the model of the inertial measurement unit and calibrate the inertial measurement unit and the monocular camera;

[0120] The time synchronization module is used to use the time signal generated by the clock source as the data acquisition synchronization point to synchronize the time of the inertial measurement unit and the monocular camera;

[0121] A pointing module is used to place a target in front of the monocular camera or to make the monocular camera point to the target;

[0122] The shooting module is used to shoot the target with the monocular camera and inertial measurement unit after running time synchronization to obtain N frames of target images and corresponding IMU data;

[0123] The alignment module is used to align each frame of image and the corresponding IMU data according to the timestamp using an interpolation algorithm to generate a data set with the same timestamp;

[0124] The correction module is used to analyze the target image in the data set, track the movement speed of the feature points in the pixel plane, calculate the drone speed based on the IMU data, estimate and correct the time deviation of the monocular camera based on the difference between the movement speed and the drone speed, and implement dynamic compensation.

[0125] Figure 2 This is a flowchart of the UAV IMU-assisted correction of the monocular camera to maintain system time synchronization in the specific implementation method one. The clock module is responsible for receiving and sending time signals. The host computer part includes sub-functions such as correction time, iterative calculation, feature matching, and feature point extraction; the FPGA includes three main modules: calibration module, trigger module, and timestamp and data splicing module. In this process, image data and IMU data packets are involved as inputs, and finally generate camera trigger signals and IMU trigger signals as output results. The whole process connects various parts through different lines to form a complete logical relationship.

[0126] Figure 3 The device architecture diagram of the drone IMU-assisted correction monocular camera to maintain system time synchronization proposed in an embodiment of the present application is shown. The clock module provides two signals downward: a clock signal and a time signal. Starting from the host computer, the IMU data packet and timestamp receive information from the FPGA driver board and pass this information to the monocular camera. There are three branches connected to the inertial measurement unit IMU, namely "IMU trigger signal", "IMU data packet" and "camera trigger signal". The FPGA driver board can receive image data and convert it into IMU data packets and various trigger signals.

[0127] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A time synchronization method for UAV IMU-assisted correction of monocular camera, characterized in that: The method is applied to a time synchronization device, and the method comprises: Pre-set the inertial measurement unit model and calibrate the inertial measurement unit and monocular camera; The time signal generated by the clock source is used as the data acquisition synchronization point to synchronize the time of the inertial measurement unit and the monocular camera; Place the target in front of the monocular camera or point the monocular camera at the target; The monocular camera and inertial measurement unit after running time synchronization shoot the target to obtain N frames of target images and corresponding IMU data; Use the interpolation algorithm to align each frame image and the corresponding IMU data according to the timestamp to generate a data set with the same timestamp; The ORB algorithm is used to detect and extract feature points from each frame of the image in the data set; Based on feature points, the FLANN feature matching algorithm is used to quickly match the same or corresponding feature points between consecutive frames and construct the motion trajectory of the feature points; In the motion trajectory, the pixel coordinate difference of the matched feature point between two consecutive frames is calculated, and then divided by the corresponding time interval to obtain the motion speed of the feature point in the pixel plane; Compare the difference between the moving speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit to estimate the time deviation; Use the time offset to correct the timestamps of the camera frames and / or IMU data, gradually correcting the time synchronization; The motion speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit are iterated, and the time deviation is updated in each iteration. When the difference reaches the minimum value, the dynamic compensation is completed.

2. The time synchronization method according to claim 1, characterized in that: The model of the inertial measurement unit is: ； ； in represents the estimated angular velocity, represents the estimated acceleration, represents the angular velocity of the body at time k, represents the acceleration of the body at time k, and is the time-varying bias, Internal errors of the sensor caused by various physical factors, and All are measurement noise.

3. The time synchronization method according to claim 1, characterized in that: The steps of using the time signal generated by the clock source as the data acquisition synchronization point to synchronize the inertial measurement unit and the monocular camera include: Extract the time accuracy of the clock source to calibrate the inertial measurement unit and monocular camera; The time signal generated by the clock source is sent to the inertial measurement unit and the monocular camera for decoding to generate a time stamp; The inertial measurement unit and the monocular camera are controlled to achieve time synchronization based on the time-stamped synchronization point.

4. The time synchronization method according to claim 1, characterized in that: The method further comprises: In the case of signal transmission delay, the linear time-invariant system model is used to adjust the filter weights according to the transfer function so that the sum of the squares of the errors between the output signal and the expected signal is minimized.

5. The time synchronization method according to claim 1, characterized in that: The steps before using the interpolation algorithm to align each frame image and the corresponding IMU data according to the timestamp to generate a data set with the same timestamp include: Perform data cleaning on N frames of target images and corresponding IMU data to remove outliers and noise; The timestamps of N frames of target images and the corresponding IMU data are unified to the same time base, and the missing data points are interpolated using adjacent data points.

6. The time synchronization method according to claim 1, characterized in that: The steps of using an interpolation algorithm to align each frame of the image and the corresponding IMU data according to the timestamp to generate a data set with the same timestamp include: Use interpolation algorithms to align each frame of image and the corresponding IMU data with a uniform time grid; Generates datasets with the same timestamp and ensures that the datasets are consistent in time.

7. The time synchronization method according to claim 6, characterized in that: The interpolation algorithm uses quadratic polynomial interpolation.

8. A time synchronization device for UAV IMU-assisted correction of monocular camera, characterized in that: The device comprises: The calibration module is used to pre-set the model of the inertial measurement unit and calibrate the inertial measurement unit and the monocular camera; The time synchronization module is used to use the time signal generated by the clock source as the data acquisition synchronization point to synchronize the time of the inertial measurement unit and the monocular camera; A pointing module is used to place a target in front of the monocular camera or to make the monocular camera point to the target; The shooting module is used to shoot the target with the monocular camera and inertial measurement unit after running time synchronization to obtain N frames of target images and corresponding IMU data; The alignment module is used to align each frame of image and the corresponding IMU data according to the timestamp using an interpolation algorithm to generate a data set with the same timestamp; The correction module is used to detect and extract feature points from each frame image in the data set using the ORB algorithm; Based on feature points, the FLANN feature matching algorithm is used to quickly match the same or corresponding feature points between consecutive frames and construct the motion trajectory of the feature points; In the motion trajectory, the pixel coordinate difference of the matched feature point between two consecutive frames is calculated, and then divided by the corresponding time interval to obtain the motion speed of the feature point in the pixel plane; Compare the difference between the moving speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit to estimate the time deviation; Use the time offset to correct the timestamps of the camera frames and / or IMU data, gradually correcting the time synchronization; The motion speed of the feature point in the pixel plane and the speed provided by the inertial measurement unit are iterated, and the time deviation is updated in each iteration. When the difference reaches the minimum value, the dynamic compensation is completed.

Images