A method and system for time synchronization of visual-inertial sensors based on motion consistency.

Abstract

This invention relates to the field of navigation technology and proposes a visual inertial sensor time synchronization method and system based on motion consistency. The method calculates the camera angular velocity from acquired camera images; extracts the IMU average angular velocity information from acquired IMU data; constructs a direction-weighted matrix based on the relative amplitude of the camera angular velocity; performs dimensionality reduction on the camera angular velocity and IMU average angular velocity using principal component analysis; then, employs fast outlier removal and interpolation optimization based on random projection; matches the camera frame and IMU sensor angular velocity data through correlation; and performs interpolation optimization to obtain the time deviation for time alignment. This method improves the accuracy, stability, and real-time performance of time synchronization by preprocessing and extracting angular velocity data, using data dimensionality reduction and random projection to accelerate the time alignment process, and dynamically correcting the time deviation.

Description

Technical Field

[0001] This invention relates to the field of navigation-related technologies, and more specifically, to a method and system for synchronizing the time of a visual inertial sensor based on motion consistency. Background Technology

[0002] The statements in this section provide only background information related to the present invention and do not necessarily constitute prior art.

[0003] With the widespread application of mobile robots in industrial manufacturing, logistics, smart homes, and autonomous driving, the requirements for their autonomous operation capabilities in complex environments are increasing. High-precision positioning and environmental perception technologies have become crucial for achieving stable operation. To address the limitations of single sensors in terms of accuracy, stability, and environmental adaptability, multi-sensor fusion solutions that integrate information from multiple heterogeneous sensors are gradually becoming mainstream. Visual inertial navigation systems, as a typical example, combine the advantages of cameras and inertial measurement units (IMUs) and have demonstrated good performance in numerous practical applications. One of their core challenges lies in achieving high-precision time alignment between camera and IMU data.

[0004] In existing visual inertial navigation systems, common time synchronization methods mainly fall into two categories: hardware-triggered synchronization and software timestamp-based soft synchronization. However, these technical solutions all have certain limitations and shortcomings in practical applications, mainly in the following aspects:

[0005] (1) High hardware cost and complex deployment: Hardware-triggered synchronization methods usually require the camera and IMU to have a unified trigger interface or clock source, which not only increases the complexity of hardware design, but also puts forward high requirements on the compatibility and cost between devices. In resource-constrained embedded platforms or consumer devices, it is often difficult to achieve accurate hardware synchronization, which limits the flexibility of its promotion and deployment.

[0006] (2) Soft synchronization relies on operating system scheduling, and the timestamp accuracy is limited: Currently, multi-sensor systems generally use timestamp matching or message filtering to achieve data alignment. However, this method is highly dependent on operating system scheduling and communication link stability. The timestamp of sensor data often records the time when the data arrives at the system rather than the actual sampling time, which can easily introduce uncertainty delays. Secondly, some systems use linear interpolation to resample IMU data to match image frames, but this interpolation process is only performed in the time domain, while the actual motion of the sensor is often nonlinear, especially under violent actions such as turning and acceleration. Interpolation cannot accurately restore the intermediate state, and the error is amplified with the intensity of the motion. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes a visual-inertial sensor time synchronization method and system based on motion consistency. By combining camera image feature point matching and IMU angular velocity data, it solves the problems of traditional soft synchronization and resampling methods in nonlinear motion and computationally burdensome scenarios. This method extracts angular velocity data through preprocessing, accelerates the time alignment process using data dimensionality reduction and random projection, dynamically corrects time deviations, improves the accuracy, stability, and real-time performance of time synchronization, reduces hardware costs, and enhances the system's robustness in complex motion scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] One or more embodiments provide a visual-inertial sensor time synchronization method based on motion consistency, comprising the following steps:

[0010] For the acquired camera images, feature points between image frames are extracted and rotation relationships are estimated to calculate the camera angular velocity; for the acquired IMU data, an IMU sliding window is established and integration is performed to obtain the IMU average angular velocity information.

[0011] Based on the relative amplitude of camera angular velocity, a direction weighting matrix is ​​constructed, and principal component analysis is used to reduce the dimensionality of camera angular velocity and IMU average angular velocity;

[0012] For the dimensionality reduction matrix of the angular velocity data after dimensionality reduction, a fast outlier removal and interpolation optimization based on random projection is adopted. The camera frame and IMU sensor angular velocity data are matched by correlation, and the interpolation optimization is performed to obtain the time deviation to achieve time alignment.

[0013] The time deviation estimate is updated based on the correlation between camera frames and the corresponding IMU angular velocities.

[0014] One or more embodiments provide a visual-inertial sensor time synchronization system based on motion consistency, including:

[0015] The angular velocity information extraction module is configured to extract inter-frame feature points and estimate rotation relationships for acquired camera images, and calculate camera angular velocity; for acquired IMU data, it establishes an IMU sliding window and performs integration to obtain IMU average angular velocity information.

[0016] The dimension reduction module is configured to construct a direction-weighted matrix based on the relative amplitude of the camera angular velocity and use principal component analysis to reduce the dimension of the camera angular velocity and the IMU average angular velocity.

[0017] The time alignment module is configured to perform a dimensionality reduction matrix on the dimensionality-reduced angular velocity data. It adopts fast outlier removal and interpolation optimization based on random projection, matches the camera frame and IMU sensor angular velocity data through correlation, and performs interpolation optimization to obtain the time deviation for time alignment.

[0018] The post-processing module is configured to update the time deviation estimate based on the correlation between the camera frame and the corresponding IMU angular velocity.

[0019] One or more embodiments provide a motion-consistency-based visual-inertial sensor time synchronization system, including: a visual-inertial sensor and a processor;

[0020] Visual inertial sensors include cameras and IMUs;

[0021] The processor is configured to perform the steps of the motion consistency-based visual inertial sensor time synchronization method described above.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention constructs a motion-consistency-based time synchronization mechanism, achieving precise alignment of visual and IMU data purely in software without the need for hardware trigger signals and high-precision timestamps. Its core advantages lie in its lack of reliance on additional hardware, adapting to low-cost configurations with ordinary industrial cameras and inertial measurement units. First, feature points are extracted from adjacent image frames to estimate camera rotation changes and calculate camera angular velocity. The camera angular velocity is then calculated using rotational relationships extracted from the image sequence, enabling the system to acquire stable visual motion information even without external references, providing a precise input foundation for time alignment. Then, dimensionality reduction and sliding correlation optimization strategies are used to reduce computational complexity, improving algorithm efficiency and real-time performance. Its structural design exhibits good adaptability to high-speed motion, system load fluctuations, and heterogeneous hardware and software environments, significantly improving the synchronization accuracy, robustness, and deployment flexibility of multi-sensor fusion positioning systems.

[0024] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0026] Figure 1 This is a flowchart of the visual inertial sensor time synchronization method according to Embodiment 1 of the present invention;

[0027] Figure 2This is a schematic diagram of the windows constructed when extracting angular velocity information in Embodiment 1 of the present invention;

[0028] Figure 3 This is a schematic diagram of the process for obtaining angular velocity from sensor data to be aligned, according to Embodiment 1 of the present invention.

[0029] Figure 4 This is a schematic diagram of the dimensionality reduction and time alignment process in Embodiment 1 of the present invention;

[0030] Figure 5 This is a flowchart of the post-processing method of Embodiment 1 of the present invention;

[0031] Figure 6 These are comparison diagrams of the alignment results of camera and IMU timestamps obtained by different methods in the experiment of Example 1 of the present invention; wherein, (a) is the alignment result of camera and IMU timestamps obtained by using the timestamp alignment method; and (b) is the alignment result of camera and IMU timestamps obtained by using the motion consistency-based visual inertial sensor time synchronization method proposed in Example 1 of the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0034] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0035] Explanation of technical terms:

[0036] Matching point: refers to the same physical point or spatial feature point that is identified and corresponds to in multiple frames of images through methods such as feature extraction and descriptor calculation.

[0037] The essential matrix is ​​a 3×3 matrix that describes the geometric relationship between two cameras with known intrinsic parameters (usually a single camera in different time frames). It encodes the relative rotation and translation information between the cameras and is used to describe the epipolar constraints of spatial points in the two frames. The epipolar constraint means that the projection point x1 of a spatial point in the first frame and the corresponding point x2 in the second frame must fall on the epipolar line determined by the epipolar geometry of the projection point in the first frame.

[0038] Singular Value Decomposition (SVD): Singular value decomposition is a general matrix decomposition technique used to decompose any matrix into three parts, thereby revealing its structural characteristics.

[0039] Lie algebras: Lie algebras are linear approximations of Lie groups around their identity element (i.e., the "initial state"), and are a mathematical tool for describing continuous transformations (such as rotations and translations). They transform nonlinear attitude space problems into operations in linear space, thus facilitating numerical optimization and error modeling.

[0040] Example 1

[0041] In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 5 As shown, a time synchronization method for a visual inertial sensor based on motion consistency includes the following steps:

[0042] Step 1: For the acquired camera images, extract inter-frame feature points and estimate rotation relationships, and calculate camera angular velocity; for the acquired IMU data, establish an IMU sliding window and perform integration to obtain IMU average angular velocity information.

[0043] Step 2: Construct a direction weighting matrix based on the relative amplitude of the camera angular velocity, and use principal component analysis to reduce the dimensionality of the camera angular velocity and the IMU average angular velocity;

[0044] Step 3: For the dimension reduction matrix of the angular velocity data after dimension reduction, fast outlier removal and interpolation optimization based on random projection are adopted. The camera frame and IMU sensor angular velocity data are matched by correlation, and the time deviation for time alignment is obtained by interpolation optimization.

[0045] Step 4: Based on the correlation between camera frames and corresponding IMU angular velocities, update the time deviation estimate;

[0046] The above implementation achieves high-precision time synchronization by analyzing the consistency of angular velocity between the camera and the IMU. During operation, feature points are first extracted from adjacent image frames to estimate the camera's rotational changes and calculate the camera's angular velocity. The camera's angular velocity is then calculated using the rotational relationships extracted from the image sequence, enabling the system to acquire stable visual motion information even without external references, thus providing a precise input basis for time alignment.

[0047] Meanwhile, angular velocity data within a sliding window is extracted from the high-frequency IMU, and the local average angular velocity is calculated by integration. The sliding window is then used to perform integration processing based on the IMU data, effectively capturing the motion trend of the IMU within the corresponding time period, improving the temporal consistency with the camera's angular velocity, and making the matching more accurate.

[0048] Subsequently, in order to reduce the computational load and extract the main motion trends, a direction weighted matrix was constructed using the relative amplitude of the camera angular velocity, and the Principal Component Analysis (PCA) method was used to reduce the dimensionality of the data, making the feature representation more compact, simplifying the data structure, effectively reducing the computational burden, and improving the real-time performance of the system.

[0049] After removing outliers by performing a random projection algorithm on the dimensionality-reduced data, matching is achieved by calculating the correlation between the angular velocity sequences of the IMU and the camera. Then, interpolation is used to optimize the time deviation between the two. The random projection method enables efficient outlier identification and removal, improving matching accuracy and robustness. It ensures that the interpolation process is based only on highly correlated data points, reducing the impact of mismatches. Based on correlation analysis and linear interpolation strategies, the time offset between the camera and the IMU is accurately estimated, and the synchronization state is continuously corrected during system operation to ensure long-term operational accuracy. Finally, the time deviation estimate is updated based on the correlation between image frames and the average angular velocity of the IMU, achieving dynamic synchronization adjustment. A sliding update mechanism ensures that time alignment dynamically adapts to changes in scene and device status, thereby enhancing the system's stability in highly dynamic or complex environments.

[0050] The time synchronization method described in this embodiment uses the consistency of sensor angular velocity as the optimization target for time deviation estimation, and solves and corrects the time deviation between the two sensors in real time. It does not rely on hardware synchronization signals or require high-precision timestamp support, and can continuously and dynamically adjust the synchronization deviation during system operation, significantly improving the accuracy and robustness of camera and IMU time alignment. It is particularly suitable for multi-sensor fusion positioning systems in high-speed motion, system load fluctuation, and heterogeneous hardware and software environments. By implementing time synchronization purely in software, it avoids the use of dedicated hardware trigger modules or unified clock sources, making it suitable for ordinary industrial cameras and inertial measurement units. This reduces the integration cost of multi-sensor systems, simplifies system deployment, and preserves the accuracy of time alignment.

[0051] In the time deviation estimation process, this embodiment introduces a dimensionality reduction strategy to simplify the expression of angular velocity features, reduce the computational complexity of the matching process, reduce system resource consumption, and improve the operating efficiency of the synchronization module. It is suitable for high-frequency data streams and resource-constrained platform environments.

[0052] Step 1 is the data preprocessing step;

[0053] In step 1, for the acquired camera images, inter-frame feature points are extracted and rotation relationships are estimated, and the camera angular velocity is calculated, including the following steps:

[0054] Step 11: Create a main window for the frames of camera image data to be processed, which includes three consecutive frames of image data.

[0055] Step 12: For any two frames in the main window, perform feature point matching and calculate the distance between each pair of matching points; any two frames for feature point matching can include the first frame and the second frame, the second frame and the third frame, or the first frame and the third frame.

[0056] Optionally, in this embodiment, the main window includes three frames of image data. The matching points between the first frame and the second frame are extracted as follows:

[0057] ;

[0058] in, These are the normalized coordinates of the i-th feature point in the first frame of the image; These are the feature points that match in the second frame; N represents the number of matching points in the image. Each pair of matching points has a corresponding descriptor distance. .

[0059] Step 13: Sort the distances of all matching point pairs in ascending order, take the median as the reference distance, and calculate the confidence weight of each matching point based on the reference distance. ;

[0060] Sort all matching distances in ascending order. To avoid errors caused by minimum and maximum values, incorrect matches can be removed. The median of all matching distances can then be used. The confidence weight of the matching point is calculated using the reference distance as a weighting factor. Defined as:

[0061] ;

[0062] in, The scaling parameter of the weighting function is defined as follows: ; This is the distance tolerance coefficient, with an empirical value of 2, used to eliminate incorrect matches.

[0063] Step 14: Based on the confidence weights of the matching points, construct and solve the essential matrix estimation optimization problem to obtain the essential matrix; the essential matrix estimation optimization problem is as follows:

[0064] ;

[0065] Where E is the essential matrix;

[0066] Step 15: Perform singular value decomposition based on the estimated essential matrix to obtain the rotation matrix of any two frames of images;

[0067] Similarly, by solving the rotation matrix from the first frame to the second frame using the above steps, we can obtain the rotation matrices from the second frame to the third frame and from the first frame to the third frame.

[0068] Step 16: Establish an optimization objective function based on the consistency relationship between rotation matrices, and solve for the absolute rotation matrix from the first frame to the third frame in the main window;

[0069] Considering the influence of various noises, an objective function can be established. :

[0070] ;

[0071] in, This represents the rotation matrix from the first frame to the third frame obtained through step 15; This represents the relative rotation matrix from the first frame to the second frame; This represents the relative rotation matrix from the second frame to the third frame; all the parameters mentioned above are observations. Let be the variable to be optimized, representing the absolute rotation matrix in the third frame with the first frame as the world coordinate system. The goal is to find the optimal solution for this rotation matrix by minimizing the objective function. Describe the Lie algebra of the rotation matrix; This represents the difference between the rotation matrices after computing the Lie algebraic mapping. These constraints are fused using an optimization algorithm to obtain the optimal solution. This refers to the change in the rotation matrix from the first frame to the third frame in the main window.

[0072] Step 17: Obtain the rotational angular velocity vector, i.e., the camera angular velocity, by approximating the calculation using the derivative of the absolute rotation matrix;

[0073] Define the time interval between the first frame and the third frame as Then, the three components of the rotational angular velocity vector can be approximated by the derivative of the rotation matrix:

[0074] ;

[0075] in, This represents the element in the 3rd row and 2nd column of the matrix; the meanings of other symbols follow the same logic. Thus, the angular velocities in the three directions displayed in the main window are obtained.

[0076] The above implementation method, without relying on additional sensor information, can extract camera angular velocity information solely from image sequences, providing a foundation for subsequent matching with IMU data. The use of a confidence weight mechanism improves the robustness of feature matching and enhances the accuracy of rotation estimation; while the three-frame structure design helps capture more stable rotation trends and effectively suppresses interference from instantaneous jitter. Furthermore, extracting camera rotation through matrix factorization and consistency optimization improves the control over rotation accuracy, providing a more reliable angular velocity data foundation for time synchronization, thereby improving the overall alignment effect of the visual-inertial system.

[0077] In step 1, the acquired IMU data is discrete raw angular velocity data, which is processed into continuous angular velocity data, i.e., IMU average angular velocity, after integration.

[0078] In step 1, an IMU sliding window is established and integrated based on the acquired IMU data to obtain the IMU average angular velocity information (which can be simply referred to as IMU angular velocity). This includes the following steps:

[0079] Step 101: Establish an IMU data extension window with a time boundary set before the time of the first frame image captured by the camera and a time boundary set after the time of the third frame image.

[0080] Specifically, the duration can be set to 1 second; for IMU data, firstly, an extended window is created 1 second before the first frame of the camera and 1 second after the third frame. This extended window contains all IMU data within this time period.

[0081] Step 102: Determine the sliding window size based on the ratio of the sampling frequencies of the IMU and the camera sensor, and construct the IMU sliding window within the extended window;

[0082] A sliding window for the IMU is created within the extended window. Its size is related to the acquisition frequency of the camera and the IMU. The size of the IMU sliding window is defined as follows:

[0083] ;

[0084] in, This indicates the size of the IMU sliding window, that is, the number of IMU data points contained in the window. , These represent the sampling frequencies of the IMU and camera sensor, respectively. This indicates rounding down to the nearest integer. See the diagram for each window. Figure 2 As shown.

[0085] Step 103: Perform numerical integration on the IMU angular velocity data within the IMU sliding window to obtain the IMU average angular velocity vector;

[0086] Since the angular velocities within the window are discrete, they cannot reflect the average angular velocity over the corresponding time period for the camera. Therefore, it is necessary to numerically integrate the discrete angular velocities to obtain the average angular velocity of the IMU sliding window. The formula is as follows:

[0087] ;

[0088] in, This represents the average angular velocity vector obtained from the IMU sliding window, where T represents the total duration of the sliding window. Let represent the angular velocity of the j-th IMU data point within the sliding window, and m represent the number of IMU sampling points in the current sliding window. This represents the time interval between time j and time j+1.

[0089] In the above implementation, to address the problem that traditional interpolation methods cannot accurately model state changes under nonlinear motions such as acceleration and turning, an IMU extended window is established by strictly corresponding to the time range of the visual frame, ensuring the basic time alignment between data and improving fusion accuracy. The sliding window mechanism effectively utilizes high-frequency IMU data, enhancing the representativeness of angular velocity information while ensuring real-time performance and reducing the interference of single-point noise on the results. The integration operation reflects the dynamic trend within a local time period, making the obtained average angular velocity more stable, which helps to conduct more accurate correlation analysis with the visual angular velocity, thereby improving the accuracy and robustness of the entire time synchronization algorithm.

[0090] The above preprocessing steps for obtaining the angular velocities of the camera and IMU sensor are illustrated in the diagram below. Figure 3 As shown in the figure. In step 1 of this embodiment, feature points are first extracted from the camera image data, matched, and an essential matrix estimation optimization problem is constructed. The rotation matrix is ​​obtained through singular value decomposition, the time interval is calculated, and then the angular velocity is extracted. IMU data is obtained through an extended window, and the average angular velocity of the sliding window is obtained through numerical integration. By constructing a three-frame window rotation model and combining it with the IMU sliding window, a locally robust angular velocity extraction mechanism is achieved. An innovative three-frame image window structure is designed, and angular velocity information is derived through continuous inter-frame rotation estimation and paired with the average angular velocity integrated within the IMU sliding window. This method can stably extract short-term motion features without relying on high-precision timestamps and accurately invert synchronization deviations, realizing a highly robust alignment mechanism that integrates structural, motion, and temporal information.

[0091] Since the same sensors on the market have different sampling frequencies, and with the development of society, the sampling frequencies of cameras and IMUs are getting higher and higher, the problem of data redundancy will become more and more serious. Meanwhile, time synchronization requires real-time performance. In order to further enhance the efficiency of time alignment, the three-dimensional angular velocity vector is reduced in dimension.

[0092] In step 2, the steps for data dimensionality reduction are as follows: a direction weighting matrix is ​​constructed based on the relative amplitude of the camera angular velocity, and principal component analysis is used to reduce the dimensionality of the camera angular velocity and the IMU average angular velocity; after weighting and removing the mean from the original data through principal component analysis, a covariance matrix is ​​established and eigenvalue decomposition is performed to extract the first two principal axis directions, thereby achieving effective dimensionality reduction of the three-dimensional angular velocity vector.

[0093] In step 2, a direction weighting matrix is ​​constructed based on the relative magnitude of the camera angular velocity, including the following steps:

[0094] Step 21: Construct an angular velocity matrix based on the obtained IMU average angular velocity information and camera angular velocity; define the angular velocity matrix. for:

[0095] ;

[0096] In this matrix, the first row represents the angular velocity data in three directions obtained by the camera, while the remaining rows represent the average angular velocity of the IMU obtained by each IMU sliding window. M represents the sum of the number of camera angular velocity vectors and IMU average angular velocity vectors.

[0097] In angular velocity analysis, motion in different directions may have different levels of importance. Among the three-axis angular velocities observed by a camera, a larger value in that direction usually indicates more intense motion and a greater impact on visual changes. Therefore, constructing a weighted matrix using camera angular velocities can both amplify the influence of the dominant direction and minimize information loss during subsequent dimensionality reduction.

[0098] Step 22: Calculate the weight vectors of the three axes based on the camera angular velocity, and construct a normalized direction weighted vector. Specifically:

[0099] ;

[0100] in, This indicates taking the absolute value. , and Let these represent the weighted values ​​in the x, y, and z directions, respectively, satisfying... .

[0101] Step 23: Based on the normalized direction weighted vector, construct the corresponding weighted diagonal matrix as follows:

[0102] ;

[0103] Step 24: Apply the weight matrix to each row to obtain the weighted angular velocity matrix. As a directional weighted matrix:

[0104] ;

[0105] Step 2 involves performing principal component analysis on the obtained directional weighted matrix, including the following steps:

[0106] Step 24: For the direction-weighted matrix, calculate the mean vector of each column, perform a mean-removal operation, and obtain the centered matrix. ;

[0107] To eliminate the offset of the data along each axis and ensure that the subsequent dimensionality reduction data can accurately extract the main direction reflecting the true trend of change, this embodiment calculates the mean vector of each column and performs a mean-removal operation to obtain a centered matrix. :

[0108] ;

[0109] in, This represents an M-row, M-column vector consisting entirely of 1s;

[0110] The column mean vector is represented as:

[0111] ;

[0112] Step 25: Based on the centered matrix Establish the covariance matrix :

[0113] ;

[0114] Step 26: Convert the covariance matrix Perform eigenvalue decomposition and take the first two principal axes. Dimensionality reduction is performed to obtain the final dimensionality-reduced matrix. :

[0115] ;

[0116] in, The eigenvectors of the covariance matrix represent the principal direction of the data. For each corresponding eigenvalue, the variance is represented by the first two principal axes. Dimensionality reduction is performed to obtain the final dimensionality-reduced matrix. as follows:

[0117] ;

[0118] In the formula, a and b represent the elements after dimensionality reduction.

[0119] In this embodiment, a weighted matrix is ​​used to reduce the dimensionality of the angular velocity data, thereby reducing redundancy and enhancing the influence of important directions. Changes in angular velocity can reflect rapid attitude changes, making it more suitable for highly dynamic environments.

[0120] Step 3 is the time alignment step. By establishing the time alignment relationship between the angular velocity data of different sensors through the correlation between motion information (angular velocity), rather than relying on the system timestamp, the error problem of traditional soft synchronization is bypassed.

[0121] In step 3, for the dimensionality reduction matrix of the angular velocity data, a fast outlier removal and interpolation optimization based on random projection is adopted. By matching the camera frame and IMU sensor angular velocity data through correlation, and performing interpolation optimization, a method to achieve time alignment and time deviation is obtained, including the following steps:

[0122] Step 31: For the dimensionality-reduced angular velocity data, outliers are removed based on random projection, that is, the dimensionality-reduced angular velocity matrix is ​​removed using random projection. The outliers in the data are identified, and the projection results of each point are used to construct an identifier vector to establish an index relationship;

[0123] Step 311: Design a set of random vectors based on random projection to map the two-dimensional data points in the dimensionality reduction matrix to multiple label spaces, and obtain a one-dimensional scalar for each data point;

[0124] To further improve the efficiency of time alignment, for the two-dimensional data points in the dimensionality reduction matrix, outliers are first removed to reduce data complexity. Then, a set of random vectors based on random projection is designed to map the point set, transforming the two-dimensional points into multiple label spaces. L sets of random vectors are designed. ,Will Each row in the data is used as a two-dimensional point. For each point, along the constructed L Each of the random vectors is projected onto a one-dimensional scalar. :

[0125] ;

[0126] In the formula, and These represent the first and second random vectors, respectively. l Group 1 and 1 l The element in the second column of the group;

[0127] Step 312: Concatenate and fuse the one-dimensional scalars of each data point in different directions to obtain the identifier vector of each data point;

[0128] Specifically, each point in the whole L The aggregation of one-dimensional scalar projections in each direction can form its... L Dimensional identifier vector:

[0129] ;

[0130] Step 313: Based on the label vector obtained by projection, calculate the Euclidean distance between each IMU data point and the camera data point in the label space, and identify and remove outliers in the dimension reduction matrix based on the distance.

[0131] For camera dimensionality reduction point Calculate its corresponding IMU identifier To establish index relationships:

[0132] ;

[0133] In the formula, Represents Euclidean distance; Let represent the projection vectors of the camera data points and IMU data points in the label space, respectively. Indicates the filtering threshold;

[0134] Points exceeding the screening threshold are considered outliers and removed. Since the mean reflects the approximate average distance between the mean points of the data distribution, and the standard deviation measures the dispersion of the data, this embodiment further uses the mean and standard deviation to adaptively adjust the sensitivity threshold. The calculation formula is:

[0135] ;

[0136] In the formula, N represents the number of identifiers. This represents a custom parameter that controls the sensitivity of the screening process.

[0137] Step 32: After removing most outliers, the correlation calculation method is used to determine the IMU data point that best matches the camera angular velocity, and the time deviation is solved based on the interpolation optimization method to finally achieve time alignment between the two sensors.

[0138] The camera frames and IMU sensor angular velocity data are matched by correlation, and interpolation optimization is performed to obtain the time deviation for time alignment, including the following steps;

[0139] Step 321: After removing the vast majority of outliers, use correlation calculation methods to determine the IMU data points most relevant to the camera angular velocity dimensionality reduction data;

[0140] Specifically, the formula for calculating the correlation between the camera dimensionality reduction point and the IMU dimensionality reduction point is as follows:

[0141] ;

[0142] In the formula, This represents the dimensionality reduction matrix after outlier removal. The i-th point in Then it represents a dimension reduction matrix. The first row of data forms a two-dimensional point, which corresponds to the point after dimensionality reduction of the camera's angular velocity. When The closer the two data points are to 1, the more related they are.

[0143] Step 322: After determining the IMU data point that best matches the camera angular velocity, use the median timestamp of the IMU sliding window corresponding to the IMU data point as a reference, and perform linear interpolation calculation on the IMU data within a set time offset range. Calculate the difference between the camera angular velocity and the IMU angular velocity under different offsets, and find the offset time corresponding to the smallest difference. Perform time alignment;

[0144] In this embodiment, the IMU angular velocity refers to the angular velocity data initially acquired by the sensor, i.e., the IMU data;

[0145] Specifically, the method for solving the offset time is established by minimizing the difference between two sets of angular velocities under different offsets. The objective function is as follows:

[0146] ;

[0147] In the formula, This represents the angular velocity vector composed of the elements in the first row of the angular velocity matrix H, i.e., the time interval at the midpoint of the main window. The corresponding camera angular velocity; This indicates the midpoint of the IMU sliding window. The corresponding angular velocity vector is directly obtained from the IMU; the objective function that is minimized by linear interpolation is the corresponding time deviation. Because of the high density of IMU data, camera timestamps can be corrected. The entire process is as follows: Figure 4 As shown.

[0148] This embodiment constructs a cross-modal motion constraint model based on angular velocity consistency to achieve accurate time deviation estimation. Unlike traditional synchronization methods that rely on hardware triggering or timestamp matching, this embodiment proposes for the first time to estimate time deviation based on the consistency constraint of camera and IMU angular velocities during motion, establishing a correlation function between visual angular velocity and IMU angular velocity as the optimization objective function. This method can bypass the interference of system scheduling delay and timestamp error, achieving sensor alignment from the essence of motion. It is particularly suitable for high-frequency, violent motion environments, breaking through the technical bottleneck of traditional time-domain interpolation methods being unable to accurately reproduce nonlinear motion states.

[0149] In the above implementation, a combination of dimensionality reduction and fast matching indexing mechanisms improves the efficiency of high-frequency angular velocity feature matching. Facing the computational pressure brought by high-frequency sensor data, this embodiment introduces a direction-sensitive dimensionality reduction strategy based on weight analysis, effectively compressing three-dimensional angular velocity into a two-dimensional feature representation, and combining it with a multi-vector random projection indexing mechanism for fast matching. This design significantly reduces the matching dimensionality and computational complexity, enabling the time synchronization process to run in real-time on embedded low-power platforms. It breaks through the efficiency bottleneck of traditional methods in high-frequency streaming data processing, improving the algorithm's engineering adaptability and practical value.

[0150] Furthermore, it also includes updating the time bias estimate based on the correlation between camera frames and the corresponding IMU average angular velocity. Specifically: based on the time bias... For the corresponding current frame image of the camera, extract the average angular velocity data of the IMU sliding window at the current time, the previous time, and the next time corresponding to the current frame image, and compare the correlation between the average angular velocity of the IMU at the previous time, the current time, and the next time with the angular velocity corresponding to the current frame image. If the correlation of the current frame is the strongest, then keep the existing time deviation; otherwise, re-execute the dimensionality reduction in step 2 and the time alignment in step 3, and update the time deviation estimate.

[0151] Under normal circumstances, the time deviation of the same sensor is roughly the same at each moment. To further reduce computational redundancy, after completing one time deviation correction, the correlation of the IMU angular velocity data corresponding to the current camera frame image is calculated, and compared with the IMU data corresponding to the previous and next frames respectively. If the correlation between the current frame and the corresponding IMU data is significantly higher than the correlation with the adjacent frames before and after, then the current time alignment is considered to be accurate, and no further correction is needed.

[0152] The correlation formula between camera frame images and corresponding IMU average angular velocity data is as follows:

[0153] ;

[0154] In the formula, To indicate correlation, the camera angular velocity vector is defined as follows: , Represents camera angular velocity The average value; the IMU angular velocity vector is defined as ; This represents the average value of the IMU angular velocity vector;

[0155] If the above conditions cannot be met, the data dimensionality reduction and time alignment steps are repeated to estimate the time deviation at the new time point. The post-processing procedure is as follows: Figure 5 As shown.

[0156] Compared with traditional time synchronization methods (such as timestamp alignment), this embodiment significantly improves the time synchronization accuracy, real-time performance, and robustness of multi-sensor systems in complex environments through an online time offset calibration mechanism based on motion consistency. Experimental verification was conducted to confirm the effectiveness, and the details are as follows:

[0157] This implementation uses the M2DGR-door02 open-source dataset for verification. This dataset contains multi-sensor measurement data from a real mobile platform. The IMU and camera acquire data independently, and the timestamps naturally have discrepancies, which is consistent with the unaligned multi-sensor synchronization scenario handled in this invention.

[0158] Figure 6 (a) and Figure 6 Figure (b) shows the alignment of camera and IMU timestamps under two different time synchronization strategies: where, Figure 6 (a) in the text corresponds to the timestamp alignment method. Figure 6 (b) in this embodiment corresponds to the motion consistency-based online calibration method, namely, the motion consistency-based visual inertial sensor time synchronization method. Given that the IMU device has a high sampling frequency (approximately 200Hz), data within the time interval of 70 seconds to 71.2 seconds was selected for visualization to facilitate analysis of the time alignment effect. During this time period, the vehicle's motion state is relatively complex and exhibits strong dynamic changes, which can comprehensively reflect the alignment performance and robustness of the time synchronization mechanism in real-world scenarios.

[0159] In traditional timestamp alignment methods, IMU data and camera frames are simply matched according to their original timestamps. Figure 6 As can be observed in (a), there is a certain degree of mismatch in the time correspondence between camera frames and IMU sampling points, especially in scenarios with low image frame rates and high IMU frequencies. This problem is more pronounced and can easily lead to a decrease in the accuracy of subsequent data fusion. In this embodiment, the motion consistency between the camera's preceding and following frames and the angular velocity of the IMU sliding window is used to estimate the time deviation online, and the correspondence between camera frames and IMU measurements is adjusted accordingly. Figure 6 In (b), it is clear that the alignment between the camera timestamp and the IMU frame time is more reasonable, especially in... Figure 6 (a) and Figure 6 The red box in (b) indicates that the motion consistency-based time synchronization method can more accurately align IMU data compared to traditional methods, thus providing a more stable and accurate foundation for state estimation and feature alignment in multi-sensor fusion.

[0160] In summary, the timestamp analysis results verify the effectiveness of this embodiment in real-world unaligned data scenarios, and it can significantly improve the synchronization accuracy and motion modeling consistency of the system.

[0161] Example 2

[0162] Based on Embodiment 1, this embodiment provides a visual-inertial sensor time synchronization system based on motion consistency, including:

[0163] The angular velocity information extraction module is configured to extract inter-frame feature points and estimate rotation relationships for acquired camera images, and calculate camera angular velocity; for acquired IMU data, it establishes an IMU sliding window and performs integration to obtain IMU average angular velocity information.

[0164] The dimension reduction module is configured to construct a direction-weighted matrix based on the relative amplitude of the camera angular velocity and use principal component analysis to reduce the dimension of the camera angular velocity and the IMU average angular velocity.

[0165] The time alignment module is configured to perform a dimensionality reduction matrix on the dimensionality-reduced angular velocity data. It adopts fast outlier removal and interpolation optimization based on random projection, matches the camera frame and IMU sensor angular velocity data through correlation, and performs interpolation optimization to obtain the time deviation for time alignment.

[0166] The post-processing module is configured to update the time deviation estimate based on the correlation between the camera frame and the corresponding IMU angular velocity.

[0167] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0168] Example 3

[0169] Based on Embodiment 1, this embodiment provides a visual inertial sensor time synchronization system based on motion consistency, including: a visual inertial sensor and a processor;

[0170] Visual inertial sensors include cameras and IMUs;

[0171] The processor is configured to perform the steps of the motion consistency-based visual inertial sensor time synchronization method described in Example 1.

[0172] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0173] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual-inertial sensor time synchronization method based on motion consistency, characterized in that, Includes the following steps: For the acquired camera images, feature points between image frames are extracted and rotation relationships are estimated to calculate the camera angular velocity; for the acquired IMU data, an IMU sliding window is established and integration is performed to obtain the IMU average angular velocity information. Constructing a direction-weighted matrix based on the relative magnitude of the camera angular velocity includes the following steps: An angular velocity matrix is ​​constructed based on the obtained IMU average angular velocity information and camera angular velocity. The weight vectors of the three axes are calculated based on the camera angular velocity, and a normalized weighted vector of directions is constructed. Construct the corresponding weighted diagonal matrix based on the normalized directional weighted vector; Apply the weighted diagonal matrix to each row to obtain the weighted angular velocity matrix, which is then used as the direction weighting matrix. Principal component analysis is used to reduce the dimensionality of the directional weighted matrix; the steps include: For the direction-weighted matrix, calculate the mean vector of each column, perform a mean-removal operation, and obtain the centered matrix. Based on the centered matrix, construct the covariance matrix; The covariance matrix is ​​decomposed into eigenvalues, and the first two principal axes are used to reduce the dimension, resulting in the final dimension-reduced matrix. For the dimensionality reduction matrix of the angular velocity data, a fast outlier removal and interpolation optimization based on random projection is adopted. The camera frames and IMU sensor angular velocity data are matched by correlation, and interpolation optimization is performed to obtain the time deviation for time alignment. The process includes the following steps: After removing outliers, the correlation calculation method was used to determine the IMU data points most relevant to the dimensionality reduction data of the camera angular velocity. After determining the IMU data points that best match the camera, the median timestamp of the corresponding IMU sliding window is used as a reference. Linear interpolation is then performed on the IMU data within a set time offset range. The difference between the two sets of angular velocities under different offsets is calculated, and the offset time corresponding to the smallest difference is found. Perform time alignment; The time deviation estimate is updated based on the correlation between camera frames and the corresponding IMU angular velocities.

2. The visual-inertial sensor time synchronization method based on motion consistency as described in claim 1, characterized in that, For the acquired camera images, extract inter-frame feature points and estimate rotation relationships, and calculate the camera angular velocity, including the following steps: Create a main window for the frames of camera image data to be processed, which includes three consecutive frames of image data; For any two frames of images in the main window, perform feature point matching and calculate the distance between each pair of matching points; Sort the distances of all matching point pairs in ascending order, take the median as the reference distance, and calculate the confidence weight of each matching point based on the reference distance; Based on the confidence weights of the matching points, an optimization problem for estimating the essential matrix is ​​constructed, and the essential matrix is ​​obtained by solving it. Singular value decomposition is performed based on the estimated essential matrix to obtain the rotation matrix of any two frames of images. An optimization objective function is established based on the consistency relationship between rotation matrices, and the absolute rotation matrix from the first frame to the third frame in the main window is obtained by solving it. The rotational angular velocity vector, i.e., the camera angular velocity, is obtained by approximating the calculation using the derivative of the absolute rotation matrix.

3. The visual-inertial sensor time synchronization method based on motion consistency as described in claim 1, characterized in that, For the acquired IMU data, an IMU sliding window is established and integration is performed to obtain the IMU average angular velocity information, including the following steps: An IMU data expansion window is established with a time boundary set before the time of the first frame image captured by the camera and a time boundary set after the time of the third frame image. The size of the sliding window is determined based on the ratio of the sampling frequencies of the IMU and the camera sensor, and the IMU sliding window is constructed within the extended window. Numerical integration is performed on the IMU angular velocity data within the IMU sliding window to obtain the IMU average angular velocity vector.

4. The visual-inertial sensor time synchronization method based on motion consistency as described in claim 1, characterized in that: For the dimensionality reduction matrix of the dimensionality-reduced angular velocity data, outliers are removed based on random projection, including the following steps: Design a set of projection vectors based on random projection to map two-dimensional data points in the dimensionality reduction matrix to multiple label spaces, obtaining a one-dimensional scalar for each data point; The one-dimensional scalars of each data point in different directions are concatenated and merged to obtain the identifier vector of each data point; Based on the label vector obtained from projection, calculate the Euclidean distance between each IMU data point and the camera data point in the label space, and identify and remove outliers in the dimension reduction matrix based on the distance.

5. The visual-inertial sensor time synchronization method based on motion consistency as described in claim 1, characterized in that: It also includes updating the time deviation estimate based on the correlation between camera frames and the corresponding IMU average angular velocity. Specifically, based on the current frame image of the camera corresponding to the time deviation, the average angular velocity data of the IMU sliding window corresponding to the current frame image at the current time, the previous time, and the next time are extracted, and the correlation between the IMU average angular velocity at the previous time, the current time, and the next time and the corresponding angular velocity of the current frame image is compared. If the correlation of the current frame is the largest, the existing time deviation is maintained; otherwise, the dimensionality is reduced and time is aligned again, and the time deviation estimate is updated.

6. A visual-inertial sensor time synchronization system based on motion consistency, characterized in that, include: The angular velocity information extraction module is configured to extract inter-frame feature points and estimate rotation relationships for the acquired camera images, and calculate the camera angular velocity. For the acquired IMU data, an IMU sliding window is established and integration is performed to obtain the IMU average angular velocity information; The dimension reduction module is configured to construct a direction-weighted matrix based on the relative magnitude of the camera angular velocity, and includes the following steps: An angular velocity matrix is ​​constructed based on the obtained IMU average angular velocity information and camera angular velocity. The weight vectors of the three axes are calculated based on the camera angular velocity, and a normalized weighted vector of directions is constructed. Construct the corresponding weighted diagonal matrix based on the normalized directional weighted vector; Apply the weighted diagonal matrix to each row to obtain the weighted angular velocity matrix, which is then used as the direction weighting matrix. Principal component analysis is used to reduce the dimensionality of the directional weighted matrix; the steps include: For the direction-weighted matrix, calculate the mean vector of each column, perform a mean-removal operation, and obtain the centered matrix. Based on the centered matrix, construct the covariance matrix; The covariance matrix is ​​decomposed into eigenvalues, and the first two principal axes are used to reduce the dimension, resulting in the final dimension-reduced matrix. The time alignment module is configured to perform a dimensionality reduction matrix on the dimensionality-reduced angular velocity data. It employs fast outlier removal and interpolation optimization based on random projection, matching camera frames and IMU sensor angular velocity data through correlation analysis, and then performing interpolation optimization to obtain the time deviation for time alignment. The module includes the following steps: After removing outliers, the correlation calculation method was used to determine the IMU data points most relevant to the dimensionality reduction data of the camera angular velocity. After determining the IMU data points that best match the camera, the median timestamp of the corresponding IMU sliding window is used as a reference. Linear interpolation is then performed on the IMU data within a set time offset range. The difference between the two sets of angular velocities under different offsets is calculated, and the offset time corresponding to the smallest difference is found. Perform time alignment; The post-processing module is configured to update the time deviation estimate based on the correlation between the camera frame and the corresponding IMU angular velocity.

7. A visual-inertial sensor time synchronization system based on motion consistency, characterized in that, include: Visual inertial sensors and processors; Visual inertial sensors include cameras and IMUs; The processor is configured to perform the steps of the motion consistency-based visual inertial sensor time synchronization method according to any one of claims 1-5.

Images