Evaluation method and system for camera-inertial measurement unit alignment

Abstract

This application provides an evaluation method and system for camera-inertial measurement unit (IMU) alignment. The evaluation method includes: acquiring visual image stream data from the camera and measurement data from the IMU; performing time synchronization evaluation based on the visual image stream data and measurement data to obtain a time synchronization evaluation index; performing extrinsic parameter consistency evaluation based on the visual image stream data from the camera and measurement data from the IMU to obtain an extrinsic parameter consistency evaluation index; determining a comprehensive evaluation index based on the time synchronization evaluation index and the extrinsic parameter consistency evaluation index; comparing the comprehensive evaluation index with a target threshold, and determining the camera-inertial measurement unit alignment evaluation result based on the comparison result; and establishing a unified quantitative standard for evaluation through a multi-index fusion scoring system. Even in scenarios where motion capture or laser mapping is unavailable, alignment quality scores can be directly obtained, supporting comparisons of different algorithms or parameter configurations and significantly reducing testing costs.

Description

Technical Field

[0001] This application relates to the field of visual-inertial integrated sensing technology, specifically to an evaluation method and system for aligning a camera with an inertial measurement unit. Background Technology

[0002] In a visual-inertial (VI) sensing system, the complementarity between the camera and the inertial measurement unit (IMU) is achieved through a tightly coupled visual-inertial odometry structure, where the camera provides high-precision geometric constraints, while the IMU provides high-frequency angular velocity and linear acceleration, effectively overcoming the limitations of the camera's limited frame rate, light sensitivity, and the IMU's pure integration drift.

[0003] In related technologies, offline calibration toolchains (such as Kalibr) achieve integrated calibration by jointly optimizing camera intrinsic parameters, distortion, IMU noise and zero bias, spatiotemporal extrinsic parameters, and time offset. However, they rely on specific motion stimuli and target visibility, and data quality directly affects the results. Online self-calibration technology incorporates extrinsic parameters and time offset into the VIO sliding window optimization state, enhancing long-term robustness, but faces the problem of slow convergence or bias caused by state coupling.

[0004] While Kalibr and similar methods are relatively mature in calibration and optimization, their evaluation process is severely limited by truth-based reliance. Truth-based evaluation requires the use of motion capture systems, laser maps, or high-precision odometry to calculate metrics such as absolute trajectory error (ATE) and relative pose error (RPE), which are often unavailable in real-world outdoor or uncontrolled environments. Existing truth-free evaluation methods (such as reprojection error statistics, pre-integration residual norm, and cross-correlation / frequency domain coherence analysis) can reflect calibration quality through self-consistency, but their reliance on a single metric and lack of a unified baseline (e.g., depending solely on gravity direction consistency) makes it difficult to objectively quantify the merits of different spatiotemporal calibration schemes, and even more so to comprehensively assess the system's drift suppression capabilities over long-term operation. Therefore, there is an urgent need to establish a systematic truth-free evaluation framework and develop unified evaluation standards to assess the alignment quality between the camera and the inertial measurement unit. Summary of the Invention

[0005] This application provides an evaluation method and system for camera-inertial measurement unit (IMU) alignment. By establishing a unified quantitative standard through a multi-index fusion scoring system, it achieves quantitative and comparable camera-inertial measurement unit alignment evaluation. Therefore, even in scenarios where motion capture or laser mapping is unavailable, alignment quality scores can be directly obtained, supporting comparisons of different algorithms or parameter configurations and significantly reducing testing costs.

[0006] The camera-inertial measurement unit (IMU) alignment evaluation method of this application includes: acquiring visual image stream data of the camera and measurement data of the IMU; performing time synchronization evaluation based on the visual image stream data and the measurement data to obtain a time synchronization evaluation index; performing extrinsic parameter consistency evaluation based on the visual image stream data of the camera and the measurement data of the IMU to obtain an extrinsic parameter consistency evaluation index; determining a comprehensive evaluation index based on the time synchronization evaluation index and the extrinsic parameter consistency evaluation index; comparing the comprehensive evaluation index with a target threshold, and determining the camera-inertial measurement unit alignment evaluation result based on the comparison result.

[0007] In some implementations, acquiring the visual image stream data of the camera and the measurement data of the inertial measurement unit includes: acquiring the visual image stream data of the camera and the measurement data of the inertial measurement unit by online subscription or offline import of a preset configuration file; and dynamically updating the visual image stream data and the measurement data acquired at the current moment according to a predetermined duration based on a sliding window caching module.

[0008] In some embodiments, the step of performing time synchronization evaluation based on the visual image stream data and the measurement data to obtain a time synchronization evaluation index includes: determining the camera continuous frame rotation difference based on the visual image stream data; converting the camera continuous frame rotation difference into visual angular velocity; analyzing the visual angular velocity and the gyroscope data in the visual image stream data based on frequency domain coherence analysis to determine the synchronization confidence level; and determining the frequency domain coherence parameter in the time synchronization evaluation index based on the synchronization confidence level.

[0009] In some implementations, the step of performing time synchronization evaluation based on the visual image stream data and the measurement data to obtain a time synchronization evaluation index includes: determining the camera continuous frame rotation difference based on the visual image stream data; converting the camera continuous frame rotation difference into visual angular velocity; cross-correlating the visual angular velocity and the gyroscope data in the visual image stream data to extract the peak position and the peak energy corresponding to the peak position; and determining the cross-correlation peak parameter in the time synchronization evaluation index based on the peak energy corresponding to the peak position.

[0010] In some embodiments, the step of performing extrinsic parameter consistency evaluation based on the visual image stream data of the camera and the measurement data of the inertial measurement unit to obtain extrinsic parameter consistency evaluation index includes: pre-integrating the measurement data based on the synchronization time offset between the camera and the inertial measurement unit and the extrinsic parameters of the inertial measurement unit to obtain rotational residual, velocity residual, position residual and covariance; and determining the pre-integrated residual parameter in the extrinsic parameter consistency evaluation index based on the distribution mean of the rotational residual, the distribution mean of the velocity residual, the distribution mean of the position residual and the distribution mean of the covariance.

[0011] In some embodiments, the step of performing an external parameter consistency assessment based on the visual image stream data of the camera and the measurement data of the inertial measurement unit to obtain an external parameter consistency assessment index includes: determining the gravity direction based on the visual image stream data of the camera and the measurement data of the inertial measurement unit; comparing the gravity direction with the calibration direction to determine the gravity direction consistency parameter in the external parameter consistency assessment index.

[0012] In some embodiments, the step of performing extrinsic parameter consistency evaluation based on the visual image stream data of the camera and the measurement data of the inertial measurement unit to obtain an extrinsic parameter consistency evaluation index includes: determining the tracked feature points of each frame of the image based on the visual image stream data; reprojecting the tracked feature points of each frame of the image to determine the mean distribution of pixel error and mean square error; and determining the extrinsic parameter consistency evaluation index based on the mean distribution of pixel error and mean square error.

[0013] In some implementations, determining the comprehensive evaluation index based on the time synchronization evaluation index and the external parameter consistency evaluation index includes: weighting and summing at least one parameter in the time synchronization evaluation index and at least one parameter in the external parameter consistency evaluation index to determine the comprehensive evaluation index.

[0014] In some embodiments, the evaluation method for camera-inertial measurement unit alignment further includes: providing a visualization interface that can be configured to generate a formatted report; the visualization interface displays real-time updated visual image stream data and measurement data, time synchronization evaluation metrics, extrinsic parameter consistency evaluation metrics, comprehensive evaluation metrics, and the evaluation results of camera-inertial measurement unit alignment.

[0015] The camera-inertial measurement unit (IMU) alignment evaluation system of this application includes a data acquisition module configured to acquire visual image stream data from the camera and measurement data from the IMU; a time synchronization evaluation module configured to perform time synchronization evaluation based on the visual image stream data and the measurement data to obtain a time synchronization evaluation index; an extrinsic parameter consistency evaluation module configured to perform extrinsic parameter consistency evaluation based on the visual image stream data from the camera and the measurement data from the IMU to obtain an extrinsic parameter consistency evaluation index; and a comprehensive evaluation module configured to determine a comprehensive evaluation index based on the time synchronization evaluation index and the extrinsic parameter consistency evaluation index. The comprehensive evaluation module is further configured to compare the comprehensive evaluation index with a target threshold and determine the camera-inertial measurement unit alignment evaluation result based on the comparison result.

[0016] This application embodiment acquires visual image stream data from the camera and measurement data from the inertial measurement unit (IMU); performs time synchronization evaluation based on the visual image stream data and the measurement data to obtain a time synchronization evaluation index; performs extrinsic parameter consistency evaluation based on the visual image stream data from the camera and the measurement data from the IMU to obtain an extrinsic parameter consistency evaluation index; determines a comprehensive evaluation index based on the time synchronization evaluation index and the extrinsic parameter consistency evaluation index; compares the comprehensive evaluation index with a target threshold, and determines the camera-IMU alignment evaluation result based on the comparison result. A unified quantitative standard is established through a multi-index fusion scoring system to achieve quantitative and comparable camera-IMU alignment evaluation. Therefore, even in scenarios where motion capture or laser mapping is unavailable, alignment quality scores can be directly obtained, supporting comparisons of different algorithms or parameter configurations and significantly reducing testing costs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the evaluation method for aligning a camera with an inertial measurement unit provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram illustrating the process of dynamically updating the data stream using a sliding window caching module provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of a time synchronization evaluation process provided for an embodiment of this application.

[0021] Figure 4 Another flowchart illustrating the time synchronization evaluation provided for embodiments of this application.

[0022] Figure 5 This is a schematic diagram of a process for evaluating the consistency of external parameters provided in an embodiment of this application.

[0023] Figure 6 This is another schematic diagram of the process for evaluating the consistency of external parameters provided in the embodiments of this application.

[0024] Figure 7 A schematic diagram of the structure of an evaluation system for aligning a camera with an inertial measurement unit.

[0025] Figure 8 A schematic diagram of the workflow of an evaluation system for aligning a camera with an inertial measurement unit. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] This application provides an evaluation method and system for aligning a camera with an inertial measurement unit (IMU).

[0028] In a visual-inertial (VI) sensing system, the complementarity between the camera and the inertial measurement unit (IMU) is achieved through a tightly coupled visual-inertial odometry structure, where the camera provides high-precision geometric constraints, while the IMU provides high-frequency angular velocity and linear acceleration, effectively overcoming the limitations of the camera's limited frame rate, light sensitivity, and the IMU's pure integration drift.

[0029] In related technologies, offline calibration toolchains (such as Kalibr) achieve integrated calibration by jointly optimizing camera intrinsic parameters, distortion, IMU noise and zero bias, spatiotemporal extrinsic parameters, and time offset. However, they rely on specific motion stimuli and target visibility, and data quality directly affects the results. Online self-calibration technology incorporates extrinsic parameters and time offset into the VIO sliding window optimization state, enhancing long-term robustness, but faces the problem of slow convergence or bias caused by state coupling.

[0030] While Kalibr and similar methods are relatively mature in calibration and optimization, their evaluation process is severely limited by truth-based reliance. Truth-based evaluation requires the use of motion capture systems, laser maps, or high-precision odometry to calculate metrics such as absolute trajectory error (ATE) and relative pose error (RPE), which are often unavailable in real-world outdoor or uncontrolled environments. Existing truth-free evaluation methods (such as reprojection error statistics, pre-integration residual norm, and cross-correlation / frequency domain coherence analysis) can reflect calibration quality through self-consistency, but their reliance on a single metric and lack of a unified baseline (e.g., depending solely on gravity direction consistency) makes it difficult to objectively quantify the merits of different spatiotemporal calibration schemes, and even more so to comprehensively assess the system's drift suppression capabilities over long-term operation. Therefore, there is an urgent need to establish a systematic truth-free evaluation framework and develop unified evaluation standards to assess the alignment quality between the camera and the inertial measurement unit.

[0031] The solutions provided in this application relate to the field of visual-inertial integrated sensing technology, and are specifically illustrated through the following embodiments. Detailed descriptions are provided below. It should be noted that the order of description of the following embodiments is not intended to limit the priority of the embodiments.

[0032] The following describes the evaluation method for camera and inertial measurement unit alignment provided by exemplary embodiments of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown for the purpose of understanding the principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0033] Figure 1 This is a schematic diagram of an evaluation system for aligning a camera and an inertial measurement unit, provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the evaluation method for aligning a camera with an inertial measurement unit includes: Step 010: Acquire the camera's visual image stream data and the inertial measurement unit's measurement data; Step 020: Based on the visual image stream data and measurement data, perform time synchronization evaluation to obtain time synchronization evaluation indicators; Step 030: Based on the camera's visual image stream data and the inertial measurement unit's measurement data, perform an external parameter consistency evaluation to obtain the external parameter consistency evaluation index; Step 040: Determine the comprehensive evaluation index based on the time synchronization evaluation index and the external parameter consistency evaluation index; Step 050: Compare the comprehensive evaluation index with the target threshold, and determine the evaluation result of the camera and inertial measurement unit alignment based on the comparison result.

[0034] Specifically, calculating the time synchronization metric quantifies the timestamp deviation between the visual image stream and the IMU data, while extrinsic parameter consistency assessment quantifies the spatial transformation relationship between the camera and the IMU. The comprehensive evaluation metric normalizes both the time synchronization and extrinsic parameter consistency metrics, replacing external references with relevant parameters from these metrics. This allows for alignment quality assessment without requiring motion capture or laser mapping of the scene. Furthermore, the comprehensive evaluation metric avoids the failure of a single metric, improving the reliability of the evaluation method.

[0035] Thus, this application embodiment establishes a unified quantitative standard through a multi-index fusion scoring system to achieve quantitative and comparable camera-inertial measurement unit alignment evaluation. Therefore, even in scenarios where motion capture or laser maps are unavailable, alignment quality scores can be directly obtained, supporting comparisons of different algorithms or parameter configurations and significantly reducing testing costs.

[0036] Figure 2 This is a schematic diagram illustrating the process of dynamically updating the data stream using a sliding window caching module provided in an embodiment of this application. Figure 2 As shown, in some embodiments, step 010 above includes: Step 011: Obtain the camera's visual image stream data and the inertial measurement unit's measurement data by subscribing online or importing the preset configuration file offline; Step 012: Based on the sliding window caching module, dynamically update the visual image stream data and measurement data acquired at the current moment according to the predetermined duration.

[0037] Specifically, the input interface for acquiring data streams supports both online and offline modes. In online mode, the ROS2 framework is plug-and-play, allowing for the construction of modular evaluation nodes that can be directly deployed within the system. It can directly connect to existing robots and autonomous driving systems for real-time data stream analysis, and flexibly configure and obtain results through ROS parameters and topic interfaces. In offline mode, real-time data streams can be simulated by parsing CSV files (timestamps, image paths, and raw IMU data). The sliding window caching module can save the most recent N seconds (configurable) of image and IMU data, ensuring the timeliness of the analysis data. It can dynamically output time synchronization and extrinsic parameter quality scores online, making it suitable for long-duration tasks.

[0038] Thus, this application embodiment introduces a real-time sliding window analysis framework, embedding time synchronization and external parameter consistency evaluation into the sliding window mechanism, calculating and updating indicators in real time, realizing online quality monitoring and dynamic diagnosis, so that time synchronization and external parameter quality can be dynamically tracked during system operation, and time offset changes or external parameter inaccuracies can be detected in real time during long-term tasks such as unmanned vehicles and robots, thereby improving system stability and maintenance efficiency.

[0039] Figure 3 This is a schematic diagram of a time synchronization evaluation process provided for an embodiment of this application. Figure 3 As shown, in some embodiments, step 020 above includes: Step 0211: Determine the rotation difference between consecutive camera frames based on the visual image stream data; Step 0212: Convert the rotation difference of consecutive camera frames into visual angular velocity; Step 0213: Based on frequency domain coherence analysis, analyze the gyroscope data in the visual angular velocity and visual image stream data to determine the synchronization confidence. Step 0214: Determine the frequency domain coherence parameters in the time synchronization evaluation index based on the synchronization confidence level.

[0040] Specifically, the logarithmic mapping is obtained from the relative rotation matrices of adjacent camera frames. A frequency domain coherence analysis was performed on the IMU gyroscope measurement ωimu to reflect the linear correlation between the two signals in the frequency domain. The synchronization confidence level was obtained by averaging the linear correlation between the two signals within the effective frequency band of the IMU.

[0041] Synchronization confidence can be used as a cross-correlation peak parameter to quantify time synchronization evaluation indicators. The higher the synchronization confidence, the better the time synchronization between visual image stream data and measurement data, and the higher the time synchronization evaluation indicator.

[0042] Thus, the synchronization confidence obtained through cross-correlation analysis can quantify the time synchronization evaluation index.

[0043] Figure 4 Another flowchart illustrating the time synchronization evaluation provided for embodiments of this application is shown. Figure 4 As shown, in some embodiments, step 020 above further includes: Step 0221: Determine the rotation difference between consecutive camera frames based on the visual image stream data; Step 0222: Convert the rotation difference of consecutive camera frames into visual angular velocity; Step 0223: Cross-correlate the visual angular velocity and the gyroscope data in the visual image stream data to extract the peak position and the peak energy corresponding to the peak position; Step 0224: Determine the cross-correlation peak parameter in the time synchronization evaluation index based on the peak energy corresponding to the peak position.

[0044] Specifically, the logarithmic mapping is obtained from the relative rotation matrices of adjacent camera frames. A sliding cross-correlation algorithm is performed between the IMU gyroscope measurement ωimu and the data, resulting in CCF(τ) = ∫ωvis(t)ωimu(t+τ)dt. The maximum peak position Δt^=argmaxτCCF(τ) and the peak energy Epeak=max(CCF) are then found. The peak energy can be used as a cross-correlation peak parameter to quantify the time synchronization evaluation index. A larger peak energy indicates better time synchronization between the visual image stream data and the measurement data, and a higher time synchronization evaluation index.

[0045] Thus, the peak energy obtained through cross-correlation analysis can be used to quantify the time synchronization evaluation index.

[0046] Traditional time synchronization assessments rely on hardware triggers, external clocks, or ground truth trajectories; they cannot be quantified when these conditions are missing.

[0047] In the implementation of step 020 above, the time offset and synchronization confidence are automatically calculated through cross-correlation analysis of visual angular velocity and IMU gyroscope, plus maximum coherence (MSC). Therefore, even without hardware synchronization or external time reference, synchronization deviation can be quickly detected, solving the common problem of missing time reference in existing engineering deployments.

[0048] Figure 5 This is a schematic diagram of a process for evaluating the consistency of external parameters provided in an embodiment of this application. Figure 5 As shown, in some embodiments, step 030 above includes: Step 0311: Based on the synchronization time offset between the camera and the inertial measurement unit and the extrinsic parameters of the inertial measurement unit, pre-integrate the measurement data to obtain the rotational residual, velocity residual, position residual and covariance; Step 0312: Determine the pre-integral residual parameters in the external parameter consistency evaluation index based on the distribution mean of the rotation residual, the distribution mean of the velocity residual, the distribution mean of the position residual, and the distribution mean of the covariance.

[0049] Specifically, there is a time offset Δt between IMU measurement and visual image stream data acquisition. Direct integration would lead to repeated calculations of gravity and initial state. Therefore, pre-integration of the measurement data can be performed using a Lie group pre-integration model. By integrating the IMU data using the current external participation time offset, rotation pre-integration, velocity pre-integration, and position pre-integration can be obtained.

[0050] The logarithmic difference between the relative rotation and rotation pre-integration of IMU data is the rotation residual rR, which reflects the rotation inconsistency between visual and IMU estimates. The logarithmic difference between the relative velocity and velocity pre-integration of IMU data is the velocity residual rv, which reflects the velocity inconsistency between visual and IMU estimates. The logarithmic difference between the relative position and position pre-integration of IMU data is the position residual rp, which reflects the position inconsistency between visual and IMU estimates. Covariance can normalize the rotation residual rR, velocity residual rv, and position residual rp, avoiding excessive dependence on a single residual.

[0051] In this way, multiple sets of data can be collected to calculate the rotational residuals, velocity residuals, position residuals, and covariance. The mean distributions of the rotational residuals, velocity residuals, position residuals, and covariance corresponding to these multiple sets of data can then be calculated, quantifying the external parameter consistency evaluation index. The greater the deviation of the residual distribution from the ideal zero mean, the smaller the external parameter consistency evaluation index.

[0052] Figure 6 This is another schematic diagram illustrating the external parameter consistency evaluation process provided in an embodiment of this application. For example... Figure 6 As shown, in some embodiments, step 030 above further includes: Step 0321: Determine the direction of gravity based on the camera's visual image stream data and the measurement data from the inertial measurement unit; Step 0322: Compare the gravity direction with the calibration direction to determine the gravity direction consistency parameter in the external parameter consistency evaluation index.

[0053] Specifically, the attitude matrix in the world coordinate system obtained after vision-IMU fusion contains implicit gravity vector information, which is then transformed into the gravity direction through the rotation matrix of the world coordinate system.

[0054] The calibrated gravity direction can be [0,0,−1] or the gravity direction configured by other calibration equipment. Calculate the angle between the gravity direction and the calibration direction to determine the gravity direction consistency parameter. The larger the angle between the gravity direction and the calibration direction, the smaller the gravity direction consistency parameter, the smaller the external parameter consistency evaluation index, and the worse the external parameter consistency.

[0055] Thus, by calculating the angle between the direction of gravity and the calibration direction, the greater the angle, the more accurate the assessment of the consistency of the direction of gravity can be.

[0056] In some embodiments, step 030 above further includes: Step 0331: Based on the visual image stream data, determine the tracked feature points for each frame of the image; Step 0332: Reproject the tracked feature points of each frame image to determine the mean distribution of pixel error and mean square error; Step 0333: Determine the reprojection parameter in the external parameter consistency evaluation index based on the mean distribution of pixel error and mean square error.

[0057] Specifically, stable feature points (such as ORB, SIFT, etc.) are extracted and tracked from the visual image stream for each frame, forming a matching point set between consecutive frames. The tracked 3D feature points are projected onto the 2D image plane according to the current camera pose to obtain the theoretical pixel coordinates. The pixel-level Euclidean distance between the theoretical projected points and the actual observed points is calculated. The error distribution of all feature points is statistically analyzed, and the mean and mean square error of the pixel errors are calculated as quantitative indicators of the central tendency of the distribution.

[0058] Thus, based on the distribution characteristics of the pixel error mean and mean square error (such as low mean + low variance), the reprojection parameters in the extrinsic consistency index can be generated, and online evaluation of extrinsic consistency can be achieved through dynamic reprojection error analysis.

[0059] In the implementation of step 030 above, direct and quantitative extrinsic parameter evaluation is provided through pre-integral residual decomposition (rotation / velocity / position), gravity direction consistency angle statistics, and pixel error mean and mean square error distribution characteristics. Therefore, extrinsic parameter rotation drift or instability can be detected earlier, avoiding misuse that leads to a decrease in overall positioning accuracy.

[0060] In some implementations, step 40 includes: weighting and summing at least one parameter in the time synchronization evaluation index and at least one parameter in the external parameter consistency evaluation index to determine a comprehensive evaluation index.

[0061] Specifically, the selection of indicators for constructing a comprehensive evaluation system using a multi-indicator fusion mechanism includes one or more parameters such as time synchronization indicators and external parameter consistency indicators.

[0062] The parameters for the time synchronization index include the selected time offset (through cross-correlation) and synchronization confidence (such as signal coherence MSC). The parameters for the extrinsic consistency index include the gravity angle (gravity alignment error), the mean reprojection error (spatial pose deviation), and the variance of the pre-integrated residual (motion continuity). The parameters are linearly weighted according to the configured weights wi to form the total score. As a comprehensive evaluation indicator.

[0063] In some implementations, step 50 above can determine the evaluation result of the camera and inertial measurement unit alignment by setting a three-level threshold: if Statal ≥ 0.85 → qualified (alignment accuracy meets real-time fusion requirements), 0.6 ≤ Statal < 0.85 → warning (calibration parameters need to be monitored or optimized), Statal < 0.6 → unqualified (external parameters or time synchronization failure, triggering recalibration).

[0064] In some implementations, the evaluation method for camera-inertial measurement unit alignment further includes: It provides a visual interface that can be configured to generate formatted reports; The visualization interface displays real-time updated visual image stream data and measurement data, time synchronization evaluation indicators, external parameter consistency evaluation indicators, comprehensive evaluation indicators, and evaluation results of camera and inertial measurement unit alignment.

[0065] Specifically, this application provides a unified automated report generation (JSON metrics + charts), which outputs cross-correlation curves, residual distributions, and score summaries with a single click, compatible with both offline and online scenarios. This significantly simplifies the engineering debugging process, reduces manual analysis time, and improves efficiency and reusability.

[0066] Figure 7 A schematic diagram of the structure of an evaluation system 100 for aligning a camera with an inertial measurement unit. (See diagram below.) Figure 7 As shown, the evaluation system 100 for aligning a camera with an inertial measurement unit includes: The data acquisition module 110 is configured to acquire visual image stream data from the camera and measurement data from the inertial measurement unit; The time synchronization evaluation module 120 is configured to perform time synchronization evaluation based on visual image stream data and measurement data to obtain time synchronization evaluation indicators. The external parameter consistency evaluation module 130 is configured to perform external parameter consistency evaluation based on the camera's visual image stream data and the inertial measurement unit's measurement data to obtain external parameter consistency evaluation indicators. The comprehensive evaluation module 140 is configured to determine a comprehensive evaluation index based on a time synchronization evaluation index and an external parameter consistency evaluation index. The comprehensive evaluation module 140 is also configured to compare the comprehensive evaluation index with a target threshold and determine the evaluation result of the camera and inertial measurement unit alignment based on the comparison result.

[0067] The camera-inertial measurement unit (IMU) alignment evaluation system provided in this application embodiment also includes a computer device. Exemplarily, the camera-inertial measurement unit alignment evaluation method of this application embodiment can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, personal computer (PC), smart vehicle terminal, etc. The terminal can also include a client, which can be a video client, shopping application client, reading application client, browser client, or instant messaging client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. However, it is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.

[0068] Figure 8 A schematic diagram illustrating the workflow of an evaluation system 100 for aligning a camera with an inertial measurement unit. (See diagram below.) Figure 8 As shown, the computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments, which will not be repeated here.

[0069] In some embodiments, the processor may invoke software programs and modules stored in memory to execute the aforementioned evaluation method for aligning the camera with the inertial measurement unit.

[0070] In some embodiments, the computer device may be integrated into a terminal or server that has storage and a processor and thus computing power, or the computer device may be the terminal or server.

[0071] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0072] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0074] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0076] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0079] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0080] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of evaluating alignment of a camera and an inertial measurement unit, the method comprising: include: Acquire the visual image stream data from the camera and the measurement data from the inertial measurement unit; Based on the visual image stream data and the measurement data, a time synchronization evaluation is performed to obtain a time synchronization evaluation index. Based on the visual image stream data from the camera and the measurement data from the inertial measurement unit, an external parameter consistency evaluation is performed to obtain an external parameter consistency evaluation index. Based on the time synchronization evaluation index and the external parameter consistency evaluation index, a comprehensive evaluation index is determined; The comprehensive evaluation index is compared with the target threshold, and the evaluation result of the camera and inertial measurement unit alignment is determined based on the comparison result.

2. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The acquisition of the visual image stream data from the camera and the measurement data from the inertial measurement unit includes: The camera's visual image stream data and the inertial measurement unit's measurement data can be obtained by subscribing online or importing preset configuration files offline. Based on the sliding window caching module, the visual image stream data and the measurement data acquired at the current moment are dynamically updated according to a predetermined duration.

3. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The step of performing time synchronization evaluation based on the visual image stream data and the measurement data to obtain time synchronization evaluation indicators includes: Based on the visual image stream data, determine the camera's consecutive frame rotation difference; Convert the rotation difference of consecutive camera frames into visual angular velocity; Based on frequency domain coherence analysis, the visual angular velocity and the gyroscope data in the visual image stream data are analyzed to determine the synchronization confidence. Based on the synchronization confidence level, determine the frequency domain coherence parameters in the time synchronization evaluation index.

4. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The step of performing time synchronization evaluation based on the visual image stream data and the measurement data to obtain time synchronization evaluation indicators includes: Based on the visual image stream data, determine the camera's consecutive frame rotation difference; Convert the rotation difference of consecutive camera frames into visual angular velocity; Cross-correlation is performed between the visual angular velocity and the gyroscope data in the visual image stream data to extract the peak position and the peak energy corresponding to the peak position. Based on the peak energy corresponding to the peak position, the cross-correlation peak parameter in the time synchronization evaluation index is determined.

5. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The step of performing an external parameter consistency evaluation based on the visual image stream data from the camera and the measurement data from the inertial measurement unit to obtain an external parameter consistency evaluation index includes: Based on the synchronization time offset between the camera and the inertial measurement unit and the extrinsic parameters of the inertial measurement unit, the measurement data is pre-integrated to obtain the rotational residual, velocity residual, position residual and covariance; Based on the mean distribution of the rotational residual, the mean distribution of the velocity residual, the mean distribution of the position residual, and the mean distribution of the covariance, the pre-integral residual parameter in the external parameter consistency evaluation index is determined.

6. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The step of performing an external parameter consistency evaluation based on the visual image stream data from the camera and the measurement data from the inertial measurement unit to obtain an external parameter consistency evaluation index includes: The direction of gravity is determined based on the visual image stream data from the camera and the measurement data from the inertial measurement unit. The gravity direction and the calibration direction are compared to determine the gravity direction consistency parameter in the external parameter consistency evaluation index.

7. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The step of performing an external parameter consistency evaluation based on the visual image stream data from the camera and the measurement data from the inertial measurement unit to obtain an external parameter consistency evaluation index includes: Based on the visual image stream data, the feature points tracked in each frame of the image are determined; Reproject the tracked feature points of each frame image to determine the mean distribution of pixel error and mean square error; The extrinsic parameter consistency evaluation index is determined based on the mean distribution of the pixel error and the mean square error.

8. The evaluation method for camera and inertial measurement unit alignment as described in claim 1, characterized in that, The determination of the comprehensive evaluation index based on the time synchronization evaluation index and the external parameter consistency evaluation index includes: The comprehensive evaluation index is determined by weighted summation of at least one parameter in the time synchronization evaluation index and at least one parameter in the external parameter consistency evaluation index.

9. The evaluation method for camera and inertial measurement unit alignment as described in any one of claims 1-8, characterized in that, The evaluation method for aligning the camera with the inertial measurement unit also includes: Provides a visualization interface that can be configured to generate formatted reports; The visualization interface displays the real-time updated visual image stream data and measurement data, the time synchronization evaluation index, the external parameter consistency evaluation index, the comprehensive evaluation index, and the evaluation results of the camera and inertial measurement unit alignment.

10. An evaluation system for aligning a camera with an inertial measurement unit, characterized in that, include: The data acquisition module is configured to acquire visual image stream data from the camera and measurement data from the inertial measurement unit; The time synchronization evaluation module is configured to perform time synchronization evaluation based on the visual image stream data and the measurement data to obtain a time synchronization evaluation index. The extrinsic parameter consistency evaluation module is configured to perform extrinsic parameter consistency evaluation based on the visual image stream data of the camera and the measurement data of the inertial measurement unit, so as to obtain extrinsic parameter consistency evaluation index. The comprehensive evaluation module is configured to determine a comprehensive evaluation index based on the time synchronization evaluation index and the external parameter consistency evaluation index. The comprehensive evaluation module is also configured to compare the comprehensive evaluation index with a target threshold and determine the evaluation result of the camera and inertial measurement unit alignment based on the comparison result.

Images