VR / AR SLAM system online self-calibration method and device based on condition triggering and medium

Abstract

The invention relates to a VR / AR SLAM system on-line self-calibration method and device based on condition triggering and a medium, the running state of an SLAM system is monitored, when a self-calibration triggering strategy is met, a self-calibration condition monitoring process is entered, and whether sensor data at the current moment meet a preset self-calibration execution condition or not is evaluated in real time; if yes, on-line joint optimization is carried out on system parameters based on visual observation data and inertial measurement unit data in the sliding window; the invention discloses a method-based implementation device and a medium. According to the method, the safety and success rate of the self-calibration process are remarkably improved, precision improvement and system stability are considered, jamming or delay of a positioning thread is avoided, smooth operation on consumer-level computing power equipment is ensured, the environmental adaptability and robustness of the system are greatly improved, a progressive fine adjustment strategy is adopted, and the self-calibration precision is improved. Smoothness of parameter updating and long-term stability of the system are ensured, core functions are not interfered while precision is improved, and user experience is ensured.

Description

Technical Field

[0001] This invention relates to the technical field of general image data processing or generation, and in particular to a condition-triggered online self-calibration method, apparatus and medium for VR / AR SLAM systems. Background Technology

[0002] With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), robotics, and drones, Simultaneous Localization and Mapping (SLAM) systems based on vision or vision-inertial fusion have become the core foundation for achieving high-precision spatial perception and interaction. The positioning accuracy and robustness of SLAM systems fundamentally depend on the accurate calibration of the internal and inter-sensor geometric relationships, namely, camera intrinsic parameters, inter-camera extrinsic parameters, and IMU-camera extrinsic parameters.

[0003] In existing technologies, especially for consumer and industrial mobile devices, mainstream calibration schemes generally follow a "factory calibration - static use" model. Before leaving the factory, the device typically acquires a set of calibration parameters in a controlled laboratory environment using offline calibration algorithms. These parameters are pre-installed in the device firmware and used by the SLAM algorithm throughout its lifecycle. In subsequent operation, this set of calibration parameters is assumed to be fixed and absolutely accurate, and all calculations based on visual geometry and sensor fusion are grounded in this set.

[0004] However, the above model has significant problems in real-world long-term application scenarios, including:

[0005] (1) Slight vibrations and collisions during transportation and daily use of the equipment, as well as changes in ambient temperature and humidity, may cause microscopic deformation of the camera lens module or IMU installation position, which in turn leads to a mismatch between the preset calibration parameters and the actual physical relationship. This drift is usually slow and cumulative, and will directly introduce system errors, manifested as drift of positioning trajectory, map scale distortion or misalignment of augmented reality content.

[0006] (2) Factory calibration is usually completed in ideal, textured scenes, but the actual operating environment of the device varies greatly, including weak textures, dynamic objects, and drastic changes in lighting. When the sensor is running in a degraded environment, the performance of the algorithm based on fixed parameters may decrease, but the system itself has no ability to adaptively adjust the parameters.

[0007] (3) Most of them do not have the ability to automatically detect calibration errors and re-optimize parameters during equipment operation. Once the parameters are inaccurate, users often have to resort to complex manual recalibration, which greatly consumes the user experience. The threshold is too high for ordinary consumers and it is difficult to perform this frequently during long-term use of the equipment.

[0008] (4) The lack of rigorous triggering and constraint mechanisms during online optimization makes it dangerous and unstable to perform online optimization at any time, which can easily lead to pathological optimization problems, incorrect parameter estimation, or even cause the SLAM system to crash. Summary of the Invention

[0009] This invention solves the problems existing in the prior art and provides a condition-triggered online self-calibration method, device and medium for VR / AR SLAM systems.

[0010] The technical solution adopted in this invention is an online self-calibration method for VR / AR SLAM systems based on condition triggering. This method monitors the operating status of the SLAM system and, when the self-calibration triggering strategy is met, enters the self-calibration condition monitoring process. It evaluates in real time whether the sensor data at the current moment meets the preset self-calibration execution conditions. If the conditions are met, the system parameters are jointly optimized online based on the visual observation data and inertial measurement unit data within the sliding window.

[0011] Preferably, the self-calibration triggering strategy includes the SLAM system's cumulative running time exceeding a first threshold or the SLAM system's reprojection positioning error exceeding a second threshold within a consecutive number of time windows.

[0012] Preferably, the self-calibration condition monitoring process includes motion detection, feature point quality detection, and environmental staticity detection;

[0013] The self-calibration execution conditions include motion sufficiency conditions, characteristic point quality conditions, and environmental static conditions.

[0014] Preferably, the motion sufficiency condition is that within the sliding window, the rotation increment and translation increment of the camera pose corresponding to multiple consecutive frames of images both exceed the corresponding motion threshold, and the distribution of the translation vector satisfies the diversity requirement.

[0015] The feature point quality condition is that the number of tracked feature points in the image exceeds a number threshold, and the uniformity of the distribution of the feature points in the image exceeds a distribution threshold.

[0016] The environmental static condition is that the proportion of static feature points determined based on the semantic segmentation results of the image exceeds the static threshold, and the average reprojection error within the current sliding window is lower than the error threshold.

[0017] Preferably, the evaluation of the environmental staticity condition further includes inputting keyframe features or map structure within the sliding window into the scene understanding model to obtain a scene change score; if the scene change score is higher than the change threshold, it is determined that the environmental staticity condition for triggering self-calibration is met.

[0018] Preferably, the online joint optimization employs a sliding window-based visual-inertial beam adjustment, which optimizes parameters, camera pose, and 3D coordinates of feature points by minimizing visual reprojection error and IMU measurement residuals.

[0019] Preferably, when the number of cached data frames reaches a predetermined calculation trigger threshold, a portion of the data is pre-calculated to obtain intermediate results; the intermediate results are saved, and the cached data that has been calculated is released.

[0020] A condition-triggered online self-calibration device for a VR / AR SLAM system includes:

[0021] The precondition monitoring module is used to monitor the operating status of the SLAM system and enter the selfcalibration condition monitoring process when the selfcalibration trigger strategy is met. In the process, the sensor data at the current moment is evaluated in real time to see if the preset selfcalibration execution conditions are met.

[0022] The online self-calibration module is used to perform online joint optimization of system parameters based on data within a sliding window when the self-calibration execution conditions are met.

[0023] Preferably, it also includes a data buffer and sliding window module, which is used to maintain image feature data, IMU data and camera pose data in the sliding window, and to perform data calculation in advance and release the cache of calculated data when the number of data caches reaches a predetermined calculation trigger threshold.

[0024] A computer-readable storage medium for storing program code for executing the condition-triggered online self-calibration method for a VR / AR SLAM system.

[0025] This invention relates to a condition-triggered online self-calibration method, apparatus, and medium for a VR / AR SLAM system. The method monitors the operating status of the SLAM system. When a self-calibration trigger strategy is met, a self-calibration condition monitoring process is initiated, and the sensor data at the current moment is evaluated in real time to determine whether the preset self-calibration execution conditions are met. If met, the system parameters are jointly optimized online based on visual observation data and inertial measurement unit data within a sliding window. The method is implemented using an apparatus and medium.

[0026] The beneficial effects of this invention are as follows:

[0027] (1) Constructing a precondition monitoring system can accurately determine the timing and safety conditions for calibration, and trigger it only when the data is reliable and the constraints are sufficient. This fundamentally prevents parameter divergence or system crash caused by blind optimization in degraded scenarios, significantly improves the safety and success rate of the self-calibration process, and balances accuracy improvement and system stability.

[0028] (2) Not only does it maintain the sliding window data, but it also introduces a pre-calculation strategy. When the final calibration is triggered, there is no need to perform large-scale centralized calculations. The optimization calculation is evenly distributed on the timeline, smoothing the peak computing power demand of the system, avoiding stuttering or delay of the positioning thread, and ensuring smooth operation on consumer-grade computing power devices.

[0029] (3) By integrating deep learning semantic understanding, the environmental applicability and judgment intelligence level are improved, and the calibrable and non-calibrable environments can be more accurately identified, making the self-calibration trigger decision closer to the logic of the complex real world, and greatly improving the environmental adaptability and robustness of the system.

[0030] (4) Adopt a progressive fine-tuning strategy to ensure the smoothness of parameter updates and the long-term stability of the system, effectively track and compensate for the slow drift of sensor parameters, and prevent the positioning trajectory from jumping due to excessive single optimization, so as to maintain the consistency and stability of user experience in the process of continuously improving long-term accuracy.

[0031] (5) Adopting a modular and decoupled system architecture, the self-calibration and SLAM main process are efficiently coordinated, so that the resource-intensive calibration calculation does not affect the key real-time positioning and mapping threads, improving accuracy without interfering with core functions and ensuring user experience. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating a specific implementation of the present invention;

[0034] Figure 3 This is a schematic block diagram of the device structure of the present invention, wherein the dashed lines indicate intermittent triggering. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0036] This invention relates to an online self-calibration method for a condition-triggered VR / AR SLAM system, the method comprising the following steps:

[0037] (1) Monitor the operating status of the SLAM system. When the self-calibration trigger strategy is met, enter the self-calibration condition monitoring process.

[0038] (2) Real-time evaluation of whether the sensor data at the current moment meets the preset self-calibration execution conditions;

[0039] (3) If satisfied, the system parameters are jointly optimized online based on the visual observation data and inertial measurement unit data within the sliding window.

[0040] The method will be described in detail below with reference to the embodiments.

[0041] (1) Monitor the operating status of the SLAM system. When the self-calibration trigger strategy is met, enter the self-calibration condition monitoring process.

[0042] The self-calibration triggering strategy includes situations where the cumulative running time of the SLAM system exceeds a first threshold, such as when the equipment's manufacturing time exceeds M months.

[0043] Or the reprojection positioning error of the SLAM system exceeds the second threshold within a consecutive number of time windows.

[0044]

[0045] in, This represents the total number of observation points within the current frame or sliding window. Let be the two-dimensional pixel coordinates of the i-th feature point in the current frame image. Let be the theoretical pixel coordinates of the i-th feature point;

[0046] Once self-calibration is triggered, self-calibration condition monitoring will be enabled.

[0047] (2) Real-time evaluation of whether the sensor data at the current moment meets the preset self-calibration execution conditions;

[0048] The self-calibration condition monitoring process includes motion detection, feature point quality detection, and environmental staticity detection.

[0049] The self-calibration execution conditions include motion sufficiency conditions, characteristic point quality conditions, and environmental static conditions.

[0050] (2-1) Exercise detection and conditions for adequate exercise

[0051] The motion sufficiency condition is that, within the sliding window, the rotation increment and translation increment of the camera pose corresponding to multiple consecutive frames of images both exceed the corresponding motion threshold, and the distribution of the translation vector satisfies the diversity requirement.

[0052] Specifically, including:

[0053] (2-1-1) Rotation angle

[0054] The pose output by the SLAM system is statistically analyzed, the rotation increment is calculated, and the rotation angle between multiple consecutive frames of data is calculated. If the rotation angle of multiple consecutive frames exceeds the threshold, it is stored in the cache module for later use.

[0055] (2-1-2) Translation distance

[0056] The pose output by the SLAM system is statistically analyzed, the translation increment is calculated, and the translation distance between multiple consecutive frames of data is calculated. If the translation of multiple consecutive frames exceeds the threshold, the data is stored in the cache module for later use.

[0057] (2-1-3) Analysis of Motion Diversity

[0058] Principal Component Analysis (PCA) is used to analyze the translation vector distribution of the most recent N frames. If the variance of the eigenvector direction is too small, it belongs to the degenerate motion of pure translation and does not meet the condition. In this case, the data is not stored in the cache.

[0059] (2-2) Feature point quality detection and feature point quality conditions

[0060] The feature point quality condition is that the number of tracked feature points in the image exceeds a number threshold, and the uniformity of the distribution of the feature points in the image exceeds a distribution threshold.

[0061] Specifically, including:

[0062] (2-2-1) Number of 2D feature points

[0063] Count the number of effectively tracked 2D feature points in each frame; the number must meet the required threshold.

[0064] (2-2-2) Uniformity of Characteristic Distribution

[0065] Divide the image into a grid and calculate the variance of the feature points. If the feature distribution is too concentrated, the condition is not met.

[0066] (2-2-3) Tracking stability

[0067] The statistical feature point lifecycle (number of frames) and matching repetition rate need to reach the required threshold.

[0068] (2-3) Environmental staticity testing and environmental staticity conditions

[0069] The environmental staticity condition is that the proportion of static feature points determined based on the semantic segmentation results of the image exceeds the static threshold. This means identifying dynamic objects, key static structures, and texture-rich areas in the scene to improve the accuracy of trigger judgment. Semantic segmentation networks, such as SegFormer and Mask R-CNN, are used to perform pixel-level classification on each frame of the image. Feature points corresponding to dynamic categories (such as people, animals, and vehicles) are marked as "dynamic features" to calculate the proportion of dynamic features. The weight of static structures (walls, desktops, pillars, etc.) is increased and they are given priority for BA optimization. If the proportion of dynamic features is too high or the number of static features is insufficient, the condition is not met.

[0070] Furthermore, the average reprojection error within the current sliding window is lower than the error threshold. Specifically, the statistical reprojection error... If the reprojection error is less than a certain threshold, it indicates that the tracking is stable and the environment is static.

[0071] Furthermore, the evaluation of environmental staticity conditions also includes inputting keyframe features or map structures within a sliding window into the scene understanding model to obtain a scene change score. If the scene change score is higher than a change threshold, it is determined that the environmental staticity condition for triggering self-calibration is met. Specifically, a large model is used to perform a higher-level understanding of scene changes, such as judging changes in scene structure, lighting, and abnormal motion patterns. Keyframe features or SLAM map structures are input into a lightweight visual Transformer or a multimodal LLM to output a Scene Change Score (SCS). When the SCS is high enough to reach a preset value, it indicates that the environment has changed significantly, triggering self-calibration; otherwise, it is not triggered.

[0072] (3) If satisfied, the system parameters are jointly optimized online based on the visual observation data and inertial measurement unit data within the sliding window;

[0073] The online self-calibration module is responsible for automatically optimizing the calibration parameters of the camera and IMU when the preconditions are met, so as to ensure the long-term positioning accuracy and stability of the SLAM system.

[0074] The online joint optimization adopts a sliding window-based visual-inertial beam adjustment, which optimizes parameters, camera pose and feature point 3D coordinates by minimizing visual reprojection error and IMU measurement residual.

[0075] Specifically, the system parameters are jointly optimized using visual information and inertial measurement data, including camera intrinsic parameters, extrinsic parameters between cameras, extrinsic parameters between IMU and camera, and time delay. The optimization goal is to reduce visual reprojection error and IMU motion residual, thereby improving positioning accuracy and system stability.

[0076] To ensure real-time performance, the module employs a sliding window mechanism, optimizing only the frame data from the most recent period. As the window moves forward, the optimization results are continuously updated, allowing the parameters to become progressively more refined with system use. It also performs incremental adjustments to long-term statistical feature offsets and minor drifts in the stereo baseline to prevent drastic parameter fluctuations from affecting system stability. Furthermore, it uses smoothing strategies (such as exponential weighted averaging) to control the parameter update amplitude, ensuring a stable and reliable optimization process.

[0077] Furthermore, when the number of cached data frames reaches a predetermined calculation trigger threshold, some data is pre-calculated to obtain intermediate results; the intermediate results are saved and the cached data that has been calculated is released. Pre-calculation can avoid centralized calculation of the system, achieve the effect of distributed computing, and reduce system load.

[0078] Specifically, for the data of the most recent N frames, including key point features of image frames, IMU data, camera pose and timestamps, a sliding window is formed for precondition monitoring and online self-calibration calculation. When the number of data cached reaches a certain threshold, such as M frames, M < N, partial calculations are performed on the data within the sliding window, such as local reprojection error, IMU residuals, and local BA. The calculation results and intermediate results, such as locally optimized camera pose, 3D coordinates of feature points, and IMU residuals, are saved. The processed data is released from the cache to avoid centralized large-scale calculations and improve the real-time performance of the system.

[0079] The distributed computing preprocessing of data here allows the system to distribute the computational load evenly over time, rather than processing a large amount of data all at once in a single frame or when triggering self-calibration. This can be combined with multi-threaded or GPU parallel computing to improve efficiency, which is especially beneficial for low-computing-power devices. It can reduce instantaneous CPU / GPU usage and ensure real-time SLAM performance. Under this processing, the calculated frame data can be stored as an intermediate state, which is convenient for the online self-calibration module to use directly without repeated calculations. When the sliding window moves forward, only the necessary key frame and feature point data are retained, and the historical cache is released.

[0080] The present invention also relates to an online self-calibration device for a condition-triggered VR / AR SLAM system, comprising:

[0081] The precondition monitoring module is used to monitor the operating status of the SLAM system and enter the selfcalibration condition monitoring process when the selfcalibration trigger strategy is met. In the process, the sensor data at the current moment is evaluated in real time to see if the preset selfcalibration execution conditions are met.

[0082] The online self-calibration module is used to perform online joint optimization of system parameters based on data within a sliding window when the self-calibration execution conditions are met.

[0083] It also includes a data buffer and sliding window module, which is used to maintain image feature data, IMU data and camera pose data in the sliding window, and to perform data calculation in advance and release the cached data when the number of data caches reaches a predetermined calculation trigger threshold.

[0084] The present invention also relates to a computer-readable storage medium for storing program code for executing the condition-triggered online self-calibration method for VR / AR SLAM systems.

[0085] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0090] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A condition-triggered online self-calibration method for a VR / AR SLAM system, characterized in that: The system monitors the operating status of the SLAM system. When the self-calibration trigger strategy is met, it enters the self-calibration condition monitoring process to evaluate in real time whether the sensor data at the current moment meets the preset self-calibration execution conditions. If it does, the system parameters are jointly optimized online based on the visual observation data and inertial measurement unit data within the sliding window.

2. The online self-calibration method for a VR / AR SLAM system based on condition triggering according to claim 1, characterized in that: The self-calibration triggering strategy includes the SLAM system's cumulative running time exceeding a first threshold or the SLAM system's reprojection positioning error exceeding a second threshold within a consecutive number of time windows.

3. The online self-calibration method for a VR / AR SLAM system based on condition triggering according to claim 1, characterized in that: The self-calibration condition monitoring process includes motion detection, feature point quality detection, and environmental staticity detection. The self-calibration execution conditions include motion sufficiency conditions, characteristic point quality conditions, and environmental static conditions.

4. The online self-calibration method for a VR / AR SLAM system based on condition triggering according to claim 3, characterized in that: The motion sufficiency condition is that, within the sliding window, the rotation increment and translation increment of the camera pose corresponding to multiple consecutive frames of images both exceed the corresponding motion threshold, and the distribution of the translation vector satisfies the diversity requirement. The feature point quality condition is that the number of tracked feature points in the image exceeds a number threshold, and the uniformity of the distribution of the feature points in the image exceeds a distribution threshold. The environmental static condition is that the proportion of static feature points determined based on the semantic segmentation results of the image exceeds the static threshold, and the average reprojection error within the current sliding window is lower than the error threshold.

5. The online self-calibration method for a VR / AR SLAM system based on condition triggering according to claim 4, characterized in that: The evaluation of the environmental staticity condition also includes inputting keyframe features or map structure within the sliding window into the scene understanding model to obtain a scene change score; if the scene change score is higher than the change threshold, it is determined that the environmental staticity condition for triggering self-calibration is met.

6. The online self-calibration method for a VR / AR SLAM system based on condition triggering according to claim 1, characterized in that: The online joint optimization employs a sliding window-based visual-inertial beam adjustment, which optimizes parameters, camera pose, and 3D coordinates of feature points by minimizing visual reprojection error and IMU measurement residuals.

7. The online self-calibration method for a VR / AR SLAM system based on condition triggering according to claim 1, characterized in that: When the number of cached data frames reaches the predetermined calculation trigger threshold, some data is pre-calculated to obtain intermediate results; the intermediate results are saved, and the cached data that has been calculated is released.

8. An online self-calibration device for a condition-triggered VR / AR SLAM system, characterized in that: include: The precondition monitoring module is used to monitor the operating status of the SLAM system and enter the selfcalibration condition monitoring process when the selfcalibration trigger strategy is met. In the process, the sensor data at the current moment is evaluated in real time to see if the preset selfcalibration execution conditions are met. The online self-calibration module is used to perform online joint optimization of system parameters based on data within a sliding window when the self-calibration execution conditions are met.

9. The online self-calibration device for a condition-triggered VR / AR SLAM system according to claim 8, characterized in that: It also includes a data buffer and sliding window module, which is used to maintain image feature data, IMU data and camera pose data in the sliding window, and to perform data calculation in advance and release the cached data when the number of data caches reaches a predetermined calculation trigger threshold.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the online self-calibration method for a VR / AR SLAM system based on condition triggering as described in any one of claims 1 to 7.

Images