Vision and inertia sensing fusion positioning method facing shielding scene

Abstract

The invention discloses a vision and inertia sensing fusion positioning method for an occlusion scene, and relates to the technical field of computer processing, and the method comprises the following steps: S01, carrying out the feature extraction of an obtained vision image sequence and inertia measurement unit data, and obtaining an image feature and an inertia feature; s02, fusing the image features and the inertial features to obtain a system state vector; s03, performing state estimation on the system state vector through a multi-state constraint Kalman filtering framework, and determining a current pose state of the system; and S04, constructing the multi-state constraint Kalman filtering framework based on a sliding window mechanism, and performing discretization processing on a system state vector in a filtering process. According to the method, the problem of positioning failure of a traditional visual inertial system in a shielding scene is solved through a cooperative mechanism of inertial data confidence evaluation and semantic segmentation static feature enhancement.

Description

Technical Field

[0001] This invention relates to the field of computer processing technology, and in particular to a visual and inertial sensing fusion localization method for occluded scenes. Background Technology

[0002] In the field of visual-inertial fusion positioning technology, existing methods mainly revolve around the complementarity of information from multiple sensors to overcome the inherent limitations of a single sensor. Visual sensors can provide rich environmental feature information, but they are prone to feature tracking loss under conditions of drastic changes in lighting, rapid movement, or scene occlusion. While inertial measurement units (IMUs) can provide high-frequency motion measurements unaffected by the environment, their inherent integral drift problem causes errors to accumulate and diverge over time. This complementarity in sensor characteristics has spurred the rapid development of visual-inertial odometry (VIO) technology, resulting in two main technical approaches: filtering-based and optimization-based.

[0003] Filtering-based methods, exemplified by multi-state constrained Kalman filtering, construct geometric constraints by maintaining historical camera poses within a sliding window, achieving efficient state updates without adding feature points to the state vector. However, this method relies on sufficient motion stimuli during system initialization, leading to slow and unstable initialization processes in occluded scenarios with frequent visual interruptions. Optimization-based methods, such as OKVIS, jointly estimate system states and feature positions through nonlinear optimization. While achieving high accuracy, they are computationally intensive, making it difficult to meet the stringent real-time requirements of occluded environments. In recent years, deep learning techniques have been introduced into the field of visual-inertial fusion, reducing reliance on manual parameter tuning through end-to-end learning. However, these methods require extensive training with labeled data, and their generalization ability in real-world occluded scenarios remains insufficient.

[0004] Especially in occluded scenarios, traditional methods face three prominent challenges: the transient disappearance of feature points leads to interruptions in visual measurements, image quality degradation causes difficulties in feature matching, and increased uncertainty in motion models leads to degraded state estimation. Existing improvements include the ROVIO algorithm, which uses a direct method based on image grayscale for tracking. It is more robust to feature loss but sensitive to changes in illumination. There are also multi-sensor fusion frameworks that combine photoelectric and infrared information to improve perception capabilities under low-light conditions. However, at the feature processing level, traditional methods often lack effective evaluation of feature point stability. In occluded environments, a large number of unstable feature points are included in the calculation, which not only increases the computational burden but also leads to a decrease in tracking reliability. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention provides a visual and inertial sensing fusion localization method for occluded scenes, comprising the following steps: S01. Feature extraction is performed on the acquired visual image sequence and inertial measurement unit data to obtain image features and inertial features, wherein: The image features include stable feature points and feature descriptors, and the inertial features include measurements from accelerometers and gyroscopes; S02. The image features and the inertial features are fused to obtain the system state vector; S03. The system state vector is estimated using a multi-state constrained Kalman filter framework to determine the current pose state of the system. S04. The multi-state constrained Kalman filter framework is constructed based on a sliding window mechanism. During the filtering process, the system state vector is first discretized and mapped to multiple time-stamped state nodes. A sliding window of fixed length is constructed based on these state nodes. Multi-view observations of feature points are used as constraints. State updates are performed on the sliding window to obtain the optimal estimate. Specifically, this includes: The tracking length of the corresponding feature point is obtained and its relationship with a preset threshold is determined. When the tracking length of the feature point does not reach the preset threshold, the inertial measurement sequence corresponding to this feature point is retrieved. By statistically analyzing the change amplitude of the maximum and minimum values ​​of the angular velocity data in the sequence, it is determined whether the change amplitude exceeds the threshold corresponding to this type of motion. When the change amplitude is greater than the threshold, it indicates that the current feature point has sufficient motion excitation, and the feature point is retained for state update.

[0006] Preferably, the feature extraction of the acquired visual image sequence and inertial measurement unit data in step S01 includes extracting stable feature points, feature descriptors and quality weights of the image respectively, and merging the stable feature points and the feature descriptors to obtain the image features. The quality weight is used to indicate the reliability level corresponding to the image features.

[0007] Preferably, the fusion of the image features and the inertial features in step S02 includes: filtering the image features and the inertial features based on the ranking of the quality weights, removing the image features and the inertial features with corresponding timestamps that do not meet the ranking requirements, adjusting the weights of the image features and the corresponding inertial features based on the quality weights, and fusing the adjusted image features and inertial features through the state transition equation.

[0008] Preferably, before extracting stable feature points, feature descriptors, and quality weights from the image, each frame in the visual image sequence needs to be preprocessed, including: The content in the image that matches the preset dynamic feature template is removed, and the image is classified into multiple image groups according to the number of feature points. Each image group corresponds to a different feature point density level. Based on the feature point distribution, each image group is further classified into multiple different image subgroups, and the similarity between the feature points in the current image subgroup and the feature templates in the current group is calculated. If the calculated similarity is higher than the predetermined threshold, the feature point is identified as a stable feature point; otherwise, the feature point is removed. Then, the coordinate information of the corresponding feature points is added to the coordinate list of the feature group and the corresponding feature template is updated to form the final feature point sequence; The feature point sequence is represented as a list of coordinates containing multiple feature points.

[0009] Preferably, extracting stable feature points from the image includes: performing stability evaluation on multiple coordinate lists, converting each feature point in each coordinate list sequence into a fixed-length stability vector, and integrating the corresponding stability vectors to convert the coordinate list sequence into a feature stability sequence.

[0010] Preferably, obtaining the quality weight includes: A confidence assessment model for inertial measurement data is established. The reliability of the inertial data is evaluated by calculating the condition number of the variance and covariance matrix of the inertial measurement values ​​within a continuous time window. When a large number of visual features are detected to be missing, the process noise covariance matrix of the filter is dynamically adjusted according to the results of the confidence assessment model. When the confidence level of the inertial data is higher than the first threshold, the original process noise covariance matrix is ​​maintained. When the confidence level of the inertial data is lower than the first threshold but higher than the second threshold, the process noise covariance matrix is ​​increased proportionally; when the confidence level of the inertial data is lower than the second threshold, the system switches to a pure vision-assisted mode and uses only visual features for state updates.

[0011] Preferably, the step of removing content from the image that matches a preset dynamic feature template, and classifying the image into multiple image groups based on the number of feature points, with each image group corresponding to a different feature point density level, includes: A semantic segmentation network is used to perform scene analysis on the input image, identifying dynamic object regions and static background regions; Based on the segmentation results, feature points are extracted from the static background region first during the feature extraction stage, and higher quality weights are assigned to feature points from the static region. Establish a static region feature point database, and create more stable tracking associations across different frames through feature point matching; When a large amount of dynamic occlusion is detected in the scene, the feature extraction strategy is automatically adjusted, and feature extraction resources are concentrated on the static background area.

[0012] Preferably, the multi-state constrained Kalman filter framework is constructed based on a sliding window mechanism. During the filtering process, the system state vector is first discretized, including: Divide the IMU data into time windows and calculate the motion excitation index for each window; Set angular velocity threshold and variance threshold. When the angular velocity is greater than the threshold or the variance is greater than the threshold, the initialization process is triggered. By utilizing the orthogonality of the IMU's three-axis acceleration, the average acceleration of the previous window is normalized to obtain the projection of the gravity direction in the IMU coordinate system. A unit coordinate system is then constructed through Schmidt orthogonalization.

[0013] The present invention has at least the following beneficial effects: by introducing the collaborative work of an adaptive filtering mechanism based on inertial data confidence and a semantic segmentation static region feature enhancement mechanism, the system can intelligently allocate sensor weights in a dynamic occlusion environment with severe visual information loss, give priority to using high-quality static background feature points, and combine the most reliable inertial data for state estimation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a visual and inertial sensing fusion localization method for occluded scenes provided in Embodiment 1 of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] Example 1

[0019] This embodiment provides a visual and inertial sensing fusion localization method for occluded scenes. The method includes the following steps: Figure 1 As shown: S01. Feature extraction is performed on the acquired visual image sequence and inertial measurement unit data to obtain image features and inertial features, wherein: Image features include stable feature points and feature descriptors, while inertial features include measurements from accelerometers and gyroscopes. Furthermore, the above-mentioned feature extraction of the acquired visual image sequence and inertial measurement unit data includes extracting stable feature points, feature descriptors and quality weights of the image respectively, and merging the stable feature points and feature descriptors to obtain image features. The quality weight is used to indicate the reliability of the image features.

[0020] Specifically, by processing visual image sequences and inertial measurement unit (IMU) data in parallel, feature information with different characteristics is extracted. For visual images, the system first extracts stable feature points in the image using feature detection algorithms such as ORB or SIFT. These feature points are mainly distributed in regions with rich texture and obvious corners. Subsequently, a corresponding feature descriptor is generated for each feature point for subsequent feature matching and tracking. To improve the system's robustness in occluded scenarios, the concept of quality weight is introduced. This weight is calculated by comprehensively evaluating multiple factors, including the tracking length of the feature point, pixel discrimination, and the semantic stability of its region (e.g., whether it belongs to a static background). Finally, the system merges stable feature points and feature descriptors to form complete image features, while the quality weight serves as metadata to identify the reliability of each image feature.

[0021] Furthermore, we obtain quality weights, including: A confidence assessment model for inertial measurement data is established. The reliability of the inertial data is evaluated by calculating the condition number of the variance and covariance matrix of the inertial measurement values ​​within a continuous time window. When a large number of visual features are detected to be missing, the process noise covariance matrix of the filter is dynamically adjusted based on the results of the confidence assessment model. When the confidence level of the inertial data is higher than the first threshold, the original process noise covariance matrix is ​​maintained. When the confidence level of the inertial data is lower than the first threshold but higher than the second threshold, the process noise covariance matrix is ​​increased proportionally; when the confidence level of the inertial data is lower than the second threshold, the system switches to a pure vision-assisted mode and uses only visual features for state updates.

[0022] Specifically, by continuously acquiring inertial measurement data through a sliding time window, calculating the variance characteristics of accelerometer and gyroscope measurements within the window, and analyzing the condition number of the covariance matrix, a multi-dimensional confidence assessment model is constructed. This model can keenly identify changes in the reliability of inertial data: when a large number of visual features are missing, the system dynamically adjusts the filtering parameters based on the real-time confidence level, implementing a three-level processing strategy—when the confidence level is above the first threshold, the original process noise covariance matrix is ​​maintained; when it is between the first and second thresholds, the noise covariance is increased proportionally to reduce the weight of inertial data; when it is below the second threshold, it completely switches to a pure vision-assisted mode, relying solely on the retained reliable visual features for state updates. By quantitatively assessing the quality of inertial data and establishing a graded response strategy, the error accumulation problem caused by the over-reliance on inertial data in traditional fusion algorithms during visual degradation is effectively solved. This dynamic and variable trust level management not only improves the system's fault tolerance in complex scenarios such as occlusion, but also achieves precise complementarity between the advantages of vision and inertial sensing, ensuring that optimal positioning accuracy can be maintained even when the performance of any sensor degrades, thereby significantly enhancing the adaptability and reliability of the entire positioning system in real-world environments.

[0023] S02. The image features and inertial features are fused to obtain the system state vector.

[0024] Furthermore, the above-mentioned image features and inertial features are fused, including: sorting the image features and inertial features based on quality weights, removing image features and corresponding timestamp inertial features that do not meet the sorting requirements, adjusting the weights of the image features and corresponding inertial features based on quality weights, and fusing the adjusted image features and inertial features through a state transition equation.

[0025] Specifically, based on pre-calculated quality weights, all image features at the current moment are sorted and filtered, and low-quality feature points with weights below a set threshold and their corresponding timestamps of inertial measurement data are removed. Next, for the filtered feature data, the system performs differentiated processing according to their respective quality weights, adjusting the contribution of different features in state estimation through a weighted method. Finally, these weighted visual observations and inertial measurement values ​​are input together into a predefined state transition equation, and the deep fusion of multi-source sensor data is completed through the prediction-update process of extended Kalman filtering, ultimately outputting an optimized system state vector.

[0026] S03. Estimate the system state vector using a multi-state constrained Kalman filter framework to determine the current pose state of the system. Specifically, the state vector, containing parameters such as position, velocity, attitude, and sensor bias, is input into an extended Kalman filter framework based on multi-state constraints. A set of historical camera pose states is maintained within a sliding window, and geometric constraints are formed using observations of feature points across multiple adjacent frames. When new visual observation data arrives, the system constructs multi-view observation equations using null projection technology and eliminates outlier observations based on Mahalanobis distance. Finally, the predicted state is fused with the observation residuals using the Kalman gain matrix to update the system and obtain the optimal current system pose state estimate. By mining the deep geometric constraints between historical poses and current observations, the accuracy and smoothness of the state estimation are significantly improved. The introduction of multi-state constraints enables the system to effectively utilize the spatiotemporal continuity of visual features. Even in cases of temporary occlusion where some features are temporarily lost, the system's observability can still be maintained through observations in other frames. This solves the tracking loss problem that easily occurs in traditional visual inertial odometry in occluded scenarios, providing a reliable technical guarantee for stable positioning in complex environments.

[0027] S04. The multi-state constrained Kalman filter framework is constructed based on a sliding window mechanism. During the filtering process, the system state vector is first discretized and mapped to multiple time-stamped state nodes. A sliding window of fixed length is constructed based on these state nodes. Multi-view observations of feature points are used as constraints. State updates are performed on the sliding window to obtain the optimal estimate. Specifically, this includes: The tracking length of the corresponding feature point is obtained and its relationship with a preset threshold is determined. When the tracking length of the feature point does not reach the preset threshold, the inertial measurement sequence corresponding to this feature point is retrieved. By statistically analyzing the change amplitude of the maximum and minimum values ​​of the angular velocity data in the sequence, it is determined whether the change amplitude exceeds the threshold corresponding to this type of motion. When the change amplitude is greater than the threshold, it indicates that the current feature point has sufficient motion excitation, and the feature point is retained for state update.

[0028] Furthermore, the multi-state constrained Kalman filter framework in the above embodiments is constructed based on a sliding window mechanism. During the filtering process, the system state vector is first discretized, including: Divide the IMU data into time windows and calculate the motion excitation index for each window; Set angular velocity threshold and variance threshold. When the angular velocity is greater than the threshold or the variance is greater than the threshold, the initialization process is triggered. By utilizing the orthogonality of the IMU's three-axis acceleration, the average acceleration of the previous window is normalized to obtain the projection of the gravity direction in the IMU coordinate system. A unit coordinate system is then constructed through Schmidt orthogonalization.

[0029] Specifically, the continuous-time system state vector is discretized and sampled, mapping it to discrete state nodes corresponding to multiple timestamps. A fixed-length sliding window is then constructed based on these nodes. During state updates, the system acquires the tracking length of each feature point and compares it with a preset threshold. When the tracking length of a feature point does not meet the threshold requirement, the system retrieves the inertial measurement sequence for the corresponding time period. By statistically analyzing the extreme value changes of the angular velocity data in the sequence, it determines whether the changes exceed the standard threshold for that type of motion. If the change exceeds the threshold, it indicates that the feature point has sufficient motion excitation and is retained for state updates; otherwise, it is discarded. Quantitative analysis of motion excitation effectively identifies stable feature points with continuous tracking value, avoiding estimation bias caused by transient or false features. This significantly improves the accuracy of state estimation and the robustness of the system in occluded scenarios while maintaining the computational efficiency of the sliding window.

[0030] Secondly, in the above embodiments, IMU data is divided into preset time windows, and motion excitation indicators, including statistical characteristics such as mean and variance of angular velocity, are calculated for each window. By setting dual judgment criteria of angular velocity and variance thresholds, the system immediately triggers the initialization process when the angular velocity of any window exceeds the set threshold or the angular velocity variance exceeds the variance threshold. During the initialization execution phase, the system fully utilizes the orthogonality of the IMU's three-axis acceleration, normalizes the average acceleration data of the previous window, accurately extracts the projection vector of the gravity direction in the IMU coordinate system, and then uses the Schmitt orthogonalization method to construct a complete unit coordinate system, establishing an accurate initial reference frame for subsequent state estimation. Through dual threshold judgment and windowing processing, fast and reliable system initialization is achieved. Especially in occluded scenarios, this mechanism can keenly capture limited motion excitation opportunities, establishing a stable coordinate system using simple acceleration observations, significantly improving the system's startup success rate and initialization efficiency in harsh environments. This lays a solid foundation for subsequent multi-state constrained Kalman filtering, ensuring that the entire positioning system can quickly enter a stable working state under various complex conditions.

[0031] Example 2

[0032] Based on the above embodiment one, before performing step S02 to extract stable feature points, feature descriptors, and quality weights of the image, it is necessary to preprocess each frame of the visual image sequence, including: The content that matches the preset dynamic feature template in the image is removed, and the image is classified into multiple image groups according to the number of feature points. Each image group corresponds to a different feature point density level. Based on the feature point distribution, each image group is further classified into multiple different image subgroups, and the similarity between the feature points in the current image subgroup and the feature templates in the current group is calculated. If the calculated similarity is higher than the predetermined threshold, the feature point is identified as a stable feature point; otherwise, the feature point is removed. Then, the coordinate information of the corresponding feature points is added to the coordinate list of the feature group and the corresponding feature template is updated to form the final feature point sequence; The feature point sequence is represented as a list of coordinates containing multiple feature points.

[0033] Specifically, the system employs a multi-level filtering mechanism to refine the raw visual data. First, the system matches the input image against a pre-defined dynamic feature template, identifying and removing features related to dynamic objects (such as pedestrians and vehicles) to reduce interference from dynamic occlusions at the source. Next, the system performs preliminary classification of the images based on the number of detected feature points, dividing them into high, medium, and low density groups, each with a corresponding processing strategy. Further, the system subdivides each image group, dividing it into multiple image subgroups based on the spatial distribution of feature points, and calculates the structural similarity between feature points within each subgroup and the current group's feature template. When the similarity between a feature point and the template exceeds a preset threshold, the feature point is identified as a stable feature point and retained; feature points that do not reach the threshold are directly discarded. Finally, the system incorporates the coordinates of the filtered stable feature points into the corresponding feature coordinate list, while dynamically updating the feature template to maintain its timeliness, resulting in an optimized feature point sequence.

[0034] Furthermore, in the above embodiments, extracting stable feature points of an image includes: performing stability evaluation on multiple coordinate lists, converting each feature point in each coordinate list sequence into a fixed-length stability vector, integrating the corresponding stability vectors, and converting the coordinate list sequence into a feature stability sequence.

[0035] Specifically, the system historically tracks the coordinate trajectory of each feature point in a continuous frame sequence, calculating multi-dimensional indicators such as its positional variance, motion consistency, and geometric constraints with other feature points within a sliding time window. Then, based on these indicators, a pre-defined stability evaluation model transforms the complex motion characteristics of each feature point into a fixed-length numerical stability vector, where each dimension of the vector corresponds to a specific stability evaluation element. Finally, the system integrates the stability vectors corresponding to all feature points to construct a complete feature stability sequence, achieving a mapping transformation from raw coordinate data to a stable feature space. By quantifying the spatiotemporal motion characteristics of feature points into comparable stability indicators, a scientific decision-making basis is provided for subsequent feature selection and fusion. This transformation not only effectively identifies and eliminates temporary feature points that are geographically significant but have poor stability, but also ensures that the system prioritizes the most persistent and reliable features for state estimation when encountering occlusion or rapid movement. This significantly improves the robustness and positioning accuracy of the visual-inertial system in complex scenes, solving the problem of unstable feature tracking caused by reliance on raw coordinate data in traditional methods.

[0036] Furthermore, content matching the preset dynamic feature template in the image is removed, and the image is classified into multiple image groups based on the number of feature points. Each image group corresponds to a different feature point density level, including: A semantic segmentation network is used to perform scene analysis on the input image, identifying dynamic object regions and static background regions; Based on the segmentation results, feature points are extracted from the static background region first during the feature extraction stage, and higher quality weights are assigned to feature points from the static region. Establish a static region feature point database, and create more stable tracking associations across different frames through feature point matching; When a large amount of dynamic occlusion is detected in the scene, the feature extraction strategy is automatically adjusted, and feature extraction resources are concentrated on the static background area.

[0037] Specifically, in the image preprocessing stage, the system uses a semantic segmentation network to perform deep scene understanding on the input image, accurately identifying pixel-level boundaries between dynamic object regions and static background regions. Based on the segmentation results, the system intelligently prioritizes collecting feature points from static background regions during feature extraction, assigning higher quality weights to these feature points from static regions. Simultaneously, a dedicated database of static region feature points is established, and persistent and stable tracking associations are built across different frames through feature descriptor matching. When a large amount of dynamic occlusion is detected in the scene, the system automatically adjusts its feature extraction strategy, concentrating limited computing resources on static background regions to ensure sufficient stable feature points are obtained even under dynamic interference environments. This semantic-level scene understanding enables intelligent selection of feature acquisition, effectively avoiding interference from dynamic objects on feature tracking. Furthermore, the established static feature database provides a reliable data foundation for long-term stable visual localization.

[0038] Example 3

[0039] This invention provides a non-transitory computer-readable storage medium storing at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the following steps: Feature extraction is performed on the acquired visual image sequence and inertial measurement unit data to obtain image features and inertial features; The system state vector is obtained by fusing image features with inertial features; The system state vector is estimated using a multi-state constrained Kalman filter framework to determine the current pose state of the system. The multi-state constrained Kalman filter framework is constructed based on the sliding window mechanism. In the filtering process, the system state vector is first discretized and mapped to state nodes with multiple timestamps. A sliding window with a fixed length is constructed based on the state nodes. The state is updated and the optimal estimate is obtained by using the multi-view observation of feature points as constraints on the sliding window.

[0040] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0041] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0042] Example 4

[0043] This invention provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program segment, and the at least one instruction or the at least one program segment is loaded and executed by the processor to implement the following steps: Feature extraction is performed on the acquired visual image sequence and inertial measurement unit data to obtain image features and inertial features; The system state vector is obtained by fusing image features with inertial features; The system state vector is estimated using a multi-state constrained Kalman filter framework to determine the current pose state of the system. The multi-state constrained Kalman filter framework is constructed based on the sliding window mechanism. In the filtering process, the system state vector is first discretized and mapped to state nodes with multiple timestamps. A sliding window with a fixed length is constructed based on the state nodes. The state is updated and the optimal estimate is obtained by using the multi-view observation of feature points as constraints on the sliding window.

[0044] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A visual and inertial sensing fusion localization method for occluded scenes, characterized in that, Includes the following steps: S01. Feature extraction is performed on the acquired visual image sequence and inertial measurement unit data to obtain image features and inertial features, wherein: The image features include stable feature points and feature descriptors, and the inertial features include measurements from accelerometers and gyroscopes; S02. The image features and the inertial features are fused to obtain the system state vector; S03. The system state vector is estimated using a multi-state constrained Kalman filter framework to determine the current pose state of the system. S04. The multi-state constrained Kalman filter framework is constructed based on a sliding window mechanism. During the filtering process, the system state vector is first discretized and mapped to multiple time-stamped state nodes. A sliding window of fixed length is constructed based on these state nodes. Multi-view observations of feature points are used as constraints. State updates are performed on the sliding window to obtain the optimal estimate. Specifically, this includes: The tracking length of the corresponding feature point is obtained and its relationship with a preset threshold is determined. When the tracking length of the feature point does not reach the preset threshold, the inertial measurement sequence corresponding to this feature point is retrieved. By statistically analyzing the change amplitude of the maximum and minimum values ​​of the angular velocity data in the sequence, it is determined whether the change amplitude exceeds the threshold corresponding to this type of motion. When the change amplitude is greater than the threshold, it indicates that the current feature point has sufficient motion excitation, and the feature point is retained for state update.

2. The visual and inertial sensing fusion localization method for occluded scenes according to claim 1, characterized in that, The step S01, which involves extracting features from the acquired visual image sequence and inertial measurement unit data, includes extracting stable feature points, feature descriptors, and quality weights from the image, and merging the stable feature points and feature descriptors to obtain the image features. The quality weights are used to indicate the reliability level corresponding to the image features.

3. The visual and inertial sensing fusion localization method for occluded scenes according to claim 2, characterized in that, The fusion of image features and inertial features in step S02 includes: filtering the image features and inertial features based on the ranking of the quality weights, removing the image features and inertial features with corresponding timestamps that do not meet the ranking requirements, adjusting the weights of the image features and corresponding inertial features based on the quality weights, and fusing the adjusted image features and inertial features through the state transition equation.

4. The visual and inertial sensing fusion localization method for occluded scenes according to claim 2, characterized in that, Before extracting stable feature points, feature descriptors, and quality weights from the image, each frame in the visual image sequence needs to be preprocessed, including: The content in the image that matches the preset dynamic feature template is removed, and the image is classified into multiple image groups according to the number of feature points. Each image group corresponds to a different feature point density level. Based on the feature point distribution, each image group is further classified into multiple different image subgroups, and the similarity between the feature points in the current image subgroup and the feature templates in the current group is calculated. If the calculated similarity is higher than a predetermined threshold, the feature point is identified as a stable feature point; otherwise, the feature point is removed. Then, the coordinate information of the corresponding feature points is added to the coordinate list of the feature group and the corresponding feature template is updated to form the final feature point sequence; The feature point sequence is represented as a list of coordinates containing multiple feature points.

5. The visual and inertial sensing fusion localization method for occluded scenes according to claim 4, characterized in that, Extracting stable feature points from the image includes: performing stability evaluation on multiple coordinate lists, converting each feature point in each coordinate list sequence into a fixed-length stability vector, and integrating the corresponding stability vectors to convert the coordinate list sequence into a feature stability sequence.

6. The visual and inertial sensing fusion localization method for occluded scenes according to claim 2, characterized in that, The quality weights are obtained by: A confidence assessment model for inertial measurement data is established. The reliability of the inertial data is evaluated by calculating the condition number of the variance and covariance matrix of the inertial measurement values ​​within a continuous time window. When a large number of visual features are detected to be missing, the process noise covariance matrix of the filter is dynamically adjusted according to the results of the confidence assessment model. When the confidence level of the inertial data is higher than the first threshold, the original process noise covariance matrix is ​​maintained. When the confidence level of the inertial data is lower than the first threshold but higher than the second threshold, the process noise covariance matrix is ​​increased proportionally; when the confidence level of the inertial data is lower than the second threshold, the system switches to a pure vision-assisted mode and uses only visual features for state updates.

7. The visual and inertial sensing fusion localization method for occluded scenes according to claim 4, characterized in that, The process involves removing content from the image that matches a preset dynamic feature template, and then classifying the image into multiple image groups based on the number of feature points. Each image group corresponds to a different feature point density level, including: A semantic segmentation network is used to analyze the scene of the input image and identify dynamic object regions and static background regions. Based on the segmentation results, feature points are extracted from the static background region first during the feature extraction stage, and higher quality weights are assigned to feature points from the static region. Establish a static region feature point database, and create more stable tracking associations across different frames through feature point matching; When a large amount of dynamic occlusion is detected in the scene, the feature extraction strategy is automatically adjusted, and feature extraction resources are concentrated on the static background area.

8. The visual and inertial sensing fusion localization method for occluded scenes according to claim 1, characterized in that, The multi-state constrained Kalman filter framework is constructed based on a sliding window mechanism. During the filtering process, the system state vector is first discretized, including: Divide the IMU data into time windows and calculate the motion excitation index for each window; Set angular velocity threshold and variance threshold. When the angular velocity is greater than the threshold or the variance is greater than the threshold, the initialization process is triggered. By utilizing the orthogonality of the IMU's three-axis acceleration, the average acceleration of the previous window is normalized to obtain the projection of the gravity direction in the IMU coordinate system. A unit coordinate system is then constructed through Schmidt orthogonalization.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the visual and inertial sensing fusion localization method for occluded scenes as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the visual and inertial sensing fusion localization method for occluded scenes as described in any one of claims 1 to 8.

Images