All-day infrared inertial slam method and device for dynamic environment

Abstract

The embodiment of the present application provides an all-weather infrared inertial SLAM method and device for a dynamic environment, which first determines the motion trajectory of a target unmanned aerial vehicle based on a thermal infrared image sequence of a target scene; secondly, based on any current key frame in a key frame set corresponding to the thermal infrared image sequence: performs IMU pre-integration on the current key frame to generate pose estimation of the target unmanned aerial vehicle; selects visual pose information corresponding to the current time from the motion trajectory; and performs visual-inertial fusion processing on the visual pose information and the pose estimation; then, according to visual re-projection residuals, IMU pre-integration residuals, and marginalization residuals, performs BA optimization processing on the key frame set according to a preset sliding window; finally, performs global closed-loop optimization processing on the optimized key frame set, and outputs a global SLAM map. Thus, accurate and robust positioning of the unmanned aerial vehicle can be realized based on the thermal infrared image, and the accuracy of SLAM map positioning is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to an all-weather infrared inertial SLAM method and apparatus for dynamic environments. Background Technology

[0002] Existing visible light vision-based real-time localization and mapping (SLAM) frameworks are mature in stable environments. However, some environments exhibit extreme differences in light distribution, dynamic changes in illumination, or visual obstructions such as dust, fog, and smoke. This visual degradation always reduces the reliability of SLAM state estimation solutions. Compared to visible light cameras, thermal infrared cameras have all-weather perception capabilities and offer significant advantages in visually degraded scenarios.

[0003] Current thermal-inertial odometry (TIO) solutions are essentially improvements on conventional visual-inertial odometry (VIO) for image feature processing in SLAM state estimation solutions. However, using thermal infrared cameras within existing VIO frameworks presents compatibility issues; for example, images captured by thermal infrared cameras typically have low resolution and contrast; due to their indistinguishability from the surrounding thermal radiation, rich visually observable texture information, such as color and stripes, is lost in thermal images; and thermal infrared cameras require non-uniformity correction or flat-field correction during operation to eliminate imaging effects between consecutive frames caused by data interruptions due to accumulated non-zero mean noise. These problems lead most research to treat thermal infrared camera information as a supplement to other sensor data for fusion.

[0004] Furthermore, most visual SLAM systems operate based on the assumption of static scenes. However, in the real world, SLAM systems operate with a wide variety of dynamic objects, and many visual SLAM methods still face potential risks when interacting with real-world environments containing various dynamic objects. Feature points located on dynamic objects may lead to incorrect matching, and the presence of dynamic objects can cause the algorithm to generate incorrect data correlations, reducing the accuracy of pose estimation in the SLAM system. In addition, in practical applications, temporary static objects are static when observed but moving when not observed. These objects may cause false alarms during loop closure detection, leading to critical failures. Related research has addressed this problem by using deep learning-assisted methods to detect regions of dynamic objects, but deep learning-assisted methods are limited to predefined objects and therefore have certain limitations. Moreover, some researchers have incorporated the dynamic properties of objects into the model to optimize the framework; however, geometry-based methods require precise camera pose, so they can only handle a limited portion of dynamic objects. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, embodiments of the present invention provide an all-weather infrared inertial SLAM method and apparatus for dynamic environments; this method can achieve accurate and robust positioning of UAVs around the clock based on thermal infrared images, thereby improving the accuracy of SLAM map positioning.

[0006] According to a first aspect of the present invention, an all-weather infrared inertial SLAM method for dynamic environments is provided. The method includes: determining the motion trajectory of a target unmanned aerial vehicle (UAV) based on a thermal infrared image sequence of a target scene; wherein the motion trajectory includes visual pose information of the target UAV at different times; generating a pose estimate of the target UAV based on any current keyframe in a keyframe set corresponding to the thermal infrared image sequence, performing pre-integration based on the current keyframe and IMU data of the previous keyframe adjacent to the current keyframe; selecting visual pose information corresponding to the current time from the motion trajectory; and performing visual-inertial fusion processing on the visual pose information and the pose estimate to obtain a visual reprojection residual and an IMU pre-integration residual; performing BA optimization processing on the keyframe set according to a preset sliding window based on the visual reprojection residual, the IMU pre-integration residual, and the marginalization residual, and outputting an optimized keyframe set; and performing global closed-loop optimization processing on the optimized keyframe set to output a global SLAM map.

[0007] Optionally, the step of performing BA optimization processing on the keyframe set according to a preset sliding window based on the visual reprojection residual, IMU pre-integration residual, and marginalization residual, and outputting an optimized keyframe set, includes: for any current keyframe within the preset sliding window: using a regularization factor and a weight momentum factor to correct the visual reprojection residual corresponding to the current keyframe, obtaining a corrected visual reprojection residual; constructing a BA optimization model based on the marginalization residuals corresponding to all current keyframes within the preset sliding window, and the corrected visual reprojection residual and IMU pre-integration residual corresponding to each current keyframe; when the BA optimization model tends to a minimum, obtaining the static features corresponding to each current keyframe within the preset sliding window, and generating optimized keyframes; and generating an optimized keyframe set based on the optimized keyframes corresponding to each preset sliding window in the keyframe set.

[0008] Optionally, determining the motion trajectory of the target UAV based on the thermal infrared image sequence of the target scene includes: for any thermal infrared image in the thermal infrared image sequence of the target scene: performing feature extraction processing on the thermal infrared image based on a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; performing feature matching and tracking processing on all thermal infrared images in the thermal infrared image sequence based on the descriptors corresponding to the feature points to generate a tracking feature point cloud; performing visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; and determining the motion trajectory of the target UAV based on the three-dimensional feature point cloud.

[0009] Optionally, the method further includes: selecting thermal infrared images from the thermal infrared image sequence with a number of tracking feature points greater than a preset threshold as keyframes to obtain a keyframe set.

[0010] Optionally, the step of performing global loop closure optimization on the optimized keyframe set to output a global SLAM map includes: dividing the optimized keyframe set into several keyframe groups based on tracking feature points; for any keyframe group in the several keyframe groups: selecting keyframes with loop closure relationships to the current keyframe from the keyframe group, and clustering the loop closure relationships to obtain cluster groups; obtaining several cluster groups based on the cluster groups corresponding to each keyframe group in the several keyframe groups; determining the average similarity corresponding to the cluster group based on the similarity of the loop closure relationships corresponding to each loop closure relationship in the cluster group; selecting the two cluster groups with the highest average similarity from the several cluster groups as hypothetical clusters, and using the two hypothetical clusters to perform BA optimization on the current keyframe; selecting the hypothetical cluster with the highest weight from the BA optimization results for global pose optimization to generate a global SLAM map.

[0011] Optionally, the step of selecting keyframes with loop relationships to the current keyframe from the keyframe group and clustering these loop relationships to obtain cluster groups includes: using the DBoW2 bag-of-words model to identify keyframes similar to the current keyframe from the keyframe group; and using keyframes with similarity greater than a preset threshold in the identification results as associated keyframes to obtain at least one associated keyframe; determining the Euclidean distance between the associated keyframe and the current keyframe based on their relative pose information; establishing loop relationships between the associated keyframes with Euclidean distance less than a preset threshold and the current keyframe to obtain at least one loop relationship; and clustering the at least one loop relationship to obtain cluster groups.

[0012] Optionally, based on tracking feature points, the optimized keyframe set is divided into several keyframe groups; including: for any target keyframe in the optimized keyframe set: if the number of tracking feature points shared between the first keyframe adjacent to the target keyframe and the target keyframe is not less than a preset threshold, then the first keyframe is added to the group corresponding to the target keyframe; and the first keyframe is used as the next target keyframe. If the number of tracking feature points shared between the second keyframe adjacent to the first keyframe and the first keyframe is not less than a preset threshold, then the second keyframe is added to the group of the target keyframe, until the number of tracking feature points shared between two adjacent keyframes is less than the preset threshold, then the addition of the target keyframe group ends, and a target keyframe group is generated; based on several target keyframe groups in the optimized keyframe set, several keyframe groups are generated.

[0013] Optionally, feature extraction processing is performed on the thermal infrared image based on a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; including: performing feature extraction processing on the thermal infrared image based on a GhostNet network to generate extracted features; performing weighted processing on the extracted features based on a long-distance attention mechanism to output a feature point cloud; and performing descriptive processing on each feature point in the feature point cloud to generate a descriptor corresponding to the feature point.

[0014] According to a second aspect of the present invention, an all-weather infrared inertial SLAM device for dynamic environments is also provided, comprising: a determination module, configured to determine the motion trajectory of a target UAV based on a thermal infrared image sequence of a target scene; wherein the motion trajectory includes visual pose information of the target UAV at different times; a fusion processing module, configured to: generate a pose estimate of the target UAV based on any current keyframe in a keyframe set corresponding to the thermal infrared image sequence, performing pre-integration based on the current keyframe and IMU data of the previous keyframe adjacent to the current keyframe; selecting visual pose information corresponding to the current time from the motion trajectory; and performing visual-inertial fusion processing on the visual pose information and the pose estimate to obtain a visual reprojection residual and an IMU pre-integration residual; a BA optimization processing module, configured to perform BA optimization processing on the keyframe set according to a preset sliding window based on the visual reprojection residual, the IMU pre-integration residual, and the marginalization residual, and output an optimized keyframe set; and a closed-loop optimization processing module, configured to perform global closed-loop optimization processing on the optimized keyframe set and output a global SLAM map.

[0015] According to a third aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the method described in the first aspect.

[0016] This invention provides an all-weather infrared inertial SLAM method and apparatus for dynamic environments. The method first determines the motion trajectory of a target UAV based on a sequence of thermal infrared images of the target scene; wherein the motion trajectory includes visual pose information of the target UAV at different times. Second, based on any current keyframe in the keyframe set corresponding to the thermal infrared image sequence: pre-integration is performed on the current keyframe and the IMU data of the previous keyframe adjacent to the current keyframe to generate a pose estimate of the target UAV; visual pose information corresponding to the current time is selected from the motion trajectory; and visual-inertial fusion processing is performed on the visual pose information and the pose estimate to obtain visual reprojection residuals and IMU pre-integration residuals; then, based on the visual reprojection residuals, IMU pre-integration residuals, and edge-integration residuals, BA optimization processing is performed on the keyframe set according to a preset sliding window to output an optimized keyframe set; finally, global closed-loop optimization processing is performed on the optimized keyframe set to output a global SLAM map. This embodiment uses a thermal infrared camera as a sensor in the SLAM system and optimizes keyframes in the acquired thermal infrared image sequence based on visual-inertial fusion technology and BA optimization technology. This allows features of dynamic objects that deviate significantly from motion priors to be discarded when estimating the UAV pose, improving the accuracy of SLAM map localization. Subsequently, by performing global closed-loop optimization on the optimized keyframe set, closed-loop detection interference from temporary static objects can be rejected, thereby eliminating global cumulative drift error and providing more reliable support for UAV autonomous navigation and localization, improving the robustness and accuracy of SLAM system state estimation. Attached Figure Description

[0017] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0018] Figure 1 This is a flowchart illustrating an all-weather infrared inertial SLAM method for dynamic environments provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the GhostNet V2 network structure provided in an embodiment of the present invention;

[0020] Figure 3 This is a flowchart illustrating the global closed-loop optimization process for an optimized keyframe set according to an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of a preset sliding window robust BA optimization process provided in an embodiment of the present invention;

[0022] Figure 5 A dynamic environment thermal infrared inertial SLAM system framework for a neural network front-end is provided in one embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the structure of an all-weather infrared inertial SLAM device for dynamic environments provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Visible light SLAM systems suffer from poor robustness in low-light or dark environments, while thermal infrared cameras can distinguish objects based on temperature differences, which is particularly useful in complex environments such as hot fog, rain, or low field-of-view conditions. Using thermal infrared cameras as sensors in SLAM systems not only makes them unaffected by changes in lighting but also allows the system to acquire crucial information for localization and map building in low-light or even complete darkness, resulting in higher performance stability. However, infrared imaging also suffers from drawbacks such as limited planar texture, low contrast, and low signal-to-noise ratio. Addressing the challenges of feature extraction from infrared images, which is difficult and time-consuming, requires reducing computational load and parameter count while maintaining feature extraction accuracy.

[0026] Furthermore, visual SLAM typically assumes a static environment, and dynamic objects may be mistakenly identified as part of the environment, leading to map drift or localization errors. To address these issues, robust SLAM methods are needed to handle undefined dynamic objects that cannot be handled by learning-based or vision-based methods alone. In real-world scenarios, there are also temporary static objects that are static when observed but moving when not observed. Ensuring that changes in dynamic objects do not affect the map quality and localization accuracy of static parts becomes a major challenge in maintaining map consistency.

[0027] like Figure 1 The diagram shown is a flowchart illustrating an all-weather infrared inertial SLAM method for dynamic environments provided by an embodiment of the present invention. Figure 2 The diagram shown is a schematic of the GhostNet V2 network structure provided in an embodiment of the present invention.

[0028] An all-weather infrared inertial SLAM method for dynamic environments includes at least the following steps:

[0029] S101, Based on the thermal infrared image sequence of the target scene, determine the motion trajectory of the target UAV; wherein, the motion trajectory includes the visual pose information of the target UAV at different times;

[0030] S102, Based on any current key frame in the key frame set corresponding to the thermal infrared image sequence: pre-integrate based on the current key frame and the IMU data of the previous key frame adjacent to the current key frame to generate the pose estimate of the target UAV; select the visual pose information corresponding to the current moment from the motion trajectory; and perform visual-inertial fusion processing on the visual pose information and pose estimate to obtain the visual reprojection residual and the IMU pre-integration residual.

[0031] S103, based on the visual reprojection residual, IMU pre-integration residual, and marginalization residual, perform BA optimization on the keyframe set according to the preset sliding window, and output the optimized keyframe set.

[0032] S104 performs global closed-loop optimization on the optimized keyframe set and outputs a global SLAM map.

[0033] In S101, based on preset rules or algorithm models, the visual pose information of the target UAV at different times is determined according to the thermal infrared image sequence of the target scene. Then, based on the visual pose information at different times, the motion trajectory of the target UAV is generated.

[0034] For example: For any thermal infrared image in the thermal infrared image sequence of the target scene: the thermal infrared image is processed by feature extraction based on a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; based on the descriptor corresponding to the feature point, feature matching and tracking processing is performed on all thermal infrared images in the thermal infrared image sequence to generate a tracking feature point cloud; visual SFM processing is performed on the tracking feature point cloud to generate a three-dimensional feature point cloud; based on the three-dimensional feature point cloud, the motion trajectory of the target UAV is determined.

[0035] Furthermore, feature extraction processing is performed on the thermal infrared image based on a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; including: performing feature extraction processing on the thermal infrared image based on a GhostNet network to generate extracted features; performing weighted processing on the extracted features based on a long-distance attention mechanism to output a feature point cloud; and performing descriptive processing on each feature point in the feature point cloud to generate a descriptor corresponding to the feature point.

[0036] For example, SuperPoint is an end-to-end feature point and descriptor extraction network that employs an encoder-decoder structure similar to semantic segmentation networks. Its workflow involves taking a complete image as input, extracting deep features through a shared encoder, and then outputting feature points and their descriptors through two decoders. While the original SuperPoint network used a VGG-like encoder with a simple structure, its large number of layers and channels resulted in high computational cost and parameter count. To overcome the shortcomings of the original SuperPoint network's VGG architecture, particularly the high parameter count and computational cost, GhostNetV2 was used to replace the original VGG encoding layers. GhostNet's overall structure consists of GhostBottlenecks and other structures, significantly reducing computational requirements while maintaining accuracy. GhostNetV2 is formed by introducing a long-range attention mechanism based on GhostNet.

[0037] like Figure 2 The diagram shows the GhostNet V2 network structure provided in this embodiment of the invention. When GhostNetV2 processes an input image (e.g., 224×224×3), it first passes through a 3×3 convolutional block containing convolution, normalization, and activation functions. Multiple Ghost Bottlenecks are then stacked to obtain a 7×7×160 feature layer. Next, a 1×1 convolutional block is used to adjust the number of channels, resulting in a 7×7×960 feature layer. Global average pooling is then performed, followed by another 1×1 convolutional block to obtain a 1×1×1280 feature layer. Finally, a fully connected layer is used for classification. The advantage of this structure is that GhostNetV2 introduces a long-range attention mechanism on top of GhostNet, further enhancing representation capabilities and achieving better performance while reducing the number of parameters.

[0038] In steps S102 and S103, a set of keyframes corresponding to the thermal infrared image sequence is determined based on preset rules or algorithm models. For example: thermal infrared images with a number of tracking feature points greater than a preset threshold are selected from the thermal infrared image sequence as keyframes, thus obtaining a set of keyframes.

[0039] For example, the pose estimation and visual pose information of the target UAV at each current moment are visually inertial aligned to ensure that the scale information in pure vision matches the IMU measurement results. This combined approach improves the overall accuracy and robustness of the SFM system and has significant implications for practical applications.

[0040] For example, the step of performing BA optimization processing on the keyframe set according to a preset sliding window based on the visual reprojection residual, IMU pre-integration residual, and marginalization residual, and outputting an optimized keyframe set includes: for any current keyframe within the preset sliding window: using a regularization factor and a weight momentum factor to correct the visual reprojection residual corresponding to the current keyframe, obtaining a corrected visual reprojection residual; constructing a BA optimization model based on the marginalization residuals corresponding to all current keyframes within the preset sliding window, and the corrected visual reprojection residual and IMU pre-integration residual corresponding to each current keyframe; when the BA optimization model tends to a minimum, obtaining the static features corresponding to each current keyframe within the preset sliding window, and generating optimized keyframes; and generating an optimized keyframe set based on the optimized keyframes corresponding to each preset sliding window in the keyframe set.

[0041] The application of robust BA optimization with a pre-defined sliding window in SLAM systems aims to improve the accuracy of pose estimation while effectively handling outliers in dynamic scenes. The following section will provide a detailed explanation using the formulas of the robust BA optimization model.

[0042] ① Visual Inertia BA Optimization Model

[0043] In visual-inertial state estimation based on a visual-inertial navigation system (VINS), maximum a posteriori (MAP) estimation is achieved by minimizing the sum of the prior and Mahalanobis distance norms of all measurement residuals. The MAP optimization model for visual-inertial systems is defined as follows:

[0044]

[0045] Where, ρ H (.) represents the Huber loss function; r p , and These represent marginalization residuals, IMU pre-integration residuals, and visual reprojection residuals, respectively. For IMU observations and feature point observations; H p Let P represent the measurement estimation matrix for marginalization, and let X represent the covariance of each term. X represents the pose information of the current keyframe. As the proportion of outliers in a dynamic scene increases, the Huber loss function can no longer completely reject the residuals of outliers, and the system cannot function successfully. In other words, the Huber loss function is used to handle these residuals to enhance the model's robustness to outliers.

[0046] ② Regularization factor

[0047] For simplicity, item, The terms are omitted and represented as follows: To robustly estimate pose while rejecting anomalous features, a novel residual term inspired by Black-Rangarajan duality is constructed. :

[0048]

[0049] Among them, w j ∈[0,1] represents each feature f j The corresponding weights determine w j Features f close to 1 j It is a static feature; Φ(w) is a constant parameter; j ) is the weight w j The regularization factor is defined as follows:

[0050] φ(w j ) = 1 - w j Equation (3);

[0051] ③ Weighted momentum factor

[0052] When motion becomes intense, IMU pre-integration becomes inaccurate, leading to inaccurate pose estimation. In this case, the feature reprojection residuals of static objects become large; these features will be ignored during BA (Balanced Assortment) through regularization factors, even if the previous weights are close to 1, resulting in inaccurate BA results. An additional factor, the weight momentum factor, is constructed to ensure that the previously estimated feature weights are unaffected by intense motion.

[0053] Because the features are tracked continuously, each feature f j Use its previous weight n were performed j This is a secondary optimization. To ensure the current weights tend to remain at [a certain level / value]... And with n j The increase in the magnitude of this trend will increase the weighted momentum factor ψ(w) j The design is as follows:

[0054]

[0055] The corrected visual reprojection residual can be obtained. as follows:

[0056]

[0057] in, This represents a constant parameter used to adjust the effect of the momentum factor on BA.

[0058] Adopting (5) Replacing the Huber norm in the visual reprojection residual term in (1), the robust BA optimization model can therefore be expressed as:

[0059]

[0060] This embodiment addresses visual reprojection residuals by introducing feature weights and using pre-integrated IMU data to calculate IMU pre-integration residuals. Each feature is assigned a weight, which is updated and optimized by introducing a weight momentum factor and a regularization factor. During optimization, the weight momentum factor uses the weights of previously tracked features, while the regularization factor is adjusted based on the weights of all features within the current preset sliding window. This strategy is solved through alternating optimization. Since the pose estimation X of the current keyframe can be estimated using IMU pre-integration and the previously optimized state, the feature weights are first optimized based on the estimated state. Therefore, features with large visual reprojection residuals start with smaller weights to reduce their impact on the overall optimization. The optimization steps are repeated until both the state and weights converge. During this process, the weights of outlier features are reduced, making their losses smoother. The regularization factor, by adaptively adjusting the weights, effectively filters out outliers, but does not completely ignore them during optimization. By employing weights and regularization factors inspired by BR duality, the impact of features with high reprojection errors on pose estimation can be reduced while maintaining state estimation performance. This approach enhances the robustness of pose estimation, making the optimization process more stable and accurate, thereby improving the system's performance in complex environments. Overall, this optimization strategy demonstrates good performance in handling dynamic scenes and disturbances, contributing to improved accuracy and reliability of visual inertial odometry.

[0061] In S104, the optimized keyframe set is subjected to global closed-loop optimization based on preset rules or algorithm models, and a global SLAM map is output.

[0062] Performing global closed-loop optimization on the optimized keyframe set effectively groups the closed-loop constraints, reducing the interference of temporary static objects on the global closed-loop optimization, thereby improving the robustness and accuracy of the SLAM system. This method not only improves the optimization efficiency of the optimized keyframe set but also better handles the interference of dynamic and static objects in complex environments, providing more reliable support for autonomous navigation and robot localization.

[0063] This embodiment uses a thermal infrared camera as the sensor for the SLAM system. A lightweight SuperPoint neural network with a GhostNetV2 encoding structure is employed to extract and match feature points from the thermal infrared images. This reduces the computational load and number of parameters in the SLAM system while maintaining the accuracy of image feature extraction. By adding a regularization factor and a weight momentum factor to the BA optimization model within a preset sliding window, and using the weights of previously tracked features in the weight momentum factor and the weights of all features in the current window in the regularization factor, features of dynamic objects that significantly deviate from the motion prior are discarded when estimating the UAV pose, improving the accuracy of UAV localization. In global loop closure optimization, loop closure constraints are grouped into multiple hypotheses. This rejects loop detection interference from temporary static objects, eliminates global cumulative drift error, and generates a global SLAM map. This provides more reliable support for UAV autonomous navigation and localization, improving the robustness and accuracy of SLAM system state estimation.

[0064] Loop closure detection within the SLAM framework can eliminate accumulated errors; however, false positive loop closure features caused by temporary static objects can lead to global loop closure optimization failure. In fact, features from temporary and real static objects may exist in the same keyframe. Therefore, a robust global loop closure optimization method is designed to group loop closure constraints into multiple hypotheses to reject loop closure detection interference from temporary static objects. Loops from the same features are grouped, even if they come from different keyframes, with each group using only one weight, thus achieving faster optimization.

[0065] like Figure 3 The diagram shown is a flowchart illustrating the global closed-loop optimization process for the optimized keyframe set according to an embodiment of the present invention.

[0066] Perform global closed-loop optimization on the optimized keyframe set, which includes at least the following steps:

[0067] S301, based on the tracking feature points, divide the optimized keyframe set into several keyframe groups;

[0068] S302, For any keyframe group among several keyframe groups: Select keyframes that have a loop relationship with the current keyframe from the keyframe group, and cluster the loop relationship to obtain a cluster group;

[0069] S303, Based on the clustering group corresponding to each keyframe group in several keyframe groups, several clustering groups are obtained; Based on the similarity corresponding to each loop relationship in the clustering group, the average similarity corresponding to the clustering group is determined.

[0070] S304: Select the two clusters with the highest average similarity from several cluster groups as hypothetical clusters, and use the two hypothetical clusters to perform BA optimization on the current keyframe.

[0071] S205. Select the hypothesis cluster with the highest weight from the BA optimization results and perform global pose optimization to generate a global SLAM map.

[0072] In S301, for example, based on tracking feature points, the optimized keyframe set is divided into several keyframe groups; including: for any target keyframe in the optimized keyframe set: if the number of tracking feature points shared between the first keyframe adjacent to the target keyframe and the target keyframe is not less than a preset threshold, then the first keyframe is added to the group corresponding to the target keyframe; and the first keyframe is used as the next target keyframe. If the number of tracking feature points shared between the second keyframe adjacent to the first keyframe and the first keyframe is not less than a preset threshold, then the second keyframe is added to the group of the target keyframe, until the number of tracking feature points shared between two adjacent keyframes is less than the preset threshold, then the grouping of the target keyframe ends, and a target keyframe group is generated; based on several target keyframe groups in the optimized keyframe set, several keyframe groups are generated.

[0073] In S302, for example, keyframes with loop relationships to the current keyframe are selected from the keyframe group, and the loop relationships are clustered to obtain cluster groups; this includes: using the DBoW2 bag-of-words model to identify keyframes similar to the current keyframe from the keyframe group; and using keyframes with similarity greater than a preset threshold in the identification results as associated keyframes to obtain at least one associated keyframe; determining the Euclidean distance between the associated keyframe and the current keyframe based on the relative pose information between the two; establishing loop relationships between the associated keyframes with Euclidean distance less than the preset threshold and the current keyframe to obtain at least one loop relationship; and clustering the at least one loop relationship to obtain cluster groups.

[0074] In S303, for any loop relationship in the cluster group: the similarity between the current keyframe and keyframes in the loop relationship is obtained, and this similarity is used as the similarity of the loop relationship. The average similarity of the loop relationships corresponding to each loop relationship in the cluster group is summed and averaged to obtain the average similarity of the cluster group.

[0075] In S304, since there are several keyframe groups preceding the current keyframe, when detecting loop closures in the current keyframe, it is necessary to perform loop closure detection on each of the keyframe groups, resulting in several clusters. These clusters are then sorted in descending order of their average similarity, and the top two clusters in the sort are taken as the hypothetical clusters.

[0076] Before grouping loop closures, adjacent keyframes sharing a minimum number of tracking features must be grouped. Starting from the i-th camera frame C... i The initial group is defined as

[0077]

[0078] Where α represents the minimum number of tracking features, Indicates from C i To C k For simplicity, the feature set for inter-frame tracking is grouped into Group(C). i ) is denoted as G i Then, multi-hypothesis clustering is performed. For the current keyframe, DBoW2 bag-of-words recognition and keyframe grouping G are used. i Each keyframe C in k Similar keyframes C m If there are no similar keyframes, skip C. k After identifying at most 3 distinct m's for k, in C k Feature matching between these keyframes yields the relative pose T. If the features used for matching come from the same object, even if the matched C... k and C m Unlike other loops, the estimated poses of the matched loops will also be located close to each other. Therefore, by calculating the Euclidean distance between the loop poses, similar closed loops with small Euclidean distances can be clustered. Each cluster group can be called a hypothesis. To reduce computational cost, the first two hypotheses are used to group the keyframes into G groups. i These two assumptions are denoted as clustering. and

[0079] The BR dual method is used to cluster the two hypotheses to construct a BR optimization model, which is shown in the following equation:

[0080]

[0081] in, Indicates two adjacent keyframes C i and C i+1 Local poses between; It is in a closed loop and Ck The relative poses between them; and P L Let represent the covariance of the local pose and the closed loop, respectively; Φ is a constant parameter, representing the regularization factor of the loop closure. i Defined as follows

[0082]

[0083] in To ensure that the weights are not affected by the number of cyclic closures in the hypothetical clusters, the weights are divided by the cardinality of each hypothetical cluster. Then, Equation (9) is optimized in the same way as (7), and finally only the hypothetical clusters with higher weights are used for global pose optimization to eliminate global cumulative drift error and form a global trajectory map.

[0084] This embodiment uses a thermal infrared camera as a sensor for the SLAM system, which is unaffected by changes in lighting. Even in low light or complete darkness, the system can still acquire key information for localization and map building, exhibiting higher performance stability. A lightweight SuperPoint neural network with a GhostNetV2 encoding structure is used for feature point extraction, reducing computational and parameter requirements while maintaining image feature extraction accuracy. A robust VI-SLAM method is employed to handle undefined dynamic objects that cannot be addressed by learning-based or vision-based methods alone. A novel bundle adjustment (BA) method is used, utilizing the regularization factor of IMU pre-integration and considering the momentum factor of each weight's previous state to cover temporary inaccuracies in pre-integration. The weight momentum factor uses the weights of previously tracked features, while the regularization factor uses the weights of all features in the current window, simultaneously estimating camera pose and discarding features of dynamic objects that significantly deviate from the motion prior. A robust global optimization method is used to group loop closure constraints into multiple hypotheses to reject loop detection interference from temporary static objects, eliminate global cumulative drift error, form a global trajectory map, ensure map updates, and maintain global consistency. This method not only improves the optimization efficiency of the optimized keyframe set, but also better handles the interference of dynamic and static objects in complex environments, providing more reliable support for autonomous navigation and robot localization; thus, it improves the robustness and accuracy of the SLAM system.

[0085] like Figure 4 The diagram shown is a schematic of a preset sliding window robust BA optimization process provided in an embodiment of the present invention.

[0086] Each feature f has a weight used for the visual reprojection residual (i.e., the visual residual); the IMU pre-integration residual (i.e., the IMU residual) is obtained through IMU pre-integration. The weights of the features are tracked using a previously preset sliding window (i.e., the previous window) in the weight momentum factor (i.e., the momentum factor), while the weights of all features in the current preset sliding window (i.e., the current window) are used in the regularization factor. Each weight is optimized through the regularization factor and momentum factor. An alternating optimization approach is used. Since the current state X can be estimated from the IMU pre-integration and the previously optimized state, the weights are optimized first based on the estimated state. Therefore, features with large visual reprojection errors start with smaller weights. As the optimization steps are repeated until the state and weights converge, the weights of outlier features are reduced, and their losses become flatter. The regularization factor effectively filters out outliers by adaptively adjusting the weights, but it does not completely ignore outliers during the optimization process. The robust BA optimization model optimizes the visual reprojection residual, IMU pre-integration residual, marginalization residual, momentum factor, and regularization factor. This reduces the impact of visual reprojection error on features with relatively high estimated state while maintaining state estimation performance.

[0087] The following section provides a detailed description of the all-weather infrared inertial SLAM method for dynamic environments provided in this embodiment, using specific application scenarios as examples.

[0088] An all-weather infrared inertial SLAM method for dynamic environments includes at least the following steps:

[0089] S1, for any thermal infrared image in the thermal infrared image sequence of the target scene: feature extraction processing is performed on the thermal infrared image based on the GhostNet network to generate extracted features; the extracted features are weighted based on a long-range attention mechanism to output a feature point cloud; each feature point in the feature point cloud is described to generate a descriptor corresponding to the feature point. Based on the descriptors corresponding to the feature points, feature matching and tracking processing is performed on all thermal infrared images in the thermal infrared image sequence to generate a tracking feature point cloud; the tracking feature point cloud is processed by visual SFM to generate a three-dimensional feature point cloud; based on the three-dimensional feature point cloud, the motion trajectory of the target UAV is determined; wherein, the motion trajectory includes the visual pose information of the target UAV at different times;

[0090] S2, Select thermal infrared images with more than a preset threshold number of tracking feature points from the thermal infrared image sequence as key frames to obtain a key frame set;

[0091] S3, based on any current keyframe in the keyframe set corresponding to the thermal infrared image sequence: pre-integrate based on the current keyframe and the IMU data of the previous keyframe adjacent to the current keyframe to generate the pose estimate of the target UAV; select the visual pose information corresponding to the current moment from the motion trajectory; and perform visual-inertial fusion processing on the visual pose information and the pose estimate to obtain the visual reprojection residual and the IMU pre-integration residual;

[0092] S4, for any current keyframe within a preset sliding window: the visual reprojection residual corresponding to the current keyframe is corrected using a regularization factor and a weight momentum factor to obtain the corrected visual reprojection residual; based on the marginalization residuals corresponding to all current keyframes within the preset sliding window, as well as the corrected visual reprojection residual and IMU pre-integration residual corresponding to each current keyframe, a BA optimization model is constructed; when the BA optimization model tends to its minimum, the static features corresponding to each current keyframe within the preset sliding window are obtained, and optimized keyframes are generated; based on the optimized keyframes corresponding to each preset sliding window in the keyframe set, an optimized keyframe set is generated;

[0093] S5, for any target keyframe in the optimized keyframe set: if the number of shared tracking feature points between the first keyframe adjacent to the target keyframe and the target keyframe is not less than a preset threshold, then the first keyframe is added to the group corresponding to the target keyframe; and the first keyframe is used as the next target keyframe. If the number of shared tracking feature points between the second keyframe adjacent to the first keyframe and the first keyframe is not less than a preset threshold, then the second keyframe is added to the group of the target keyframe. This process continues until the number of shared tracking feature points between two adjacent keyframes is less than a preset threshold, then the grouping of the target keyframe is terminated, and a target keyframe group is generated.

[0094] S6. Based on the optimized keyframe set and several target keyframe groups, generate several keyframe groups. For any keyframe group among the several keyframe groups: use the DBoW2 bag-of-words model to identify keyframes similar to the current keyframe from the keyframe group; and take the keyframes with similarity greater than a preset threshold in the identification results as associated keyframes to obtain at least one associated keyframe; determine the Euclidean distance between the associated keyframe and the current keyframe based on the relative pose information between the associated keyframe and the current keyframe; establish a loop relationship between the associated keyframe with an Euclidean distance less than a preset threshold and the current keyframe to obtain at least one loop relationship; cluster the at least one loop relationship to obtain a cluster group;

[0095] S7. Based on the clustering group corresponding to each keyframe group in the plurality of keyframe groups, a plurality of clustering groups are obtained; based on the similarity of the loop closure relationship corresponding to each loop closure relationship in the clustering group, the average similarity corresponding to the clustering group is determined; the clustering groups with the highest average similarity among the plurality of clustering groups are selected as hypothetical clustering groups, and the two hypothetical clustering groups are used to perform BA optimization processing on the current keyframe; the hypothetical clustering group with the highest weight is selected from the BA optimization processing results to perform global pose optimization processing to generate a global SLAM map.

[0096] This embodiment uses a thermal infrared camera as the sensor for the SLAM system. A lightweight SuperPoint neural network with a GhostNetV2 encoding structure is designed to extract and match feature points from the thermal infrared images. This reduces computational cost and parameter count while maintaining image feature extraction accuracy. In the sliding window optimization model, a regularization factor and a weight momentum factor are added. The weight momentum factor uses the weights of previously tracked features, while the regularization factor uses the weights of all features in the current window. This allows for simultaneous estimation of camera pose and discarding features of dynamic objects that significantly deviate from the motion prior. In the global loop closure optimization, loop closure constraints are grouped into multiple hypothetical clusters to reject loop closure detection interference from temporary static objects, eliminating global cumulative drift error and forming a global SLAM map.

[0097] like Figure 5 The figure shows a framework for a dynamic environment thermal infrared inertial SLAM system with a neural network front-end provided in an embodiment of the present invention.

[0098] The system includes a front-end measurement and processing module, a sliding window optimization module, and a global optimization module.

[0099] The front-end measurement and processing module first uses the Superpoint network to extract and track features from the thermal infrared image sequence acquired by the thermal infrared camera; then it performs visual SFM processing on the feature extraction and tracking results; next, it obtains the set of key frames corresponding to the thermal infrared image sequence based on the visual SFM processing results; then, for any key frame in the key frame set, it obtains the IMU data corresponding to the key frame from the inertial measurement unit; and performs IMU pre-integration on the IMU data; finally, it performs visual-inertial alignment processing based on the IMU pre-integration result and the visual SFM processing result, and outputs the visual reprojection residual and the IMU pre-integration residual.

[0100] The sliding window optimization module first corrects the visual reprojection residual using regularization and weight momentum factors to generate a corrected visual reprojection residual. Then, it constructs a BA optimization model based on the corrected visual reprojection residual, IMU pre-integration residual, and marginalization residual. Next, it performs dynamic robust BA optimization on keyframes within the sliding window to obtain keyframes with static features. Finally, it obtains the optimized keyframe set based on all keyframes with static features within each sliding window.

[0101] The global optimization module first divides the optimized keyframe set into several keyframe groups; then, for the received current keyframe, it uses DBoW2 bag-of-words to match the current keyframe and the keyframe group; and performs multi-hypothesis clustering based on the matching results; next, it selectively optimizes the loop closure hypothesis in the multi-hypothesis clustering results to obtain the correct loop closure; finally, it generates a global SLAM map based on the correct loop closure.

[0102] like Figure 6 The diagram shown is a schematic representation of an all-weather infrared inertial SLAM device for dynamic environments provided in an embodiment of the present invention.

[0103] An all-weather infrared inertial SLAM device for dynamic environments, the device 600 includes: a determination module 601, used to determine the motion trajectory of a target UAV based on a thermal infrared image sequence of the target scene; wherein the motion trajectory includes visual pose information of the target UAV at different times; a fusion processing module 602, used to generate a pose estimate of the target UAV based on any current keyframe in the keyframe set corresponding to the thermal infrared image sequence, performing pre-integration based on the current keyframe and the IMU data of the previous keyframe adjacent to the current keyframe; selecting the visual pose information corresponding to the current time from the motion trajectory; and performing visual-inertial fusion processing on the visual pose information and the pose estimate to obtain a visual reprojection residual and an IMU pre-integration residual; a BA optimization processing module 603, used to perform BA optimization processing on the keyframe set according to a preset sliding window based on the visual reprojection residual, the IMU pre-integration residual, and the marginalization residual, and output an optimized keyframe set; and a closed-loop optimization processing module 604, used to perform global closed-loop optimization processing on the optimized keyframe set and output a global SLAM map.

[0104] In a preferred embodiment of this example, the BA optimization processing module includes: a correction processing unit, used to correct the visual reprojection residual corresponding to any current keyframe within a preset sliding window using a regularization factor and a weight momentum factor, to obtain a corrected visual reprojection residual; a first generation unit, used to construct a BA optimization model based on the marginalization residuals corresponding to all current keyframes within the preset sliding window, and the corrected visual reprojection residual and IMU pre-integration residual corresponding to each current keyframe; when the BA optimization model tends to a minimum, obtaining the static features corresponding to each current keyframe within the preset sliding window, and generating optimized keyframes; and a second generation unit, used to generate an optimized keyframe set based on the optimized keyframes corresponding to each preset sliding window in the keyframe set.

[0105] In a preferred embodiment of this example, the determining module includes: a first generation unit, configured to perform feature extraction processing on any thermal infrared image in the thermal infrared image sequence of the target scene based on a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; a second generation unit, configured to perform feature matching and tracking processing on all thermal infrared images in the thermal infrared image sequence based on the descriptors corresponding to the feature points to generate a tracking feature point cloud; a third generation unit, configured to perform visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; and a determining unit, configured to determine the motion trajectory of the target UAV based on the three-dimensional feature point cloud.

[0106] In a preferred embodiment of this invention, the device further includes a selection module, used to select thermal infrared images from the thermal infrared image sequence whose number of tracking feature points is greater than a preset threshold as keyframes, thereby obtaining a keyframe set.

[0107] In a preferred embodiment of this example, the closed-loop optimization processing module includes: a partitioning unit, used to partition the optimized keyframe set into several keyframe groups based on tracking feature points; a clustering unit, used to select keyframes with loop closure relationships to the current keyframe from any keyframe group among the several keyframe groups, and cluster the loop closure relationships to obtain cluster groups; a determination unit, used to obtain several cluster groups based on the cluster groups corresponding to each keyframe group among the several keyframe groups; and to determine the average similarity corresponding to the cluster group based on the similarity of the loop closure relationships corresponding to each loop closure relationship in the cluster group; a BA optimization processing unit, used to select the two cluster groups with the highest average similarity from the several cluster groups as hypothetical clusters, and use the two hypothetical clusters to perform BA optimization processing on the current keyframe; and a global optimization processing unit, used to select the hypothetical cluster with the highest weight from the BA optimization processing results for global pose optimization processing to generate a global SLAM map.

[0108] In a preferred embodiment of this example, the clustering unit includes: an identification subunit, used to identify keyframes similar to the current keyframe from the keyframe group using the DPoW2 bag-of-words model; and to identify keyframes with a similarity greater than a preset threshold in the identification results as associated keyframes, thereby obtaining at least one associated keyframe; a determination subunit, used to determine the Euclidean distance between the associated keyframe and the current keyframe based on the relative pose information between the two; an establishment subunit, used to establish loop relationships between the associated keyframes with an Euclidean distance less than the preset threshold and the current keyframe, thereby obtaining at least one loop relationship; and a clustering subunit, used to cluster the at least one loop relationship to obtain a cluster group.

[0109] In a preferred embodiment of this example, the partitioning unit includes: a first generation subunit, configured to, for any target keyframe in the optimized keyframe set: if the number of shared tracking feature points between a first keyframe adjacent to the target keyframe and the target keyframe is not less than a preset threshold, add the first keyframe to the group corresponding to the target keyframe; and use the first keyframe as the next target keyframe; if the number of shared tracking feature points between a second keyframe adjacent to the first keyframe and the first keyframe is not less than a preset threshold, add the second keyframe to the group of the target keyframe, until the number of shared tracking feature points between two adjacent keyframes is less than the preset threshold, then end the grouping of the target keyframe and generate a target keyframe group; and a second generation subunit, configured to generate several keyframe groups based on several target keyframe groups in the optimized keyframe set.

[0110] In a preferred embodiment of this example, the first generation unit includes: a feature extraction subunit, used to perform feature extraction processing on the thermal infrared image based on the GhostNet network to generate extracted features; a weighted processing subunit, used to perform weighted processing on the extracted features based on a long-distance attention mechanism to output a feature point cloud; and a description processing subunit, used to perform description processing on each feature point in the feature point cloud to generate a descriptor corresponding to the feature point.

[0111] The above-described apparatus can execute the all-weather infrared inertial SLAM method for dynamic environments provided in an embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the all-weather infrared inertial SLAM method for dynamic environments. Technical details not described in detail in this embodiment can be found in the all-weather infrared inertial SLAM method for dynamic environments provided in an embodiment of the present invention.

[0112] The present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the all-weather infrared inertial SLAM method for dynamic environments described in the present invention.

[0113] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0114] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0115] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to the following embodiments of this application described in the "Exemplary Methods" section above.

[0116] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0118] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0119] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0122] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0123] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An all-time infrared inertial SLAM method for dynamic environments, characterized in that, The method comprises the following steps: determining the motion trajectory of the target UAV based on the target scene thermal infrared image sequence; wherein the motion trajectory comprises the visual pose information corresponding to the target UAV at different time; based on any current key frame in the key frame set corresponding to the thermal infrared image sequence: based on the current key frame and the IMU data of the last key frame adjacent to the current key frame, pre-integrating to generate the pose estimation of the target UAV; selecting the visual pose information corresponding to the current time from the motion trajectory; and performing visual-inertial fusion processing on the visual pose information and the pose estimation to obtain visual re-projection residuals and IMU pre-integration residuals; for any current key frame in the preset sliding window: using the regularization factor and the weight momentum factor to correct the visual re-projection residuals corresponding to the current key frame, to obtain the corrected visual re-projection residuals; based on the marginalization residuals of all current key frames in the preset sliding window, and the corrected visual re-projection residuals and IMU pre-integration residuals corresponding to each current key frame, constructing a BA optimization model; when the BA optimization model tends to be the minimum, obtaining the static features corresponding to each current key frame in the preset sliding window, generating optimized key frames; based on the optimized key frames corresponding to each preset sliding window in the key frame set, generating an optimized key frame set; performing global loop closure optimization processing on the optimized key frame set, and outputting a global SLAM map.

2. The method of claim 1, wherein, The method for determining the motion trajectory of the target UAV based on the target scene thermal infrared image sequence comprises the following steps: for any thermal infrared image in the target scene thermal infrared image sequence: based on the lightweight SuperPoint network, performing feature extraction processing on the thermal infrared image to generate a feature point cloud and a descriptor corresponding to each feature point; based on the descriptors corresponding to the feature points, performing feature matching and tracking processing on all thermal infrared images in the thermal infrared image sequence to generate a tracking feature point cloud; performing visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; based on the three-dimensional feature point cloud, determining the motion trajectory of the target UAV.

3. The method of claim 2, wherein, The method further comprises the following steps: selecting a thermal infrared image with a number of tracking feature points greater than a preset threshold from the thermal infrared image sequence as a key frame to obtain a key frame set.

4. The method of claim 2, wherein, The method for performing global loop closure optimization processing on the optimized key frame set to output a global SLAM map comprises the following steps: based on the tracking feature points, dividing the optimized key frame set into a plurality of key frame groups; for any key frame group in the plurality of key frame groups: selecting a key frame having a loop relationship with the current key frame from the key frame group, and clustering the loop relationship to obtain a cluster group; based on the cluster group corresponding to each key frame group in the plurality of key frame groups, obtaining a plurality of cluster groups; based on the loop relationship similarity corresponding to each loop relationship in the cluster group, determining the average similarity corresponding to the cluster group; selecting, from the several clustering groups, a clustering group with the top two average similarities as a hypothesis clustering group, and performing BA optimization processing on the current key frame by using the two hypothesis clustering groups; selecting, from the BA optimization processing result, a hypothesis clustering group with the highest weight to perform global pose optimization processing, and generating a global SLAM map.

5. The method of claim 4, wherein, The selecting, from the key frame groups, a key frame having a loop relationship with the current key frame, and clustering the loop relationship to obtain a clustering group, includes: identifying, from the key frame groups, a key frame similar to the current key frame by using a DBoW2 bag-of-words model; and taking a key frame with a similarity greater than a preset threshold in the identification result as an associated key frame to obtain at least one associated key frame; determining an Euclidean distance between the associated key frame and the current key frame based on relative pose information between the associated key frame and the current key frame; establishing a loop relationship between the associated key frame and the current key frame, to obtain at least one loop relationship, wherein the associated key frame has an Euclidean distance less than a preset threshold from the current key frame; clustering the at least one loop relationship to obtain a clustering group.

6. The method of claim 4, wherein, Based on the tracking feature points, the optimized key frame set is divided into several key frame groups; including: For any target key frame in the optimized key frame set: if the number of tracking feature points shared between the first key frame adjacent to the target key frame and the target key frame is not less than a preset threshold, the first key frame is added to the grouping corresponding to the target key frame; and the first key frame is taken as the next target key frame, if the number of tracking feature points shared between the second key frame adjacent to the first key frame and the first key frame is not less than a preset threshold, the second key frame is added to the grouping of the target key frame, until the number of tracking feature points shared between two adjacent key frames is less than a preset threshold, the grouping addition of the target key frame is ended, and a target key frame grouping is generated; Based on the several target key frame groupings in the optimized key frame set, several key frame groupings are generated.

7. The method of claim 2, wherein, Based on the lightweight SuperPoint network, the thermal infrared image is subjected to feature extraction processing to generate a feature point cloud and a descriptor corresponding to each feature point; including: Based on the GhostNet network, the thermal infrared image is subjected to feature extraction processing to generate extracted features; Based on the long-distance attention mechanism, the extracted features are subjected to weighting processing to output a feature point cloud; Each feature point in the feature point cloud is described to generate a descriptor corresponding to the feature point.

8. An all-weather infrared inertial SLAM device for dynamic environments, characterized by, The determination module is configured to determine a motion trajectory of a target unmanned aerial vehicle based on a sequence of thermal infrared images of a target scene; wherein the motion trajectory includes visual pose information corresponding to the target unmanned aerial vehicle at different times; ​ The fusion processing module is configured to, based on any current key frame in a key frame set corresponding to the thermal infrared image sequence: generate a pose estimation of the target UAV based on pre-integration of IMU data of the current key frame and a previous key frame adjacent to the current key frame; select visual pose information corresponding to a current time from the motion trajectory; and perform visual-inertial fusion processing on the visual pose information and the pose estimation to obtain visual re-projection residuals and IMU pre-integration residuals. The BA optimization processing module includes: a correction processing unit configured to, for any current key frame in a preset sliding window: correct visual re-projection residuals corresponding to the current key frame by using a regularization factor and a weight momentum factor to obtain corrected visual re-projection residuals; a first generation unit configured to construct a BA optimization model based on edge residuals of all current key frames in the preset sliding window, and the corrected visual re-projection residuals and the IMU pre-integration residuals corresponding to each of the current key frames; and obtain static features corresponding to each of the current key frames in the preset sliding window when the BA optimization model tends to be the minimum, and generate optimized key frames; and a second generation unit configured to generate an optimized key frame set based on the optimized key frames corresponding to each of the preset sliding windows in the key frame set. The loop optimization processing module is configured to perform global loop optimization processing on the optimized key frame set, and output a global SLAM map. 9.A computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the method of any one of claims 1-7.

Images