Vision-inertial odometer method and device based on multi-modal image fusion

Abstract

The application provides a visual inertial odometer method and device based on multi-modal image fusion, and the method comprises the following steps: for any current thermal infrared image in a thermal infrared image sequence of a target scene: registering the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image; performing image fusion processing on the current quasi-thermal infrared image and the current visible light image to generate a current fusion image; for any current key frame in the fusion image sequence: performing visual inertial alignment processing on the current visual pose information corresponding to the current key frame and the IMU pre-integration output result to output aligned image data; performing nonlinear optimization processing on each aligned image data in the key frame sequence according to a preset sliding window to output the real-time pose state of the first device. Therefore, the image information of different wave bands can be fused, so that the fusion image with full-scene and full-time information is obtained, and the robustness and precision of the VIO system in a complex environment are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to a visual inertial odometry method and apparatus based on multimodal image fusion. Background Technology

[0002] While visual-inertial odometry (VIO) technology has made significant progress in many applications, visible light image-based inertial VIO degrades in scenarios with extreme differences in light distribution, dynamic changes in illumination, or visual obstructions such as dust, fog, and smoke, reducing the reliability of visual VIO state estimation solutions. Therefore, the core challenge facing visual-inertial odometry is how to achieve efficient localization and high-precision state estimation and map building in complex environments with unstable features.

[0003] To address these issues, researchers have been exploring ways to improve the robustness and accuracy of VIO systems by fusing multiple modal sensors (such as cameras, LiDAR, and IMUs). Compared to visible light cameras, thermal infrared cameras offer all-weather sensing capabilities, providing significant advantages for applications in visually degraded scenarios. Current thermal-inertial odometry (TIO) solutions are largely improvements on conventional visual-inertial odometry, relying on image feature processing. However, using thermal infrared cameras within existing VIO frameworks presents several challenges: the image data captured by thermal infrared cameras typically has low resolution and contrast; and because it is indistinguishable from the surrounding thermal radiation, much visually observable texture information, such as color and stripes, is lost in thermal images. Furthermore, thermal infrared cameras require non-uniformity correction or flat-field correction during operation to eliminate imaging effects between consecutive frames caused by data interruptions due to accumulated non-zero mean noise.

[0004] To overcome the problems of single-modal image-based visual image integration (VIO), some studies have coupled thermal infrared (TIR) ​​images with visible light images for VIO. The VIO front-end uses a dual-spectral image channel composed of a visible light camera and a TIR camera, performing feature extraction and tracking in parallel. A front-end coordinator calculates the scores of the TIR and visible light modalities based on feature quality, selecting the optimal frame for single-modal input, or assigning different numbers of feature points to both modalities, while simultaneously inputting the data to the back-end for state estimation. This method can improve the stability of the VIO system in complex lighting environments, but the dual-spectral image channel at the front end consumes more computational resources and introduces unnecessary lag during modal switching, affecting real-time performance. With the advancement of computer vision technology and deep learning data fusion networks, the fusion of TIR and visible light images has rapidly developed. Fusion of TIR and visible light images can effectively overcome the challenges posed by changes in lighting. In some low-texture environments, infrared images can compensate for the deficiencies of visible light images. Therefore, directly using the fused image for VIO processing is a novel approach. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, embodiments of the present invention provide a visual inertial odometry method and apparatus based on multimodal image fusion; this method can fuse image information from two different bands to improve the robustness and accuracy of the VIO system in complex environments.

[0006] According to a first aspect of the present invention, a visual inertial odometry method based on multimodal image fusion is provided, applied to a first device; comprising: for any current thermal infrared image in a thermal infrared image sequence of a target scene: registering the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image; performing image fusion processing on the current quasi-thermal infrared image and the current visible light image using a PSFusion network to generate a current fused image; for any current keyframe in a keyframe sequence corresponding to the fused image sequence: performing visual inertial alignment processing on the current visual pose information corresponding to the current keyframe and the IMU pre-integration output result, and outputting aligned image data; performing nonlinear optimization processing on the aligned image data of each current keyframe in the keyframe sequence according to a preset sliding window, and outputting the real-time pose state of the first device.

[0007] Optionally, registering the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image includes: performing image distortion correction processing on the current thermal infrared image and the current visible light image respectively to generate a distorted thermal infrared image and a distorted visible light image; performing downsampling processing on the distorted thermal infrared image to generate a thermal infrared image with the same resolution as the distorted visible light image; performing feature matching processing on the visible light image and the thermal infrared image with the same resolution respectively to generate visible light features and thermal infrared features with matching relationships; performing model estimation based on the visible light features and the thermal infrared features to generate a homography matrix; and performing reprojection transformation processing on the current thermal infrared image based on the homography matrix to generate the current quasi-thermal infrared image.

[0008] Optionally, the step of using the PSFusion network to perform image fusion processing on the current quasi-thermal infrared image and the current visible light image to generate a current fused image includes: performing feature fusion processing on the current visible light image and the current quasi-thermal infrared image at different scales to generate surface detail fusion features and semantic fusion features; performing sparse semantic perception processing on the semantic fusion features to generate semantic information; performing scene reconstruction processing on the surface detail fusion features to generate scene features; performing fusion processing on the semantic information and the scene features based on a channel-spatial attention mechanism to generate semantic scene fusion features; the semantic scene fusion features are then output as a fused image via an image fusion path; or, the semantic scene fusion features are processed via a scene fidelity path and then output as a fused image via an image fusion path.

[0009] Optionally, the method further includes: performing feature extraction processing on the current visible light image and the current quasi-thermal infrared image at different scales to obtain first surface features and first residual features at different scales corresponding to the current visible light image; and second surface features and second residual features at different scales corresponding to the current quasi-thermal infrared image; performing surface detail fusion processing on the first surface features, second surface features, first residual features at a specified scale, and second residual features at a specified scale to generate surface detail fusion features; and performing deep semantic fusion processing on the first residual features and second residual features at different scales to generate semantic fusion features.

[0010] Optionally, the method further includes: performing feature extraction processing on the fused image sequence corresponding to the thermal infrared image sequence based on a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; performing feature matching and tracking processing on all fused images in the fused image sequence based on the descriptors corresponding to the feature points to generate a tracking feature point cloud; and selecting fused images from the fused image sequence with a number of tracking feature points greater than a preset threshold as keyframes to obtain a keyframe sequence.

[0011] Optionally, the method further includes: acquiring the tracking feature point cloud corresponding to the current keyframe; performing visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; and determining the current visual pose information of the first device based on the three-dimensional feature point cloud.

[0012] Optionally, the aligned image data includes: visual reprojection residual, IMU pre-integration residual, and pose estimation value; the step of performing nonlinear optimization processing on the aligned image data of each current key frame in the keyframe sequence according to a preset sliding window to output the real-time pose state of the first device includes: for any current key frame in the keyframe sequence: using a regularization factor and a weight momentum factor to correct the visual reprojection residual corresponding to the current key frame to obtain a corrected visual reprojection residual; constructing a BA optimization model based on the marginalization residuals corresponding to all current key frames in the preset sliding window and the corrected visual reprojection residuals and IMU pre-integration residuals corresponding to each current key frame; and using the BA optimization model to optimize the pose estimation value corresponding to each current key frame in the preset sliding window to generate the real-time pose state of the first device.

[0013] Optionally, the feature extraction process for the fused image sequence corresponding to the thermal infrared image sequence based on the lightweight SuperPoint network, generating a feature point cloud and a descriptor corresponding to each feature point, includes: for any fused image in the fused image sequence corresponding to the thermal infrared image sequence: performing feature extraction processing on the fused image sequence corresponding to the thermal infrared image sequence based on the GhostNet network to generate extracted features; performing weighted processing on the extracted features based on a long-range attention mechanism to output a feature point cloud; and performing descriptive processing on each feature point in the feature point cloud to generate a descriptor corresponding to the feature point.

[0014] According to a second aspect of the present invention, a visual inertial odometry device based on multimodal image fusion is also provided, applied to a first device; comprising: an image fusion processing module, configured to: register the current thermal infrared image to a visible light image for any current thermal infrared image in a thermal infrared image sequence of a target scene, generating a current quasi-thermal infrared image; and perform image fusion processing on the current quasi-thermal infrared image and the current visible light image using a PSFusion network to generate a current fused image; a visual inertial alignment processing module, configured to: perform visual inertial alignment processing on the current visual pose information corresponding to the current key frame and the IMU pre-integration output result for any current key frame in a key frame sequence corresponding to the fused image sequence, and output aligned image data; and a nonlinear optimization processing module, configured to perform nonlinear optimization processing on the aligned image data of each current key frame in the key frame sequence according to a preset sliding window, and output the real-time pose state of the first device.

[0015] According to a third aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the method described in the first aspect.

[0016] This invention provides a visual inertial odometry (VIO) method based on multimodal image fusion, applied to a first device. The method includes: First, for any current thermal infrared image in a thermal infrared image sequence of a target scene, registering the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image; then, using a PSFusion network to perform image fusion processing on the current quasi-thermal infrared image and the current visible light image to generate a current fused image; Second, for any current keyframe in a keyframe sequence corresponding to the fused image sequence, performing visual inertial alignment processing on the current visual pose information and IMU pre-integration output result corresponding to the current keyframe, outputting aligned image data; then, performing nonlinear optimization processing on the aligned image data of each current keyframe in the keyframe sequence according to a preset sliding window, outputting the real-time pose state of the first device. In this embodiment, after image registration of the visible light image and the thermal infrared image, the image is input into the PSFusion network for image fusion; thus, image information from different bands can be fused to obtain a fused image rich in information across all scenes and all times, thereby improving the robustness and accuracy of the VIO system in complex environments. Attached Figure Description

[0017] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0018] Figure 1This is a flowchart illustrating a visual inertial odometry method based on multimodal image fusion according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the process of image fusion processing of the current quasi-thermal infrared image and the current visible light image using a PSFusion network according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the PSFusion network framework provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of a visual inertial odometry device based on multimodal image fusion according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] like Figure 1 The diagram shown is a flowchart of a visual inertial odometry method based on multimodal image fusion provided in an embodiment of the present invention.

[0024] A visual-inertial odometry method based on multimodal image fusion, applied to a first device, includes at least the following steps:

[0025] S101, for any current thermal infrared image in the thermal infrared image sequence of the target scene: register the current thermal infrared image to the visible light image to generate the current quasi-thermal infrared image; use the PSFusion network to perform image fusion processing on the current quasi-thermal infrared image and the current visible light image to generate the current fused image;

[0026] S102, for any current key frame in the key frame sequence corresponding to the fused image sequence: perform visual inertial alignment processing on the current visual pose information and IMU pre-integration output result corresponding to the current key frame, and output the aligned image data.

[0027] S103, perform nonlinear optimization processing on the aligned image data of each current key frame in the key frame sequence according to the preset sliding window, and output the real-time pose state of the first device.

[0028] Here, the first device can be a drone or a humanoid robot.

[0029] In S101, the current thermal infrared image is registered to the visible light image based on the algorithm model or preset rules to generate the current quasi-thermal infrared image.

[0030] For example, image distortion correction is performed on the current thermal infrared image and the current visible light image to generate a distorted thermal infrared image and a distorted visible light image, respectively; the distorted thermal infrared image is downsampled to generate a thermal infrared image with the same resolution as the distorted visible light image; feature matching is performed on the visible light image and the thermal infrared image with the same resolution to generate visible light features and thermal infrared features with matching relationships; model estimation is performed based on the visible light features and the thermal infrared features to generate a homography matrix; and the current thermal infrared image is reprojected based on the homography matrix to generate a current quasi-thermal infrared image.

[0031] Specifically, due to the differences in imaging mechanisms, lenses, and resolutions between visible light cameras and thermal infrared cameras, pixel matching is required between the images to map similar features in the final fused image. Before registering the two types of images, distortion correction and sampling preprocessing are necessary. The purpose of image distortion correction is to establish a geometric model of the camera imaging based on the intrinsic, extrinsic, and distortion parameters obtained from camera calibration, determining the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image. Because of the differences in resolution and external environmental conditions between multimodal images, directly fusing images of different resolutions does not yield ideal results. To address the resolution mismatch between visible light and thermal infrared images, bilinear interpolation is used to upsample the thermal infrared image, making it conform to the size of the visible light image. Then, image registration is performed, which can be roughly divided into four steps, as follows: First, feature detection is performed on the visible light image and the thermal infrared image, mainly detecting the corresponding similar features to be fused; second, feature matching is performed, which matches the similar features detected in the image sequence; then, the model parameters are estimated, and the thermal infrared image transformation relationship is calculated using the visible light image as a reference image. This step is used to find the parameters to be used and the type of estimation model, such as the homography matrix; finally, the thermal infrared image is reprojected and registered to the visible light image. This step mainly uses the parameters estimated in the previous step for transformation and calculates the reprojection error.

[0032] After image registration is completed, thermal infrared and visible light images of the same resolution are obtained, and the scene content in the infrared and visible light images is consistent. Then, a fused image is obtained based on the PSFusion network algorithm. The main structure of the PSFusion network is based on the classic convolutional neural network (CNN), but a multi-module fusion mechanism is added to the basic structure to integrate the feature extraction results of different levels and different features.

[0033] In S102, a keyframe sequence corresponding to the fused image sequence is determined based on a model algorithm or preset rules. For example, for any fused image in the fused image sequence corresponding to the thermal infrared image sequence: feature extraction processing is performed on the fused image sequence corresponding to the thermal infrared image sequence based on the GhostNet network to generate extracted features; the extracted features are weighted based on a long-range attention mechanism to output a feature point cloud; each feature point in the feature point cloud is described to generate a descriptor corresponding to the feature point; based on the descriptors corresponding to the feature points, feature matching and tracking processing are performed on all fused images in the fused image sequence to generate a tracking feature point cloud; fused images with a number of tracking feature points greater than a preset threshold are selected as keyframes from the fused image sequence to obtain a keyframe sequence.

[0034] For example, SuperPoint is an end-to-end feature point and descriptor extraction network that employs an encoder-decoder structure similar to semantic segmentation networks. Its workflow involves taking a complete image as input, extracting deep features through a shared encoder, and then outputting feature points and their descriptors through two decoders. While the original SuperPoint network used a VGG-like encoder with a simple structure, its large number of layers and channels resulted in high computational cost and parameter count. To overcome the shortcomings of the original SuperPoint network's VGG architecture, particularly the high parameter count and computational cost, GhostNetV2 was used to replace the original VGG encoding layers. GhostNet's overall structure consists of GhostBottlenecks and other structures, significantly reducing computational requirements while maintaining accuracy. GhostNetV2 is formed by introducing a long-range attention mechanism based on GhostNet.

[0035] When GhostNetV2 processes an input image (e.g., 224×224×3), it first passes the image through a 3×3 convolutional block containing convolution, normalization, and activation functions. Multiple Ghost Bottlenecks are then stacked to obtain a 7×7×160 feature layer. Next, a 1×1 convolutional block is used to adjust the number of channels, resulting in a 7×7×960 feature layer. Global average pooling is then performed, followed by another 1×1 convolutional block to obtain a 1×1×1280 feature layer. Finally, a fully connected layer is used for classification. The advantage of this architecture is that GhostNetV2 introduces a long-range attention mechanism on top of GhostNet, further enhancing representation capabilities and achieving better performance while reducing the number of parameters.

[0036] For example, the current visual pose information of the first device is determined based on preset rules or model algorithms. Specifically, the tracking feature point cloud corresponding to the current keyframe is obtained; the tracking feature point cloud is subjected to visual SFM processing to generate a three-dimensional feature point cloud; and the current visual pose information of the first device is determined based on the three-dimensional feature point cloud.

[0037] Pre-integration is performed on the IMU data of the current keyframe and the previous keyframe adjacent to the current keyframe to generate the pose estimate of the first device (i.e., the IMU pre-integration output). The pose estimate of the first device at each current time step is then visually and inertially aligned to ensure that the scale information in pure vision matches the IMU pre-integration output. This combined approach improves the overall accuracy and robustness of the SFM system and has significant implications for practical applications.

[0038] Therefore, the VIO system front end uses a lightweight Superpoint network to complete feature point detection and descriptor decoding of the fused image, and then uses visual SFM (Structure From Motion) to obtain the current visual pose information of the initial keyframe; at the same time, the IMU pre-integration output is used for visual inertial alignment, so that the scale information in pure vision can be matched with the IMU measurement.

[0039] In S103, based on preset rules or algorithms, the aligned image data of each current key frame in the key frame sequence is nonlinearly optimized according to a preset sliding window, and the real-time pose state of the first device is output.

[0040] For example, the aligned image data includes: visual reprojection residual, IMU pre-integration residual, and pose estimation value; the step of performing nonlinear optimization processing on the aligned image data of each current key frame in the keyframe sequence according to a preset sliding window to output the real-time pose state of the first device includes: for any current key frame in the keyframe sequence: using a regularization factor and a weight momentum factor to correct the visual reprojection residual corresponding to the current key frame to obtain a corrected visual reprojection residual; constructing a BA optimization model based on the marginalization residuals corresponding to all current key frames in the preset sliding window and the corrected visual reprojection residuals and IMU pre-integration residuals corresponding to each current key frame; and using the BA optimization model to optimize the pose estimation value corresponding to each current key frame in the preset sliding window to generate the real-time pose state of the first device.

[0041] The application of robust BA optimization with a pre-defined sliding window in the VIO system aims to improve the accuracy of pose estimation while effectively handling outliers in dynamic scenes. The following section will provide a detailed explanation using the formulas of the robust BA optimization model.

[0042] ① Visual Inertia BA Optimization Model

[0043] In visual-inertial state estimation based on a visual-inertial navigation system (VINS), maximum a posteriori (MAP) estimation is achieved by minimizing the sum of the prior and Mahalanobis distance norms of all measurement residuals. The MAP optimization model for visual-inertial systems is defined as follows:

[0044]

[0045] Where, ρ H (·) represents the Huber loss function; r p , and These represent marginalization residuals, IMU pre-integration residuals, and visual reprojection residuals, respectively. For IMU observations and feature point observations; H p Let P represent the measurement estimation matrix for marginalization, and let X represent the covariance of each term. X represents the pose information of the current keyframe. As the proportion of outliers in a dynamic scene increases, the Huber loss function can no longer completely reject the residuals of outliers, and the system cannot function successfully. In other words, the Huber loss function is used to handle these residuals to enhance the model's robustness to outliers.

[0046] ② Regularization factor

[0047] For simplicity, item, The terms are omitted and represented as follows: To robustly estimate pose while rejecting anomalous features, a novel residual term inspired by Black-Rangarajan duality is constructed. :

[0048]

[0049] Among them, w j ∈[0,1] represents each feature f j The corresponding weights determine w j Features f close to 1 j It is a static feature; Φ(w) is a constant parameter; j ) is the weight w j The regularization factor is defined as follows:

[0050]

[0051] ③ Weighted momentum factor

[0052] When motion becomes intense, IMU pre-integration becomes inaccurate, leading to inaccurate pose estimation. In this case, the feature reprojection residuals of static objects become large; these features will be ignored during BA (Balanced Assortment) through regularization factors, even if the previous weights are close to 1, resulting in inaccurate BA results. An additional factor, the weight momentum factor, is constructed to ensure that the previously estimated feature weights are unaffected by intense motion.

[0053] Because the features are tracked continuously, each feature f j Use its previous weight n were performed j This is a secondary optimization. To ensure the current weights tend to remain at [a certain level / value]... And with n j The increase in the magnitude of this trend will increase the weighted momentum factor Ψ(w) j The design is as follows:

[0054]

[0055] The corrected visual reprojection residual can be obtained. as follows:

[0056]

[0057] in, This represents a constant parameter used to adjust the effect of the momentum factor on BA.

[0058] Adopting (5) Replacing the Huber norm in the visual reprojection residual term in (1), the robust BA optimization model can therefore be expressed as:

[0059]

[0060] Where, r p , and These represent marginalization residuals, IMU pre-integration residuals, and visual reprojection residuals, respectively. For IMU pre-integration output and current visual pose information; H p This represents the measurement estimation matrix for marginalization, where P represents the covariance of each term; Φ(w j ) is the weight w j The regularization factor is defined as Φ(w j ) = 1 - w j Ψ(w j ) is the weighted momentum factor, defined as

[0061] This embodiment addresses visual reprojection residuals by introducing feature weights and using pre-integrated IMU data to calculate IMU pre-integration residuals. Each feature is assigned a weight, which is updated and optimized by introducing a weight momentum factor and a regularization factor. During optimization, the weight momentum factor uses the weights of previously tracked features, while the regularization factor is adjusted based on the weights of all features within the current preset sliding window. This strategy is solved through alternating optimization. Since the pose estimation X of the current keyframe can be estimated using IMU pre-integration and the previously optimized state, the feature weights are first optimized based on the estimated state. Therefore, features with large visual reprojection residuals start with smaller weights to reduce their impact on the overall optimization. The optimization steps are repeated until both the state and weights converge. During this process, the weights of outlier features are reduced, making their losses smoother. The regularization factor, by adaptively adjusting the weights, effectively filters out outliers, but does not completely ignore them during optimization. By employing weights and regularization factors inspired by BR duality, the impact of features with high reprojection errors on pose estimation can be reduced while maintaining state estimation performance. This approach enhances the robustness of pose estimation, making the optimization process more stable and accurate, thereby improving the system's performance in complex environments. Overall, this optimization strategy demonstrates good performance in handling dynamic scenes and disturbances, contributing to improved accuracy and reliability of visual inertial odometry.

[0062] In this embodiment, after image registration of visible light and thermal infrared images, the images are input into the PSFusion network for image fusion. Then, based on the lightweight SuperPoint network, feature point clouds are extracted from the fused image, and matching and tracking are performed to generate tracking feature point clouds. Finally, within a sliding window, nonlinear optimization is used to perform tight coupling optimization on the visual pose information obtained from feature observation and the IMU pre-integration output results. As a result, not only can image information from different bands be fused to obtain a fused image with rich information across all scenes and all times, but the stability of feature extraction and matching in low-texture environments can also be improved. In scenarios with drastic changes in lighting or visual obstructions such as dust, fog, and smoke, high-precision positioning and mapping can be maintained, thereby improving the robustness and accuracy of the VIO system in complex environments.

[0063] Furthermore, this implementation uses a thermal infrared sensor in a VIO system, employing a lightweight SuperPoint neural network with a GhostNetV2 encoding structure to extract and match feature points from thermal infrared images. This reduces the computational load and parameter count of the VIO system while maintaining the accuracy of image feature extraction. By adding a regularization factor and a weight momentum factor to the BA optimization model within a preset sliding window, and using the weights of previously tracked features in the weight momentum factor and the weights of all features in the current window in the regularization factor, features of dynamic objects that significantly deviate from the motion prior can be discarded when estimating the pose of the first device, thereby improving the positioning accuracy of the first device.

[0064] like Figure 2 The diagram shown is a schematic representation of the process of image fusion processing of the current quasi-thermal infrared image and the current visible light image using a PSFusion network according to an embodiment of the present invention.

[0065] Using the PSFusion network to perform image fusion processing on the current quasi-thermal infrared image and the current visible light image, the process includes at least the following steps;

[0066] S201, perform feature fusion processing at different scales on the current visible light image and the current quasi-thermal infrared image respectively to generate surface detail fusion features and semantic fusion features;

[0067] S202, Perform sparse semantic perception processing on semantic fusion features to generate semantic information;

[0068] S203, Perform scene reconstruction processing on the surface detail fusion features to generate scene features;

[0069] S204, based on the channel-space attention mechanism, fuses semantic information and scene features to generate semantic scene fusion features;

[0070] S205, the semantic scene fusion feature is output as a fused image after passing through the image fusion path; or, the semantic scene fusion feature is processed by the scene fidelity path and then output as a fused image after passing through the image fusion path.

[0071] In S202, for example, feature extraction processing at different scales is performed on the current visible light image and the current quasi-thermal infrared image respectively to obtain first surface features and first residual features at different scales corresponding to the current visible light image; and second surface features and second residual features at different scales corresponding to the current quasi-thermal infrared image; surface detail fusion processing is performed on the first surface features, second surface features, first residual features at a specified scale, and second residual features at a specified scale to generate surface detail fusion features; deep semantic fusion processing is performed on the first residual features and second residual features at different scales to generate semantic fusion features.

[0072] Specifically, after image registration is completed, thermal infrared and visible light images of the same resolution can be obtained, and the scene content in the thermal infrared and visible light images is consistent. The fused image is then obtained using the PSFusion network algorithm, the PSFusion network framework of which is as follows: Figure 3 As shown, the PSFusion network comprises a fusion scene reconstruction branch and a sparse semantic prediction branch, both of which share a multi-scale feature extraction network. The PSFusion network first extracts sufficient semantic features through the sparse semantic prediction branch, and then uses a feature-level semantic injection module to progressively inject semantic information constrained by multiple semantic perception tasks into the fusion scene reconstruction branch. Simultaneously, the scene fidelity path in the fusion scene reconstruction branch is designed to ensure that the fused features contain the complete information required to reconstruct the source image. Furthermore, contrast masks and salient object masks are used to construct the fusion loss to maintain the visually comprehensive information of the fusion result. This results in a fused image containing rich semantic information, which is friendly and robust to high-level vision tasks, demonstrating the potential of image-level fusion in high-level vision tasks and better meeting semantic requirements. The scene restoration branch includes a scene fidelity path and an image fusion path. The two paths share the Continuous Progressive Semantic Injection Module (PSIM), Dense Scene Reconstruction Module (DSRM), Semantic Injection Module (SIM), and Dense Scene Reconstruction Module. The sparse semantic prediction branch consists of the Sparse Semantic Awareness Module (S2PM) and the Sparse Semantic Awareness Path (S2P2). S2P2 consists of three task-specific output heads used to perceive sparse semantics from different perspectives. The semantic information output by the sparse semantic prediction branch includes: boundary segmentation results, semantic segmentation results, and binary segmentation results.

[0073] Considering that shallow features contain rich details and structural information, a shallow feature fusion module (SDFM) based on a channel-spatial attention mechanism is used to integrate shallow features. This module concatenates infrared and visible features in the channel dimension and then feeds them into a channel attention module consisting of convolution and pooling operations to generate attention weights. These weights are then used to weight the original features through element-wise multiplication, and the resulting features are added to the original features of another branch to enhance their representation. The enhanced features are concatenated along the channel dimension and fed into parallel channel attention and spatial attention modules to generate the final fusion weights. A deep semantic fusion module (PSFM) based on cross-attention is then used to integrate the deep features.

[0074] This embodiment first uses the PSFusion network algorithm to obtain a real-time, all-day, and information-rich fused image of visible light and thermal infrared light. Then, it performs VIO state estimation based on the information-rich fused image of visible light and thermal infrared light. This can improve the stability of feature extraction and matching in low-texture environments and achieve accurate and robust localization and mapping.

[0075] The visual inertial odometry method based on multimodal image fusion provided in this embodiment will be described in detail below with reference to specific application scenarios.

[0076] The visual-inertial odometry method based on multimodal image fusion includes at least the following steps:

[0077] S1, for any current thermal infrared image in the thermal infrared image sequence of the target scene: perform image distortion correction processing on the current thermal infrared image and the current visible light image respectively to generate a distorted thermal infrared image and a distorted visible light image; perform downsampling processing on the distorted thermal infrared image to generate a thermal infrared image with the same resolution as the distorted visible light image; perform feature matching processing on the visible light image and the thermal infrared image with the same resolution respectively to generate visible light features and thermal infrared features with matching relationship; perform model estimation based on the visible light features and the thermal infrared features to generate a homography matrix; perform reprojection transformation processing on the current thermal infrared image based on the homography matrix to generate a current quasi-thermal infrared image.

[0078] S2, perform feature extraction processing at different scales on the current visible light image and the current quasi-thermal infrared image respectively to obtain first surface features and first residual features at different scales corresponding to the current visible light image; and second surface features and second residual features at different scales corresponding to the current quasi-thermal infrared image; perform surface detail fusion processing on the first surface features, second surface features, first residual features at a specified scale, and second residual features at a specified scale to generate surface detail fusion features; perform deep semantic fusion processing on the first residual features and second residual features at different scales to generate semantic fusion features. Perform sparse semantic perception processing on the semantic fusion features to generate semantic information; perform scene reconstruction processing on the surface detail fusion features to generate scene features; perform fusion processing on the semantic information and the scene features based on a channel-spatial attention mechanism to generate semantic scene fusion features; and output a fused image after the semantic scene fusion features are processed through an image fusion path.

[0079] S3, for any fused image in the fused image sequence corresponding to the thermal infrared image sequence: Feature extraction processing is performed on the fused image sequence corresponding to the thermal infrared image sequence based on the GhostNet network to generate extracted features; the extracted features are weighted based on a long-range attention mechanism to output a feature point cloud; each feature point in the feature point cloud is described to generate a descriptor corresponding to the feature point. Based on the descriptors corresponding to the feature points, feature matching and tracking processing are performed on all fused images in the fused image sequence to generate a tracking feature point cloud; fused images with a number of tracking feature points greater than a preset threshold are selected as keyframes to obtain a keyframe sequence.

[0080] S4, for any current keyframe in the keyframe sequence corresponding to the fused image sequence: obtain the tracking feature point cloud corresponding to the current keyframe; perform visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; determine the current visual pose information of the first device based on the three-dimensional feature point cloud; perform visual inertial alignment processing on the current visual pose information corresponding to the current keyframe and the IMU pre-integration output result to output aligned image data; wherein, the aligned image data includes: visual reprojection residual, IMU pre-integration residual, and pose estimation value. The visual reprojection residual corresponding to the current keyframe is corrected using a regularization factor and a weight momentum factor to obtain a corrected visual reprojection residual; a BA optimization model is constructed based on the marginalization residuals corresponding to all current keyframes within the preset sliding window and the corrected visual reprojection residuals and IMU pre-integration residuals corresponding to each current keyframe;

[0081] S5 uses the BA optimization model to optimize the pose estimation value corresponding to each current keyframe within the preset sliding window, generating the real-time pose state of the first device.

[0082] like Figure 4 The diagram shown is a structural schematic of a visual inertial odometry device based on multimodal image fusion according to an embodiment of the present invention.

[0083] A visual inertial odometry device based on multimodal image fusion is applied to a first device. The device 400 includes: an image fusion processing module 401, used for any current thermal infrared image in a thermal infrared image sequence of a target scene: registering the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image; and using a PSFusion network to perform image fusion processing on the current quasi-thermal infrared image and the current visible light image to generate a current fused image; a visual inertial alignment processing module 402, used for any current keyframe in a keyframe sequence corresponding to the fused image sequence: performing visual inertial alignment processing on the current visual pose information corresponding to the current keyframe and the IMU pre-integration output result of the current keyframe, and outputting aligned image data; and a nonlinear optimization processing module 403, used for performing nonlinear optimization processing on the aligned image data of each current keyframe in the keyframe sequence according to a preset sliding window, and outputting the real-time pose state of the first device.

[0084] In a preferred embodiment of this example, the image fusion processing module includes: an image distortion correction unit, used to perform image distortion correction processing on the current thermal infrared image and the current visible light image respectively, generating a distortion-corrected thermal infrared image and a distortion-corrected visible light image; a sampling unit, used to perform downsampling processing on the distortion-corrected thermal infrared image, generating a thermal infrared image with the same resolution as the distortion-corrected visible light image; a feature matching processing unit, used to perform feature matching processing on the visible light image and the thermal infrared image with the same resolution respectively, generating visible light features and thermal infrared features with matching relationship; a model estimation unit, used to perform model estimation based on the visible light features and the thermal infrared features, generating a homography matrix; and a reprojection transformation processing unit, used to perform reprojection transformation processing on the current thermal infrared image based on the homography matrix, generating a current quasi-thermal infrared image.

[0085] In a preferred embodiment of this example, the image fusion processing module further includes: a feature fusion processing unit, used to perform feature fusion processing on the current visible light image and the current quasi-thermal infrared image at different scales to generate surface detail fusion features and semantic fusion features; a semantic sensing unit, used to perform sparse semantic perception processing on the semantic fusion features to generate semantic information; a scene reconstruction unit, used to perform scene reconstruction processing on the surface detail fusion features to generate scene features; an information fusion processing unit, used to perform fusion processing on the semantic information and the scene features based on a channel-spatial attention mechanism to generate semantic scene fusion features; and an output unit, used to output a fused image after the semantic scene fusion features are processed through an image fusion path; or, the semantic scene fusion features are processed through a scene fidelity path and then output through an image fusion path to generate a fused image.

[0086] In a preferred embodiment of this example, the image fusion processing module further includes: a first generation unit, which performs feature extraction processing on the current visible light image and the current quasi-thermal infrared image at different scales to obtain first surface features and first residual features at different scales corresponding to the current visible light image; and second surface features and second residual features at different scales corresponding to the current quasi-thermal infrared image; a second generation unit, which performs surface detail fusion processing on the first surface features, second surface features, first residual features at a specified scale, and second residual features at a specified scale to generate surface detail fusion features; and a third generation unit, which performs deep semantic fusion processing on the first residual features and second residual features at different scales to generate semantic fusion features.

[0087] In a preferred embodiment of this invention, the device further includes: a feature extraction module, used to perform feature extraction processing on the fused image sequence corresponding to the thermal infrared image sequence based on a lightweight SuperPoint network, generating a feature point cloud and a descriptor corresponding to each feature point; a first generation module, used to perform feature matching and tracking processing on all fused images in the fused image sequence based on the descriptors corresponding to the feature points, generating a tracking feature point cloud; and a selection module, used to select fused images from the fused image sequence with a number of tracking feature points greater than a preset threshold as keyframes, thereby obtaining a keyframe sequence.

[0088] In a preferred embodiment of this invention, the device further includes: an acquisition module for acquiring the tracking feature point cloud corresponding to the current keyframe; a second generation module for performing visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; and a determination module for determining the current visual pose information of the first device based on the three-dimensional feature point cloud.

[0089] In a preferred embodiment of this example, the aligned image data includes: visual reprojection residual, IMU pre-integration residual, and pose estimation value; the nonlinear optimization processing module includes: a correction processing unit, used to correct the visual reprojection residual corresponding to any current key frame in the key frame sequence using a regularization factor and a weight momentum factor to obtain a corrected visual reprojection residual; a construction unit, used to construct a BA optimization model based on the marginalization residuals corresponding to all current key frames within the preset sliding window and the corrected visual reprojection residual and IMU pre-integration residual corresponding to each current key frame; and an optimization processing unit, used to optimize the pose estimation value corresponding to each current key frame within the preset sliding window using the BA optimization model to generate the real-time pose state of the first device.

[0090] In a preferred embodiment of this example, the feature extraction module includes: a feature extraction unit, configured to perform feature extraction processing on any fused image in the fused image sequence corresponding to the thermal infrared image sequence based on a GhostNet network to generate extracted features; a weighted processing unit, configured to perform weighted processing on the extracted features based on a long-distance attention mechanism to output a feature point cloud; and a description processing unit, configured to perform description processing on each feature point in the feature point cloud to generate a descriptor corresponding to the feature point.

[0091] The above-described apparatus can execute the visual inertial odometry method based on multimodal image fusion provided in an embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the visual inertial odometry method based on multimodal image fusion. Technical details not described in detail in this embodiment can be found in the visual inertial odometry method based on multimodal image fusion provided in an embodiment of the present invention.

[0092] The present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the visual inertial odometry method based on multimodal image fusion described in the present invention.

[0093] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0094] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0095] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to the following embodiments of this application described in the "Exemplary Methods" section above.

[0096] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0097] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0098] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0099] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0100] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0101] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0102] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0103] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0104] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual-inertial odometer method based on multi-modal image fusion, characterized in that, The application is applied to a first device and comprises the following steps: For any current thermal infrared image in a thermal infrared image sequence of a target scene: registering the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image; performing image fusion processing on the current quasi-thermal infrared image and a current visible light image by using a PSFusion network to generate a current fusion image; Performing feature extraction processing on a fusion image sequence corresponding to the thermal infrared image sequence by using a lightweight SuperPoint network to generate a feature point cloud and a descriptor corresponding to each feature point; performing feature matching and tracking processing on all fusion images in the fusion image sequence based on the descriptor corresponding to each feature point to generate a tracking feature point cloud; selecting a fusion image with a number of tracking feature points greater than a preset threshold from the fusion image sequence as a key frame to obtain a key frame sequence; For any current key frame in the key frame sequence corresponding to the fusion image sequence: performing visual-inertial alignment processing on current visual pose information and an IMU pre-integration output result corresponding to the current key frame to output aligned image data; Performing nonlinear optimization processing on the aligned image data of each current key frame in the key frame sequence according to a preset sliding window to output a real-time pose state of the first device.

2. The method of claim 1, wherein, The registration of the current thermal infrared image to the visible light image to generate the current quasi-thermal infrared image comprises the following steps: Performing image de-distortion processing on the current thermal infrared image and the current visible light image respectively to generate a de-distorted thermal infrared image and a de-distorted visible light image; Performing up-sampling processing on the de-distorted thermal infrared image to generate a thermal infrared image with the same resolution as the de-distorted visible light image; Performing feature matching processing on the visible light image and the thermal infrared image with the same resolution respectively to generate visible light features and thermal infrared features with a matching relationship; Performing model estimation based on the visible light features and the thermal infrared features to generate a homography matrix; Performing re-projection transformation processing on the current thermal infrared image based on the homography matrix to generate the current quasi-thermal infrared image.

3. The method of claim 1, wherein, The image fusion processing on the current quasi-thermal infrared image and the current visible light image by using the PSFusion network to generate the current fusion image comprises the following steps: Performing feature fusion processing of different scales on the current visible light image and the current quasi-thermal infrared image respectively to generate surface detail fusion features and semantic fusion features; Performing sparse semantic perception processing on the semantic fusion features to generate semantic information; Performing scene reconstruction processing on the surface detail fusion features to generate scene features; Performing fusion processing on the semantic information and the scene features based on a channel-space attention mechanism to generate semantic scene fusion features; The semantic scene fusion features are output as a fusion image via an image fusion path, or the semantic scene fusion features are processed via a scene fidelity path and then output as a fusion image via an image fusion path.

4. The method of claim 3, wherein, The application further comprises the following steps: respectively, to the current visible light image and the current quasi-thermal infrared image to obtain different scale first surface features and different scale first residual features corresponding to the current visible light image; and different scale second surface features and different scale second residual features corresponding to the current quasi-thermal infrared image; performing surface detail fusion processing on the different scale first surface features, the different scale second surface features, the specified scale first residual features, and the specified scale second residual features to generate surface detail fusion features; performing depth semantic fusion processing on the different scale first residual features and the different scale second residual features to generate semantic fusion features.

5. The method of claim 1, wherein, Further comprising: obtaining a tracking feature point cloud corresponding to the current key frame; performing visual SFM processing on the tracking feature point cloud to generate a three-dimensional feature point cloud; determining current visual pose information of the first device based on the three-dimensional feature point cloud.

6. The method of claim 1, wherein, The aligned image data includes visual re-projection residuals, IMU pre-integration residuals, and pose estimates; the nonlinear optimization processing of the aligned image data of each current key frame in the key frame sequence according to the preset sliding window outputs the real-time pose state of the first device; including: For any current key frame in the key frame sequence: using a regularization factor and a weight momentum factor to correct the visual re-projection residuals corresponding to the current key frame to obtain corrected visual re-projection residuals; Based on the marginalization residuals corresponding to all current key frames within the preset sliding window and the corrected visual re-projection residuals and the IMU pre-integration residuals corresponding to each current key frame, a BA optimization model is constructed; using the BA optimization model to optimize the pose estimates corresponding to each current key frame within the preset sliding window to generate the real-time pose state of the first device.

7. The method of claim 1, wherein, The light-weight SuperPoint network performs feature extraction processing on the fusion image sequence corresponding to the thermal infrared image sequence to generate a feature point cloud and a descriptor corresponding to each feature point; including: For any fusion image in the fusion image sequence corresponding to the thermal infrared image sequence: performing feature extraction processing on the fusion image sequence corresponding to the thermal infrared image sequence based on the GhostNet network to generate extracted features; performing weighting processing on the extracted features based on a long-distance attention mechanism to output a feature point cloud; performing description processing on each feature point in the feature point cloud to generate a descriptor corresponding to the feature point.

8. A visual-inertial odometry device based on multi-modal image fusion, characterized in that, Applied to a first device; including: An image fusion processing module is configured to, for any current thermal infrared image in a thermal infrared image sequence of a target scene: register the current thermal infrared image to a visible light image to generate a current quasi-thermal infrared image; and perform image fusion processing on the current quasi-thermal infrared image and a current visible light image using a PSFusion network to generate a current fusion image; The feature extraction module is configured to perform feature extraction processing on the fusion image sequence corresponding to the thermal infrared image sequence based on a lightweight SuperPoint network, to generate a feature point cloud and a descriptor corresponding to each feature point; The first generation module is configured to perform feature matching and tracking processing on all fusion images in the fusion image sequence based on the descriptor corresponding to the feature point, to generate a tracking feature point cloud; The selection module is configured to select a fusion image with a number of tracking feature points greater than a preset threshold from the fusion image sequence as a key frame, to obtain a key frame sequence; The visual-inertial alignment processing module is configured to, for any current key frame in the key frame sequence corresponding to the fusion image sequence: perform visual-inertial alignment processing on current visual pose information corresponding to the current key frame and an IMU pre-integration output result, to output aligned image data; The nonlinear optimization processing module is configured to perform nonlinear optimization processing on the aligned image data of each current key frame in the key frame sequence according to a preset sliding window, to output a real-time pose state of the first device. 9.A computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images