A visual-inertial localization method based on cross-modal matching of visible light and infrared light

Abstract

A visual-inertial localization method based on cross-modal matching of visible light and infrared, belonging to the field of multi-source fusion navigation technology, is presented. This invention addresses the problems of poor localization performance and loop closure detection failure in existing technologies under visual degradation environments. It combines the rich texture features of visible light cameras with the illumination-independent characteristics of infrared cameras to construct a visual-inertial localization system based on cross-modal matching of visible light and infrared. A common deep neural network extracts visual feature points and descriptors from visible light and thermal infrared images respectively, providing feature confidence scores and proposing a classification management method for cross-modal feature points. Then, the depth of landmark points is recovered using two-view triangulation, and loop closure detection is performed based on the cosine similarity of bag-of-words vectors. The position and attitude of the visible light and infrared cameras are jointly optimized using depth estimates, loop closure detection results, and inertial pre-integration measurements. This method can be applied to the field of multi-source fusion navigation.

Description

Technical Field

[0001] This invention belongs to the field of multi-source fusion navigation technology, specifically relating to a visual inertial positioning method that uses visible light and infrared cross-modal matching. Background Technology

[0002] Visual-inertial oscillatory (VIO) positioning is a positioning technique that combines visual and inertial information, and it is widely used in robotics, autonomous driving, augmented reality (AR), and virtual reality (VR). Traditional VIO methods typically rely on visual input from a single modality (such as visible light images). However, factors such as lighting variations, occlusion, and dynamic objects in complex environments can lead to the loss of information from the visible light modality, thus affecting positioning performance. Therefore, infrared images, with their unique advantages under low-light conditions, are being used for cross-modal matching with visible light images, making it an important research direction for enhancing the robustness of VIO.

[0003] Patent application CN117036218A discloses a method for fusing infrared and visible light images based on discrete wavelet transform. This method extracts approximate and detail components from both infrared and visible light images by performing discrete wavelet decomposition on each. Subsequently, a monochrome thresholding fusion strategy is used to perform a weighted average of these two components to generate the fused result. Finally, an inverse discrete wavelet transform is performed on the weighted fusion result to generate the final fused image, and the camera's motion state is estimated based on a visual odometry method and this fused image.

[0004] Patent application CN116977369A employs two-dimensional discrete wavelet transform to decompose infrared and visible light images of the environment separately, obtaining approximate and detail components. Subsequently, a monochrome thresholding fusion technique is used to weight the approximate and detail components of the infrared and visible light images to generate a fused image. Next, an improved R-CNN network is designed and trained using the fused image as input data to obtain a prediction model. Finally, this prediction model is used to process the test data, inferring the camera's motion trajectory to complete the visual odometry task.

[0005] The paper (Qin L, Wu C, Kong X, et al. BVT-SLAM: A Binocular Visible-Thermal Sensors SLAM System in Low-Light Environments[J].IEEE Sensors Journal,2024,24(7):11599-11609.) proposes a binocular visible-infrared SLAM system (BVT-SLAM). This system improves the BRIEF descriptor and, based on a binocular system composed of a visible light camera and an infrared camera, can adaptively switch between monocular and binocular modes. It also proposes a highly efficient method for matching visible light and thermal image feature points. Based on BDBRIEF, it can effectively complete multispectral stereo matching in the BVT-SLAM system, while enabling the system to run in real time. However, the improved BRIEF descriptor is a manually tuned descriptor, which is difficult to cover all possible variations and anomalies, resulting in poor generalization ability of the model in new scenes.

[0006] In summary, although existing technologies have made some progress in the fusion of infrared and visible light images, problems such as poor positioning performance and loop closure detection failure still exist in visual degradation environments due to the failure of visible light cameras. Therefore, it is necessary to propose a new visual inertial positioning method based on cross-modal matching of visible light and infrared to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to solve the problems of poor positioning performance and loop closure detection failure in existing technologies under visual degradation environments, and to propose a visual inertial positioning method based on cross-modal matching of visible light and infrared.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a visual inertial positioning method for cross-modal matching of visible light and infrared, the method specifically including the following steps:

[0009] Step 1: Trigger the visible light camera, infrared camera, and IMU measurement unit with the same pulse signal. The visible light camera and infrared camera are both installed on the unmanned system. Then read the images acquired by the visible light camera, the images acquired by the infrared camera, and the IMU measurement sequence from the IMU measurement unit after triggering.

[0010] Using the timestamps of images from the visible light camera as a reference, a set of IMU measurement sequences is obtained from the timestamp of the previous visible light camera image to the timestamp of the current visible light camera image. If the time difference between the current visible light image and the current infrared image is within t0, the current visible light image, the current infrared image, and the obtained set of IMU measurement sequences are packaged into a data set and processed in binocular inertial mode, i.e., step 2 is continued. If the time difference between the current visible light image and the current infrared image is greater than or equal to t0, the image with the earlier timestamp is selected from the current visible light image and the current infrared image and processed in monocular inertial mode.

[0011] Step 2: Use a common deep neural network to extract the descriptor of each pixel in the current frame visible light image and the current frame infrared image, and determine the feature points in the current frame visible light image and the current frame infrared image based on the descriptor score;

[0012] Step 3: Determine whether the current frame of the visible light image is a key frame based on the visible light key frame image in the sliding window, and determine whether the current frame of the infrared image is a key frame based on the infrared key frame image in the sliding window.

[0013] If neither the current frame's visible light image nor the current frame's infrared image is a keyframe, then no processing is required for the current frame's visible light image and the current frame's infrared image, and no addition to the current frame's visible light image and the current frame's infrared image to the sliding window is needed. Return to step 1.

[0014] If the current frame's visible light image or the current frame's infrared image is a keyframe, then the current frame's visible light image is added as a keyframe to the sliding window, and the current frame's infrared image is added as a keyframe to the sliding window.

[0015] The feature points in the current frame's visible light image and the current frame's infrared image are matched according to the descriptor to obtain the feature point matching results. Based on the feature point matching results, all binocular landmark points in the current frame's visible light image and the current frame's infrared image are obtained. Feature point matching is performed between the current frame's visible light image and the previous frame's keyframe, and feature point matching is performed between the current frame's infrared image and the previous frame's keyframe, to obtain the corresponding monocular landmark points in the current frame's visible light image and the current frame's infrared image.

[0016] The reprojection errors of the binocular landmarks on the current frame visible light image and the current frame infrared image, the reprojection errors of the monocular landmarks corresponding to the current frame visible light image on the current frame visible light image and the key frame of the previous visible light image, and the reprojection errors of the monocular landmarks corresponding to the current frame infrared image on the current frame infrared image and the key frame of the previous infrared image are calculated respectively.

[0017] Continue to step 4;

[0018] Step 4: Construct bag-of-words vectors for the visible light image and the infrared image of the current frame, respectively; specifically:

[0019] Create a visual vocabulary V containing K visual words. For any feature point in the current frame of the visible light image, calculate the cosine distance between the descriptor of the feature point and each visual word in the visual vocabulary V. Assign the visual word corresponding to the minimum cosine distance to the feature point. Similarly, assign visual words to each feature point in the current frame of the visible light image. Then count the number of times each visual word appears in the current frame of the visible light image. Use the number of times each visual word appears to form the bag-of-words vector of the current frame of the visible light image.

[0020] Similarly, the bag-of-words vector of the current frame infrared image is obtained;

[0021] Step 5: Construct an IMU pre-integration constraint model using the set of IMU measurement sequences located between the timestamp of the previous visible light image keyframe and the timestamp of the current visible light image.

[0022] Step 6: Perform loop closure detection based on the bag-of-words vectors corresponding to the historical visible light image keyframes, historical infrared image keyframes, and the current frame visible light image within the sliding window, and obtain the loop closure detection results.

[0023] Step 7: Establish a global objective function based on the reprojection error, IMU pre-integration constraint model, and loop closure detection results; solve the global objective function to obtain the visual inertial positioning results of the visible light camera and the infrared camera.

[0024] Return to step 1.

[0025] Furthermore, the common deep neural network includes a first adaptor, a second adaptor, an encoder, and a decoder; wherein:

[0026] The input to the first adaptor is the visible light image of the current frame. The output of the first adaptor is used as the input to the encoder. The encoder outputs the descriptor of each pixel in the visible light image of the current frame. The output of the encoder is used as the input to the decoder. The decoder outputs the feature score of each pixel in the visible light image of the current frame.

[0027] The input to the second adaptor is the current frame infrared image. The output of the second adaptor is used as the input to the encoder. The encoder outputs the descriptor of each pixel in the current frame infrared image. The output of the encoder is then used as the input to the decoder. The decoder outputs the feature score of each pixel in the current frame infrared image.

[0028] The loss function used during training of a common deep neural network for:

[0029]

[0030] in, For the descriptor loss function, For peak loss function, Let λ be the feature score loss function, and λ be a hyperparameter.

[0031]

[0032] in, Represents all pixels in a visible light image. Find the expected value, D N and D′ N These represent descriptor subsets of pixels in visible light and infrared images, respectively.

[0033]

[0034] Among them, {d i ,d i′} is a descriptor for a pair of matched pixels in a visible light image and an infrared image, d j It is set D N In and d i The most recent descriptor, d k It is set D′ N In and d i′ The most recent descriptor, d m It is set D′ N In and d i The most recent non-matching descriptor, d n It is set D N In and d i′ The nearest non-matching descriptor, θ(·,·) represents the angle between the two descriptor vectors;

[0035] c(s i ,s i ′)=s i s i′

[0036] Among them, s i It is the d output of the decoder. i The score, s i′ It is the d output of the decoder. i′ The score;

[0037]

[0038] Where p represents a sliding image patch, D N [p[i]] represents the descriptor of the i-th pixel within the sliding image block p in the visible light image, D′ w D′ represents the infrared image after affine transformation of w. w [p[i]] represents the descriptor of the i-th pixel within the sliding image block p in the infrared image after the affine transformation of w. For D of all pixels within the sliding image block p N [p[i]] T D′ w [p[i]] Calculate the expectation, where S is the score set of descriptors for all pixels in the visible light image, S′ w S is the score set of all pixels in the infrared image after the affine transformation of w, and S[p] is the score matrix of pixels in the sliding image block p in set S. w [p] is the set S′ w The score matrix of pixels within a sliding image block p; For all sliding image blocks, b(p;D) N ,D′ w )×(1-S[p] T S′ w [p]) Calculate the expected value;

[0039]

[0040] Where I is the edge image of the visible light image, AP represents the average pooling layer, MP represents the max pooling layer, AP(S) represents the output of S after the average pooling layer, AP(S)[i] represents the value of the i-th pixel in AP(S), MP(S) represents the output of S after the max pooling layer, and (1-MP(S))[i] represents the value of the i-th pixel in 1-MP(S). Let AP(S)[i] represent the sum of all pixels in the visible light image. 2 +(1-MP(S))[i]) 2 Find the expectation, where S[i] represents the score of the descriptor of the i-th pixel in S, and M(I) is the processing result of the edge image I. Represents all pixels Seeking expectations;

[0041]

[0042] in, It is the Laplacian operator for images, ||·|| denotes the calculation of the 2-norm. express The value of the i-th pixel in the function x represents the independent variable of the function.

[0043] Further, the step of determining the feature points in the current frame's visible light image and the current frame's infrared image based on the descriptor score specifically involves:

[0044] For the current frame of the visible light image, the descriptor scores of each pixel in the visible light image are sorted from largest to smallest. The top 20% of pixels are then used as feature points, and the number of feature points c is compared with the threshold value.

[0045] If the number of feature points c is less than the set threshold, then continue to select the pixels with the highest descriptor scores from the sorting results as feature points until the number of feature points reaches the set threshold; if the number of feature points c is greater than or equal to the set threshold, then directly select the top 20% of the pixels as feature points.

[0046] Similarly, feature points in the current frame of the infrared image are obtained.

[0047] Furthermore, the specific process of step 3 is as follows:

[0048] Step 31: Determine whether the time difference between the current visible light image and the keyframe of the previous visible light image exceeds the time threshold;

[0049] If the time threshold is not exceeded, the visible light image of the current frame is not a key frame;

[0050] If the time threshold is exceeded, feature matching is performed between the current visible light image and the keyframe of the previous visible light image to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current visible light image is added to the keyframe list of visible light images; otherwise, the current visible light image is not a keyframe.

[0051] Determine whether the time difference between the current frame of the infrared image and the keyframe of the previous infrared image exceeds the time threshold.

[0052] If the time threshold is not exceeded, the current frame of infrared image is not a key frame;

[0053] If the time threshold is exceeded, feature matching is performed between the current frame infrared image and the keyframe of the previous infrared image to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current frame infrared image is added to the keyframe list of infrared images; otherwise, the current frame infrared image is not a keyframe.

[0054] If neither the current frame's visible light image nor the current frame's infrared image is a keyframe, then no processing is required for the current frame's visible light image and the current frame's infrared image, and no addition to the current frame's visible light image and the current frame's infrared image to the sliding window is needed. Return to step 1.

[0055] If the current frame's visible light image or the current frame's infrared image is a keyframe, then the current frame's visible light image is added as a keyframe to the sliding window, and the current frame's infrared image is added as a keyframe to the sliding window.

[0056] Then proceed to step 32;

[0057] Step 32: Use the similar projection matching method to match the feature points in the current frame's visible light image and the current frame's infrared image to obtain all successfully matched feature point pairs;

[0058] Step 33: Obtain the points on the map corresponding to each successfully matched feature point pair in Step 32. For any successfully matched feature point pair (x1, y1) and (x2, y2), denote the points on the map corresponding to the feature point pair as p. Calculate the reprojection error of point p on the current frame of the visible light image based on the keyframe of the visible light image in the sliding window, and calculate the reprojection error of point p on the current frame of the infrared image based on the keyframe of the infrared image in the sliding window.

[0059] If the reprojection error of point p on both the visible light image and the infrared image of the current frame is less than 2 pixels, then p will be added to the set of binocular landmarks as a binocular landmark point.

[0060] Then, each pair of feature points that were successfully matched in step 31 is treated as a monocular landmark, and the monocular landmark is added to the monocular landmark set.

[0061] Step 34: Calculate the reprojection error of each monocular landmark corresponding to the current visible light image on the current visible light image and the keyframe of the previous visible light image, the reprojection error of each monocular landmark corresponding to the current infrared image on the current infrared image and the keyframe of the previous infrared image, and the reprojection error of each binocular landmark corresponding to the current visible light image and the current infrared image on the current visible light image and the current infrared image, respectively.

[0062] Furthermore, the specific process of step 32 is as follows:

[0063] For any feature point a in the visible light image of the current frame, the feature point a is transformed and projected onto the infrared image of the current frame using calibration parameters. The feature point b with the smallest descriptor distance to feature point a is found in the infrared image of the current frame. Then, the feature point b is transformed and projected onto the visible light image of the current frame using calibration parameters. The feature point with the smallest descriptor distance to feature point b is found in the visible light image of the current frame.

[0064] If feature point a has the smallest descriptor distance to feature point b in the current frame of the visible light image, then feature point a and feature point b are successfully matched.

[0065] If the feature point with the smallest descriptor distance to feature point b in the current frame of the visible light image is not a, then feature point a and feature point b will not be matched.

[0066] Similarly, each feature point on the visible light image of the current frame is processed separately to obtain all successfully matched feature point pairs.

[0067] Further, obtaining the points on the map corresponding to each successfully matched feature point pair in step 32 specifically involves:

[0068] For any pair of successfully matched feature points (x1, y1) and (x2, y2):

[0069]

[0070] Where (x1, y1) and (x2, y2) are the coordinates of feature points on the visible light image and the infrared image of the current frame, respectively. This represents the j-th row of the projection matrix corresponding to the visible light image of the current frame. This represents the j-th row of the projection matrix corresponding to the current frame infrared image, and p is the landmark point on the map corresponding to (x1, y1) and (x2, y2).

[0071] Similarly, the points corresponding to each successfully matched feature point pair in step 32 are obtained on the map.

[0072] Further, in step 34, the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image is calculated on the current frame visible light image and the current frame infrared image. The specific process is as follows:

[0073] For the pixels (x1, y1) and (x2, y2) corresponding to the stereo landmark point p, transform the stereo landmark point p into the visible light camera coordinate system, and label the coordinates of point p in the visible light camera coordinate system as follows:

[0074]

[0075] in, Let be a rotation matrix. It is a translation matrix;

[0076]

[0077] in, This is the extrinsic parameter matrix between the visible light camera and the IMU. It is the rotation matrix of the k-th visible light image keyframe within the sliding window relative to the (k-1)-th visible light image keyframe within the sliding window. It is the translation matrix of the k-th visible light image keyframe within the sliding window relative to the (k-1)-th visible light image keyframe within the sliding window;

[0078] Will Incorporate the pinhole camera model π(·):

[0079]

[0080] In the formula, f x and f y These represent the two focal lengths of a visible light camera, c and c. x and c y These represent the optical center offsets of the two axes of a visible light camera. express The coordinates projected onto the visible light image, x Z express Z-axis direction coordinates;

[0081] Calculate a reprojection error of the binocular landmark point p based on the visible light pixel (x1, y1):

[0082]

[0083] in, This is due to reprojection error;

[0084] Transform the binocular landmark point p to the infrared camera coordinate system, and label the coordinates of point p in the infrared camera coordinate system as follows:

[0085]

[0086] in, Let be a rotation matrix. It is a translation matrix;

[0087]

[0088] in, This is the extrinsic parameter matrix between the infrared camera and the IMU. It is the rotation matrix of the k-th infrared image keyframe within the sliding window relative to the (k-1)-th infrared image keyframe within the sliding window. It is the translation matrix of the k-th infrared image keyframe within the sliding window relative to the (k-1)-th infrared image keyframe within the sliding window;

[0089] Will Incorporate the pinhole camera model π(·):

[0090]

[0091] In the formula, fx ′ and f y ′ represent the two focal lengths of the infrared camera, and c represents the focal length of the infrared camera. x ′ and c′ y These represent the two-axis optical center offsets of the infrared camera. express The coordinates projected onto the infrared image, x′ Z express Z-axis direction coordinates;

[0092] Calculate another reprojection error of the binocular landmark point p based on (x2, y2):

[0093]

[0094] in, This is the reprojection error.

[0095] Furthermore, the specific process of step 5 is as follows:

[0096] Step 51: Calculate the pre-integral variable relationship between the IMU measurement sequence at time i′ and the IMU measurement sequence at time j′:

[0097]

[0098] Where ~ represents the measured value, It is the measured value of IMU acceleration in the IMU measurement sequence at time k′. It is the measured value of the IMU angular velocity in the IMU measurement sequence at time k′. Let be the gyroscope offset in the IMU measurement sequence at time k′. The accelerometer offset in the IMU measurement sequence at time k′ is... Let be the random walk noise of the gyroscope in the IMU measurement sequence at time k′. Let ΔR be the random walk noise of the accelerometer in the IMU measurement sequence at time k′, Δt be the time difference between the IMU measurement sequence at time i′ and time k′, and ΔR be the random walk noise of the accelerometer at time k′. i′k′ Let Δv be the pre-integral variable for the rotation between time i′ and time k′. i′k′ Let be the pre-integral variable of the velocity between time i′ and time k′. The measured values ​​of the pre-integrated variables for rotation between time i′ and time j′. The measured value of the pre-integrated variable of velocity between time i′ and time j′. Let Exp be the measured value of the pre-integrated variable at the position between time i′ and time j′, and let Exp denote the exponential mapping of the Lie algebra.

[0099] Step 52: Construct the IMU pre-integration constraint model

[0100]

[0101] Where, Δp i′j′ Let be the pre-integral variable representing the position between time i′ and time k′. Let ΔR be the gyroscope offset in the IMU measurement sequence at time j′. i′j′ Let Δv be the pre-integral variable for the rotation between time i′ and time j′. i′j′ Let be the pre-integral variable of the velocity between time i′ and time j′. Let be the gyroscope offset in the IMU measurement sequence at time i′. Let be the accelerometer offset in the IMU measurement sequence at time j′. Let be the accelerometer offset in the IMU measurement sequence at time i′. Let Log[·] be the variable to be optimized, and let Log[·] denote the logarithmic mapping of the Lie group.

[0102] Furthermore, the specific process of step 6 is as follows:

[0103] For the current visible light image, find the key frames with the same visual words as the current visible light image from the key frames of the historical visible light image and the key frames of the historical infrared image within the sliding window, and then calculate the cosine similarity between the bag-of-words vector of the current visible light image and the bag-of-words vector of each key frame.

[0104] The cosine similarity between two bag-of-words vectors is calculated as follows:

[0105]

[0106] Where H1 and H2 are two bag-of-words vectors, "·" represents the vector dot product, and ||·|| represents the Euclidean norm of the vector;

[0107] If there is a keyframe with a cosine similarity to the bag-of-words vector of the current visible light image that is higher than a threshold, then a loop closure is detected in the current visible light image. The keyframe with a cosine similarity higher than the threshold is the keyframe detected by the loop closure in the current visible light image.

[0108] If there is no keyframe with a bag-of-words vector cosine similarity higher than the threshold of the current frame's visible light image, then no loop closure is detected in the current frame's visible light image.

[0109] Similarly, loop closure detection is performed on the current frame of the infrared image.

[0110] Furthermore, the global objective function is:

[0111]

[0112] Where, X0={X B Let {X} be the sum of the state variables to be optimized in the system, and X = {X} C ,X′ C}, X C Let X′ represent the set of rotation and translation matrices of keyframes from a visible light camera within a sliding window. C r represents the set of rotation and translation matrices of the infrared camera keyframes within the sliding window. p To marginalize the residual, H p The marginalized Hessian matrix, where B represents the set of IMU measurement times. Let C represent the matrix used to calculate the IMU residual information, where C is the set of binocular landmarks in all visible light and infrared images within the sliding window, and Q is the set of monocular landmarks in all visible light and infrared images within the sliding window. The information matrix represents the camera reprojection error, and L is the set of all landmark points corresponding to the visible light image and infrared image matched during the loop closure detection process and located within the sliding window.

[0113] The beneficial effects of this invention are:

[0114] This invention combines the rich texture features of visible light cameras with the illumination-independent characteristics of infrared cameras to construct a visual-inertial positioning system that uses cross-modal matching between visible light and infrared. A common deep neural network is used to extract visual feature points and descriptors from both visible light and thermal infrared images, providing feature confidence levels and proposing a classification management method for cross-modal feature points. Then, the depth of landmark points is recovered using two-view triangulation, and loop closure detection is performed based on the cosine similarity of bag-of-words vectors. The position and attitude of the visible light and infrared cameras are jointly optimized using this depth estimate, loop closure detection results, and inertial pre-integration measurements. This invention effectively solves the problems of poor positioning performance and loop closure detection failure in existing technologies under visual degradation environments. Attached Figure Description

[0115] Figure 1 This is a flowchart of a visual inertial positioning method for cross-modal matching of visible light and infrared light according to the present invention;

[0116] Figure 2 It is a diagram of a common deep neural network architecture; Detailed Implementation

[0117] Specific implementation method one: Combining Figure 1 This embodiment describes a visual-inertial localization method based on visible light and infrared cross-modal matching. The method specifically includes the following steps:

[0118] Step 1: Trigger the visible light camera, infrared camera, and IMU measurement unit with the same pulse signal (when the IMU measurement unit is a model that does not support external triggering, the IMU measurement value at the time of the visible light camera image timestamp can be obtained by interpolation). The visible light camera and infrared camera are both installed on the unmanned system. Then read the image acquired by the visible light camera, the image acquired by the infrared camera, and the IMU measurement sequence from the IMU measurement unit after triggering.

[0119] Using the timestamps of images from a visible light camera as a reference, a set of IMU measurement sequences is obtained from the timestamp of the previous visible light camera image to the timestamp of the current visible light camera image. If the time difference between the current visible light image and the current infrared image is within t0 (3ms in this invention), the current visible light image, the current infrared image, and the obtained set of IMU measurement sequences are packaged into a data set and processed in binocular inertial mode, i.e., step 2 is continued. If the time difference between the current visible light image and the current infrared image is greater than or equal to t0, the image with the earlier timestamp is selected from the current visible light image and the current infrared image and processed in monocular inertial mode.

[0120] Step 2: Use a common deep neural network to extract the descriptor of each pixel in the current frame visible light image and the current frame infrared image, and determine the feature points in the current frame visible light image and the current frame infrared image based on the descriptor score;

[0121] Step 3: Determine whether the current frame of the visible light image is a key frame based on the visible light key frame image in the sliding window, and determine whether the current frame of the infrared image is a key frame based on the infrared key frame image in the sliding window.

[0122] If neither the current frame's visible light image nor the current frame's infrared image is a keyframe, then no processing is required for the current frame's visible light image and the current frame's infrared image, and no addition to the current frame's visible light image and the current frame's infrared image to the sliding window is needed. Return to step 1.

[0123] If the current frame's visible light image or the current frame's infrared image is a keyframe, then the current frame's visible light image is added as a keyframe to the sliding window, and the current frame's infrared image is added as a keyframe to the sliding window.

[0124] The feature points in the current frame's visible light image and the current frame's infrared image are matched according to the descriptor to obtain the feature point matching results. Based on the feature point matching results, all binocular landmark points in the current frame's visible light image and the current frame's infrared image are obtained. Feature point matching is performed between the current frame's visible light image and the previous frame's keyframe, and feature point matching is performed between the current frame's infrared image and the previous frame's keyframe, to obtain the corresponding monocular landmark points in the current frame's visible light image and the current frame's infrared image.

[0125] The reprojection errors of the binocular landmarks on the current frame visible light image and the current frame infrared image, the reprojection errors of the monocular landmarks corresponding to the current frame visible light image on the current frame visible light image and the key frame of the previous visible light image, and the reprojection errors of the monocular landmarks corresponding to the current frame infrared image on the current frame infrared image and the key frame of the previous infrared image are calculated respectively.

[0126] Continue to step 4;

[0127] Step 4: Construct bag-of-words vectors for the visible light image and the infrared image of the current frame, respectively; specifically:

[0128] Create a visual vocabulary V containing K visual words. For any feature point in the current frame of the visible light image, calculate the cosine distance between the descriptor of the feature point and each visual word in the visual vocabulary V. Assign the visual word corresponding to the minimum cosine distance to the feature point. Similarly, assign visual words to each feature point in the current frame of the visible light image. Then count the number of times each visual word appears in the current frame of the visible light image. Use the number of times each visual word appears to form the bag-of-words vector of the current frame of the visible light image.

[0129] Similarly, the bag-of-words vector of the current frame infrared image is obtained;

[0130] Step 5: Construct an IMU pre-integration constraint model using the set of IMU measurement sequences located between the timestamp of the previous visible light image keyframe and the timestamp of the current visible light image.

[0131] Step 6: Perform loop closure detection based on the bag-of-words vectors corresponding to the historical visible light image keyframes, historical infrared image keyframes, and the current frame visible light image within the sliding window, and obtain the loop closure detection results.

[0132] Step 7: Based on the reprojection error, IMU pre-integration constraint model and loop closure detection results (constructing the loop closure detection error model in the global objective function based on the key frames detected by loop closure), establish the global objective function, solve the global objective function, and obtain the visual inertial positioning results of the visible light camera and infrared camera for each frame within the sliding window;

[0133] Return to step 1.

[0134] Specific Implementation Method Two: Combining Figure 2 This embodiment is described below. This embodiment differs from Specific Embodiment One in that the common deep neural network includes a first adaptor, a second adaptor, an encoder, and a decoder; wherein:

[0135] The input to the first adaptor is the visible light image of the current frame. The output of the first adaptor is used as the input to the encoder. The encoder outputs the descriptor of each pixel in the visible light image of the current frame. The output of the encoder is used as the input to the decoder. The decoder outputs the feature score of each pixel in the visible light image of the current frame.

[0136] The input to the second adaptor is the current frame infrared image. The output of the second adaptor is used as the input to the encoder. The encoder outputs the descriptor of each pixel in the current frame infrared image. The output of the encoder is then used as the input to the decoder. The decoder outputs the feature score of each pixel in the current frame infrared image.

[0137] The loss function used during training of a common deep neural network for:

[0138]

[0139] in, For the descriptor loss function, For peak loss function, λ is the feature score loss function, and λ is a hyperparameter (taken as 0.5 in this invention);

[0140]

[0141] in, Represents all pixels in a visible light image. Find the expected value, D N and D′ N These represent descriptor subsets of pixels in visible light and infrared images, respectively.

[0142]

[0143] Among them, {d i ,d i′} is a descriptor for a pair of matched pixels in a visible light image and an infrared image, d j It is set D N In and d i The most recent descriptor, d k It is set D′ N In and d i′ The most recent descriptor, d m It is set D′ N In and d i The most recent non-matching descriptor, d n It is set D N In and d i′ The nearest non-matching descriptor, θ(·,·) represents the angle between the two descriptor vectors, with d i and d j For example,

[0144] c(s i ,s i ′)=s i s i′

[0145] Among them, s i It is the d output of the decoder. i The score, s i′ It is the d output of the decoder. i′ The score;

[0146]

[0147] Where p represents a sliding image patch, D N [p[i]] represents the descriptor of the i-th pixel within the sliding image block p in the visible light image, D′ w Let D′ represent the infrared image after the affine transformation of w (the affine transformation of w is a conventional viewpoint transformation, essentially a matrix multiplication). w [p[i]] represents the descriptor of the i-th pixel within the sliding image block p in the infrared image after the affine transformation of w. For D of all pixels within the sliding image block p N [p[i]] T D′ w [p[i]] Calculate the expectation, where S is the score set of descriptors for all pixels in the visible light image, S′ w S is the score set of all pixels in the infrared image after the affine transformation of w, and S[p] is the score matrix of pixels in the sliding image block p in set S. w [p] is the set S′ w The score matrix of pixels within a sliding image block p; For all sliding image blocks, b(p;D)N ,D′ w )×(1-S[p] T S′ w [p]) Calculate the expected value;

[0148]

[0149] Where I is the edge image of the visible light image, AP represents the average pooling layer, MP represents the max pooling layer, AP(S) represents the output of S after the average pooling layer, AP(S)[i] represents the value of the i-th pixel in AP(S), MP(S) represents the output of S after the max pooling layer, and (1-MP(S))[i] represents the value of the i-th pixel in 1-MP(S). Let AP(S)[i] represent the sum of all pixels in the visible light image. 2 +(1-MP(S))[i]) 2 Find the expectation, where S[i] represents the score of the descriptor of the i-th pixel in S, and M(I) is the processing result of the edge image I. Represents all pixels Seeking expectations; and The calculation process is similar;

[0150]

[0151] in, It is the Laplacian operator for images, ||·|| denotes the calculation of the 2-norm. express The value of the i-th pixel in the function x represents the independent variable of the function.

[0152] The other steps and parameters are the same as in Specific Implementation Method 1.

[0153] In this invention, the operation of the first adaptor is as follows:

[0154] Within the first adaptor, the input image first passes through the first convolutional layer and the first ReLU activation function layer in sequence;

[0155] The output of the first ReLU activation function layer then passes through the second convolutional layer and the second ReLU activation function layer in sequence; the outputs of the first ReLU activation function layer and the second ReLU activation function layer are added together to obtain the sum a;

[0156] The summed result 'a' is passed through the third convolutional layer and the third ReLU activation function layer in sequence; the output of the third ReLU activation function layer is passed through the fourth convolutional layer and the fourth ReLU activation function layer in sequence, and then the outputs of the third ReLU activation layer and the fourth ReLU activation function layer are added together to obtain the summed result 'b'.

[0157] The summed result b is passed through the fifth convolutional layer and the fifth ReLU activation function layer in sequence; the output of the fifth ReLU activation function layer is passed through the sixth convolutional layer and the sixth ReLU activation function layer in sequence, and then the outputs of the sixth ReLU activation function layer and the fifth ReLU activation function layer are added together to obtain the summed result c, which is used as the output of the first adaptor.

[0158] The second adaptor works in the same way as the first adaptor.

[0159] The encoder consists of three concatenated convolutional layers. By adjusting the number of channels, it outputs deep features of a fixed dimension. The input to the encoder is the feature map from the adaptor, and the output of the encoder is a 128-dimensional feature vector.

[0160] The decoder is the output module of a common deep neural network. Its function is to decode the high-dimensional feature map from the encoder into the importance or confidence of each pixel. The decoder consists of a first branch and a second branch. The input of the first branch comes from the feature vector output by the encoder. In the first branch, it first passes through a convolutional layer with a kernel size of 1×1, and then passes through a softplus activation function to map the output of the convolutional layer to the positive value range and normalize it to (0,1). The input of the second branch is also from the feature vector from the encoder. The input passes through three concatenated convolutional modules in sequence. Each convolutional module includes a convolutional layer, a batch normalization (BN) layer, and a ReLU activation function layer connected in sequence. The output of the last convolutional module passes through three concatenated convolutional units in sequence. Each convolutional unit includes a convolutional layer, a softmax layer, a BN layer, a ReLU activation function layer, an instance normalization (IN) layer, and a ReLU activation function layer. The output of the last convolutional unit then passes through a convolutional layer and a softmax layer. The output of the softmax layer is used as the output of the second branch. The outputs of the first branch and the second branch are multiplied pixel by pixel to obtain the feature score output by the decoder.

[0161] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the step of determining the feature points in the current frame's visible light image and the current frame's infrared image based on descriptor scores specifically involves:

[0162] For the current frame of the visible light image, the descriptor scores of each pixel in the visible light image are sorted from largest to smallest. The top 20% of pixels are then used as feature points, and the number of feature points c is compared with a threshold (set to 1024 in this invention).

[0163] If the number of feature points c is less than the set threshold, then continue to select the pixels with the highest descriptor scores from the sorting results as feature points until the number of feature points reaches the set threshold; if the number of feature points c is greater than or equal to the set threshold, then directly select the top 20% of the pixels as feature points.

[0164] Similarly, feature points in the current frame of the infrared image are obtained.

[0165] Other steps and parameters are the same as in specific implementation method one or two.

[0166] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the specific process of step 3 is as follows:

[0167] Step 31: Determine whether the time difference between the current visible light image and the keyframe of the previous visible light image exceeds the time threshold (in this invention, the time threshold is designed to be 1 second);

[0168] If the time threshold is not exceeded, the visible light image of the current frame is not a key frame;

[0169] If the time threshold is exceeded, the current frame visible light image is matched with the previous visible light image keyframe (i.e., the keyframe in history that is closest to the current frame visible light image in time) to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current frame visible light image is added to the list of keyframes of visible light images; otherwise, the current frame visible light image is not a keyframe.

[0170] Determine whether the time difference between the current frame of the infrared image and the key frame of the previous infrared image exceeds a time threshold (in this invention, the time threshold is designed to be 1 second);

[0171] If the time threshold is not exceeded, the current frame of infrared image is not a key frame;

[0172] If the time threshold is exceeded, the current frame infrared image is matched with the previous infrared image keyframe (i.e., the keyframe that is closest to the current frame infrared image in time among the historical keyframes) to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current frame infrared image is added to the list of keyframes of infrared images; otherwise, the current frame infrared image is not a keyframe.

[0173] If neither the current frame's visible light image nor the current frame's infrared image is a keyframe, then no processing is required for the current frame's visible light image and the current frame's infrared image, and no addition to the current frame's visible light image and the current frame's infrared image to the sliding window is needed. Return to step 1.

[0174] If the current frame's visible light image or the current frame's infrared image is a keyframe, then the current frame's visible light image is added as a keyframe to the sliding window, and the current frame's infrared image is added as a keyframe to the sliding window.

[0175] Then proceed to step 32;

[0176] Step 32: Match the feature points in the current frame's visible light image and the current frame's infrared image to obtain all successfully matched feature point pairs;

[0177] Step 33: Obtain the points on the map corresponding to each successfully matched feature point pair in Step 32. For any successfully matched feature point pair (x1, y1) and (x2, y2), denote the points on the map corresponding to the feature point pair as p. Calculate the reprojection error of point p on the current frame of the visible light image based on the keyframe of the visible light image in the sliding window, and calculate the reprojection error of point p on the current frame of the infrared image based on the keyframe of the infrared image in the sliding window.

[0178] If the reprojection error of point p on both the visible light image and the infrared image of the current frame is less than 2 pixels, then p will be added to the set of binocular landmarks as a binocular landmark point.

[0179] Then, each pair of feature points that were successfully matched in step 31 is treated as a monocular landmark, and the monocular landmark is added to the monocular landmark set.

[0180] Step 34: Calculate the reprojection error of each monocular landmark corresponding to the current visible light image on the current visible light image and the keyframe of the previous visible light image, the reprojection error of each monocular landmark corresponding to the current infrared image on the current infrared image and the keyframe of the previous infrared image, and the reprojection error of each binocular landmark corresponding to the current visible light image and the current infrared image on the current visible light image and the current infrared image, respectively.

[0181] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0182] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the specific process of step 32 is as follows:

[0183] For any feature point a in the visible light image of the current frame, the feature point a is transformed and projected onto the infrared image of the current frame using calibration parameters. The feature point b with the smallest descriptor distance to feature point a is found in the infrared image of the current frame. Then, the feature point b is transformed and projected onto the visible light image of the current frame using calibration parameters. The feature point with the smallest descriptor distance to feature point b is found in the visible light image of the current frame.

[0184] If feature point a has the smallest descriptor distance to feature point b in the current frame of the visible light image, then feature point a and feature point b are successfully matched.

[0185] If the feature point with the smallest descriptor distance to feature point b in the current frame of the visible light image is not a, then feature point a and feature point b will not be matched successfully (at this time, both feature point a and feature point b will be regarded as monocular landmarks).

[0186] Similarly, each feature point on the visible light image of the current frame is processed separately to obtain all successfully matched feature point pairs.

[0187] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0188] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that, specifically, obtaining the points on the map corresponding to each successfully matched feature point pair in step 32 is as follows:

[0189] For any pair of successfully matched feature points (x1, y1) and (x2, y2):

[0190]

[0191] Where (x1, y1) and (x2, y2) are the coordinates of feature points on the visible light image and the infrared image of the current frame, respectively. This represents the j-th row of the projection matrix corresponding to the visible light image of the current frame. This represents the j-th row of the projection matrix corresponding to the current frame infrared image, and p is the landmark point on the map corresponding to (x1, y1) and (x2, y2).

[0192] Similarly, the points corresponding to each successfully matched feature point pair in step 32 are obtained on the map.

[0193] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0194] In this invention, based on epipolar geometry constraints, the projection matrix corresponding to the visible light image and the projection matrix corresponding to the infrared image of the current frame are calculated using all successfully matched feature point pairs on the current frame visible light image and the current frame infrared image. Then, based on the two obtained projection matrices, the points corresponding to each successfully matched feature point pair on the map are obtained.

[0195] For a monocular landmark point corresponding to the current visible light image, the projection matrix corresponding to the current visible light image and the projection matrix corresponding to the previous visible light image are calculated using all successfully matched feature points from the current and previous visible light images. Then, based on the obtained two projection matrices, the point corresponding to each group of monocular landmark points on the map is obtained. Similarly, the point corresponding to the monocular landmark point corresponding to the current infrared image is calculated on the map.

[0196] In this invention, the process of calculating the corresponding points on the map for successfully matched feature point pairs is referred to as restoring the depth of landmark points using two-view triangulation.

[0197] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One to Six in that, in step 34, the reprojection error of each binocular landmark corresponding to the current frame visible light image and the current frame infrared image is calculated on the current frame visible light image and the current frame infrared image. The specific process is as follows:

[0198] For the pixels (x1, y1) and (x2, y2) corresponding to the stereo landmark point p, transform the stereo landmark point p into the visible light camera coordinate system, and label the coordinates of point p in the visible light camera coordinate system as follows:

[0199]

[0200] in, Let be a rotation matrix. It is a translation matrix;

[0201]

[0202] in, This is the extrinsic parameter matrix between the visible light camera and the IMU (obtainable by any calibration algorithm). It is the rotation matrix of the k-th visible light image keyframe within the sliding window relative to the (k-1)-th visible light image keyframe within the sliding window. It is the translation matrix of the k-th visible light image keyframe within the sliding window relative to the (k-1)-th visible light image keyframe within the sliding window.

[0203] Will Incorporate the pinhole camera model π(·):

[0204]

[0205] In the formula, f x and f y These represent the two focal lengths of a visible light camera, c and c. x and c y These represent the optical center offsets of the two axes of a visible light camera. express The coordinates projected onto the visible light image, x Z express The Z-axis coordinate (i.e., the Z-axis coordinate in the visible light camera coordinate system);

[0206] Calculate a reprojection error of the binocular landmark point p based on the visible light pixel (x1, y1):

[0207]

[0208] in, This is due to reprojection error;

[0209] Transform the binocular landmark point p to the infrared camera coordinate system, and label the coordinates of point p in the infrared camera coordinate system as follows:

[0210]

[0211] in, Let be a rotation matrix. It is a translation matrix;

[0212]

[0213] in, This is the extrinsic parameter matrix between the infrared camera and the IMU. It is the rotation matrix of the k-th infrared image keyframe within the sliding window relative to the (k-1)-th infrared image keyframe within the sliding window. It is the translation matrix of the k-th infrared image keyframe within the sliding window relative to the (k-1)-th infrared image keyframe within the sliding window;

[0214] Will Incorporate the pinhole camera model π(·):

[0215]

[0216] In the formula, f x ′ and f y ′ represent the two focal lengths of the infrared camera, and c represents the focal length of the infrared camera. x ′ and c′ y These represent the two-axis optical center offsets of the infrared camera. express The coordinates projected onto the infrared image, x′ Z express Z-axis direction coordinates;

[0217] Calculate another reprojection error of the binocular landmark point p based on (x2, y2):

[0218]

[0219] in, This is the reprojection error.

[0220] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0221] Similarly, the method of this embodiment is used to calculate the reprojection error of each monocular landmark corresponding to the current frame of the visible light image on the current frame of the visible light image and the previous frame of the visible light image, and the reprojection error of each monocular landmark corresponding to the current frame of the infrared image on the current frame of the infrared image and the previous frame of the infrared image.

[0222] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the specific process of step 5 is as follows:

[0223] Step 51: Calculate the pre-integral variable relationship between the IMU measurement sequence at time i′ and the IMU measurement sequence at time j′:

[0224]

[0225] Where ~ represents the measured value, It is the measured value of IMU acceleration in the IMU measurement sequence at time k′. It is the measured value of the IMU angular velocity in the IMU measurement sequence at time k′. Let be the gyroscope offset in the IMU measurement sequence at time k′. The accelerometer offset in the IMU measurement sequence at time k′ is... Let be the random walk noise of the gyroscope in the IMU measurement sequence at time k′. Let ΔR be the random walk noise of the accelerometer in the IMU measurement sequence at time k′, Δt be the time difference between the IMU measurement sequence at time i′ and time k′, and ΔR be the random walk noise of the accelerometer at time k′. i′k′ Let Δv be the pre-integral variable for the rotation between time i′ and time k′. i′k′ Let be the pre-integral variable of the velocity between time i′ and time k′. The measured values ​​of the pre-integrated variables for rotation between time i′ and time j′. The measured value of the pre-integrated variable of velocity between time i′ and time j′. Let Exp be the measured value of the pre-integrated variable at the position between time i′ and time j′, and let Exp denote the exponential mapping of the Lie algebra.

[0226]

[0227] The i′ time represents the timestamp corresponding to the previous visible light image, and the j′ time represents the timestamp corresponding to the current visible light image.

[0228] Step 52: Construct the IMU pre-integration constraint model

[0229]

[0230] Where, Δp i′j′ Let be the pre-integral variable representing the position between time i′ and time k′. Let ΔR be the gyroscope offset in the IMU measurement sequence at time j′. i′j′ Let Δv be the pre-integral variable for the rotation between time i′ and time j′. i′j′ Let be the pre-integral variable of the velocity between time i′ and time j′. Let be the gyroscope offset in the IMU measurement sequence at time i′. Let be the accelerometer offset in the IMU measurement sequence at time j′. Let be the accelerometer offset in the IMU measurement sequence at time i′. Let Log[·] be the variable to be optimized, and let Log[·] denote the logarithmic mapping of the Lie group.

[0231] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0232] To describe the constraints of inertial measurement unit (IMU) data on pose estimation, this step integrates the IMU data into the optimization framework by constructing a pre-integration model, thereby reducing the computational burden.

[0233]

[0234] Where R is the rotation matrix and I is the 3×3 identity matrix.

[0235] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that the specific process of step 6 is as follows:

[0236] For the current visible light image, find the key frames with the same visual words as the current visible light image from the key frames of the historical visible light image and the key frames of the historical infrared image within the sliding window, and then calculate the cosine similarity between the bag-of-words vector of the current visible light image and the bag-of-words vector of each key frame.

[0237] The cosine similarity between two bag-of-words vectors is calculated as follows:

[0238]

[0239] Where H1 and H2 are two bag-of-words vectors, "·" represents the vector dot product, and ||·|| represents the Euclidean norm of the vector;

[0240] If there exists a keyframe whose bag-of-words vector cosine similarity to the current frame's visible light image is higher than a threshold (this keyframe can be either an infrared image keyframe or a visible light image keyframe, and the keyframe whose cosine similarity to the current frame's visible light image is higher than the threshold may not be unique), then a loop closure is detected in the current frame's visible light image. The keyframe whose cosine similarity is higher than the threshold is the keyframe detected by the loop closure in the current frame's visible light image.

[0241] If there is no keyframe with a bag-of-words vector cosine similarity higher than the threshold of the current frame's visible light image, then no loop closure is detected in the current frame's visible light image.

[0242] Similarly, loop closure detection is performed on the current frame of the infrared image.

[0243] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0244] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Six in that the global objective function is:

[0245]

[0246] Where, X0={X B Let {X} be the sum of the state variables to be optimized in the system, and X = {X} C ,X′ C}, X C Let X′ represent the set of rotation and translation matrices of keyframes from a visible light camera within a sliding window. C r represents the set of rotation and translation matrices of the infrared camera keyframes within the sliding window. p To marginalize the residual, H p The marginalized Hessian matrix is ​​given, where B represents the set of IMU measurement times (all IMU measurement times within the timestamp range of the sliding window). Let C represent the matrix used to calculate the IMU residual information, where C is the set of binocular landmarks in all visible light and infrared images within the sliding window, and Q is the set of monocular landmarks in all visible light and infrared images within the sliding window. The information matrix represents the camera reprojection error, and L is the set of all landmark points corresponding to the visible light image and infrared image matched during the loop closure detection process and located within the sliding window.

[0247] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0248] It should be noted that the third to fifth terms in the global objective function are calculated using a method similar to that used to calculate the reprojection error. In this invention, a fixed-size sliding window is set for the image, and the termination timestamp of the sliding window is the timestamp of the current frame. When a historical image exceeds the timestamp range of the sliding window, the landmark points in the historical image are removed from sets C, Q, and L. In other words, the camera pose is calculated using all frame and feature point information within the window, as well as IMU pre-integration information.

[0249] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual-inertial positioning method based on cross-modal matching of visible light and infrared, characterized in that, The method specifically includes the following steps: Step 1: Trigger the visible light camera, infrared camera, and IMU measurement unit with the same pulse signal. The visible light camera and infrared camera are both installed on the unmanned system. Then read the images acquired by the visible light camera, the images acquired by the infrared camera, and the IMU measurement sequence from the IMU measurement unit after triggering. Using the timestamps of images from the visible light camera as a reference, a set of IMU measurement sequences is obtained from the timestamp of the previous visible light camera image to the timestamp of the current visible light camera image. If the time difference between the current visible light image and the current infrared image is within t0, the current visible light image, the current infrared image, and the obtained set of IMU measurement sequences are packaged into a data set and processed in binocular inertial mode, i.e., step 2 is continued. If the time difference between the current visible light image and the current infrared image is greater than or equal to t0, the image with the earlier timestamp is selected from the current visible light image and the current infrared image and processed in monocular inertial mode. Step 2: Use a common deep neural network to extract the descriptor of each pixel in the current frame visible light image and the current frame infrared image, and determine the feature points in the current frame visible light image and the current frame infrared image based on the descriptor score; Step 3: Determine whether the current frame of the visible light image is a key frame based on the visible light key frame image in the sliding window, and determine whether the current frame of the infrared image is a key frame based on the infrared key frame image in the sliding window. If neither the current frame's visible light image nor the current frame's infrared image is a keyframe, then no processing is required for the current frame's visible light image and the current frame's infrared image, and no addition to the current frame's visible light image and the current frame's infrared image to the sliding window is needed. Return to step 1. If the current frame's visible light image or the current frame's infrared image is a keyframe, then the current frame's visible light image is added as a keyframe to the sliding window, and the current frame's infrared image is added as a keyframe to the sliding window. The feature points in the current frame's visible light image and the current frame's infrared image are matched according to the descriptor to obtain the feature point matching results. Based on the feature point matching results, all binocular landmark points in the current frame's visible light image and the current frame's infrared image are obtained. Feature point matching is performed between the current frame's visible light image and the previous frame's keyframe, and feature point matching is performed between the current frame's infrared image and the previous frame's keyframe, to obtain the corresponding monocular landmark points in the current frame's visible light image and the current frame's infrared image. The reprojection errors of the binocular landmarks on the current frame visible light image and the current frame infrared image, the reprojection errors of the monocular landmarks corresponding to the current frame visible light image on the current frame visible light image and the key frame of the previous visible light image, and the reprojection errors of the monocular landmarks corresponding to the current frame infrared image on the current frame infrared image and the key frame of the previous infrared image are calculated respectively. Continue to step 4; Step 4: Construct bag-of-words vectors for the visible light image and the infrared image of the current frame, respectively; specifically: Create a visual vocabulary V containing K visual words. For any feature point in the current frame of the visible light image, calculate the cosine distance between the descriptor of the feature point and each visual word in the visual vocabulary V. Assign the visual word corresponding to the minimum cosine distance to the feature point. Similarly, assign visual words to each feature point in the current frame of the visible light image. Then count the number of times each visual word appears in the current frame of the visible light image. Use the number of times each visual word appears to form the bag-of-words vector of the current frame of the visible light image. Similarly, the bag-of-words vector of the current frame infrared image is obtained; Step 5: Construct an IMU pre-integration constraint model using the set of IMU measurement sequences located between the timestamp of the previous visible light image keyframe and the timestamp of the current visible light image. Step 6: Perform loop closure detection based on the bag-of-words vectors corresponding to the historical visible light image keyframes, historical infrared image keyframes, and the current frame visible light image within the sliding window, and obtain the loop closure detection results. Step 7: Establish a global objective function based on the reprojection error, IMU pre-integration constraint model, and loop closure detection results; solve the global objective function to obtain the visual inertial positioning results of the visible light camera and the infrared camera. Return to step 1.

2. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 1, characterized in that, The common deep neural network includes a first adaptor, a second adaptor, an encoder, and a decoder; wherein: The input to the first adaptor is the visible light image of the current frame. The output of the first adaptor is used as the input to the encoder. The encoder outputs the descriptor of each pixel in the visible light image of the current frame. The output of the encoder is used as the input to the decoder. The decoder outputs the feature score of each pixel in the visible light image of the current frame. The input to the second adaptor is the current frame infrared image. The output of the second adaptor is used as the input to the encoder. The encoder outputs the descriptor of each pixel in the current frame infrared image. The output of the encoder is then used as the input to the decoder. The decoder outputs the feature score of each pixel in the current frame infrared image. The loss function used during training of a common deep neural network for: in, For the descriptor loss function, For peak loss function, Let λ be the feature score loss function, and λ be the hyperparameter. in, This represents the expression for (c(s) of all pixels in a visible light image. i ,s i′ )) Find the expected value, D N and D′ N These represent descriptor subsets of pixels in visible light and infrared images, respectively. Among them, {d i ,d i′ } is a descriptor for a pair of matched pixels in a visible light image and an infrared image, d j It is set D N In and d i The most recent descriptor, d k It is set D′ N In and d i′ The most recent descriptor, d m It is set D′ N In and d i The most recent non-matching descriptor, d n It is set D N In and d i′ The nearest non-matching descriptor, θ(·,·) represents the angle between the two descriptor vectors; c(s i ,s i ′)=s i s i′ Among them, s i It is the d output of the decoder. i The score, s i′ It is the d output of the decoder. i′ The score; Where p represents a sliding image patch, D N [p[i]] represents the descriptor of the i-th pixel within the sliding image block p in the visible light image, D′ w D′ represents the infrared image after affine transformation of w. w [p[i]] represents the descriptor of the i-th pixel within the sliding image block p in the infrared image after the affine transformation of w. For D of all pixels within the sliding image block p N [p[i]] T D′ w [p[i]] Calculate the expectation, where S is the score set of descriptors for all pixels in the visible light image, S′ w S is the score set of all pixels in the infrared image after the affine transformation of w, and S[p] is the score matrix of pixels in the sliding image block p in set S. w [p] is the set S′ w The score matrix of pixels within a sliding image block p; For all sliding image blocks, b(p;D) N ,D′ w )×(1-S[p] T S′ w [p]) Calculate the expected value; Where I is the edge image of the visible light image, AP represents the average pooling layer, MP represents the max pooling layer, AP(S) represents the output of S after the average pooling layer, AP(S)[i] represents the value of the i-th pixel in AP(S), MP(S) represents the output of S after the max pooling layer, and (1-MP(S))[i] represents the value of the i-th pixel in 1-MP(S). Let AP(S)[i] represent the sum of all pixels in the visible light image. 2 +(1-MP(S))[i]) 2 Find the expectation, where S[i] represents the score of the descriptor of the i-th pixel in S, and M(I) is the processing result of the edge image I. Represents all pixels Seeking expectations; in, It is the Laplacian operator for images, ||·|| denotes the calculation of the 2-norm. express The value of the i-th pixel in the function x represents the independent variable of the function.

3. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 1, characterized in that, The step of determining feature points in the current frame's visible light image and the current frame's infrared image based on descriptor scores specifically involves: For the current frame of the visible light image, the descriptor scores of each pixel in the visible light image are sorted from largest to smallest. The top 20% of pixels are then used as feature points, and the number of feature points c is compared with the threshold value. If the number of feature points c is less than the set threshold, then continue to select the pixels with the highest descriptor scores from the sorting results as feature points until the number of feature points reaches the set threshold; if the number of feature points c is greater than or equal to the set threshold, then directly select the top 20% of the pixels as feature points. Similarly, feature points in the current frame of the infrared image are obtained.

4. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 3, characterized in that, The specific process of step 3 is as follows: Step 31: Determine whether the time difference between the current visible light image and the keyframe of the previous visible light image exceeds the time threshold; If the time threshold is not exceeded, the visible light image of the current frame is not a key frame; If the time threshold is exceeded, feature matching is performed between the current visible light image and the keyframe of the previous visible light image to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current visible light image is added to the keyframe list of visible light images; otherwise, the current visible light image is not a keyframe. Determine whether the time difference between the current frame of the infrared image and the keyframe of the previous infrared image exceeds the time threshold. If the time threshold is not exceeded, the current frame of infrared image is not a key frame; If the time threshold is exceeded, feature matching is performed between the current frame infrared image and the keyframe of the previous infrared image to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current frame infrared image is added to the keyframe list of infrared images; otherwise, the current frame infrared image is not a keyframe. If neither the current frame's visible light image nor the current frame's infrared image is a keyframe, then no processing is required for the current frame's visible light image and the current frame's infrared image, and no addition to the current frame's visible light image and the current frame's infrared image to the sliding window is needed. Return to step 1. If the current frame's visible light image or the current frame's infrared image is a keyframe, then the current frame's visible light image is added as a keyframe to the sliding window, and the current frame's infrared image is added as a keyframe to the sliding window. Then proceed to step 32; Step 32: Match the feature points in the current frame's visible light image and the current frame's infrared image to obtain all successfully matched feature point pairs; Step 33: Obtain the points on the map corresponding to each successfully matched feature point pair in Step 32. For any successfully matched feature point pair (x1, y1) and (x2, y2), denote the points on the map corresponding to the feature point pair as p. Calculate the reprojection error of point p on the current frame of the visible light image based on the keyframe of the visible light image in the sliding window, and calculate the reprojection error of point p on the current frame of the infrared image based on the keyframe of the infrared image in the sliding window. If the reprojection error of point p on both the visible light image and the infrared image of the current frame is less than 2 pixels, then p will be added to the set of binocular landmarks as a binocular landmark point. Then, each pair of feature points that were successfully matched in step 31 is treated as a monocular landmark, and the monocular landmark is added to the monocular landmark set. Step 34: Calculate the reprojection error of each monocular landmark corresponding to the current visible light image on the current visible light image and the keyframe of the previous visible light image, the reprojection error of each monocular landmark corresponding to the current infrared image on the current infrared image and the keyframe of the previous infrared image, and the reprojection error of each binocular landmark corresponding to the current visible light image and the current infrared image on the current visible light image and the current infrared image, respectively.

5. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 4, characterized in that, The specific process of step 32 is as follows: For any feature point a in the visible light image of the current frame, the feature point a is transformed and projected onto the infrared image of the current frame using calibration parameters. The feature point b with the smallest descriptor distance to feature point a is found in the infrared image of the current frame. Then, the feature point b is transformed and projected onto the visible light image of the current frame using calibration parameters. The feature point with the smallest descriptor distance to feature point b is found in the visible light image of the current frame. If feature point a has the smallest descriptor distance to feature point b in the current frame of the visible light image, then feature point a and feature point b are successfully matched. If the feature point with the smallest descriptor distance to feature point b in the current frame of the visible light image is not a, then feature point a and feature point b will not be matched. Similarly, each feature point on the visible light image of the current frame is processed separately to obtain all successfully matched feature point pairs.

6. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 5, characterized in that, The specific steps for obtaining the points on the map corresponding to each successfully matched feature point pair in step 32 are as follows: For any pair of successfully matched feature points (x1, y1) and (x2, y2): Where (x1, y1) and (x2, y2) are the coordinates of feature points on the visible light image and the infrared image of the current frame, respectively. This represents the j-th row of the projection matrix corresponding to the visible light image of the current frame. This represents the j-th row of the projection matrix corresponding to the current frame infrared image, and p is the landmark point on the map corresponding to (x1, y1) and (x2, y2). Similarly, the points corresponding to each successfully matched feature point pair in step 32 are obtained on the map.

7. The visual-inertial positioning method for cross-modal matching of visible light and infrared light according to claim 6, characterized in that, In step 34, the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image is calculated on the current frame visible light image and the current frame infrared image. The specific process is as follows: For the pixels (x1, y1) and (x2, y2) corresponding to the stereo landmark point p, transform the stereo landmark point p into the visible light camera coordinate system, and label the coordinates of point p in the visible light camera coordinate system as follows: in, Let be a rotation matrix. It is a translation matrix; in, This is the extrinsic parameter matrix between the visible light camera and the IMU. It is the rotation matrix of the k-th visible light image keyframe within the sliding window relative to the (k-1)-th visible light image keyframe within the sliding window. It is the translation matrix of the k-th visible light image keyframe within the sliding window relative to the (k-1)-th visible light image keyframe within the sliding window; Will Incorporate the pinhole camera model π(·): In the formula, f x and f y These represent the two focal lengths of a visible light camera, c and c. x and c y These represent the optical center offsets of the two axes of a visible light camera. express The coordinates projected onto the visible light image, x Z express Z-axis direction coordinates; Calculate a reprojection error of the binocular landmark point p based on the visible light pixel (x1, y1): in, This is due to reprojection error; Transform the binocular landmark point p to the infrared camera coordinate system, and label the coordinates of point p in the infrared camera coordinate system as follows: in, Let be a rotation matrix. It is a translation matrix; in, This is the extrinsic parameter matrix between the infrared camera and the IMU. It is the rotation matrix of the k-th infrared image keyframe within the sliding window relative to the (k-1)-th infrared image keyframe within the sliding window. It is the translation matrix of the k-th infrared image keyframe within the sliding window relative to the (k-1)-th infrared image keyframe within the sliding window; Will Incorporate the pinhole camera model π(·): In the formula, f x ′ and f y ′ represent the two focal lengths of the infrared camera, and c represents the focal length of the infrared camera. x ′ and c′ y These represent the optical center offsets of the two axes of the infrared camera. express The coordinates projected onto the infrared image, x′ Z express Z-axis direction coordinates; Calculate another reprojection error of the binocular landmark point p based on (x2, y2): in, This is the reprojection error.

8. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 7, characterized in that, The specific process of step 5 is as follows: Step 51: Calculate the pre-integral variable relationship between the IMU measurement sequence at time i′ and the IMU measurement sequence at time j′: Where ~ represents the measured value, It is the measured value of IMU acceleration in the IMU measurement sequence at time k′. It is the measured value of the IMU angular velocity in the IMU measurement sequence at time k′. Let be the gyroscope offset in the IMU measurement sequence at time k′. The accelerometer offset in the IMU measurement sequence at time k′ is... Let be the random walk noise of the gyroscope in the IMU measurement sequence at time k′. Let ΔR be the random walk noise of the accelerometer in the IMU measurement sequence at time k′, Δt be the time difference between the IMU measurement sequence at time i′ and time k′, and ΔR be the random walk noise of the accelerometer at time k′. i′k′ Let Δv be the pre-integral variable for the rotation between time i′ and time k′. i′k′ Let be the pre-integral variable of the velocity between time i′ and time k′. The measured values ​​of the pre-integrated variables for rotation between time i′ and time j′. The measured value of the pre-integrated variable of velocity between time i′ and time j′. Let Exp be the measured value of the pre-integrated variable at the position between time i′ and time j′, and let Exp denote the exponential mapping of the Lie algebra. Step 52: Construct the IMU pre-integration constraint model Where, Δp i′j′ Let be the pre-integral variable representing the position between time i′ and time k′. Let ΔR be the gyroscope offset in the IMU measurement sequence at time j′. i′j′ Let Δv be the pre-integral variable for the rotation between time i′ and time j′. i′j′ Let be the pre-integral variable of the velocity between time i′ and time j′. Let be the gyroscope offset in the IMU measurement sequence at time i′. Let be the accelerometer offset in the IMU measurement sequence at time j′. Let be the accelerometer offset in the IMU measurement sequence at time i′. Let Log[·] be the variable to be optimized, and let Log[·] denote the logarithmic mapping of the Lie group.

9. The visual-inertial positioning method based on visible light and infrared cross-modal matching according to claim 8, characterized in that, The specific process of step 6 is as follows: For the current visible light image, find the key frames with the same visual words as the current visible light image from the key frames of the historical visible light image and the key frames of the historical infrared image within the sliding window, and then calculate the cosine similarity between the bag-of-words vector of the current visible light image and the bag-of-words vector of each key frame. The cosine similarity between two bag-of-words vectors is calculated as follows: Where H1 and H2 are two bag-of-words vectors, "·" represents the vector dot product, and ||·|| represents the Euclidean norm of the vector; If there is a keyframe with a cosine similarity to the bag-of-words vector of the current visible light image that is higher than a threshold, then a loop closure is detected in the current visible light image. The keyframe with a cosine similarity higher than the threshold is the keyframe detected by the loop closure in the current visible light image. If there is no keyframe with a bag-of-words vector cosine similarity higher than the threshold of the current frame's visible light image, then no loop closure is detected in the current frame's visible light image. Similarly, loop closure detection is performed on the current frame of the infrared image.

10. A visual-inertial positioning method for cross-modal matching of visible light and infrared light according to claim 9, characterized in that, The global objective function is: Where, X0={X B Let {X} be the sum of the state variables to be optimized in the system, and X = {X} C ,X′ C }, X C Let X′ represent the set of rotation and translation matrices of keyframes from a visible light camera within a sliding window. C r represents the set of rotation and translation matrices of the infrared camera keyframes within the sliding window. p To marginalize the residual, H p The marginalized Hessian matrix, where B represents the set of IMU measurement times. Let C represent the matrix used to calculate the IMU residual information, where C is the set of binocular landmarks in all visible light and infrared images within the sliding window, and Q is the set of monocular landmarks in all visible light and infrared images within the sliding window. The information matrix represents the camera reprojection error, and L is the set of all landmark points corresponding to the visible light image and infrared image matched during the loop closure detection process and located within the sliding window.

Images