Visual inertial positioning method for visible light and infrared cross-modal matching

Abstract

The invention discloses a visual inertial positioning method for visible light and infrared cross-modal matching, and belongs to the technical field of multi-source fusion navigation. According to the invention, the problems of poor positioning effect and loopback detection failure in the prior art in a visual degradation environment are solved. According to the method, rich texture features of a visible light camera and features of an infrared camera which are not influenced by illumination conditions are combined, and a visual inertial positioning system with visible light and infrared cross-modal matching is constructed. Visual feature points and descriptors of a visible light image and a thermal infrared image are extracted through a public deep neural network, the confidence degree of the features is provided, classification management of cross-modal feature points is provided, then the depth of road sign points is recovered through two-view triangularization, loopback detection is carried out based on cosine similarity of bag-of-word vectors, and the visual feature points and descriptors of the visible light image and the thermal infrared image are obtained. And performing joint optimization by utilizing a depth estimation value, a loopback detection result and an inertia pre-integration measurement value to obtain positions and postures of the visible light camera and the infrared camera. The method can be applied to the field of multi-source fusion navigation.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-source fusion navigation, and in particular relates to a visual inertial positioning method for cross-modal matching of visible light and infrared. Background Art

[0002] Visual-Inertial Odometry (VIO) is a positioning technology that combines visual and inertial information. It is widely used in robotics, autonomous driving, augmented reality (AR), and virtual reality (VR). Traditional VIO methods usually rely on visual input from a single modality (such as visible light images). However, factors such as lighting changes, occlusions, and dynamic objects in complex environments may cause information loss in the visible light modality, thereby affecting positioning performance. For this reason, infrared images, with their unique advantages in low-light conditions, are matched cross-modally with visible light images, becoming an important research direction for enhancing the robustness of VIO.

[0003] A patent application with publication number CN117036218A proposes a method for fusing infrared images and visible light images based on discrete wavelet transform. The method extracts approximate components and detail components by performing discrete wavelet decomposition on infrared and visible light images respectively. Subsequently, a monochromatic threshold fusion strategy is used to perform weighted averaging on the two components to generate a fused result. Finally, a discrete wavelet inverse transform is performed on the weighted fusion result to generate the final fused image, and the motion state of the camera is estimated based on the visual odometer method and the fused image.

[0004] The patent application with publication number CN116977369A uses two-dimensional discrete wavelet transform to decompose the infrared and visible light images in the environment respectively to obtain approximate components and detail components. Subsequently, the approximate components and detail components of the infrared image and the visible light image are weighted and processed to generate a fused image using monochrome threshold fusion technology. Next, an improved RCNN network is designed, which uses the fused image as input data and is trained to obtain a prediction model. Finally, the prediction model is used to process the test data, from which the motion trajectory of the camera is inferred, thereby completing the visual odometer task.

[0005] The literature (Qin L, Wu C, Kong X, et al. BVT-SLAM: A Binocular Visible-Thermal Sensors SLAM System in Low-Light Environments [J]. IEEE Sensors Journal, 2024, 24 (7): 11599-11609.) proposed a binocular visible-infrared SLAM system (BVT-SLAM), which improved the BRIEF descriptor. Based on a binocular system consisting of a visible light camera and an infrared camera, it can adaptively switch between monocular and binocular modes, and proposed a highly efficient visible light and thermal image feature point matching method. Based on BDBRIEF, it can effectively complete multispectral stereo matching in the BVT-SLAM system, while enabling the system to run in real time. However, the improved BRIEF descriptor is a manually debugged descriptor, which is difficult to cover all possible changes and abnormal situations, resulting in poor generalization ability of the model in new scenes.

[0006] In summary, although the existing technology has made certain achievements in the fusion of infrared images and visible light images, there are still problems such as poor positioning effect and loop detection failure caused by the failure of visible light cameras in visually degraded environments. Therefore, it is very necessary to propose a new visual inertial positioning method with cross-modal matching of visible light and infrared to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to solve the problems of poor positioning effect and loop detection failure in the prior art under visually degraded environments, and to propose a visual-inertial positioning method with cross-modal matching of visible light and infrared.

[0008] The technical solution adopted by the present invention to solve the above technical problems is: a visual inertial positioning method for cross-modal matching of visible light and infrared, the method specifically comprising the following steps:

[0009] Step 1: trigger a visible light camera, an infrared camera and an IMU measurement unit by the same pulse signal, both of which are installed on the unmanned system, and then read the image collected by the visible light camera, the image collected by the infrared camera and the IMU measurement sequence from the IMU measurement unit after the trigger;

[0010] Taking the timestamp of the image from the visible light camera as a reference, obtain an IMU measurement sequence set within the timestamp of the previous visible light camera image frame to the timestamp of the current visible light camera image frame. If the time difference between the read current visible light image frame and the current infrared image frame is within t0, the current visible light image frame, the current infrared image frame and the obtained IMU measurement sequence set are packaged into a data set, and processed in binocular inertial mode, that is, continue to execute step 2; if the time difference between the read current visible light image frame and the current infrared image frame is greater than or equal to t0, select the image with an earlier timestamp from the read current visible light image frame and the current infrared image frame, and process it in monocular inertial mode;

[0011] Step 2: Use a public deep neural network to extract the descriptor of each pixel in the current visible light image and the current infrared image, and determine the feature points in the current visible light image and the current infrared image according to the descriptor scores;

[0012] Step 3: judging whether the current visible light image is a key frame according to the visible light key frame image in the sliding window, and judging whether the current infrared image is a key frame according to the infrared key frame image in the sliding window;

[0013] If the current visible light image and the current infrared image are not key frames, there is no need to process the current visible light image and the current infrared image, and there is no need to add the current visible light image and the current infrared image to the sliding window, and the process returns to step 1.

[0014] If the current visible light image or the current infrared image is a key frame, the current visible light image is added to the sliding window as a key frame, and the current infrared image is added to the sliding window as a key frame;

[0015] And according to the descriptor, the feature points in the current visible light image and the current infrared image are matched to obtain the feature point matching result, and all the binocular landmark points of the current visible light image and the current infrared image are obtained according to the feature point matching result; the current visible light image is matched with the previous visible light image key frame for feature points, and the current infrared image is matched with the previous infrared image key frame for feature points, and the monocular landmark points corresponding to the current visible light image and the current infrared image are obtained;

[0016] And respectively calculate the reprojection error of the binocular landmark points on the current visible light image and the current infrared image, the reprojection error of the monocular landmark points corresponding to the current visible light image on the current visible light image and the previous visible light image key frame, and the reprojection error of the monocular landmark points corresponding to the current infrared image on the current infrared image and the previous infrared image key frame;

[0017] Then proceed to step 4;

[0018] Step 4: Construct the bag-of-words vectors of the current frame visible light image and the current frame infrared image respectively; specifically:

[0019] Create a visual vocabulary table V containing K visual words. For any feature point in the current frame visible light image, calculate the cosine distance between the descriptor of the feature point and each visual word in the visual vocabulary table V, and assign the visual word corresponding to the minimum cosine distance to the feature point. Similarly, assign a visual word to each feature point in the current frame visible light image, and then count the number of times each visual word appears in the current frame visible light image. Use the number of times corresponding to each visual word to form a bag-of-words vector for the current frame visible light image.

[0020] Similarly, the bag-of-words vector of the current frame infrared image is obtained;

[0021] Step 5: construct an IMU pre-integration constraint model using an IMU measurement sequence set located between the timestamp of the previous visible light image key frame and the timestamp of the current frame visible light image;

[0022] Step 6: Perform loop detection based on the word bag vectors corresponding to the historical visible light image key frames, historical infrared image key frames and the current frame visible light image in the sliding window, and obtain loop detection results based on the word bag vectors corresponding to the historical visible light image key frames, historical infrared image key frames and the current frame infrared image in the sliding window.

[0023] Step 7: Establish a global objective function based on the reprojection error, the IMU pre-integration constraint model, and the loop detection result, solve the global objective function, and obtain the visual inertial positioning results of the visible light camera and the infrared camera;

[0024] Return to step 1.

[0025] Furthermore, the common deep neural network includes a first adapter, a second adapter, an encoder and a decoder; wherein:

[0026] The input of the first adapter is the visible light image of the current frame. The output of the first adapter is used as the input of the encoder. The encoder outputs the descriptor of each pixel in the visible light image of the current frame. The output of the encoder is used as the input of the decoder. The decoder outputs the feature score of each pixel in the visible light image of the current frame.

[0027] The input of the second adapter is the infrared image of the current frame. The output of the second adapter is used as the input of the encoder. The encoder outputs the descriptor of each pixel in the infrared image of the current frame. The output of the encoder is used as the input of the decoder. The decoder outputs the feature score of each pixel in the infrared image of the current frame.

[0028] Loss functions used in public deep neural network training for:

[0029]

[0030] in, is the descriptor loss function, is the peak loss function, is the feature score loss function, λ is a hyperparameter;

[0031]

[0032] in, Represents the total number of pixels in the visible light image Seeking expectations, D N and D′ N Represents the descriptor set of pixels in visible light image and infrared image respectively;

[0033]

[0034] Among them, {d i ,d i′} is a pair of matching pixel descriptors in the visible light image and the infrared image, d j It is set D N Middle and d i The nearest descriptor, d k is the set D′ N Middle and d i′ The nearest descriptor, d m is the set D′ N Middle and d i The nearest non-matching descriptor, d n It is set D N Middle and d i′ The nearest non-matching descriptor, θ(·,·) represents the angle between two descriptor vectors;

[0035] c(s i ,s i ′)=s i s i′

[0036] Among them, s i is the decoder output d i Score, s i′ is the decoder output d i′ score;

[0037]

[0038] Among them, p represents the sliding image block, D N [p[i]] represents the descriptor of the i-th pixel in the sliding image block p on the visible light image, D′ w represents the infrared image after affine transformation of w, D′ w [p[i]] represents the descriptor of the i-th pixel in the sliding image block p on the infrared image after the affine transformation w. is the D of all pixels in the sliding image block p N [p[i]] T D′ w [p[i]] is the expectation, S is the score set of all pixel descriptors on the visible light image, S′ w is the score set of the descriptors of all pixels on the infrared image after the affine transformation of w, S[p] is the score matrix of the pixels in the sliding image block p in the set S, and S′ w [p] is the set S′ w The score matrix of the pixels in the sliding image block p; is b(p; D N ,D′ w )×(1-S[p] T S′ w [p])Seek expectation;

[0039]

[0040] Where I is the edge image of the visible light image, AP represents the average pooling layer, MP represents the maximum pooling layer, AP(S) represents the output of S after the average pooling layer, AP(S)[i] represents the value of the i-th pixel in AP(S), MP(S) represents the output of S after the maximum pooling layer, (1-MP(S))[i] represents the value of the i-th pixel in 1-MP(S), Represents the AP(S)[i] for all pixels on the visible light image 2 +(1-MP(S))[i]) 2 Find the expectation, S[i] represents the score of the descriptor of the i-th pixel in S, M(I) is the processing result of edge image I, Represents all pixels Seek expectations;

[0041]

[0042] in, is the Laplacian operator of the image, ||·|| means calculating the 2-norm, express The value of the i-th pixel in the function x represents the function independent variable.

[0043] Furthermore, determining the feature points in the current frame visible light image and the current frame infrared image according to the descriptor scores is specifically as follows:

[0044] For the current frame of visible light image, the descriptor scores of each pixel in the visible light image are sorted from large to small, and then the pixels in the top 20% are taken as feature points, and the number of feature points c is compared with the threshold:

[0045] If the number of feature points c is less than the set threshold, continue to select pixels with the highest descriptor scores from the sorting results as feature points until the number of feature points reaches the set threshold; if the number of feature points c is greater than or equal to the set threshold, directly select the top 20% of pixels as feature points;

[0046] Similarly, the feature points in the current frame infrared image are obtained.

[0047] Furthermore, the specific process of step 3 is as follows:

[0048] Step 31, determining whether the time difference between the current visible light image frame and the previous visible light image key frame exceeds a time threshold;

[0049] If the time threshold is not exceeded, the current frame of visible light image is not a key frame;

[0050] If the time threshold is exceeded, the current visible light image frame is feature matched with the previous visible light image key frame to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current visible light image frame is added to the key frame list of the visible light image, otherwise, the current visible light image frame is not a key frame;

[0051] Determine whether the time difference between the current infrared image frame and the previous infrared image key frame exceeds a time threshold;

[0052] If it does not exceed the time threshold, the current frame of infrared image is not a key frame;

[0053] If the time threshold is exceeded, the current infrared image frame is feature matched with the previous infrared image key frame to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current infrared image frame is added to the key frame list of the infrared image, otherwise, the current infrared image frame is not a key frame;

[0054] If the current visible light image and the current infrared image are not key frames, there is no need to process the current visible light image and the current infrared image, and there is no need to add the current visible light image and the current infrared image to the sliding window, and the process returns to step 1.

[0055] If the current visible light image or the current infrared image is a key frame, the current visible light image is added to the sliding window as a key frame, and the current infrared image is added to the sliding window as a key frame;

[0056] Then execute step 32;

[0057] Step 32: Use a similar projection matching method to match feature points in the current frame visible light image and the current frame infrared image to obtain all successfully matched feature point pairs;

[0058] Step 33, respectively obtain the points on the map corresponding to each set of successfully matched feature point pairs in step 32, for any set of successfully matched feature point pairs (x1, y1) and (x2, y2), record the point on the map corresponding to the feature point pair as p, calculate the reprojection error of point p on the visible light image of the current frame according to the visible light image key frame in the sliding window, and calculate the reprojection error of point p on the infrared image of the current frame according to the infrared image key frame in the sliding window;

[0059] If the reprojection error of point p on the current frame visible light image and the current frame infrared image is less than 2 pixels, then p is added to the binocular landmark point set as a binocular landmark point;

[0060] Each pair of feature points successfully matched in step 31 is then used as a monocular landmark point, and the monocular landmark point is added to the monocular landmark point set;

[0061] Step 34, respectively calculate the reprojection error of each monocular landmark point corresponding to the current frame visible light image on the current frame visible light image and the key frame of the previous visible light image, the reprojection error of each monocular landmark point corresponding to the current frame infrared image on the current frame infrared image and the key frame of the previous infrared image, and the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image on the current frame visible light image and the current frame infrared image.

[0062] Furthermore, the specific process of step 32 is as follows:

[0063] For any feature point a on the current frame visible light image, the calibration parameters are used to transform and project the feature point a onto the current frame infrared image, and the feature point b with the smallest descriptor distance from the feature point a is found from the current frame infrared image. The calibration parameters are then used to transform and project the feature point b onto the current frame visible light image, and the feature point with the smallest descriptor distance from the feature point b is found from the current frame visible light image.

[0064] If the feature point a has the smallest descriptor distance with the feature point b in the current frame visible light image, then the feature point a is successfully matched with the feature point b;

[0065] If the feature point with the smallest descriptor distance to feature point b in the current frame visible light image is not a, the matching between feature point a and feature point b is unsuccessful;

[0066] Similarly, each feature point on the current frame visible light image is processed separately to obtain all successfully matched feature point pairs.

[0067] Furthermore, the points corresponding to each set of successfully matched feature point pairs on the map in step 32 are obtained respectively, specifically:

[0068] For any set of successfully matched feature point pairs (x1, y1) and (x2, y2):

[0069]

[0070] Among them, (x1, y1) and (x2, y2) are the coordinates of the feature points on the current frame visible light image and the current frame infrared image respectively. Represents the jth row of the projection matrix corresponding to the current frame visible light image, represents the jth row of the projection matrix corresponding to the infrared image of the current frame, and p is the landmark point corresponding to (x1, y1) and (x2, y2) on the map;

[0071] Similarly, the points on the map corresponding to each set of successfully matched feature point pairs in step 32 are obtained respectively.

[0072] Furthermore, in step 34, the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image on the current frame visible light image and the current frame infrared image is calculated, and the specific process is:

[0073] For the pixel points (x1, y1) and (x2, y2) corresponding to the binocular landmark point p, convert the binocular landmark point p to the visible light camera coordinate system, and mark the coordinates of point p in the visible light camera coordinate system as

[0074]

[0075] in, is the rotation matrix, is the translation matrix;

[0076]

[0077] in, is the external parameter matrix between the visible light camera and IMU, is the rotation matrix of the kth visible light image key frame in the sliding window relative to the k-1th visible light image key frame in the sliding window, is the translation matrix of the kth visible light image key frame in the sliding window relative to the k-1th visible light image key frame in the sliding window;

[0078] Will Substitute into the pinhole camera model π(·):

[0079]

[0080] In the formula, f x and f y They represent the two-axis focal lengths of the visible light camera, c x and c y They represent the two-axis optical center offset of the visible light camera, express The coordinates projected onto the visible light image, x Z express The Z-axis coordinate in ;

[0081] Calculate a reprojection error of the binocular landmark point p based on the visible light pixel point (x1, y1):

[0082]

[0083] in, is the reprojection error;

[0084] Convert the binocular landmark point p to the infrared camera coordinate system and mark the coordinates of point p in the infrared camera coordinate system as

[0085]

[0086] in, is the rotation matrix, is the translation matrix;

[0087]

[0088] in, is the external parameter matrix between the infrared camera and IMU, is the rotation matrix of the kth infrared image key frame in the sliding window relative to the k-1th infrared image key frame in the sliding window, is the translation matrix of the kth infrared image key frame in the sliding window relative to the k-1th infrared image key frame in the sliding window;

[0089] Will Substitute into the pinhole camera model π(·):

[0090]

[0091] In the formula, fx ′ and f y ′ represents the two-axis focal length of the infrared camera, c x ′ and c′ y They represent the two-axis optical center offset of the infrared camera, express Coordinate projected onto the infrared image, x′ Z express The Z-axis coordinate in ;

[0092] Calculate another reprojection error of the binocular landmark point p based on (x2, y2):

[0093]

[0094] in, is the reprojection error.

[0095] Furthermore, the specific process of step 5 is as follows:

[0096] Step 51, calculate the pre-integrated variable relationship between the IMU measurement sequence at the i′th moment and the IMU measurement sequence at the j′th moment:

[0097]

[0098] Among them, ~ represents the measured value, is the measured value of the IMU acceleration in the IMU measurement sequence at the k′th moment, is the measured value of the IMU angular velocity in the IMU measurement sequence at the k′th moment, is the gyroscope offset in the IMU measurement sequence at the k′th moment, is the accelerometer offset in the IMU measurement sequence at the k′th moment, is the random walk noise of the gyroscope in the IMU measurement sequence at the k′th moment, is the random walk noise of the accelerometer in the IMU measurement sequence at the k′th moment, Δt is the time difference between the IMU measurement sequence at the i′th moment and the IMU measurement sequence at the k′th moment, ΔR i′k′ is the pre-integrated variable rotated between the i′th moment and the k′th moment, Δv i′k′ is the pre-integrated variable of the velocity between the i′th moment and the k′th moment, is the measured value of the pre-integrated variable rotated between the i′th moment and the j′th moment, is the measured value of the pre-integrated variable of the velocity between the i′th moment and the j′th moment, is the measured value of the pre-integrated variable at the position between the i′th moment and the j′th moment, Exp represents the exponential mapping of the Lie algebra;

[0099] Step 52: Build IMU pre-integration constraint model

[0100]

[0101] Among them, Δp i′j′ is the pre-integrated variable of the position between the i′th moment and the k′th moment, is the gyroscope offset in the IMU measurement sequence at the j′th moment, ΔR i′j′ is the pre-integrated variable rotated between the i′th moment and the j′th moment, Δv i′j′ is the pre-integrated variable of the velocity between the i′th moment and the j′th moment, is the gyroscope offset in the IMU measurement sequence at the i′th moment, is the accelerometer offset in the IMU measurement sequence at the j′th moment, is the accelerometer offset in the IMU measurement sequence at the i′th moment, is the variable to be optimized, and Log[·] represents the logarithmic mapping of the Lie group.

[0102] Furthermore, the specific process of step 6 is as follows:

[0103] For the current visible light image, find the key frames with the same visual words as the current visible light image from the historical visible light image key frames and historical infrared image key frames in the sliding window, and then calculate the cosine similarity between the bag-of-words vector of the current visible light image and the bag-of-words vector of each key frame found;

[0104] Among them, the calculation method of the cosine similarity of two word bag vectors is:

[0105]

[0106] Among them, H1 and H2 are two bag-of-words vectors, “·” represents the inner product of the vector, and ||·|| represents the Euclidean norm of the vector;

[0107] If there is a key frame whose cosine similarity with the bag-of-words vector of the current frame visible light image is higher than the threshold, then the current frame visible light image detects a loop, and the key frame whose cosine similarity is higher than the threshold is the key frame detected by the loop of the current frame visible light image;

[0108] If there is no key frame whose cosine similarity of the bag-of-words vector with the current frame visible light image is higher than the threshold, no loop is detected in the current frame visible light image;

[0109] Similarly, loop closure detection is performed on the current frame infrared image.

[0110] Furthermore, the global objective function is:

[0111]

[0112] Where X0 = {X B ,X} is the sum of the state variables to be optimized, X={X C ,X′ C}, X C represents the set of rotation matrices and translation matrices of the visible light camera keyframes in the sliding window, X′ C Represents the set of rotation matrices and translation matrices of the infrared camera keyframes in the sliding window, r p is the marginalized residual, H p is the marginalized Hessian matrix, B represents the IMU measurement time set, represents the calculation of the IMU residual information matrix, C is the set of binocular landmark points in all visible light images and all infrared images in the sliding window, Q is the set of monocular landmark points in all visible light images and all infrared images in the sliding window, The information matrix representing the camera reprojection error, L is the set of all landmark points corresponding to the visible light image and infrared image that are matched in the sliding window during the loop closure detection process.

[0113] The beneficial effects of the present invention are:

[0114] The present invention combines the rich texture features of visible light cameras and the features of infrared cameras that are not affected by lighting conditions to construct a visual inertial positioning system for cross-modal matching of visible light and infrared. Through a common deep neural network, the visual feature points and descriptors of visible light images and thermal infrared images are extracted respectively, and the confidence of the features is provided. The classification management of cross-modal feature points is proposed, and then the depth of the landmark points is restored by triangulation of the two views. At the same time, loop detection is performed based on the cosine similarity of the bag-of-words vector, so that the position and posture of the visible light camera and the infrared camera are obtained by joint optimization using the depth estimation value, loop detection result and inertial pre-integration measurement value. The method of the present invention can effectively solve the problems of poor positioning effect and loop detection failure of the existing technology in visually degraded environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 It is a flow chart of a visual inertial positioning method for cross-modal matching of visible light and infrared according to the present invention;

[0116] Figure 2 It is a public deep neural network architecture diagram; DETAILED DESCRIPTION

[0117] Specific implementation method 1: Combination Figure 1 This embodiment describes a visual inertial positioning method for cross-modal matching of visible light and infrared, which specifically includes the following steps:

[0118] Step 1: trigger the visible light camera, infrared camera and IMU measurement unit by the same pulse signal (when the IMU measurement unit is a model that does not support external triggering, the IMU measurement value at the time of the visible light camera image timestamp can be obtained by interpolation method). Both the visible light camera and the infrared camera are installed on the unmanned system, and then read the image collected by the visible light camera, the image collected by the infrared camera and the IMU measurement sequence from the IMU measurement unit after the trigger;

[0119] Taking the timestamp of the image from the visible light camera as a reference, obtain an IMU measurement sequence set within the timestamp of the previous visible light camera image frame to the timestamp of the current visible light camera image frame. If the time difference between the read current visible light image frame and the current infrared image frame is within t0 (taken as 3ms in the present invention), the visible light image of the current frame, the infrared image of the current frame and the obtained IMU measurement sequence set are packaged into a data set, and processed in binocular inertial mode, that is, continue to execute step 2; if the time difference between the read current visible light image frame and the current infrared image frame is greater than or equal to t0, select the image with an earlier timestamp from the read current visible light image frame and the current infrared image frame, and process it in monocular inertial mode;

[0120] Step 2: Use a public deep neural network to extract the descriptor of each pixel in the current visible light image and the current infrared image, and determine the feature points in the current visible light image and the current infrared image according to the descriptor scores;

[0121] Step 3: judging whether the current visible light image is a key frame according to the visible light key frame image in the sliding window, and judging whether the current infrared image is a key frame according to the infrared key frame image in the sliding window;

[0122] If the current visible light image and the current infrared image are not key frames, there is no need to process the current visible light image and the current infrared image, and there is no need to add the current visible light image and the current infrared image to the sliding window, and the process returns to step 1.

[0123] If the current visible light image or the current infrared image is a key frame, the current visible light image is added to the sliding window as a key frame, and the current infrared image is added to the sliding window as a key frame;

[0124] And according to the descriptor, the feature points in the current visible light image and the current infrared image are matched to obtain the feature point matching result, and all the binocular landmark points of the current visible light image and the current infrared image are obtained according to the feature point matching result; the current visible light image is matched with the previous visible light image key frame for feature points, and the current infrared image is matched with the previous infrared image key frame for feature points, and the monocular landmark points corresponding to the current visible light image and the current infrared image are obtained;

[0125] And respectively calculate the reprojection error of the binocular landmark points on the current visible light image and the current infrared image, the reprojection error of the monocular landmark points corresponding to the current visible light image on the current visible light image and the previous visible light image key frame, and the reprojection error of the monocular landmark points corresponding to the current infrared image on the current infrared image and the previous infrared image key frame;

[0126] Then proceed to step 4;

[0127] Step 4: Construct the bag-of-words vectors of the current frame visible light image and the current frame infrared image respectively; specifically:

[0128] Create a visual vocabulary table V containing K visual words. For any feature point in the current frame visible light image, calculate the cosine distance between the descriptor of the feature point and each visual word in the visual vocabulary table V, and assign the visual word corresponding to the minimum cosine distance to the feature point. Similarly, assign a visual word to each feature point in the current frame visible light image, and then count the number of times each visual word appears in the current frame visible light image. Use the number of times corresponding to each visual word to form a bag-of-words vector for the current frame visible light image.

[0129] Similarly, the bag-of-words vector of the current frame infrared image is obtained;

[0130] Step 5: construct an IMU pre-integration constraint model using an IMU measurement sequence set located between the timestamp of the previous visible light image key frame and the timestamp of the current frame visible light image;

[0131] Step 6: Perform loop detection based on the word bag vectors corresponding to the historical visible light image key frames, historical infrared image key frames and the current frame visible light image in the sliding window, and obtain loop detection results based on the word bag vectors corresponding to the historical visible light image key frames, historical infrared image key frames and the current frame infrared image in the sliding window.

[0132] Step 7: Establish a global objective function based on the reprojection error, the IMU pre-integration constraint model and the loop detection result (build the loop detection error model in the global objective function based on the key frames detected by the loop), solve the global objective function, and obtain the visual inertial positioning results of the visible light camera and the infrared camera in each frame in the sliding window;

[0133] Return to step 1.

[0134] Specific implementation method 2: Combination Figure 2 This embodiment is different from the first embodiment in that the common deep neural network includes a first adapter, a second adapter, an encoder and a decoder; wherein:

[0135] The input of the first adapter is the visible light image of the current frame. The output of the first adapter is used as the input of the encoder. The encoder outputs the descriptor of each pixel in the visible light image of the current frame. The output of the encoder is used as the input of the decoder. The decoder outputs the feature score of each pixel in the visible light image of the current frame.

[0136] The input of the second adapter is the infrared image of the current frame. The output of the second adapter is used as the input of the encoder. The encoder outputs the descriptor of each pixel in the infrared image of the current frame. The output of the encoder is used as the input of the decoder. The decoder outputs the feature score of each pixel in the infrared image of the current frame.

[0137] Loss functions used in public deep neural network training for:

[0138]

[0139] in, is the descriptor loss function, is the peak loss function, is the feature score loss function, λ is a hyperparameter (the value is 0.5 in this invention);

[0140]

[0141] in, Represents the total number of pixels in the visible light image Seeking expectations, D N and D′ N Represents the descriptor set of pixels in visible light image and infrared image respectively;

[0142]

[0143] Among them, {d i ,d i′} is a pair of matching pixel descriptors in the visible light image and the infrared image, d j It is set D N Middle and d i The nearest descriptor, d k is the set D′ N Middle and d i′ The nearest descriptor, d m is the set D′ N Middle and d i The nearest non-matching descriptor, d n It is set D N Middle and d i′ The nearest non-matching descriptor, θ(·,·) represents the angle between two descriptor vectors, expressed as d i and d j For example,

[0144] c(s i ,s i ′)=s i s i′

[0145] Among them, s i is the decoder output d i Score, s i′ is the decoder output d i′ score;

[0146]

[0147] Among them, p represents the sliding image block, D N [p[i]] represents the descriptor of the i-th pixel in the sliding image block p on the visible light image, D′ w represents the infrared image after w affine transformation (w affine transformation is a conventional perspective transformation, which is essentially a matrix multiplication), D′ w [p[i]] represents the descriptor of the i-th pixel in the sliding image block p on the infrared image after the affine transformation w. is the D of all pixels in the sliding image block p N [p[i]] T D′ w [p[i]] is the expectation, S is the score set of all pixel descriptors on the visible light image, S′ w is the score set of the descriptors of all pixels on the infrared image after the affine transformation of w, S[p] is the score matrix of the pixels in the sliding image block p in the set S, and S′ w [p] is the set S′ w The score matrix of the pixels in the sliding image block p; is b(p; DN ,D′ w )×(1-S[p] T S′ w [p])Seek expectation;

[0148]

[0149] Where I is the edge image of the visible light image, AP represents the average pooling layer, MP represents the maximum pooling layer, AP(S) represents the output of S after the average pooling layer, AP(S)[i] represents the value of the i-th pixel in AP(S), MP(S) represents the output of S after the maximum pooling layer, (1-MP(S))[i] represents the value of the i-th pixel in 1-MP(S), Represents the AP(S)[i] for all pixels on the visible light image 2 +(1-MP(S))[i]) 2 Find the expectation, S[i] represents the score of the descriptor of the i-th pixel in S, M(I) is the processing result of edge image I, Represents all pixels Seek expectations; and The calculation process is similar to;

[0150]

[0151] in, is the Laplacian operator of the image, ||·|| means calculating the 2-norm, express The value of the i-th pixel in the function x represents the function independent variable.

[0152] The other steps and parameters are the same as those in the first embodiment.

[0153] In the present invention, the working process of the first adapter is:

[0154] In the first adapter, the input image first passes through the first convolutional layer and the first Relu activation function layer in sequence;

[0155] The output of the first Relu activation function layer then passes through the second convolutional layer and the second Relu activation function layer in sequence; the outputs of the first Relu activation function layer and the second Relu activation function layer are added to obtain an addition result a;

[0156] The addition result a is passed through the third convolution layer and the third Relu activation function layer in sequence; the output of the third Relu activation function layer is passed through the fourth convolution layer and the fourth Relu activation function layer in sequence, and then the outputs of the third Relu activation function layer and the fourth Relu activation function layer are added to obtain the addition result b;

[0157] The addition result b is passed through the fifth convolution layer and the fifth Relu activation function layer in sequence; the output of the fifth Relu activation function layer is passed through the sixth convolution layer and the sixth Relu activation function layer in sequence, and then the outputs of the sixth Relu activation function layer and the fifth Relu activation function layer are added to obtain the addition result c, and the addition result c is used as the output of the first adapter.

[0158] The working process of the second adapter is the same as that of the first adapter.

[0159] The encoder includes three convolutional layers connected in series. By adjusting the number of channels, it finally outputs deep features of fixed dimension. The input of the encoder is the feature map from the adapter, and the output of the encoder is a 128-dimensional feature vector.

[0160] The decoder is the output module of the public deep neural network, and its function is to decode the high-dimensional feature map from the encoder into the importance or confidence of each pixel. The decoder consists of a first branch and a second branch. The input of the first branch comes from the feature vector output by the encoder. In the first branch, it first passes through a convolution layer with a convolution kernel size of 1×1, and then the output of the convolution layer is mapped to a positive interval through a softplus activation function and normalized to (0,1). The input of the second branch is also the feature vector from the encoder. The input passes through three series of convolution modules in sequence. Each convolution module includes a convolution layer, a batch normalization (BN) layer, and a ReLU activation function layer connected in sequence. The output of the last convolution module passes through three series of convolution units in sequence. Each convolution unit includes a convolution layer, a softmax layer, a BN layer, a ReLU activation function layer, an instance normalization (IN) layer, and a ReLU activation function layer. The output of the last convolution unit passes through a convolution layer and a Softmax layer again, and the output of the Softmax layer is used as the output of the second branch; the output of the first branch and the output of the second branch are multiplied pixel by pixel to finally obtain the feature score output by the decoder.

[0161] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the feature points in the current frame visible light image and the current frame infrared image are determined according to the descriptor scores, specifically:

[0162] For the current frame visible light image, the descriptor scores of each pixel in the visible light image are sorted from large to small, and then the top 20% of the pixels are used as feature points, and the number of feature points c is compared with the threshold (set to 1024 in the present invention):

[0163] If the number of feature points c is less than the set threshold, continue to select pixels with the highest descriptor scores from the sorting results as feature points until the number of feature points reaches the set threshold; if the number of feature points c is greater than or equal to the set threshold, directly select the top 20% of pixels as feature points;

[0164] Similarly, the feature points in the current frame infrared image are obtained.

[0165] The other steps and parameters are the same as those in the first or second embodiment.

[0166] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that the specific process of step 3 is as follows:

[0167] Step 31, determining whether the time difference between the current visible light image frame and the previous visible light image key frame exceeds a time threshold (the time threshold is designed to be 1 s in the present invention);

[0168] If the time threshold is not exceeded, the current frame of visible light image is not a key frame;

[0169] If the time threshold is exceeded, the current visible light image is matched with the previous visible light image key frame (i.e., the key frame in the history that is closest to the current visible light image in time) to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current visible light image is added to the key frame list of the visible light image, otherwise, the current visible light image is not a key frame;

[0170] Determine whether the time difference between the current infrared image frame and the previous infrared image key frame exceeds a time threshold (the time threshold is designed to be 1 s in the present invention);

[0171] If it does not exceed the time threshold, the current frame of infrared image is not a key frame;

[0172] If the time threshold is exceeded, the current infrared image frame is matched with the key frame of the previous infrared image (i.e., the key frame in the history of key frames that is closest to the current infrared image frame in time) to obtain all the successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current infrared image frame is added to the key frame list of the infrared image, otherwise, the current infrared image frame is not a key frame;

[0173] If the current visible light image and the current infrared image are not key frames, there is no need to process the current visible light image and the current infrared image, and there is no need to add the current visible light image and the current infrared image to the sliding window, and the process returns to step 1.

[0174] If the current visible light image or the current infrared image is a key frame, the current visible light image is added to the sliding window as a key frame, and the current infrared image is added to the sliding window as a key frame;

[0175] Then execute step 32;

[0176] Step 32, matching feature points in the current frame visible light image and the current frame infrared image to obtain all successfully matched feature point pairs;

[0177] Step 33, respectively obtain the points on the map corresponding to each set of successfully matched feature point pairs in step 32, for any set of successfully matched feature point pairs (x1, y1) and (x2, y2), record the point on the map corresponding to the feature point pair as p, calculate the reprojection error of point p on the visible light image of the current frame according to the visible light image key frame in the sliding window, and calculate the reprojection error of point p on the infrared image of the current frame according to the infrared image key frame in the sliding window;

[0178] If the reprojection error of point p on the current frame visible light image and the current frame infrared image is less than 2 pixels, then p is added to the binocular landmark point set as a binocular landmark point;

[0179] Each pair of feature points successfully matched in step 31 is then used as a monocular landmark point, and the monocular landmark point is added to the monocular landmark point set;

[0180] Step 34, respectively calculate the reprojection error of each monocular landmark point corresponding to the current frame visible light image on the current frame visible light image and the key frame of the previous visible light image, the reprojection error of each monocular landmark point corresponding to the current frame infrared image on the current frame infrared image and the key frame of the previous infrared image, and the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image on the current frame visible light image and the current frame infrared image.

[0181] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0182] Specific implementation mode 5: This implementation mode is different from the specific implementation modes 1 to 4 in that the specific process of step 32 is as follows:

[0183] For any feature point a on the current frame visible light image, the calibration parameters are used to transform and project the feature point a onto the current frame infrared image, and the feature point b with the smallest descriptor distance from the feature point a is found from the current frame infrared image. The calibration parameters are then used to transform and project the feature point b onto the current frame visible light image, and the feature point with the smallest descriptor distance from the feature point b is found from the current frame visible light image.

[0184] If the feature point a has the smallest descriptor distance with the feature point b in the current frame visible light image, then the feature point a is successfully matched with the feature point b;

[0185] If the feature point with the smallest descriptor distance to feature point b in the current frame visible light image is not a, the matching between feature point a and feature point b is unsuccessful (in this case, both feature point a and feature point b are regarded as monocular landmark points);

[0186] Similarly, each feature point on the current frame visible light image is processed separately to obtain all successfully matched feature point pairs.

[0187] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0188] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the points corresponding to each set of successfully matched feature point pairs in step 32 on the map are obtained respectively, specifically:

[0189] For any set of successfully matched feature point pairs (x1, y1) and (x2, y2):

[0190]

[0191] Among them, (x1, y1) and (x2, y2) are the coordinates of the feature points on the current frame visible light image and the current frame infrared image respectively. Represents the jth row of the projection matrix corresponding to the current frame visible light image, represents the jth row of the projection matrix corresponding to the infrared image of the current frame, and p is the landmark point corresponding to (x1, y1) and (x2, y2) on the map;

[0192] Similarly, the points on the map corresponding to each set of successfully matched feature point pairs in step 32 are obtained respectively.

[0193] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0194] The present invention is based on epipolar geometry constraints and uses all successfully matched feature point pairs on the current frame visible light image and the current frame infrared image to calculate the projection matrix corresponding to the current frame visible light image and the projection matrix corresponding to the current frame infrared image, and then based on the two obtained projection matrices, the points on the map corresponding to each group of successfully matched feature point pairs are obtained.

[0195] For the monocular landmark points corresponding to the current visible light image, all the successfully matched feature point pairs on the current visible light image and the previous visible light image are used to calculate the projection matrix corresponding to the current visible light image and the projection matrix corresponding to the previous visible light image, and then the points corresponding to each group of monocular landmark points on the map are obtained based on the two projection matrices. Similarly, the points corresponding to the monocular landmark points on the map corresponding to the current infrared image are calculated.

[0196] In the present invention, the process of calculating the corresponding points of the successfully matched feature point pairs on the map is referred to as restoring the depth of the landmark points by triangulating the two views.

[0197] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that, in step 34, the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image on the current frame visible light image and the current frame infrared image is calculated, and the specific process is as follows:

[0198] For the pixel points (x1, y1) and (x2, y2) corresponding to the binocular landmark point p, convert the binocular landmark point p to the visible light camera coordinate system, and mark the coordinates of point p in the visible light camera coordinate system as

[0199]

[0200] in, is the rotation matrix, is the translation matrix;

[0201]

[0202] in, is the extrinsic parameter matrix between the visible light camera and the IMU (obtained by any calibration algorithm), is the rotation matrix of the kth visible light image key frame in the sliding window relative to the k-1th visible light image key frame in the sliding window, is the translation matrix of the kth visible light image key frame in the sliding window relative to the k-1th visible light image key frame in the sliding window,

[0203] Will Substitute into the pinhole camera model π(·):

[0204]

[0205] In the formula, f x and f y They represent the two-axis focal lengths of the visible light camera, c x and c y They represent the two-axis optical center offset of the visible light camera, express The coordinates projected onto the visible light image, x Z express The Z-axis coordinate in (i.e. the Z-axis coordinate in the visible light camera coordinate system);

[0206] Calculate a reprojection error of the binocular landmark point p based on the visible light pixel point (x1, y1):

[0207]

[0208] in, is the reprojection error;

[0209] Convert the binocular landmark point p to the infrared camera coordinate system and mark the coordinates of point p in the infrared camera coordinate system as

[0210]

[0211] in, is the rotation matrix, is the translation matrix;

[0212]

[0213] in, is the external parameter matrix between the infrared camera and IMU, is the rotation matrix of the kth infrared image key frame in the sliding window relative to the k-1th infrared image key frame in the sliding window, is the translation matrix of the kth infrared image key frame in the sliding window relative to the k-1th infrared image key frame in the sliding window;

[0214] Will Substitute into the pinhole camera model π(·):

[0215]

[0216] In the formula, f x ′ and f y ′ represents the two-axis focal length of the infrared camera, c x ′ and c′ y They represent the two-axis optical center offset of the infrared camera, express Coordinate projected onto the infrared image, x′ Z express The Z-axis coordinate in ;

[0217] Calculate another reprojection error of the binocular landmark point p based on (x2, y2):

[0218]

[0219] in, is the reprojection error.

[0220] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0221] Similarly, the method of this embodiment is used to calculate the reprojection error of each monocular landmark point corresponding to the current frame visible light image on the current frame visible light image and the previous frame visible light image, and the reprojection error of each monocular landmark point corresponding to the current frame infrared image on the current frame infrared image and the previous frame infrared image.

[0222] Specific implementation eight: This implementation is different from specific implementations one to seven in that the specific process of step 5 is as follows:

[0223] Step 51, calculate the pre-integrated variable relationship between the IMU measurement sequence at the i′th moment and the IMU measurement sequence at the j′th moment:

[0224]

[0225] Among them, ~ represents the measured value, is the measured value of the IMU acceleration in the IMU measurement sequence at the k′th moment, is the measured value of the IMU angular velocity in the IMU measurement sequence at the k′th moment, is the gyroscope offset in the IMU measurement sequence at the k′th moment, is the accelerometer offset in the IMU measurement sequence at the k′th moment, is the random walk noise of the gyroscope in the IMU measurement sequence at the k′th moment, is the random walk noise of the accelerometer in the IMU measurement sequence at the k′th moment, Δt is the time difference between the IMU measurement sequence at the i′th moment and the IMU measurement sequence at the k′th moment, ΔR i′k′ is the pre-integrated variable rotated between the i′th moment and the k′th moment, Δv i′k′ is the pre-integrated variable of the velocity between the i′th moment and the k′th moment, is the measured value of the pre-integrated variable rotated between the i′th moment and the j′th moment, is the measured value of the pre-integrated variable of the velocity between the i′th moment and the j′th moment, is the measured value of the pre-integrated variable at the position between the i′th moment and the j′th moment, Exp represents the exponential mapping of the Lie algebra;

[0226]

[0227] The i′th moment represents the timestamp corresponding to the previous visible light image frame, and the j′th moment represents the timestamp corresponding to the current visible light image frame;

[0228] Step 52: Build IMU pre-integration constraint model

[0229]

[0230] Among them, Δp i′j′ is the pre-integrated variable of the position between the i′th moment and the k′th moment, is the gyroscope offset in the IMU measurement sequence at the j′th moment, ΔR i′j′ is the pre-integrated variable rotated between the i′th moment and the j′th moment, Δv i′j′ is the pre-integrated variable of the velocity between the i′th moment and the j′th moment, is the gyroscope offset in the IMU measurement sequence at the i′th moment, is the accelerometer offset in the IMU measurement sequence at the j′th moment, is the accelerometer offset in the IMU measurement sequence at the i′th moment, is the variable to be optimized, and Log[·] represents the logarithmic mapping of the Lie group.

[0231] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0232] In order to describe the constraints of inertial measurement unit (IMU) data on pose estimation, this step integrates IMU data into the optimization framework by constructing a pre-integration model to reduce the computational burden.

[0233]

[0234] Where R is the rotation matrix and I is the 3×3 identity matrix.

[0235] Specific implementation method 9: This implementation method is different from specific implementation methods 1 to 8 in that the specific process of step 6 is as follows:

[0236] For the current visible light image, find the key frames with the same visual words as the current visible light image from the historical visible light image key frames and historical infrared image key frames in the sliding window, and then calculate the cosine similarity between the bag-of-words vector of the current visible light image and the bag-of-words vector of each key frame found;

[0237] Among them, the calculation method of the cosine similarity of two word bag vectors is:

[0238]

[0239] Among them, H1 and H2 are two bag-of-words vectors, “·” represents the inner product of the vector, and ||·|| represents the Euclidean norm of the vector;

[0240] If there is a key frame whose cosine similarity with the bag-of-words vector of the current frame visible light image is higher than the threshold (it can be an infrared image key frame or a visible light image key frame, and the key frame whose cosine similarity with the current frame visible light image is higher than the threshold may not be unique), then the current frame visible light image detects a loop, and the key frame whose cosine similarity is higher than the threshold is the key frame detected by the loop of the current frame visible light image;

[0241] If there is no key frame whose cosine similarity of the bag-of-words vector with the current frame visible light image is higher than the threshold, no loop is detected in the current frame visible light image;

[0242] Similarly, loop closure detection is performed on the current frame infrared image.

[0243] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0244] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 1 to 6 in that the global objective function is:

[0245]

[0246] Where X0 = {X B ,X} is the sum of the state variables to be optimized, X={X C ,X′ C}, X C represents the set of rotation matrices and translation matrices of the visible light camera keyframes in the sliding window, X′ C Represents the set of rotation matrices and translation matrices of the infrared camera keyframes in the sliding window, r p is the marginalized residual, H p is the marginalized Hessian matrix, B represents the IMU measurement time set (all IMU measurement times within the timestamp range of the sliding window), represents the calculation of the IMU residual information matrix, C is the set of binocular landmark points in all visible light images and all infrared images in the sliding window, Q is the set of monocular landmark points in all visible light images and all infrared images in the sliding window, The information matrix representing the camera reprojection error, L is the set of all landmark points corresponding to the visible light image and infrared image that are matched in the sliding window during the loop closure detection process.

[0247] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0248] It should be noted that the third to fifth items in the global objective function are calculated using a method similar to that for calculating the reprojection error. In the present invention, a sliding window of a fixed size is set for the image, and the termination timestamp corresponding to the sliding window is the timestamp of the current frame image. When the historical image exceeds the timestamp range of the sliding window, the landmark points in the historical image are removed from the sets C, Q and L, that is, all frames and feature point information and IMU pre-integration information in the window are used to calculate the camera pose.

[0249] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual inertial positioning method for cross-modal matching of visible light and infrared, characterized in that: The method specifically comprises the following steps: Step 1: trigger a visible light camera, an infrared camera and an IMU measurement unit by the same pulse signal, both of which are installed on the unmanned system, and then read the image collected by the visible light camera, the image collected by the infrared camera and the IMU measurement sequence from the IMU measurement unit after the trigger; Taking the timestamp of the image from the visible light camera as a reference, obtain an IMU measurement sequence set within the timestamp of the previous visible light camera image frame to the timestamp of the current visible light camera image frame. If the time difference between the read current visible light image frame and the current infrared image frame is within t0, the current visible light image frame, the current infrared image frame and the obtained IMU measurement sequence set are packaged into a data set, and processed in binocular inertial mode, that is, continue to execute step 2; if the time difference between the read current visible light image frame and the current infrared image frame is greater than or equal to t0, select the image with an earlier timestamp from the read current visible light image frame and the current infrared image frame, and process it in monocular inertial mode; Step 2: Use a public deep neural network to extract the descriptor of each pixel in the current visible light image and the current infrared image, and determine the feature points in the current visible light image and the current infrared image according to the descriptor scores; Step 3: judging whether the current visible light image is a key frame according to the visible light key frame image in the sliding window, and judging whether the current infrared image is a key frame according to the infrared key frame image in the sliding window; If the current visible light image and the current infrared image are not key frames, there is no need to process the current visible light image and the current infrared image, and there is no need to add the current visible light image and the current infrared image to the sliding window, and the process returns to step 1. If the current visible light image or the current infrared image is a key frame, the current visible light image is added to the sliding window as a key frame, and the current infrared image is added to the sliding window as a key frame; And according to the descriptor, the feature points in the current visible light image and the current infrared image are matched to obtain the feature point matching result, and all the binocular landmark points of the current visible light image and the current infrared image are obtained according to the feature point matching result; the current visible light image is matched with the previous visible light image key frame for feature points, and the current infrared image is matched with the previous infrared image key frame for feature points, and the monocular landmark points corresponding to the current visible light image and the current infrared image are obtained; And respectively calculate the reprojection error of the binocular landmark points on the current visible light image and the current infrared image, the reprojection error of the monocular landmark points corresponding to the current visible light image on the current visible light image and the previous visible light image key frame, and the reprojection error of the monocular landmark points corresponding to the current infrared image on the current infrared image and the previous infrared image key frame; Then proceed to step 4; Step 4: Construct the bag-of-words vectors of the current frame visible light image and the current frame infrared image respectively; specifically: Create a visual vocabulary table V containing K visual words. For any feature point in the current frame visible light image, calculate the cosine distance between the descriptor of the feature point and each visual word in the visual vocabulary table V, and assign the visual word corresponding to the minimum cosine distance to the feature point. Similarly, assign a visual word to each feature point in the current frame visible light image, and then count the number of times each visual word appears in the current frame visible light image. Use the number of times corresponding to each visual word to form a bag-of-words vector for the current frame visible light image. Similarly, the bag-of-words vector of the current frame infrared image is obtained; Step 5: construct an IMU pre-integration constraint model using an IMU measurement sequence set located between the timestamp of the previous visible light image key frame and the timestamp of the current frame visible light image; Step 6: Perform loop detection based on the word bag vectors corresponding to the historical visible light image key frames, historical infrared image key frames and the current frame visible light image in the sliding window, and obtain loop detection results based on the word bag vectors corresponding to the historical visible light image key frames, historical infrared image key frames and the current frame infrared image in the sliding window. Step 7: Establish a global objective function based on the reprojection error, the IMU pre-integration constraint model, and the loop detection result, solve the global objective function, and obtain the visual inertial positioning results of the visible light camera and the infrared camera; Return to step 1.

2. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 1, characterized in that: The public deep neural network includes a first adapter, a second adapter, an encoder and a decoder; wherein: The input of the first adapter is the visible light image of the current frame. The output of the first adapter is used as the input of the encoder. The encoder outputs the descriptor of each pixel in the visible light image of the current frame. The output of the encoder is used as the input of the decoder. The decoder outputs the feature score of each pixel in the visible light image of the current frame. The input of the second adapter is the infrared image of the current frame. The output of the second adapter is used as the input of the encoder. The encoder outputs the descriptor of each pixel in the infrared image of the current frame. The output of the encoder is used as the input of the decoder. The decoder outputs the feature score of each pixel in the infrared image of the current frame. Loss functions used in public deep neural network training for: in, is the descriptor loss function, is the peak loss function, is the feature score loss function, λ is a hyperparameter; in, Represents the (c(s i ,s i′ )) Seeking expectations, D N and D′ N Represents the descriptor set of pixels in visible light image and infrared image respectively; Among them, {d i ,d i′ } is a pair of matching pixel descriptors in the visible light image and the infrared image, d j It is set D N Middle and d i The nearest descriptor, d k is the set D′ N Middle and d i′ The nearest descriptor, d m is the set D′ N Middle and d i The nearest non-matching descriptor, d n It is set D N Middle and d i′ The nearest non-matching descriptor, θ(·,·) represents the angle between two descriptor vectors; c(s i ,s i ′)=s i s i′ Among them, s i is the decoder output d i Score, s i′ is the decoder output d i′ score; Among them, p represents the sliding image block, D N [p[i]] represents the descriptor of the i-th pixel in the sliding image block p on the visible light image, D′ w represents the infrared image after affine transformation of w, D′ w [p[i]] represents the descriptor of the i-th pixel in the sliding image block p on the infrared image after the affine transformation w. is the D of all pixels in the sliding image block p N [p[i]] T D′ w [p[i]] is the expectation, S is the score set of all pixel descriptors on the visible light image, S′ w is the score set of the descriptors of all pixels on the infrared image after the affine transformation of w, S[p] is the score matrix of the pixels in the sliding image block p in the set S, and S′ w [p] is the set S′ w The score matrix of the pixels in the sliding image block p; is b(p; D N ,D′ w )×(1-S[p] T S′ w [p])Seek expectation; Where I is the edge image of the visible light image, AP represents the average pooling layer, MP represents the maximum pooling layer, AP(S) represents the output of S after the average pooling layer, AP(S)[i] represents the value of the i-th pixel in AP(S), MP(S) represents the output of S after the maximum pooling layer, (1-MP(S))[i] represents the value of the i-th pixel in 1-MP(S), Represents the AP(S)[i] for all pixels on the visible light image 2 +(1-MP(S))[i]) 2 Find the expectation, S[i] represents the score of the descriptor of the i-th pixel in S, M(I) is the processing result of edge image I, Represents all pixels Seek expectations; in, is the Laplacian operator of the image, ||·|| means calculating the 2-norm, express The value of the i-th pixel in the function x represents the function independent variable.

3. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 1, characterized in that: The determining of feature points in the current frame visible light image and the current frame infrared image according to the descriptor scores is specifically as follows: For the current frame of visible light image, the descriptor scores of each pixel in the visible light image are sorted from large to small, and then the pixels in the top 20% are taken as feature points, and the number of feature points c is compared with the threshold: If the number of feature points c is less than the set threshold, continue to select pixels with the highest descriptor scores from the sorting results as feature points until the number of feature points reaches the set threshold; if the number of feature points c is greater than or equal to the set threshold, directly select the top 20% of pixels as feature points; Similarly, the feature points in the current frame infrared image are obtained.

4. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 3 is characterized in that: The specific process of step 3 is as follows: Step 31, determining whether the time difference between the current visible light image frame and the previous visible light image key frame exceeds a time threshold; If the time threshold is not exceeded, the current frame of visible light image is not a key frame; If the time threshold is exceeded, the current visible light image frame is feature matched with the previous visible light image key frame to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current visible light image frame is added to the key frame list of the visible light image, otherwise, the current visible light image frame is not a key frame; Determine whether the time difference between the current infrared image frame and the previous infrared image key frame exceeds a time threshold; If it does not exceed the time threshold, the current frame of infrared image is not a key frame; If the time threshold is exceeded, the current infrared image frame is feature matched with the previous infrared image key frame to obtain all successfully matched feature point pairs; if the number of successfully matched feature point pairs is less than the threshold, the current infrared image frame is added to the key frame list of the infrared image, otherwise, the current infrared image frame is not a key frame; If the current visible light image and the current infrared image are not key frames, there is no need to process the current visible light image and the current infrared image, and there is no need to add the current visible light image and the current infrared image to the sliding window, and the process returns to step 1. If the current visible light image or the current infrared image is a key frame, the current visible light image is added to the sliding window as a key frame, and the current infrared image is added to the sliding window as a key frame; Then execute step 32; Step 32, matching feature points in the current frame visible light image and the current frame infrared image to obtain all successfully matched feature point pairs; Step 33, respectively obtain the points on the map corresponding to each set of successfully matched feature point pairs in step 32, for any set of successfully matched feature point pairs (x1, y1) and (x2, y2), record the point on the map corresponding to the feature point pair as p, calculate the reprojection error of point p on the visible light image of the current frame according to the visible light image key frame in the sliding window, and calculate the reprojection error of point p on the infrared image of the current frame according to the infrared image key frame in the sliding window; If the reprojection error of point p on the current frame visible light image and the current frame infrared image is less than 2 pixels, then p is added to the binocular landmark point set as a binocular landmark point; Each pair of feature points successfully matched in step 31 is then used as a monocular landmark point, and the monocular landmark point is added to the monocular landmark point set; Step 34, respectively calculate the reprojection error of each monocular landmark point corresponding to the current frame visible light image on the current frame visible light image and the key frame of the previous visible light image, the reprojection error of each monocular landmark point corresponding to the current frame infrared image on the current frame infrared image and the key frame of the previous infrared image, and the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image on the current frame visible light image and the current frame infrared image.

5. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 4, characterized in that: The specific process of step 32 is as follows: For any feature point a on the current frame visible light image, the calibration parameters are used to transform and project the feature point a onto the current frame infrared image, and the feature point b with the smallest descriptor distance from the feature point a is found from the current frame infrared image. The calibration parameters are then used to transform and project the feature point b onto the current frame visible light image, and the feature point with the smallest descriptor distance from the feature point b is found from the current frame visible light image. If the feature point a has the smallest descriptor distance with the feature point b in the current frame visible light image, then the feature point a is successfully matched with the feature point b; If the feature point with the smallest descriptor distance to feature point b in the current frame visible light image is not a, the matching between feature point a and feature point b is unsuccessful; Similarly, each feature point on the current frame visible light image is processed separately to obtain all successfully matched feature point pairs.

6. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 5, characterized in that: The points corresponding to each set of successfully matched feature point pairs on the map obtained in step 32 are specifically: For any set of successfully matched feature point pairs (x1, y1) and (x2, y2): Among them, (x1, y1) and (x2, y2) are the coordinates of the feature points on the current frame visible light image and the current frame infrared image respectively. Represents the jth row of the projection matrix corresponding to the current frame visible light image, represents the jth row of the projection matrix corresponding to the infrared image of the current frame, and p is the landmark point corresponding to (x1, y1) and (x2, y2) on the map; Similarly, the points on the map corresponding to each set of successfully matched feature point pairs in step 32 are obtained respectively.

7. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 6, characterized in that: In step 34, the reprojection error of each binocular landmark point corresponding to the current frame visible light image and the current frame infrared image on the current frame visible light image and the current frame infrared image is calculated. The specific process is: For the pixel points (x1, y1) and (x2, y2) corresponding to the binocular landmark point p, convert the binocular landmark point p to the visible light camera coordinate system, and mark the coordinates of point p in the visible light camera coordinate system as in, is the rotation matrix, is the translation matrix; in, is the external parameter matrix between the visible light camera and the IMU, is the rotation matrix of the kth visible light image key frame in the sliding window relative to the k-1th visible light image key frame in the sliding window, is the translation matrix of the kth visible light image key frame in the sliding window relative to the k-1th visible light image key frame in the sliding window; Will Substitute into the pinhole camera model π(·): In the formula, f x and f y They represent the two-axis focal lengths of the visible light camera, c x and c y They represent the two-axis optical center offset of the visible light camera, express The coordinates projected onto the visible light image, x Z express The Z-axis coordinate in ; Calculate a reprojection error of the binocular landmark point p based on the visible light pixel point (x1, y1): in, is the reprojection error; Convert the binocular landmark point p to the infrared camera coordinate system and mark the coordinates of point p in the infrared camera coordinate system as in, is the rotation matrix, is the translation matrix; in, is the external parameter matrix between the infrared camera and IMU, is the rotation matrix of the kth infrared image key frame in the sliding window relative to the k-1th infrared image key frame in the sliding window, is the translation matrix of the kth infrared image key frame in the sliding window relative to the k-1th infrared image key frame in the sliding window; Will Substitute into the pinhole camera model π(·): In the formula, f x ′ and f y ′ represents the two-axis focal length of the infrared camera, c x ′ and c′ y They represent the two-axis optical center offset of the infrared camera, express Coordinate projected onto the infrared image, x′ Z express The Z-axis coordinate in ; Calculate another reprojection error of the binocular landmark point p based on (x2, y2): in, is the reprojection error.

8. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 7, characterized in that: The specific process of step 5 is as follows: Step 51, calculate the pre-integrated variable relationship between the IMU measurement sequence at the i′th moment and the IMU measurement sequence at the j′th moment: Among them, ~ represents the measured value, is the measured value of the IMU acceleration in the IMU measurement sequence at the k′th moment, is the measured value of the IMU angular velocity in the IMU measurement sequence at the k′th moment, is the gyroscope offset in the IMU measurement sequence at the k′th moment, is the accelerometer offset in the IMU measurement sequence at the k′th moment, is the random walk noise of the gyroscope in the IMU measurement sequence at the k′th moment, is the random walk noise of the accelerometer in the IMU measurement sequence at the k′th moment, Δt is the time difference between the IMU measurement sequence at the i′th moment and the IMU measurement sequence at the k′th moment, ΔR i′k′ is the pre-integrated variable rotated between the i′th moment and the k′th moment, Δv i′k′ is the pre-integrated variable of the velocity between the i′th moment and the k′th moment, is the measured value of the pre-integrated variable rotated between the i′th moment and the j′th moment, is the measured value of the pre-integrated variable of the velocity between the i′th moment and the j′th moment, is the measured value of the pre-integrated variable at the position between the i′th moment and the j′th moment, Exp represents the exponential mapping of the Lie algebra; Step 52: Build IMU pre-integration constraint model Among them, Δp i′j′ is the pre-integrated variable of the position between the i′th moment and the k′th moment, is the gyroscope offset in the IMU measurement sequence at the j′th moment, ΔR i′j′ is the pre-integrated variable rotated between the i′th moment and the j′th moment, Δv i′j′ is the pre-integrated variable of the velocity between the i′th moment and the j′th moment, is the gyroscope offset in the IMU measurement sequence at the i′th moment, is the accelerometer offset in the IMU measurement sequence at the j′th moment, is the accelerometer offset in the IMU measurement sequence at the i′th moment, is the variable to be optimized, and Log[·] represents the logarithmic mapping of the Lie group.

9. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 8, characterized in that: The specific process of step 6 is as follows: For the current visible light image, find the key frames with the same visual words as the current visible light image from the historical visible light image key frames and historical infrared image key frames in the sliding window, and then calculate the cosine similarity between the bag-of-words vector of the current visible light image and the bag-of-words vector of each key frame found; Among them, the calculation method of the cosine similarity of two word bag vectors is: Among them, H1 and H2 are two bag-of-words vectors, "·" represents the inner product of the vector, and ||·|| represents the Euclidean norm of the vector; If there is a key frame whose cosine similarity with the bag-of-words vector of the current frame visible light image is higher than the threshold, then the current frame visible light image detects a loop, and the key frame whose cosine similarity is higher than the threshold is the key frame detected by the loop of the current frame visible light image; If there is no key frame whose cosine similarity of the bag-of-words vector with the current frame visible light image is higher than the threshold, no loop is detected in the current frame visible light image; Similarly, loop closure detection is performed on the current frame infrared image.

10. The visual inertial positioning method for cross-modal matching of visible light and infrared according to claim 9, characterized in that: The global objective function is: Where X0={X B ,X} is the sum of the state variables to be optimized, X={X C ,X′ C }, X C represents the set of rotation matrices and translation matrices of the visible light camera keyframes in the sliding window, X′ C Represents the set of rotation matrices and translation matrices of the infrared camera keyframes in the sliding window, r p is the marginalized residual, H p is the marginalized Hessian matrix, B represents the IMU measurement time set, represents the calculation of the IMU residual information matrix, C is the set of binocular landmark points in all visible light images and all infrared images in the sliding window, Q is the set of monocular landmark points in all visible light images and all infrared images in the sliding window, The information matrix representing the camera reprojection error, L is the set of all landmark points corresponding to the visible light image and infrared image that are matched in the sliding window during the loop closure detection process.

Images