A multi-camera visual-inertial odometry method fusing infrared images

Abstract

The application provides a multi-camera vision-inertial odometer method for fusing infrared images, step 1: sensor layout uses front-view and rear-view cameras in combination, and the combination scheme balances the perception ability and the calculation power consumption of the environment; the visual residual error of the multi-camera system is constructed; step 2: a mode selection strategy is formulated, in the process of odometer operation, it is determined to select the visible light or infrared mode as the operation main mode; after the main mode is selected, the other mode will play an auxiliary role; step 3: after the mode selection in step 2 is determined, it is determined which information in the auxiliary mode is non-redundant and can be used for auxiliary positioning, and the fusion weight of the auxiliary information is determined; step 4: based on the tight coupling odometer, the estimation of each state quantity is carried out, the constraints provided by each sensor observation are unified into an optimization problem, and the visual-inertial tight coupling information fusion is constructed. The application fully utilizes the visible light and infrared information, reduces the redundant observation, and improves the performance and robustness of the system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of unmanned aerial vehicle / car navigation, and particularly relates to a multi-camera vision-inertial odometer method fusing infrared images. BACKGROUND

[0002] The existing visible light and infrared image fusion visual odometer method mainly includes two technical paths of a traditional visual odometer and a deep learning-based visual odometer. The traditional method generates a fusion image containing information of both visible light and infrared images by performing wavelet transform or monochrome threshold fusion on the visible light and infrared images, and then processes the fusion image by using a traditional visual odometer to realize positioning. The deep learning method constructs a multi-modal network or uses an attention mechanism to directly learn the features of visible light and infrared images and fuse them, thereby improving the positioning accuracy. Both methods need to perform image acquisition, preprocessing, feature extraction, feature matching, pose estimation, and trajectory reconstruction, and finally realize estimation and positioning of camera motion.

[0003] CN202310955029.4 provides a visible light and infrared fusion-based visual odometer method. First, the visible light and infrared images are decomposed by using discrete wavelet transform to obtain approximate components and detail components. Then, the approximate components and detail components are fused by using a monochrome threshold fusion method, and the inverse transform of the discrete wavelet transform is used to obtain a fusion image. Finally, the fusion image is input to estimate the camera state by using an optical flow feature extraction method and a feature point triangulation method. This method fuses the texture information of the visible light image and the heat source information of the infrared image.

[0004] CN 202310956658.9 provides a visible light and infrared fusion-based visual odometer method based on deep learning. The method proposes to acquire infrared images and visible light images of an environment by using a camera, and decompose the images by using two-dimensional discrete wavelet transform to obtain respective approximate components and detail components. The components are fused by using a monochrome threshold fusion method to obtain a fusion image containing detail information of the infrared image and texture information of the visible light image. Finally, an improved RCNN network is constructed, and the fusion image is input as an image frame for training to obtain a prediction model. By acquiring to-be-measured data and using the model, the trajectory of the camera motion can be predicted to realize the visual odometer function.

[0005] The above prior art mainly focuses on enhancing the performance of visual odometry through image fusion technology, without involving the fusion of multi-camera systems and multiple modalities. The methods of these two patents focus on combining the information of visible light and infrared images using discrete wavelet transform and monochrome threshold fusion methods to improve the detectability of features and the stability of tracking. However, this method may encounter challenges when dealing with complex environments or changing light conditions, as it does not take advantage of the wider field of view and rich environmental information provided by multi-camera systems, nor does it consider the possible redundant and complementary information between different modalities. Multi-camera systems can provide more comprehensive visual coverage, reduce blind areas, and improve the integrity and coverage of the map, while the fusion of multiple modalities can maintain stable feature detection and tracking capabilities under different lighting conditions. SUMMARY

[0006] To solve the above technical problems, the present application provides a multi-camera visual-inertial odometry method with infrared image fusion. By increasing the number of cameras, the system can capture environmental information from multiple perspectives, improving the perception of the scene and reducing errors through mutual verification between multiple cameras. In addition, multi-camera systems can provide wider visual coverage, reduce blind areas, and improve the integrity and coverage of the map. An adaptive visible-infrared image fusion method is also proposed, which can flexibly select and adjust the use of cameras according to the current environmental conditions. The visible-infrared odometry can operate using only the visible light modality or the infrared modality, or both modalities simultaneously. By fully utilizing visible light and infrared information, reducing redundant observations, and improving the performance and robustness of the system.

[0007] The specific technical solutions are as follows:

[0008] A multi-camera visual-inertial odometry method with infrared image fusion, comprising the following steps:

[0009] Step 1: Sensor layout uses front and rear cameras, balancing the perception ability of the environment and the consumption of computing power; constructing visual residuals of the multi-camera system;

[0010] In step 1, constructing visual residuals of the multi-camera system includes the following sub-steps:

[0011] 1.1, for the lth feature point P l , assuming that P l is observed in the ith frame and the jth frame, which provides visual constraints for the ith frame and the jth frame, the visual residual is constructed as follows:

[0012]

[0013] where, is the observed coordinate of the lth feature point in the jth frame in the camera normalized coordinate system, is the estimated value of the lth feature point in the jth frame in the camera normalized coordinate system, where z denotes the z component of a three-dimensional vector, xy denotes the xy component of a three-dimensional vector, is the back-projection formula, is the pixel coordinate of the lth feature point in the jth frame, and K is the intrinsic matrix of the camera. In the above formula, the form is denotes the rotation matrix and translation vector from the f2 coordinate system to the f1 coordinate system.

[0014] 1.2. In the process of continuous motion, visible light frames and infrared frames and camera input frames under different viewing angles are taken as independent observations, each camera runs independently, and is associated and matched with past features captured by the camera, and the pose transformation between them is associated through inertial navigation integration.

[0015] Step 2: Formulate a modal selection strategy. During the running of the odometer, decide to select the visible light or infrared mode as the main mode of operation; after the main mode is selected, the other mode will play an auxiliary role;

[0016] In step 2, the odometer modal selection strategy includes the following sub-steps:

[0017] 2.1. Select the main mode according to the current score of the mode and the historical score of the mode, including the following sub-steps:

[0018] 2.1.1. The current score of the mode is represented by the number of stable tracking feature points in the current sliding window. Only the feature points that are continuously tracked by multiple frames are considered as stable tracking feature points; the current score of the mode is a relative score compared with other modes:

[0019]

[0020] S Infrared_cur = 1-S visible_cur

[0021] N visible is the number of stable tracking feature points of the visible light mode, N Infrared is the number of stable tracking feature points of the infrared mode. S visible_cur and S Infrared_cur are the current scores of the visible light mode and the infrared mode, respectively.

[0022] 2.1.2, In the case that the current modality score is similar, leading to frequent switching between modalities, the system can fix the use of one modality by introducing the modality history score. The index is designed to smooth the short-term changes in the modality score, so that the algorithm is more inclined to choose the modality that has performed well in the past. The calculation formula of the modality history score is as follows:

[0023]

[0024] 2.1.3, The modality total score is obtained by several combinations of the two indexes, and the final modality score is:

[0025]

[0026] 2.2, The strategy of modality selection is to select the modality with the largest total score as the main modality.

[0027] Step 3: After determining the modality selection in step 2, determine which information in the auxiliary modality is non-redundant and can be used to assist positioning, and determine the fusion weight of the auxiliary information;

[0028] The auxiliary modality weight allocation in step 3 includes the following sub-steps:

[0029] 3.1, According to the feature distribution of the main modality image, the feature points in the auxiliary modality image are divided into complementary features and redundant features. Complementary features refer to features that can fill the sparse areas of main modality features, and redundant features refer to features that coincide with the rich areas of main modality features. The specific steps are as follows:

[0030] 3.1.1, The tracked feature points can be divided into main modality feature set M t and auxiliary modality feature set S t based on the main modality image feature distribution score S t of M t Assign weights.

[0031] 3.1.2, According to the probability distribution of the feature position of the modality, the feature distribution score D(l) of the auxiliary modality is calculated by taking the negative logarithm. The calculation formula of the feature distribution score D(l) of the auxiliary modality is given by the following formula. represents the Gaussian kernel function applied in the main modality image.

[0032]

[0033] 3.2, For complementary feature observations in the auxiliary modality that are higher than the preset score, add them to the optimization problem. For redundant feature observations in the auxiliary modality that are lower than the preset score, reduce their weight and add them to the optimization problem when the total number of main modality features is small. If the total number of main modality features is sufficient, ignore the redundant features and exclude them from the optimization problem.

[0034] Step 4: Based on the tight coupling odometer, the estimation of each state variable is expanded, and the constraints provided by each sensor observation are unified into an optimization problem to construct visual-inertial tight coupling information fusion.

[0035] Step 4 of constructing visual-inertial tight coupling information fusion includes the following sub-steps:

[0036] 4.1, In order to avoid too many optimization problems to be optimized, the odometer maintains a fixed capacity sliding window, and the sliding window capacity is the state variable in the past 1s.

[0037] 4.2, The variable χ optimized in the visual-inertial tight coupling odometer sliding window is as follows:

[0038] χ=[x n ,x n+1 ,...,x n+N ,λ m ,λ m+1 ,...,λ m+M ]

[0039]

[0040] Where x i contains the position, attitude, velocity and accelerometer and gyroscope bias of the IMU coordinate system in the world coordinate system at the i-th frame, λ represents the inverse depth of the feature point observed for the first time in the sliding window, n and m represent the number of the first frame and the first feature point in the sliding window, and N and M represent the number of image frames and feature points in the sliding window.

[0041] 4.3, The IMU constraint residual is constructed as:

[0042] Compared with the prior art, the present application has the following technical effects:

[0043] 1. By increasing the number of cameras, the system can capture environmental information from multiple perspectives, reducing blind areas and improving the perception ability of the scene.

[0044] 2. The design of the modal selection strategy avoids the interference of redundant information and can improve the positioning accuracy. Because the accuracy of using only one mode is higher than that of using two modes at the same time. The reason for this result is also relatively easy to explain: for example, in the daytime, visible light images usually have rich texture and stable features, while infrared images have relatively monotonous texture. In this case where visible light images are completely available, the part of the infrared image similar to the structure and texture of the visible light image belongs to redundant information. Fusing redundant information will not only waste computing resources and reduce real-time performance, but also due to the characteristics of low resolution and more noise in infrared images, integrating redundant infrared information may damage the positioning accuracy of the system.

[0045] 3、The adaptive fusion method can select visible light or infrared modal as the main modal according to the current environmental conditions, and the other modal will be used as the auxiliary modal. Since the visible light camera and the infrared camera are installed close to each other at the same viewing angle, the captured pictures have a certain overlap in the field of view, and have the potential of complementary perception. Generally speaking, for the image frames of visual SLAM, the most ideal case is that the feature points are uniformly distributed, which can enhance the observability of the carrier state and avoid information redundancy caused by feature point clustering. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the present application;

[0047] Figure 2 is a schematic diagram of the pose transformation between cameras of the present application through inertial integration. DETAILED DESCRIPTION

[0048] The specific embodiments of the present application will be described in conjunction with the accompanying drawings.

[0049] As shown in Figure 1 , a multi-camera visual-inertial odometer method fusing infrared images comprises the following steps:

[0050] Step 1: The sensor layout uses front-view and rear-view cameras in combination, which balances the environmental perception ability and the algorithm consumption. First, the visual residual of the multi-camera system is constructed, including the following sub-steps:

[0051] 1.1, for the lth feature point P l , assuming that the ith frame and the jth frame observe P l , the point provides visual constraints for the ith frame and the jth frame, and the visual residual is constructed as follows:

[0052]

[0053] wherein, is the coordinate of the lth feature point in the jth frame camera normalized coordinate system according to the observation, is the estimated value of the lth feature point in the jth frame camera normalized coordinate system, in the visual residual formula, [·] z represents taking the z component of a three-dimensional vector, [·] xy represents taking the xy component of a three-dimensional vector, is the back-projection formula, is the pixel coordinate of the lth feature point in the jth frame, K is the intrinsic matrix of the camera, and in the above formula, the form is represents the rotation matrix and translation vector from the f2 coordinate system to the f1 coordinate system.

[0054] 1.2 During continuous motion, visible light frames, infrared frames, and camera input frames from different viewpoints are treated as independent observations. Each camera operates independently and is correlated and matched with past features captured by its own camera. The pose transformations between them are correlated through inertial navigation integration, such as... Figure 2 As shown.

[0055] Step 2: Develop a mode selection strategy. During odometer operation, determine whether to select the visible light or infrared mode as the primary operating mode. Once the primary mode is selected, the other mode will play a supporting role. The odometer mode selection strategy includes the following sub-steps:

[0056] 2.1 Selecting the dominant mode based on the current modality score and the modality's historical score includes the following sub-steps:

[0057] 2.1.1 The current modality score is represented by the number of stably tracked feature points within the current sliding window. Feature points that are tracked consecutively for multiple frames are considered stably tracked. Using 4 frames as the threshold for the number of frames for stable tracking yields good results; that is, features tracked for 4 consecutive frames are considered stably tracked. The current modality score is a relative score compared to other modes.

[0058]

[0059] S Infrared_cur =1-S visible_cur

[0060] N visible For stable tracking feature points of the visible light mode, N Infrared These are feature points for infrared modal stabilization tracking. visible_cur and S Infrared_cur These are the current scores for the visible light mode and the infrared mode, respectively.

[0061] 2.1.2 Using only the current modal score introduces a degree of randomness. If the current scores of the visible light and infrared modes are similar, subtle environmental changes may cause random mode switching. To avoid this, and to ensure the system consistently uses a single mode when current modal scores are similar and frequent mode switching occurs, historical modal scores need to be introduced. An indicator is designed to smooth short-term changes in modal scores, making the algorithm more inclined to select modes that have historically performed well. The formula for calculating historical modal scores is:

[0062]

[0063] 2.1.3 The modal score is obtained from several combinations of the two indicators. The final modal score is:

[0064]

[0065] 2.2、The strategy of modal selection is to take the modal with the highest total score as the main modal.

[0066] Step 3: After determining the modal selection in step 2, it is further determined which information in the auxiliary modal is non-redundant and can be used to assist positioning, and the fusion weight of the auxiliary information is determined. The auxiliary modal weight allocation includes the following sub-steps:

[0067] 3.1、According to the feature distribution of the main modal image, the feature points in the auxiliary modal image are divided into complementary features and redundant features. Complementary features refer to features that can fill the sparse areas of main modal features, and redundant features refer to features that coincide with the rich areas of main modal features. The specific steps are as follows:

[0068] 3.1.1、The tracked feature points can be divided into main modal feature set M t and auxiliary modal feature set S t based on the feature distribution score S t of the main modal image based on M t and the weight distribution.

[0069] 3.1.2、According to the modal feature position probability distribution, the feature distribution score D(l) of the auxiliary modal is calculated by taking the negative logarithm. The calculation formula of the feature distribution score D(l) of the auxiliary modal is given by the following formula.f Mt (·) represents the Gaussian kernel function applied in the main modal image.

[0070]

[0071] 3.2、For complementary feature observations in the auxiliary modal that are higher than the preset score, add them to the optimization problem. For redundant feature observations in the auxiliary modal that are lower than the preset score, reduce their weight and add them to the optimization problem when the total number of main modal features is small. If the total number of main modal features is sufficient, you can skip the redundant features and exclude them from the optimization problem.

[0072] Step 4: Based on the tightly coupled odometry, the estimation of each state variable is expanded, and the constraints provided by each sensor observation are unified into an optimization problem to construct a visual-inertial tightly coupled information fusion, including the following sub-steps:

[0073] 4.1、To avoid too many state variables to be optimized in the optimization problem, the odometry maintains a fixed capacity sliding window. In general, the sliding window capacity is the state variable within the past 1s.

[0074] 4.2、The variables χ that need to be optimized in the visual-inertial tightly coupled odometry sliding window are as follows:

[0075] χ=[x n ,x n+1, ..., x n+N ,λ m , λ m+1 , …, λ m+M ]

[0076]

[0077] Where x i It includes the position, attitude, velocity of the IMU coordinate system in the world coordinate system at the i-th frame time, as well as the zero bias of the accelerometer and gyroscope. λ represents the inverse depth when the feature point in the sliding window is first observed. n and m represent the numbers of the first frame and the first feature point in the sliding window, respectively. N and M represent the number of image frames and the number of feature points in the sliding window, respectively.

[0078] 4.3 Construct the IMU constrained residuals as follows:

[0079]

[0080] 4.4. Combining the visual residual r constructed in step 1.1 cam We construct a joint BA optimization problem and obtain the maximum posterior probability estimate for the state variable χ by minimizing the sum of the Mahalanobis distances of the prior constraints and the residuals of each observation.

[0081]

[0082] B represents all IMU pre-integrations within the sliding window, C contains all feature points within the sliding window and their corresponding observed image frames, and ρ represents the Huber robust kernel function.

[0083]

[0084] 4.4 Solve the BA optimization problem from the previous step to obtain the pose and 3D position of the map points.

[0085] This invention introduces a multi-camera system into a visible-infrared fusion odometry system, constructing the visual residual of the multi-camera system. The visible-infrared odometry can operate using either the visible light mode or the infrared mode alone, or it can use both modes simultaneously.

[0086] The auxiliary modality weight allocation method of this invention first divides the feature points in the auxiliary modality image into complementary features and redundant features based on the feature distribution of the main modality image. Complementary features are those that can fill sparse regions of the main modality features, while redundant features are those that overlap with rich regions of the main modality features. For complementary features, a visual reprojection residual is constructed for them and added to the optimization problem. For redundant features, a lower weight can be assigned when constructing the optimization problem, or redundant features can be ignored and excluded from the optimization problem.

Claims

1. A multi-camera visual-inertial odometry method fusing infrared images, characterized in that, Includes the following steps: Step 1: Sensor layout uses a combination of front-view and rear-view cameras to balance environmental perception and computing power consumption; construct the visual residual of the multi-camera system; Step 2: Develop a mode selection strategy. During the operation of the odometer, decide whether to select the visible light or infrared mode as the primary operating mode. After the primary mode is selected, the other mode will play an auxiliary role. The odometer mode selection strategy includes the following sub-steps: 2.1 Selecting the dominant mode based on the current modality score and the modality's historical score includes the following sub-steps: 2.1.1 The current modality score is represented by the number of stably tracked feature points within the current sliding window; only feature points that are tracked continuously for multiple frames are considered stably tracked; the current modality score is a relative score compared with other modes. S Infrared_cur =1-S visible_cur N visible For stable tracking feature points of the visible light mode, N Infrared Feature points for infrared modal stabilization tracking; S visible_cur and S Infrared_cur These are the current scores for the visible light mode and the infrared mode, respectively. 2.1.2 To ensure the system consistently uses a single modality when current modality scores are similar, leading to frequent modality switching, historical modality scores are introduced. An indicator is designed to smooth short-term changes in modality scores, making the algorithm more inclined to select historically performing modes. The formula for calculating historical modality scores is: 2.1.3 The modal score is obtained from several combinations of the two indicators. The final modal score is: 2.2 The strategy for modality selection is to use the modality with the higher total score as the dominant modality; Step 3: After determining the mode selection in Step 2, determine which information in the auxiliary modes is non-redundant and can be used for auxiliary localization, and determine the fusion weight of the auxiliary information; Step 4: Estimate each state variable based on the tightly coupled odometer, unify the constraints provided by each sensor observation into an optimization problem, and construct a visual-inertial tightly coupled information fusion.

2. The multi-camera visual-inertial odometry method according to claim 1, characterized in that, Step 1 involves constructing the visual residuals of the multi-camera system, including the following sub-steps: 1.1 For the l-th feature point P l Assume that P is observed in frame i and frame j. l This point provides visual constraints for the i-th and j-th frames, and the visual residual is constructed as follows: in, To determine the coordinates of the l-th feature point in the camera's normalized coordinate system in the j-th frame, based on the observed coordinates, Let be the estimated value of the l-th feature point in the normalized coordinate system of the camera in the j-th frame. In the visual residual formula, [·] z This indicates taking the z-component of a three-dimensional vector, [·]. xy This indicates taking the xy components of a three-dimensional vector. This is the formula for back projection. Let K be the pixel coordinates of the l-th feature point in the j-th frame, and K be the intrinsic parameter matrix of the camera. The above formula is in the form of... Represents the rotation matrix and translation vector from coordinate system f2 to coordinate system f1; 1.2 During continuous motion, visible light frames, infrared frames, and camera input frames from different viewpoints are treated as independent observations. Each camera operates independently and is associated and matched with past features captured by this camera. The pose transformation between them is associated through inertial navigation integration.

3. The multi-camera visual-inertial odometry method according to claim 1, characterized in that, Step 3, auxiliary mode weight allocation, includes the following sub-steps: 3.1 Based on the feature distribution of the main modality image, the feature points in the auxiliary modality image are divided into complementary features and redundant features. Complementary features are those that can fill in the sparse regions of the main modality features, while redundant features are those that overlap with the rich regions of the main modality features. This process includes the following sub-steps: 3.1.1 The tracked feature points can be divided into a principal mode feature set M according to their respective modes. t and auxiliary modal feature set S t Based on M t The main modality image feature distribution score is S t Assign weights; 3.1.

2. Based on the modal feature location probability distribution, the feature distribution score D(l) of the auxiliary mode is calculated by taking the negative logarithm; the formula for calculating the feature distribution score D(l) of the auxiliary mode is given by the following formula; This represents the Gaussian kernel function applied in the main modality image; 3.2 For complementary feature observations in the auxiliary mode that have a score higher than the preset score, add them to the optimization problem; for redundant feature observations in the auxiliary mode that have a score lower than the preset score, reduce their weight before adding them to the optimization problem if the total number of features in the main mode is less than the preset feature threshold; if the total number of features in the main mode is sufficient, skip the redundant features and exclude them from the optimization problem.

4. The multi-camera visual-inertial odometry method according to claim 1, characterized in that, Step 4 involves constructing a tightly coupled visual-inertial information fusion system, which includes the following sub-steps: 4.1 To avoid having too many states to optimize, the odometer maintains a sliding window with a fixed capacity, the capacity of which is the state variables within the past 1 second. 4.2 The variable χ optimized in the vision-inertial tightly coupled odometer sliding window is shown below: Where x i Includes the position, attitude, velocity of the IMU coordinate system in the world coordinate system at the i-th frame time, as well as the zero bias of the accelerometer and gyroscope. λ represents the inverse depth when the feature point in the sliding window is first observed. n and m represent the numbers of the first frame and the first feature point in the sliding window, respectively. N and M represent the number of image frames and the number of feature points in the sliding window, respectively. 4.3 Construct the IMU constrained residuals as follows: 4.

4. Combining the visual residual r constructed in step 1 cam We construct a joint BA optimization problem and obtain the maximum posterior probability estimate for the state variable χ by minimizing the sum of the Mahalanobis distances of the prior constraints and the residuals of each observation. B represents all IMU pre-integrations within the sliding window, C contains all feature points within the sliding window and their corresponding observed image frames, and ρ represents the Huber robust kernel function. 4.4 Solve the BA optimization problem from the previous step to obtain the pose and 3D position of the map points.

Images