Infrared image fused multi-camera vision-inertial odometer method

Abstract

The invention provides an infrared image fused multi-camera vision-inertial odometer method, which comprises the following steps of: 1, matching a forward-looking camera and a rear-looking camera in a sensor layout, and balancing the perception capability and the computing power consumption of an environment by a matching scheme; constructing a visual residual error of the multi-camera system; 2, formulating a mode selection strategy, and determining to select a visible light mode or an infrared mode as a main operation mode in the operation process of the odometer; after the main mode is selected, the other mode plays an auxiliary role; 3, after modal selection is determined in the step 2, determining which information in the auxiliary modal is not redundant and can be used for auxiliary positioning, and determining the fusion weight of the auxiliary information; and 4, performing estimation on each state quantity based on a tight coupling speedometer, unifying constraints provided by observation of each sensor into an optimization problem, and constructing vision-inertia tight coupling information fusion. According to the method, visible light and infrared information are fully utilized, redundant observation is reduced, and the performance and robustness of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) / unmanned vehicle navigation technology, and in particular relates to a multi-camera visual-inertial odometer method for fusing infrared images. Background Art

[0002] Existing visual odometry methods that fuse visible light and infrared images are mainly divided into two technical paths: based on traditional visual odometry and based on deep learning. The traditional method generates a fused image containing information of both by performing wavelet transform or monochrome threshold fusion on visible light and infrared images, and then processes the fused image using traditional visual odometry calculation methods to achieve positioning. The deep learning method constructs a multimodal network or uses an attention mechanism to directly learn the features of visible light and infrared images and fuse them, thereby improving positioning accuracy. Both methods require steps such as image acquisition, preprocessing, feature extraction, feature matching, pose estimation, and trajectory reconstruction to ultimately achieve camera motion estimation and positioning.

[0003] CN202310955029.4 provides a visual odometer method based on visible light infrared fusion. First, the visible light and infrared images are decomposed by discrete wavelet transform to obtain approximate components and detail components respectively. Then, the approximate component and the detail component are weighted averaged and fused by the monochrome threshold fusion method, and the fused image is obtained by the inverse transform of the discrete wavelet transform. Finally, the fused image is used as input to estimate the camera state by optical flow feature extraction and feature point triangulation method. This method combines the texture information of the visible light image and the heat source information of the infrared image.

[0004] CN 202310956658.9 provides a visible light infrared fusion visual odometer method based on deep learning, which proposes to use a camera to obtain infrared images and visible light images of the environment, and decompose the two respectively through two-dimensional discrete wavelet transform to obtain their respective approximate components and detail components. The components are weighted averaged and fused using a monochrome threshold fusion method to obtain a fused image containing infrared image detail information and visible light image texture information. Finally, an improved RCNN network is constructed, and the fused image is used as an input image frame for training to obtain a prediction model. By obtaining the data to be tested and using the model, the trajectory of the camera movement can be predicted, thereby realizing the visual odometer function.

[0005] The above-mentioned prior arts mainly focus on enhancing the performance of visual odometers through image fusion technology, without involving the fusion of multi-camera systems and multiple modalities. The methods of these two patents focus on using discrete wavelet transform and monochrome threshold fusion methods to combine the information of visible light and infrared images to improve the detectability of features and the stability of tracking. However, this method may encounter challenges when dealing with complex environments or changing lighting conditions, because they do not take advantage of the wider field of view and perspective-rich environmental information provided by multi-camera systems, nor do they take into account the redundant and complementary information that may exist between different modalities. Multi-camera systems can provide more comprehensive perspective coverage, reduce blind spots, and improve map integrity and coverage, while the fusion of multiple modalities can maintain stable feature detection and tracking capabilities under different lighting conditions. Summary of the invention

[0006] In view of the above technical problems, the present invention provides a multi-camera visual-inertial odometer method that fuses infrared images. By increasing the number of cameras, the system can capture environmental information from multiple perspectives, thereby improving the perception of the scene, and the mutual verification between multiple cameras can also reduce errors. In addition, the multi-camera system can provide a wider field of view coverage, reduce blind spots, and improve the integrity and coverage of the map. At the same time, a method for adaptive fusion of visible light and infrared images is proposed. The method can flexibly select and adjust the use of the camera according to the current environmental conditions. The visible light-infrared odometer can be operated using the visible light mode or the infrared mode alone, or both modes can be used at the same time. Make full use of visible light and infrared information, reduce redundant observations, and improve the performance and robustness of the system.

[0007] The specific technical solution is:

[0008] A multi-camera visual-inertial odometer method for fusing infrared images comprises the following steps:

[0009] Step 1: The sensor layout uses a combination of front-view and rear-view cameras. The combination scheme balances the perception ability of the environment and the computing power consumption; constructs the visual residual of the multi-camera system;

[0010] In step 1, constructing the visual residual of the multi-camera system includes the following sub-steps:

[0011] 1.1. For the lth feature point P l , assuming that P is observed in the i-th frame and the j-th frame l , which provides visual constraints for the i-th frame and the j-th frame, and the visual residual is constructed as follows:

[0012]

[0013] in, is the coordinate of the lth feature point in the normalized coordinate system of the camera in the jth frame obtained according to the observation, is the estimated value of the lth feature point in the normalized coordinate system of the camera in the jth frame, where [·] z represents the z component of a three-dimensional vector, [·] xy It means taking the xy component of a three-dimensional vector. is the back projection formula, is the pixel coordinate of the lth feature point in the jth frame, K is the intrinsic parameter matrix of the camera, and the above formula is as follows Represents the rotation matrix and translation vector from the f2 coordinate system to the f1 coordinate system.

[0014] 1.2. During continuous motion, the visible light frames, infrared frames and camera input frames at different viewing angles are treated as independent observations. Each camera operates independently and is associated and matched with the past features captured by the camera. The pose transformations between them are associated through inertial navigation integration.

[0015] Step 2: Develop a mode selection strategy. During the operation of the odometer, decide whether to select the visible light or infrared mode as the main mode of operation. After the main mode is selected, the other mode will play a supporting role.

[0016] In step 2, the odometer mode selection strategy includes the following sub-steps:

[0017] 2.1. Selecting the main mode according to the current score of the mode and the historical score of the mode includes the following sub-steps:

[0018] 2.1.1. The current score of the modality is represented by the number of stable tracking feature points in the current sliding window. Feature points that are tracked continuously for multiple frames are considered stable tracking feature points; the current score of the modality is the relative score compared with other modes:

[0019]

[0020] S Infrared_cur =1-S visible_cur

[0021] N visible is the stable tracking feature point of the visible light mode, N Infrared S is the characteristic point of infrared mode stable tracking. visible_cur and S Infrared_cur are the current scores for the visible light modality and infrared modality, respectively.

[0022] 2.1.2. In order to make the system use one mode when the current mode scores are similar and the mode switches frequently, the mode history score is introduced. The design index smoothes the short-term changes of the mode score, so that the algorithm is more inclined to choose the mode with good historical performance. The mode history score calculation formula is:

[0023]

[0024] 2.1.3. The total modal score is obtained by combining the two indicators in several ways. The final modal score is:

[0025]

[0026] 2.2. The strategy of modality selection is to take the modality with the larger total score as the main modality.

[0027] Step 3: After determining the modality selection in step 2, decide which information in the auxiliary modality is non-redundant and can be used for auxiliary positioning, and determine the fusion weight of the auxiliary information;

[0028] The auxiliary modal weight allocation in step 3 includes the following sub-steps:

[0029] 3.1. According to the feature distribution of the main modality image, the feature points in the auxiliary modality image are divided into complementary features and redundant features. Complementary features refer to features that can fill the sparse areas of the main modality features, and redundant features refer to features that overlap with the rich areas of the main modality features. Specifically, it includes the following sub-steps:.

[0030] 3.1.1. The tracked feature points can be divided into the main modal feature set M according to the mode they belong to t and auxiliary modal feature set S t , based on M t The main modality image feature distribution score is S t Assign weights.

[0031] 3.1.2. According to the probability distribution of the modal feature position, the feature distribution score D(l) of the auxiliary modal is calculated by taking the negative logarithm. The calculation formula of the feature distribution score D(l) of the auxiliary modal is given by the following formula. represents the Gaussian kernel function applied to the primary modality image.

[0032]

[0033] 3.2. For complementary feature observations in the auxiliary modality that are higher than the preset score, add them to the optimization problem. For redundant feature observations in the auxiliary modality that are lower than the preset score, when the total number of features in the main modality is small, reduce their weights and add them to the optimization problem. If the total number of features in the main modality is sufficient, skip the redundant features and exclude them from the optimization problem.

[0034] Step 4: Estimate each state quantity based on the tightly coupled odometry, unify the constraints provided by each sensor observation into an optimization problem, and construct visual-inertial tightly coupled information fusion.

[0035] In step 4, visual-inertial tightly coupled information fusion is constructed, which includes the following sub-steps:

[0036] 4.1. In order to avoid the optimization problem of too many states to be optimized, the odometer maintains a sliding window of fixed capacity, and the sliding window capacity is the state quantity in the past 1s.

[0037] 4.2. The optimized variable χ in the sliding window of the visual-inertial tightly coupled odometry is as follows:

[0038] χ=[x n ,x n+1 , ..., x n+N ,λ m ,λ m+1 , ..., λ m+M ]

[0039]

[0040] where x i It contains the position, attitude, velocity, and zero bias of the IMU coordinate system in the world coordinate system at the i-th frame time. λ represents the inverse depth at which the feature point in the sliding window is first observed. n and m represent the numbers of the first frame and the first feature point in the sliding window, respectively. N and M represent the number of image frames and feature points in the sliding window, respectively.

[0041] 4.3. Construct the IMU constraint residual as:

[0042] Compared with the prior art, the present invention has the following technical effects:

[0043] 1. By increasing the number of cameras, the system can capture environmental information from multiple perspectives, reduce blind spots, and thus improve the ability to perceive the scene.

[0044] 2. Designing a modality selection strategy avoids the interference of redundant information and can improve positioning accuracy. This is because the accuracy of using only one modality is higher than using two modalities at the same time. The reason for this result is also relatively easy to explain: for example, during the day, visible light images usually have rich textures and stable features, while infrared images have relatively monotonous textures. In this case where visible light images are fully available, the parts of infrared images that are similar to the visible light image in structure and texture are redundant information. On the one hand, the fusion of redundant information will cause a waste of computing resources and reduce real-time performance. On the other hand, due to the low resolution and high noise characteristics of infrared images, the integration of infrared redundant information may damage the positioning accuracy of the system.

[0045] 3. The adaptive fusion method proposed in the present invention can choose to use visible light or infrared mode as the main mode according to the current environmental conditions, and the other mode will be used as the auxiliary mode. Since the visible light camera and the infrared camera are installed close to each other under the same viewing angle, the captured images have a certain overlap in vision and have the potential for perceptual complementarity. Generally speaking, for the image frames of visual SLAM, the uniform distribution of feature points is the most ideal, which can enhance the observability of the carrier state and avoid information redundancy caused by the clustering of feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the present invention;

[0047] Figure 2 It is a schematic diagram of the posture transformation between cameras of the present invention through inertial navigation integration association. DETAILED DESCRIPTION

[0048] The specific implementation modes of the present invention are described with reference to the accompanying drawings.

[0049] like Figure 1 As shown, a multi-camera visual-inertial odometer method for fusing infrared images includes the following steps:

[0050] Step 1: The sensor layout uses a front-view and rear-view camera combination, which balances the environment perception capability and computing power consumption. First, build the visual residual of the multi-camera system, including the following sub-steps:

[0051] 1.1. For the lth feature point P l , assuming that P is observed in the i-th frame and the j-th frame l , which provides visual constraints for the i-th frame and the j-th frame, and the visual residual is constructed as follows:

[0052]

[0053] in, is the coordinate of the lth feature point in the normalized coordinate system of the camera in the jth frame obtained according to the observation, is the estimated value of the lth feature point in the normalized coordinate system of the camera in the jth frame, where [·] z represents the z component of a three-dimensional vector, [·] xy It means taking the xy component of a three-dimensional vector. is the back projection formula, is the pixel coordinate of the lth feature point in the jth frame, K is the intrinsic parameter matrix of the camera, and the above formula is as follows Represents the rotation matrix and translation vector from the f2 coordinate system to the f1 coordinate system.

[0054] 1.2. During continuous motion, the visible light frame, infrared frame and camera input frame at different viewing angles are treated as independent observations. Each camera operates independently and is associated and matched with the past features captured by the camera. The pose transformations between them are associated through inertial navigation integration, such as Figure 2 shown.

[0055] Step 2: Develop a mode selection strategy. During the odometer operation, decide whether to select visible light or infrared mode as the main mode. After the main mode is selected, the other mode will play a supporting role. The odometer mode selection strategy includes the following sub-steps:

[0056] 2.1. Selecting the main mode according to the current score of the mode and the historical score of the mode includes the following sub-steps:

[0057] 2.1.1. The current score of the modality is represented by the number of stable tracking feature points in the current sliding window. Feature points that are tracked continuously for multiple frames are considered stable tracking feature points. When 4 frames are used as the threshold for stable feature tracking frames, it will have a better effect, that is, features that are tracked for 4 consecutive frames are stable tracking feature points. The current score of the modality is a relative score compared with other modes:

[0058]

[0059] S Infrared_cur =1-S visible_cur

[0060] N visible is the stable tracking feature point of the visible light mode, N Infrared S is the characteristic point of infrared mode stable tracking. visible_cur and S Infrared_cur are the current scores for the visible light modality and infrared modality, respectively.

[0061] 2.1.2. There is a certain degree of randomness when only using the current score of the modality. If the current scores of the visible light modality and the infrared modality are close, slight changes in the environment may cause random switching of the modalities. To avoid the above situation, when the current modal scores are similar and the modal is frequently switched, the system can use one modality fixedly. It is necessary to introduce the modal history score. The design indicator smoothes the short-term changes in the modal score, so that the algorithm is more inclined to select the modality that has performed well in history. The modal history score calculation formula is:

[0062]

[0063] 2.1.3. The total modal score is obtained by combining the two indicators in several ways. The final modal score is:

[0064]

[0065] 2.2. The strategy of modality selection is to take the modality with the larger total score as the main modality.

[0066] Step 3: After determining the modality selection in step 2, it is further determined which information in the auxiliary modality is non-redundant and can be used for auxiliary positioning, and the fusion weight of the auxiliary information is determined. The auxiliary modality weight allocation includes the following sub-steps:

[0067] 3.1. According to the feature distribution of the main modality image, the feature points in the auxiliary modality image are divided into complementary features and redundant features. Complementary features refer to features that can fill the sparse areas of the main modality features, and redundant features refer to features that overlap with the rich areas of the main modality features. Specifically, it includes the following sub-steps:.

[0068] 3.1.1. The tracked feature points can be divided into the main modal feature set M according to the mode they belong to t and auxiliary modal feature set S t , based on M t The main modality image feature distribution score is S t Assign weights.

[0069] 3.1.2. According to the probability distribution of the modal feature position, the feature distribution score D(l) of the auxiliary modal is calculated by taking the negative logarithm. The calculation formula of the feature distribution score D(l) of the auxiliary modal is given by the following formula. Mt (·) denotes the Gaussian kernel function applied in the primary modality image.

[0070]

[0071] 3.2. For complementary feature observations in the auxiliary modality that are higher than the preset score, add them to the optimization problem. For redundant feature observations in the auxiliary modality that are lower than the preset score, when the total number of features in the main modality is small, their weights can be reduced before adding them to the optimization problem. If the total number of features in the main modality is sufficient, the redundant features can be skipped and excluded from the optimization problem.

[0072] Step 4: Estimate each state quantity based on the tightly coupled odometry, unify the constraints provided by each sensor observation into an optimization problem, and construct visual-inertial tightly coupled information fusion, which includes the following sub-steps:

[0073] 4.1. To avoid the problem of too many states to be optimized, the odometer maintains a sliding window of fixed capacity. Usually, the sliding window capacity is the state quantity in the past 1s.

[0074] 4.2. The variable χ that needs to be optimized in the sliding window of the visual-inertial tightly coupled odometry is as follows:

[0075] χ=[x n ,x n+1, …, x n+N ,λ m ,λ m+1 , …, λ m+M ]

[0076]

[0077] where x i It contains the position, attitude, velocity, and zero bias of the IMU coordinate system in the world coordinate system at the i-th frame time. λ represents the inverse depth at which the feature point in the sliding window is first observed. n and m represent the numbers of the first frame and the first feature point in the sliding window, respectively. N and M represent the number of image frames and feature points in the sliding window, respectively.

[0078] 4.3. Construct the IMU constraint residual as:

[0079]

[0080] 4.4. Combined with the visual residual r constructed in step 1.1 cam , construct a joint BA optimization problem, and obtain the maximum a posteriori probability estimate for the state variable χ to be optimized by minimizing the sum of the prior constraints and the Mahalanobis distances of each observation residual:

[0081]

[0082] B represents all IMU pre-integrated quantities in the sliding window, C contains all feature points in the sliding window and their corresponding observed image frames, and ρ represents the Huber robust kernel function:

[0083]

[0084] 4.4. Solve the BA optimization problem in the previous step to obtain the pose and 3D position of the map point.

[0085] The present invention introduces a multi-camera system into the visible light-infrared fusion odometer and constructs the visual residual of the multi-camera system. The visible light-infrared odometer can be operated using the visible light mode or the infrared mode alone, or both modes can be used simultaneously.

[0086] The auxiliary modality weight allocation method of the present invention first divides the feature points in the auxiliary modality image into complementary features and redundant features according to the feature distribution of the main modality image. The complementary features refer to features that can fill the sparse areas of the main modality features, and the redundant features refer to features that overlap with the rich areas of the main modality features. For the complementary features, a visual reprojection residual is constructed for them and added to the optimization problem. For the redundant features, a lower weight can be assigned to them when constructing the optimization problem, or the redundant features can be skipped and excluded from the optimization problem.

Claims

1. A multi-camera visual-inertial odometry method for fusion of infrared images, characterized in that: The following steps are involved: Step 1: The sensor layout uses a combination of front-view and rear-view cameras. The combination scheme balances the perception ability of the environment and the computing power consumption; constructs the visual residual of the multi-camera system; Step 2: Develop a mode selection strategy. During the operation of the odometer, decide whether to select the visible light or infrared mode as the main mode of operation. After the main mode is selected, the other mode will play a supporting role. Step 3: After determining the modality selection in step 2, decide which information in the auxiliary modality is non-redundant and can be used for auxiliary positioning, and determine the fusion weight of the auxiliary information; Step 4: Estimate each state quantity based on the tightly coupled odometry, unify the constraints provided by each sensor observation into an optimization problem, and construct visual-inertial tightly coupled information fusion.

2. The multi-camera visual-inertial odometer method for fusion of infrared images according to claim 1 is characterized in that: In step 1, constructing the visual residual of the multi-camera system includes the following sub-steps: 1.

1. For the lth feature point P l , assuming that P is observed in the i-th frame and the j-th frame l , which provides visual constraints for the i-th frame and the j-th frame, and the visual residual is constructed as follows: in, is the coordinate of the lth feature point in the normalized coordinate system of the camera in the jth frame obtained according to the observation, is the estimated value of the lth feature point in the normalized coordinate system of the camera in the jth frame, where [·] z represents the z component of a three-dimensional vector, [·] xy It means taking the xy component of a three-dimensional vector. is the back projection formula, is the pixel coordinate of the lth feature point in the jth frame, K is the intrinsic parameter matrix of the camera, and the above formula is as follows Represents the rotation matrix and translation vector from the f2 coordinate system to the f1 coordinate system; 1.

2. During continuous motion, the visible light frames, infrared frames and camera input frames at different viewing angles are treated as independent observations. Each camera operates independently and is associated and matched with the past features captured by the camera. The pose transformations between them are associated through inertial navigation integration.

3. The multi-camera visual-inertial odometer method for fusion of infrared images according to claim 1 is characterized in that: In step 2, the odometer mode selection strategy includes the following sub-steps: 2.

1. Selecting the main mode according to the current score of the mode and the historical score of the mode includes the following sub-steps: 2.1.

1. The current score of the modality is represented by the number of stable tracking feature points in the current sliding window. Feature points that are tracked continuously for multiple frames are considered stable tracking feature points. The current score of the modality is the relative score compared with other modes: S Infrared_cur =1-S visible_cur N visible is the stable tracking feature point of the visible light mode, NInfrared is the stable tracking feature point of the infrared mode; S visible_cur and S Infrared_cur are the current scores for the visible light modality and infrared modality, respectively; 2.1.

2. In order to make the system use one mode in a fixed way when the current mode scores are similar and the mode switches frequently, the mode history score is introduced. The design index smoothes the short-term changes of the mode score, so that the algorithm is more inclined to select the mode with good performance in history. The mode history score calculation formula is: 2.1.

3. The total modal score is obtained by combining the two indicators in several ways. The final modal score is: 2.

2. The strategy of modality selection is to take the modality with the larger total score as the main modality.

4. The multi-camera visual-inertial odometer method for fusion of infrared images according to claim 1, characterized in that: The auxiliary modal weight allocation in step 3 includes the following sub-steps: 3.

1. According to the feature distribution of the main modality image, the feature points in the auxiliary modality image are divided into complementary features and redundant features. Complementary features refer to features that can fill the sparse areas of the main modality features, and redundant features refer to features that overlap with the rich areas of the main modality features. Specifically, the following sub-steps are included: 3.1.

1. The tracked feature points can be divided into the main modal feature set M according to the mode they belong to t and auxiliary modal feature set S t , based on M t The main modality image feature distribution score is S t Assign weights; 3.1.

2. According to the probability distribution of the modal feature position, the feature distribution score D(l) of the auxiliary modal is calculated by taking the negative logarithm. The calculation formula of the feature distribution score D(l) of the auxiliary modal is given by the following formula: represents the Gaussian kernel function applied to the primary modality image; 3.

2. For complementary feature observations in the auxiliary modality that are higher than the preset score, add them to the optimization problem; for redundant feature observations in the auxiliary modality that are lower than the preset score, when the total number of features in the main modality is less than the preset feature number threshold, reduce their weights and add them to the optimization problem; if the total number of features in the main modality is sufficient, skip the redundant features and exclude them from the optimization problem.

5. The multi-camera visual-inertial odometer method for fusion of infrared images according to claim 1, characterized in that: In step 4, visual-inertial tightly coupled information fusion is constructed, which includes the following sub-steps: 4.

1. To avoid the problem of too many states to be optimized, the odometer maintains a sliding window with a fixed capacity. The sliding window capacity is the past 1 s The state quantity within; 4.

2. The optimized variable χ in the sliding window of the visual-inertial tightly coupled odometry is as follows: x=[x n ,x n+1 ,…,x n+N ,l m ,l m+1 ,…,l m+M ] where x i Contains the position, attitude, velocity, and zero bias of the IMU coordinate system in the world coordinate system at the i-th frame time. λ represents the inverse depth at which the feature point in the sliding window is first observed. n and m represent the numbers of the first frame and the first feature point in the sliding window, respectively. N and M represent the number of image frames and feature points in the sliding window, respectively. 4.

3. Construct the IMU constraint residual as: 4.

4. Combined with the visual residual r constructed in step 1 cam , construct a joint BA optimization problem, and obtain the maximum a posteriori probability estimate for the state variable χ to be optimized by minimizing the sum of the prior constraints and the Mahalanobis distances of each observation residual: B represents all IMU pre-integrated quantities in the sliding window, C contains all feature points in the sliding window and their corresponding observed image frames, and ρ represents Hube r Robust kernel function: 4.

4. Solve the BA optimization problem in the previous step to obtain the pose and 3D position of the map point.

Images