Monocular visual-inertial odometry positioning method based on adaptive hybrid visual residual

Abstract

The present invention discloses a monocular visual inertial odometer positioning method based on adaptive hybrid visual residual, which belongs to the field of robot navigation and positioning technology. The present invention first extracts and pre-integrates the data of the monocular camera and the inertial measurement unit (IMU) respectively, and selects the strong pixel gradient area according to the feature extraction result; predicts the candidate feature point search window based on the IMU motion information, accelerates feature matching and reduces mismatching, and selects key frames at the same time; introduces a direct method model on the basis of feature extraction and matching to construct photometric error; designs a feature reliability evaluation method and adopts dynamic factors to adaptively select and optimize the hybrid visual residual projection error or photometric error in real time; adopts a sliding window-based graph optimization method to optimize the joint optimization model for pose estimation in the back-end optimization stage, so as to achieve the purpose of improving the positioning accuracy of the visual inertial odometer in low-light and weak-texture scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot navigation and positioning, and in particular relates to a monocular visual-inertial odometer positioning method based on adaptive hybrid visual residual. Background Art

[0002] Reliable positioning information and navigation systems are crucial for robots. Visual inertial odometry (VIO) and simultaneous localization and mapping (SLAM) methods based on images and inertial data have the advantages of low cost and strong autonomy, and show strong robustness and positioning accuracy. However, in complex scenes, affected by dim lighting conditions and fuzzy texture features, optical images often have problems such as blur, noise, and low contrast. VIO is prone to large positioning errors and tracking loss. In this context, neither feature-based methods nor direct methods can effectively improve the positioning accuracy and robustness of visual odometry and SLAM systems in low-light and weak-texture scenes. Summary of the invention

[0003] Purpose of the invention: Due to the complex conditions of robot motion scenes, such as blurred textures and dim lighting, VIO directly applied to low-light and weak-texture scenes still faces many difficulties. In view of the characteristics of this environment, the present invention uses a visual inertial odometer positioning method based on adaptive hybrid visual residuals of photometric error and reprojection error, which combines the direct method with the feature method, to solve the problems brought by challenging scenes and improve the accuracy of the visual positioning system.

[0004] Technical solution: To achieve the above purpose, the present invention adopts the following technical solution:

[0005] The present invention provides a monocular visual inertial odometer positioning method based on adaptive hybrid visual residual, the method comprising the following steps:

[0006] S1: The image data obtained by the monocular camera and the motion information measured by the IMU are used as input information, the corner point information is extracted from the image, and the strong pixel gradient area is selected according to the corner point position. The attitude, position and velocity changes between adjacent image frames are obtained through pre-integration;

[0007] S2: Based on the feature points of the previous frame and the pose change obtained by IMU pre-integration between the current frame and the previous frame, the position of the feature points to be matched in the current frame is predicted and a local search window is established to match potential feature points in the window;

[0008] S3: constructing a mixed visual residual consisting of a reprojection residual and a photometric residual according to the feature point matching in step S2 and the strong pixel gradient region selected in step S1;

[0009] S4: Design a feature reliability evaluation method and use a dynamic factor to adaptively select the optimized photometric residual or the mixed visual residual obtained in step S3 in real time;

[0010] S5: In the backend optimization stage, a sliding window-based graph optimization method is used to combine the inertial residual and the optimized hybrid visual residual obtained in step S4 for pose estimation.

[0011] Furthermore, the step S1 specifically includes the following sub-steps:

[0012] S1-1: Convert the RGB image obtained by the monocular camera to grayscale and extract image features based on the fast corner feature detection algorithm: randomly select pixels As the center point, 16 pixels in a circular area with a radius of 3 around the center point , calculate and center point Grayscale difference ,in, Indicates The gray value of a pixel, if the gray difference of multiple consecutive pixels is greater than the threshold , then the point is considered as a corner point, and the rectangular area with the corner point as the center and a radius of 2 is considered as a strong pixel gradient candidate area;

[0013] S1-2: For each corner point, combined with the gradient information of the image, define a rectangular area with a radius of 2 around it as a candidate strong pixel gradient area for evaluation. In the candidate strong pixel gradient area, calculate the sum of the gradient amplitudes :

[0014] ,

[0015] in, Represents pixel The gradient value at ; Indicates A rectangular area with a radius of 2 as the center point;

[0016] The sum of the gradient amplitudes obtained With a set threshold For comparison, if the sum of the gradient magnitudes in a candidate strong pixel gradient region is If it is greater than the threshold, the candidate strong pixel gradient region is considered to be a strong pixel gradient region;

[0017] S1-3: According to the flow type pre-integration theory, the attitude, position and velocity changes obtained by the IMU between adjacent frames are calculated:

[0018] ,

[0019] in, They are the camera's coordinates in the world coordinate system from Frame to The pose, velocity and translation changes of the frame; are the angular velocity and acceleration measurements, respectively; Represent the bias of the gyroscope and accelerometer respectively; represent the Gaussian white noise of the gyroscope and accelerometer respectively; is the gravitational acceleration in the world coordinate system; For the Frame to The time interval between frames; Respectively represent Frame and Frame pose; superscript Represents the transpose of a matrix; Respectively represent Frame and Frame rate; Indicates Frame to The time interval between frames; Indicates Frame to Frame pose changes; Respectively represent Frame and The position of the frame.

[0020] Furthermore, the step S2 specifically calculates the BRIEF feature descriptor based on the information of all corner points of the previous frame image, and accelerates feature matching and reduces mismatching by predicting the feature point area through IMU. And the pre-integration result between two frames Predict the camera pose for the current frame and feature points Location:

[0021] ,

[0022] ;

[0023] Assume that the feature points of the current frame The predicted two-dimensional coordinates of , define a radius of The circular local search window is expressed as:

[0024] ,

[0025] in, Represents a point in the local search window.

[0026] Furthermore, step S3 specifically includes the following sub-steps:

[0027] S3-1: Assume that there is a map point in the world coordinate system , after two frames of camera projection, point The projection on the two frames is denoted by , , through the camera's intrinsic parameter matrix and the rotation matrix , translation vector To describe the projection relationship of the point in the two frames of images:

[0028] ,

[0029] ,

[0030] in, Yes The depth of Indicate point The depth in the camera coordinate system of the second frame, The depth is 1. The horizontal and vertical coordinate values ​​of the two frames are used to construct a mathematical model for solving the camera pose between the two frames.

[0031] S3-2: In the feature point method, the matching relationship between the projections of the same spatial point in two adjacent frames is obtained by matching descriptors. In the direct method, it is necessary to find a suitable pose transformation so that after the projection point of the first frame is transformed, the grayscale error between the projection point on the second frame and the projection point on the first frame is minimized. Definition of photometric error as follows:

[0032]

[0033] in and Represent the grayscale values ​​of the projection points in the first frame and the second frame respectively. Based on the grayscale invariance assumption, the grayscale of a spatial point is constant at each viewing angle. The pose transformation is solved by minimizing the photometric error. By performing the least squares fitting on all strong pixel gradient regions, an optimization problem is constructed and the least squares objective function is set. as follows:

[0034] ,

[0035] ,

[0036] in, For the Photometric error of feature points; is the total number of feature points; and Respectively represent the first and second frames Gray value of feature points;

[0037] Based on the above analysis, the photometric residual It is expressed as:

[0038] ;

[0039] Reprojection residual It is expressed as:

[0040] ,

[0041] in, and Respectively represent Frame to The rotation matrix and translation vector of the frame; and Respectively represent Frame and The first feature points;

[0042] The hybrid visual residual constructed based on the reprojection residual and the photometric residual is expressed as:

[0043] ,

[0044] in is the Huber robust kernel function, and are the sets of reprojection residuals and photometric residuals, respectively. Represents mixed visual residual.

[0045] Furthermore, step S4 is that when the visual sensor begins to degrade due to harsh environment (such as blurred texture, dim light, etc.), the number of visual constraints will decrease due to feature loss and the average value of reprojection residual will increase due to feature tracking failure. Based on the above analysis, a feature reliability evaluation factor based on the number of feature constraints and the average residual is designed. : ,in represents the image quality assessment parameter, and:

[0046] ,

[0047] in It is The total number of feature constraints for a frame, For the The average reprojection residual of the frame, Expressed as The average reprojection residual of the frame; and The latest The average number of feature constraints and the average reprojection residual of the frames, Indicates the number of image frames; image evaluation parameters when feature points decrease will decline; is to avoid The value is too large, limiting its range of variation.

[0048] Furthermore, the step S5 specifically includes the following steps:

[0049] S5-1: According to the pre-integration result, get Frame to Inertial residual of the frame :

[0050] ,

[0051] in represents the scale factor, Respectively represent the speed and position information of the missing scale, is the gravity vector, It is The rotation matrix of the frame; , , Respectively represent Frame to The frame's attitude residual, velocity residual, and position residual; , Respectively represent Frame and The speed of frame loss scale; Indicates Frame to Frame speed changes; Indicates Frame to The time interval between frames; , Respectively represent Frame and Displacement of frame loss scale; Indicates Frame to Frame displacement changes;

[0052] S5-2: Construct a joint residual model based on mixed visual residuals, inertial residuals, and feature reliability assessment factors:

[0053] ,

[0054] in, represents the joint residual; Represents mixed visual residual; represents the inertia residual; represents the photometric residual; is the set threshold. When the feature tracking is good, the feature reliability evaluation factor Above threshold When , a hybrid visual residual and inertial residual are used for pose estimation; when feature tracking is lost, Lower than When , the features are no longer reliable, and the direct method model is switched to directly optimize the photometric residual for pose estimation.

[0055] S5-3: Based on the sliding window factor graph model, the visual-inertial nonlinear tight coupling optimization method is adopted. The variables that need to be optimized are as follows:

[0056] ,

[0057] x q =[ p b q w , q b q w , v b q w , b a q , b ω q ] ,

[0058] in, Indicates The camera pose of the frame Indicates the number of images in the sliding window; Represents the transformation matrix from the camera coordinate system to the inertial coordinate system; Indicates The inverse depth of the features; represents the number of features in the sliding window, Represents the sliding window The state of the IMU at the frame; , , Respectively represent The position, rotation, and velocity of the IMU at the frame; , denote the accelerometer and gyroscope biases, respectively; Represents the set of variables to be optimized. After combining the prior information, the optimization problem is rewritten as:

[0059] ,

[0060] in represents the optimal solution of the cost function, represents the inertia residual, represents the mixed visual residual, represents the marginalized prior residual in the sliding window; Represents mixed visual residual; The Jacobian matrix representing the prior residual; represents the covariance of the prior residuals; represents the covariance of the inertial residuals; represents the covariance of mixed visual residuals; represents the inertial measurement set; Represents IMU measurement; Represents a set of map points; Represents visual observation.

[0061] Beneficial effects: Compared with the prior art, the present invention innovatively proposes a monocular visual inertial odometry positioning method based on adaptive hybrid visual residual. This method is based on the feature method and the direct method, constructs a hybrid visual residual model including reprojection residual and photometric residual, designs feature reliability evaluation factors, models the system pose estimation as an optimization problem, and optimizes the cost function through a sliding window under the factor graph framework. The visual inertial odometry system based on hybrid visual residual will be able to achieve higher positioning accuracy in more complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is an overall block diagram of the present invention;

[0063] Figure 2 It is an underwater image taken in the Aqualoc Harbor dataset. Figure 2 In (a) and (b), there are a lot of water plants, (c) has a calibration plate and an abandoned bicycle, and (d) has an old fence. Compared with land scenes, underwater scenes have fewer features and insufficient light, which will have a negative impact on the positioning accuracy of the system.

[0064] Figure 3 This is a trajectory visualization comparison diagram of the method of the present invention and the ORB-SLAM3 method for the Aqualoc Harbor dataset 01 sequence. DETAILED DESCRIPTION

[0065] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0066] The present invention provides a monocular visual inertial odometer positioning method based on adaptive hybrid visual residual. Figure 1 As shown, specifically including:

[0067] The present invention provides a monocular visual inertial odometer positioning method based on adaptive hybrid visual residual, which simultaneously constructs a reprojection residual based on feature points and a photometric residual based on a strong pixel gradient area, and optimizes the hybrid visual residual and the inertial residual in a sliding window. The method comprises the following steps:

[0068] S1: The image data obtained by the monocular camera and the motion information measured by the IMU are used as input information, the corner point information is extracted from the image, and the strong pixel gradient area is selected according to the corner point position. The attitude, position and velocity changes between adjacent image frames are obtained through pre-integration;

[0069] S2: Based on the feature points of the previous frame and the pose change obtained by IMU pre-integration between the current frame and the previous frame, the position of the feature points to be matched in the current frame is predicted and a local search window is established to match potential feature points in the window;

[0070] S3: constructing a mixed visual residual consisting of a reprojection residual and a photometric residual according to the feature point matching in step S2 and the strong pixel gradient region selected in step S1;

[0071] S4: Design a feature reliability evaluation method and use a dynamic factor to adaptively select the optimized photometric residual or the mixed visual residual obtained in step S3 in real time;

[0072] S5: In the backend optimization stage, a sliding window-based graph optimization method is used to combine the inertial residual and the optimized hybrid visual residual obtained in step S4 for pose estimation.

[0073] In this embodiment, step S1 specifically includes the following sub-steps:

[0074] S1-1: Convert the RGB image obtained by the monocular camera to grayscale and extract image features based on the fast corner feature detection algorithm: randomly select pixels As the center point, 16 pixels in a circular area with a radius of 3 around the center point , calculate and center point Grayscale difference ,in, Indicates The gray value of a pixel, if the gray difference of multiple consecutive pixels is greater than the threshold , usually 20%, then the point is considered a corner point, and the rectangular area with the corner point as the center and a radius of 2 is considered a strong pixel gradient candidate area;

[0075] S1-2: For each corner point, combined with the gradient information of the image, define a rectangular area with a radius of 2 around it as a candidate strong pixel gradient area for evaluation. In the candidate strong pixel gradient area, calculate the sum of the gradient amplitudes :

[0076] ,

[0077] in, Represents pixel The gradient value at ; Indicates A rectangular area with a radius of 2 as the center point;

[0078] The sum of the gradient amplitudes obtained With a set threshold For comparison, if the sum of the gradient magnitudes in a candidate strong pixel gradient region is If it is greater than the threshold, the candidate strong pixel gradient region is considered to be a strong pixel gradient region;

[0079] S1-3: According to the flow type pre-integration theory, the attitude, position and velocity changes obtained by the IMU between adjacent frames are calculated:

[0080] ,

[0081] in, They are the camera's coordinates in the world coordinate system from Frame to The pose, velocity and translation changes of the frame; are the angular velocity and acceleration measurements, respectively; Represent the bias of the gyroscope and accelerometer respectively; represent the Gaussian white noise of the gyroscope and accelerometer respectively; is the gravitational acceleration in the world coordinate system; For the Frame to The time interval between frames; Respectively represent Frame and Frame pose; superscript Represents the transpose of a matrix; Respectively represent Frame and Frame rate; Indicates Frame to The time interval between frames; Indicates Frame to Frame pose changes; Respectively represent Frame and The position of the frame.

[0082] In this embodiment, the step S2 predicts the position of the feature point to be matched in the current frame and establishes a local search window. The specific method is:

[0083] The BRIEF feature descriptor is calculated based on the information of all corner points in the previous frame. The feature point area is predicted by IMU to accelerate feature matching and reduce mismatching. and location And the pre-integration result between two frames Predict the camera pose for the current frame and feature points Location:

[0084] ,

[0085] ;

[0086] Assume that the feature points of the current frame The predicted two-dimensional coordinates of , define a radius of The circular local search window is expressed as:

[0087] ,

[0088] in, Represents a point in the local search window.

[0089] Preferably, in step S3, a mixed visual residual is constructed based on a three-dimensional space point projection model and a photometric invariance assumption:

[0090] S3-1: Assume that there is a map point in the world coordinate system , after two frames of camera projection, point The projection on the two frames is denoted by , , through the camera's intrinsic parameter matrix and the rotation matrix , translation vector To describe the projection relationship of the point in the two frames of images:

[0091] ,

[0092] ,

[0093] in, Yes The depth of Indicate point The depth in the camera coordinate system of the second frame, The depth is 1. The horizontal and vertical coordinate values ​​of the two frames are used to construct a mathematical model for solving the camera pose between the two frames.

[0094] S3-2: In the feature point method, the matching relationship between the projections of the same spatial point in two adjacent frames is obtained by matching descriptors. In the direct method, it is necessary to find a suitable pose transformation so that after the projection point of the first frame is transformed, the grayscale error between the projection point on the second frame and the projection point on the first frame is minimized. Definition of photometric error as follows:

[0095] ,

[0096] in and Represent the grayscale values ​​of the projection points in the first frame and the second frame respectively. Based on the grayscale invariance assumption, the grayscale of a spatial point is constant under various viewing angles. Minimize the photometric error to solve the pose transformation, and construct an optimization problem by performing least squares fitting on all strong pixel gradient areas, setting the least squares objective function as follows:

[0097] ,

[0098] ,

[0099] in, For the Photometric error of feature points; is the total number of feature points; and Respectively represent the first and second frames The gray value of a feature point.

[0100] Based on the above analysis, the photometric residual It is expressed as:

[0101] ,

[0102] Reprojection residual It is expressed as:

[0103] ,

[0104] in, and Respectively represent Frame to The rotation matrix and translation vector of the frame; and Respectively represent Frame and The first feature points;

[0105] The hybrid visual residual constructed based on the reprojection residual and the photometric residual is expressed as:

[0106] ,

[0107] in is the Huber robust kernel function, and are the sets of reprojection residuals and photometric residuals, respectively. Represents mixed visual residual.

[0108] Preferably, when the visual sensor begins to degrade due to a harsh environment (such as blurred texture, dim lighting, etc.), the number of visual constraints will decrease due to feature loss and the average value of reprojection residuals will increase due to feature tracking failure. Based on the above analysis, a feature reliability evaluation factor based on the number of feature constraints and the average residual is designed. : ,in represents the image quality assessment parameter, and:

[0109] ,

[0110] in It is The total number of feature constraints for a frame, For the The average reprojection residual of the frame, Expressed as The average reprojection residual of the frame; and The latest The average number of feature constraints and the average reprojection residual of the frames, Indicates the number of image frames; image evaluation parameters when feature points decrease will decline; is to avoid The value is too large, limiting its range of variation.

[0111] Preferably, the joint residual model is constructed in step S5 and the nonlinear equation is optimized by sliding window in the factor graph framework:

[0112] S5-1: According to the pre-integration result, get Frame to Inertial residual of the frame :

[0113] ,

[0114] in represents the scale factor, Respectively represent the speed and position information of the missing scale, is the gravity vector, It is The rotation matrix of the frame; , , Respectively represent Frame to The frame's attitude residual, velocity residual, and position residual; , Respectively represent Frame and The speed of frame loss scale; Indicates Frame to Frame speed changes; Indicates Frame to The time interval between frames; , Respectively represent Frame and Displacement of frame loss scale; Indicates Frame to Frame displacement changes;

[0115] S5-2: Construct a joint residual model based on mixed visual residuals, inertial residuals, and feature reliability assessment factors:

[0116] ,

[0117] in, represents the joint residual; Represents mixed visual residual; represents the inertia residual; represents the photometric residual; is the set threshold. When the feature tracking is good, the feature reliability evaluation factor Above threshold When , a hybrid visual residual and inertial residual are used for pose estimation; when feature tracking is lost, Lower than When , the features are no longer reliable, and the direct method model is switched to directly optimize the photometric residual for pose estimation.

[0118] S5-3: Based on the sliding window factor graph model, the visual-inertial nonlinear tight coupling optimization method is adopted. The variables that need to be optimized are as follows:

[0119] ,

[0120] x q =[ p b q w , q b q w , v b q w , b a q , b ω q ] ,

[0121] in, Indicates The camera pose of the frame, Indicates the number of images in the sliding window; Represents the transformation matrix from the camera coordinate system to the inertial coordinate system; Indicates The inverse depth of the features; represents the number of features in the sliding window, Represents the sliding window The state of the IMU at the frame; , , Respectively represent The position, rotation, and velocity of the IMU at the frame; , denote the accelerometer and gyroscope biases, respectively; Represents the set of variables to be optimized. After combining the prior information, the optimization problem is rewritten as:

[0122] ,

[0123] in represents the optimal solution of the cost function, represents the inertia residual, represents the mixed visual residual, represents the marginalized prior residual in the sliding window; Represents mixed visual residual; The Jacobian matrix representing the prior residual; represents the covariance of the prior residuals; represents the covariance of the inertial residuals; represents the covariance of mixed visual residuals; represents the inertial measurement set; Represents IMU measurement; Represents a set of map points; Represents visual observation.

[0124] In order to verify the monocular visual-inertial odometry positioning method based on adaptive hybrid visual residual proposed in this paper, the positioning trajectory accuracy was evaluated by the evo tool based on the underwater dataset Aqualoc, and the method in this paper was verified. Figure 2 is an underwater scene image in the Aqualoc dataset. This paper conducted experiments on three sequences of Aqualoc Harbor and used the root mean square error (RMSE) of the absolute trajectory error (ATE) as the criterion for evaluating positioning accuracy. All experiments were conducted on a laptop with a CPU of i7-1360P 2.20GHz and 16G RAM. The positioning accuracy evaluation results of the compared traditional visual inertial SLAM method, namely the ORB-SLAM3 method and the method of the present invention are shown in Table 1:

[0125] Table 1. Comparison results of positioning accuracy of 3 sequences in the Aqualoc dataset between the ORB-SLAM3 method and the method of the present invention:

[0126] sequence ORB-SLAM3RMSE(m) RMSE(m) of the method of the present invention Increase (%) 01 1.100 0.957 13.00% 02 0.509 0.422 17.09% 03 1.202 0.925 23.04% average value 0.937 0.768 17.71%

[0127] Some experimental results are visualized as follows: Figure 3 As shown in Table 1 and Figure 3 It can be seen that the positioning accuracy estimation performance of the present invention is improved by an average of about 17.71% compared with the ORB-SLAM3 method in complex environments.

Claims

1. A monocular visual inertial odometry positioning method based on adaptive hybrid visual residual, characterized in that: The method simultaneously constructs a reprojection residual based on feature points and a photometric residual based on strong pixel gradient regions, and optimizes the mixed visual residual and inertial residual in a sliding window. The method includes the following steps: S1: The image data obtained by the monocular camera and the motion information measured by the IMU are used as input information, the corner point information is extracted from the image, and the strong pixel gradient area is selected according to the corner point position. The attitude, position and velocity changes between adjacent image frames are obtained through pre-integration; S2: Based on the feature points of the previous frame and the pose change obtained by IMU pre-integration between the current frame and the previous frame, the position of the feature points to be matched in the current frame is predicted and a local search window is established to match potential feature points in the window; S3: constructing a mixed visual residual consisting of a reprojection residual and a photometric residual according to the feature point matching in step S2 and the strong pixel gradient region selected in step S1; S4: Design a feature reliability evaluation method and use a dynamic factor to adaptively select the optimized photometric residual or the mixed visual residual obtained in step S3 in real time; S5: In the backend optimization stage, a sliding window-based graph optimization method is used to combine the inertial residual and the optimized hybrid visual residual obtained in step S4 for pose estimation.

2. The monocular visual inertial odometer positioning method based on adaptive hybrid visual residual according to claim 1, characterized in that: Step S1 specifically includes the following sub-steps: S1-1: Convert the RGB image obtained by the monocular camera to grayscale and extract image features based on the fast corner feature detection algorithm: randomly select pixels As the center point, 16 pixels in a circular area with a radius of 3 around the center point , calculate and center point Grayscale difference ,in, Indicates The gray value of a pixel, if the gray difference of multiple consecutive pixels is greater than the threshold , then the point is considered as a corner point, and the rectangular area with the corner point as the center and a radius of 2 is considered as a strong pixel gradient candidate area; S1-2: For each corner point, combined with the gradient information of the image, define a rectangular area with a radius of 2 around it as a candidate strong pixel gradient area for evaluation. In the candidate strong pixel gradient area, calculate the sum of the gradient amplitudes : ， in, Represents pixel The gradient value at ; Indicates A rectangular area with a radius of 2 as the center point; The sum of the gradient amplitudes obtained With a set threshold For comparison, if the sum of the gradient magnitudes in a candidate strong pixel gradient region is If it is greater than the threshold, the candidate strong pixel gradient region is considered to be a strong pixel gradient region; S1-3: According to the flow type pre-integration theory, the attitude, position and velocity changes obtained by the IMU between adjacent frames are calculated: ， in, They are the camera's coordinates in the world coordinate system from Frame to The pose, velocity and translation changes of the frame; are the angular velocity and acceleration measurements, respectively; represent the bias of the gyroscope and accelerometer respectively; represent the Gaussian white noise of the gyroscope and accelerometer respectively; is the gravitational acceleration in the world coordinate system; For the Frame to The time interval between frames; Respectively represent Frame and Frame pose; superscript Represents the transpose of a matrix; Respectively represent Frame and Frame rate; Indicates Frame to The time interval between frames; Indicates Frame to Frame pose changes; Respectively represent Frame and The position of the frame.

3. The monocular visual inertial odometer positioning method based on adaptive hybrid visual residual according to claim 2, characterized in that: Step S2 predicts the position of the feature points to be matched in the current frame and establishes a local search window. The specific method is: According to the posture of the camera coordinate system in the previous frame and location And the pre-integration result between two frames Predict the camera pose for the current frame and feature points Location: ， ， Assume that the feature points of the current frame The predicted two-dimensional coordinates of , define a radius of The circular local search window is expressed as: ， in, Represents a point in the local search window.

4. The monocular visual inertial odometer positioning method based on adaptive hybrid visual residual according to claim 3 is characterized in that: Step S3 constructs a mixed visual residual consisting of a reprojection residual and a photometric residual, as follows: S3-1: Assume that there is a map point in the world coordinate system After two frames of camera projection, point The projection on the two frames is denoted by , , through the camera's intrinsic parameter matrix and the rotation matrix , translation vector To describe the projection relationship of the point in the two frames of images: ， ， in, Yes The depth of Indicate point The depth in the camera coordinate system of the second frame, The depth is 1. The horizontal and vertical coordinate values ​​of the two frames are used to construct a mathematical model for solving the camera pose between the two frames. S3-2: Defining Photometric Error as follows: ， in and Represent the grayscale values ​​of the projection points in the first frame and the second frame respectively. Based on the grayscale invariance assumption, the grayscale of a spatial point imaged at each viewing angle is constant: minimize the photometric error to solve the pose transformation, and construct an optimization problem by performing least squares fitting on all strong pixel gradient areas, setting the least squares objective function as follows: ， ， in, For the Photometric error of feature points; is the total number of feature points; and Respectively represent the first and second frames in the Gray value of feature points; Photometric residual It is expressed as: ， and Respectively represent Frame and Frame Gray value of feature points; Reprojection residual It is expressed as: ， in, and Respectively represent Frame to The rotation matrix and translation vector of the frame; and Respectively represent Frame and The first feature points; Hybrid visual residual constructed based on reprojection residual and photometric residual It is expressed as: ， in is the Huber robust kernel function, and are the sets of reprojection residuals and photometric residuals, respectively. Represents mixed visual residual.

5. The monocular visual inertial odometer positioning method based on adaptive hybrid visual residual according to claim 4, characterized in that: The specific method of step S4 is as follows: Design a feature reliability assessment factor based on the number of feature constraints and the average residual : ,in represents the image quality assessment parameter, and: ， in It is The total number of feature constraints for a frame, For the The average reprojection residual of the frame, Expressed as The average reprojection residual of the frame; and The latest The average number of feature constraints and the average reprojection residual of the frame, h represents the number of image frames; when the feature points are reduced, the image evaluation parameters will decline; is to avoid The value is too large, limiting its range of variation.

6. The monocular visual inertial odometer positioning method based on adaptive hybrid visual residual according to claim 5, characterized in that: The specific method of step S5 is as follows: S5-1: According to the pre-integration result, get Frame to Inertial residual of the frame : ， in represents the scale factor, Respectively represent the speed and position information of the missing scale, is the gravity vector, It is The rotation matrix of the frame; , , Respectively represent Frame to The frame's attitude residual, velocity residual, and position residual; , Respectively represent Frame and The speed of frame loss scale; Indicates Frame to Frame speed changes; Indicates Frame to The time interval between frames; , Respectively represent Frame and Displacement of frame loss scale; Indicates Frame to Frame displacement changes; S5-2: Construct a joint residual model based on mixed visual residuals, inertial residuals, and feature reliability assessment factors: ， in, represents the joint residual; Represents mixed visual residual; represents the inertia residual; represents the photometric residual; is the set threshold. When the feature tracking is good, the feature reliability evaluation factor Above threshold When , a hybrid visual residual and inertial residual are used for pose estimation; when feature tracking is lost, Lower than When , the features are no longer reliable, and the direct method model is switched to directly optimize the photometric residual for pose estimation; S5-3: Based on the sliding window factor graph model, the visual-inertial nonlinear tight coupling optimization method is adopted. The variables that need to be optimized are as follows: ， ， in, Indicates The camera pose of the frame Indicates the number of images in the sliding window; Represents the transformation matrix from the camera coordinate system to the inertial coordinate system; Indicates The inverse depth of the features; represents the number of features in the sliding window, Represents the sliding window The state of the IMU at the frame; , , Respectively represent The position, rotation, and velocity of the IMU at the frame; , denote the accelerometer and gyroscope biases, respectively; Represents the set of variables to be optimized. After combining the prior information, the optimization problem is rewritten as: ， in represents the optimal solution of the cost function, represents the inertia residual, represents the mixed visual residual, represents the marginalized prior residual in the sliding window; Represents mixed visual residual; The Jacobian matrix representing the prior residual; represents the covariance of the prior residuals; represents the covariance of the inertial residuals; represents the covariance of mixed visual residuals; represents the inertial measurement set; Represents IMU measurement; Represents a set of map points; Represents visual observation.

Images