Visual-inertial slam method based on optical flow sliding window and orb fusion

Abstract

The application discloses a visual inertial SLAM method based on optical flow sliding window and ORB fusion, and relates to the technical field of computer vision and simultaneous localization and mapping. The application combines the advantages of optical flow matching and ORB feature matching, can keep stable feature tracking capability in complex environments such as weak texture, low light, light mutation, multi-scale, rotation, dynamic object occlusion and the like, and greatly improves the system robustness; only extracts ORB features in key frames, and only performs lightweight optical flow tracking in ordinary frames, so that the overall computing power consumption is greatly reduced, and the embedded platform is adapted; through long-term feature association of ORB, local inertial bundle adjustment optimization under long-term feature constraint is constructed, combined with prior pose constraint, long-term running pose drift is reduced, and the positioning accuracy is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and Simultaneous Localization and Mapping (SLAM) technology, and particularly relates to a visual inertial SLAM method based on the fusion of optical flow sliding window and ORB (Oriented FAST and Rotated BRIEF, key point detection and feature descriptor). Background Technology

[0002] Simultaneous localization and mapping (SLAM) technology is the core technology for autonomous mobile devices to achieve environmental perception and autonomous navigation. The current mainstream visual inertial SLAM technologies can be divided into feature point method, semi-direct method, and direct method according to the visual matching method. Each of them has its own technical focus and applicable scenarios, but none of them can simultaneously meet the stable operation requirements of complex environments under embedded low computing power platforms.

[0003] Feature-point methods, such as those based on ORB (Oriented Fast and Rotated BRIEF, keypoint detection and feature descriptors), typically possess excellent long-term feature association capabilities, adapting to complex scenes with varying lighting, multiple scales, rotation, and dynamic object occlusion. They also offer high localization accuracy, and some full-featured solutions include backend optimization modules, loop closure detection, and rapid relocation after feature loss. However, they generally suffer from heavy engineering architecture and high computational consumption, making them difficult to run fully on computationally limited embedded hardware. Furthermore, due to their reliance on ORB feature matching logic, feature matching has a higher probability of failure in weak texture and low-light environments.

[0004] Semi-direct methods primarily rely on optical flow for feature tracking, exhibiting high matching stability in weakly textured environments and possessing a robust VIO (Visual-Inertial Odometry) joint initialization process with fast initialization speed. Furthermore, these solutions are typically lightweight, purely VIO-based architectures, consuming minimal computational resources and offering better compatibility with embedded hardware. However, these methods only support short-term optical flow feature associations, lacking long-term feature management capabilities. Over extended periods, they suffer from significant pose drift issues, resulting in low positioning accuracy. Moreover, matching stability deteriorates under conditions of altered lighting, scale changes, and rapid rotation.

[0005] The direct method is mainly based on gray-level matching to associate visual information. It does not require the extraction and description of feature points, but only realizes inter-frame pose estimation through pixel gray-level differences. It is better than the feature point method in weak texture and repetitive texture scenes. However, this type of scheme strictly relies on the assumption of gray-level invariance, and has extremely poor anti-interference ability to sudden changes in lighting and dynamic object occlusion. Furthermore, it is more likely to lose tracking when scale changes or rotations occur, and it has no long-term feature association ability. The pose drift problem is most prominent when running for a long time. Summary of the Invention

[0006] To address the aforementioned problems in the prior art, this invention provides a visual inertial SLAM method based on optical flow sliding window and ORB fusion.

[0007] The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a visual-inertial SLAM method based on optical flow sliding window and ORB fusion, comprising: Optical flow feature matching is performed based on the matching optical flow feature points and matching ORB feature points in the previous frame's binocular image, as well as the historical target ORB map points of the region contained in the current frame's binocular image obtained in advance, to obtain the matching optical flow feature points and matching ORB feature points of the current frame. Based on the matched optical flow feature points of the current frame, generate optical flow map points for the current frame; Based on the optical flow map points of historical frames and the current frame, the corresponding historical target ORB map points of matched ORB feature points, and the pre-acquired IMU data, visual-inertial joint sliding window optimization is performed to obtain the camera pose data of the current frame. If the current frame is determined to be a keyframe, then based on the stereo image of the current frame, obtain the ORB feature points of the current frame; perform ORB feature matching on the ORB feature points of the current frame to generate the ORB map points to be optimized for the current frame. If the current frame is determined to be a key frame, local inertial bundle adjustment optimization is performed based on the ORB map points, IMU data and camera pose data in the local map including the current frame. The target ORB map points and target camera pose data in the local map are output, and the camera pose data of the next frame is corrected based on the target camera pose data of the current frame.

[0008] This invention provides a visual-inertial SLAM method based on optical flow sliding window and ORB fusion. Combining the advantages of optical flow matching and ORB feature matching, it maintains stable feature tracking capabilities in complex environments such as weak texture, low illumination, abrupt illumination changes, multi-scale, rotation, and dynamic object occlusion, significantly improving system robustness. ORB features are extracted only in keyframes, while lightweight optical flow tracking is performed in ordinary frames, resulting in a significant reduction in overall computational power compared to pure ORB solutions, making it suitable for embedded platforms. Through long-term ORB feature association, a local inertial beam adjustment optimization is constructed under long-term feature constraints. Combined with prior pose constraints, this reduces long-term pose drift, achieving positioning accuracy comparable to pure ORB solutions.

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a visual-inertial SLAM method based on optical flow sliding window and ORB fusion provided in an embodiment of the present invention. Figure 2A This is a schematic diagram of feature tracking of the left eye image according to an embodiment of the present invention; Figure 2B This is a schematic diagram of feature tracking of the right eye image according to an embodiment of the present invention; Figure 2C This is a top view of the pose trajectory of the binocular camera and the target ORB map point in an embodiment of the present invention; Figure 2D This is a plan view of the pose trajectory of the binocular camera and the target ORB map point in an embodiment of the present invention. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0012] This invention provides a visual-inertial SLAM method based on optical flow sliding window and ORB fusion. See also Figure 1 The method includes the following steps: S10. Based on the matching optical flow feature points and matching ORB feature points in the previous frame's binocular image, and the historical target ORB map points of the area contained in the current frame's binocular image obtained in advance, perform optical flow feature matching to obtain the matching optical flow feature points and matching ORB feature points of the current frame.

[0013] Optionally, step S10 may specifically include: S101. Based on the matching optical flow feature points and matching ORB feature points in the previous frame's binocular image, perform optical flow tracing between frames to obtain the first matching optical flow feature point and the first matching ORB feature point of the current frame.

[0014] S102. Perform corner detection on the unmatched regions in the current frame's binocular image to obtain the optical flow feature points to be matched in the current frame.

[0015] S103. Based on the historical target ORB map points of the region contained in the stereo image of the current frame, project the unmatched region in the stereo image of the current frame to obtain the ORB feature points to be matched.

[0016] S104. Perform left and right optical flow feature matching based on the optical flow feature points to be matched and the ORB feature points to be matched to obtain the second matching optical flow feature points and the second matching ORB feature points of the current frame.

[0017] Among them, the first matched optical flow feature point, the first matched ORB feature point, the second matched optical flow feature point, and the second matched ORB feature point of the current frame constitute the matched optical flow feature point and the matched ORB feature point of the current frame.

[0018] For example, the Lucas-Kanade (LK) optical flow method can be used to perform optical flow tracking on the matching optical flow feature points and matching ORB feature points in the previous frame's binocular image to obtain the first matching optical flow feature point and the first matching ORB feature point in the current frame. Here, the Lucas-Kanade optical flow method is a sparse optical flow algorithm that can track large motion feature points and is widely used in SLAM, target tracking, and visual odometry.

[0019] For the current frame of the stereo image, GFTT (Good Features to Track, Shi-Tomasi Corner) detection is performed on the unmatched regions other than the region where the first matched optical flow feature point is located, to obtain the optical flow feature points to be matched in the current frame, i.e., the GFTT feature points to be matched. Here, the optical flow feature points to be matched are 2D pixel features.

[0020] The historical target ORB map points of the region contained in the pre-acquired stereo image of the current frame are projected onto the unmatched region to obtain the ORB feature points to be matched. Here, the historical target ORB map points can be obtained from the target map database, which is used to store the target ORB map points obtained in each frame and the feature points generated in the process.

[0021] The optical flow feature points to be matched and the ORB feature points to be matched are subjected to left and right optical flow feature matching. The successfully matched feature points are used as the second matched optical flow feature point and the second matched ORB feature point of the current frame. Thus, the first matched optical flow feature point, the first matched ORB feature point, the second matched optical flow feature point, and the second matched ORB feature point of the current frame constitute the complete matched optical flow feature point and matched ORB feature point of the current frame. (Refer to...) Figure 2A and Figure 2B The diagrams show feature tracking illustrations for the left and right eyes, respectively. Blue and red dots represent GFTT feature points, while light blue and yellow dots represent ORB feature points. The total number of consecutive tracking frames corresponding to blue and light blue dots is less than 10, while the total number of consecutive tracking frames corresponding to red and yellow dots is greater than or equal to 10.

[0022] S20. Based on the matched optical flow feature points of the current frame, generate optical flow map points for the current frame.

[0023] For example, the matching optical flow feature points of the current frame are converted into three-dimensional spatial coordinates through triangulation to obtain the optical flow map points of the current frame.

[0024] S30. Based on the optical flow map points of historical frames and the current frame, the corresponding historical target ORB map points of the matched ORB feature points, and the pre-acquired IMU data, a visual-inertial joint sliding window optimization is performed to obtain the camera pose data of the current frame.

[0025] For example, when the collected historical frames and current frames accumulate to a target number (e.g., 20) of frame data, each frame data includes matched optical flow feature points and IMU data from the previous frame to the current frame. Visual-Inertial Odometry (VIO) joint initialization is performed to obtain initialized camera pose data, velocity data, and IMU (Inertial Measurement Unit) zero-bias parameters (including accelerometer and gyroscope zero-bias). After initialization, VIO joint sliding window optimization is performed, where the number of sliding windows can be 10. The optimization problem constructed by the VIO joint sliding window optimization of this invention includes constraints such as reprojection residuals of optical flow features and IMU pre-integration residuals, as well as reprojection residual constraints of ORB features. Least squares iterative solution is then performed, ultimately outputting the optimized camera pose data, optical flow feature inliers (i.e., GFTT inliers), and ORB inliers for the current frame. Here, the historical target ORB map points corresponding to the matched ORB feature points can be obtained from the target map database.

[0026] The sliding window optimization of this invention integrates the observation constraints of optical flow features and ORB features. Optical flow features can ensure the system's adaptability in weak texture environments, while ORB features can ensure the system's positioning stability and positioning accuracy over a longer time range.

[0027] Optionally, the total residual expression for the optimization problem corresponding to the joint visual-inertial sliding window optimization is:

[0028] in: It is the set of first state parameters to be optimized; The optimal set of first state parameters obtained by solving; The maximum frame index of the stereo image. To match the maximum index of the optical flow map points corresponding to the optical flow feature points, To match the maximum index of historical target ORB map points corresponding to ORB feature points; For the sliding window The motion state corresponding to the frame, where, The first The rotation and translation components of the camera pose data for a frame. For the first Frame rate, For the first The frame's accelerometer has zero bias. For the first The gyroscope in the frame has zero bias; For the sliding window The motion state corresponding to the frame; For the first The inverse depth of optical flow map points corresponding to each matched optical flow feature point; This is a set of optical flow feature observation pairs within a sliding window, where each observation pair includes a reference frame index. Observation Frame Index And matching optical flow feature point index .

[0029] For the first The inverse depth of the historical target ORB map points corresponding to each matched ORB feature point; This is the set of ORB feature observation pairs within the sliding window, where each observation pair includes an observation frame index. Matching ORB feature point index .

[0030] To match the reprojection residuals of optical flow feature points, and The first Frame number Pixel observations of the matching optical flow feature points and the first Frame number Pixel observations of matching optical flow feature points This is the corresponding covariance matrix; To match the reprojection residuals of ORB feature points, This is the first Frame number Pixel observations matching ORB feature points This is the corresponding covariance matrix; For IMU pre-integration residuals, For the first Frame to the IMU data of the frame This is the corresponding covariance matrix; .

[0031] Optionally, the expression for the reprojection residuals matching ORB feature points is: ; in: For camera projection functions; , The extrinsic parameters of rotation and translation from the IMU to the stereo camera are respectively indicated by superscript. Indicates transpose; It is the first The unit direction vector of the historical target ORB map point corresponding to each matched ORB feature point in the world coordinate system.

[0032] Optionally, the expression for the reprojection residual of the matched optical flow feature points is: ; ; ; in: For camera projection function, For camera inverse projection function; and For the first The matching optical flow feature points are respectively at the _th ... Frame and the 3D coordinates within a frame; , The first Frame and the The homogeneous 6-DOF pose expression for the frame is as follows: , ; The homogeneous expression for the 6-DOF extrinsic parameters from the IMU to the stereo camera is as follows: .

[0033] S40. If the current frame is determined to be a keyframe, then based on the stereo image of the current frame, obtain the ORB feature points of the current frame; perform ORB feature matching on the ORB feature points of the current frame to generate the ORB map points of the current frame.

[0034] Specifically, if the current frame is determined to be a keyframe, steps S40 and S50 are executed for the current frame. If the current frame is not a keyframe, the process returns to step S10.

[0035] Optionally, the method of the present invention further includes: if the time interval between the current frame and the previous key frame is greater than a preset time interval, or the rotation angle of the current frame relative to the previous key frame is greater than a preset angle, or the translation distance of the current frame relative to the previous key frame is greater than a preset distance, or the number of feature interior points of the current frame is less than a preset multiple of the number of interior points of the previous key frame, then the current frame is determined to be a key frame.

[0036] For example, the preset time interval can be 500ms, the preset angle can be 35 degrees, the preset distance can be 50cm, and the preset multiplier can be 0.6. This invention performs optical flow tracing on ordinary frames and extracts ORB features only in key frames, significantly reducing system computational power consumption and enabling real-time operation on performance-constrained computing platforms. Here, the number of feature in-points includes optical flow feature in-points and ORB in-points.

[0037] Optionally, in step S40, based on the stereo image of the current frame, the ORB feature points of the current frame are obtained. Specifically, this may include: extracting initial ORB feature points from the stereo image of the current frame; merging the initial ORB feature points and the matched ORB feature points to obtain the ORB feature points of the current frame. Based on this, ORB feature matching is performed on the ORB feature points of the current frame to generate ORB map points of the current frame. Specifically, this may include: when performing ORB feature matching on the ORB feature points of the current frame, only the initial ORB feature points are matched, and the matching relationship of the matched ORB feature points is reused to obtain the ORB map points of the current frame.

[0038] For example, after determining the current frame as a keyframe, initial ORB feature points are extracted from the stereo image of the current frame. These initial ORB feature points are then merged with previously obtained matched ORB feature points to obtain the ORB feature points of the current frame. Next, ORB feature matching is performed between the stereo cameras in the current frame, followed by ORB feature matching from the previous keyframe to the current frame. The successfully matched ORB feature points are then triangulated to obtain the ORB map points of the current frame. During this process, matching is only performed on the initial ORB feature points; the matched ORB feature points can directly reuse the results of previous optical flow tracing, eliminating the need for repeated matching.

[0039] This invention extracts ORB features from keyframes individually before subsequent local inertial beam adjustment optimization and reuses the matched ORB feature points obtained through optical flow matching. It eliminates the need to repeatedly perform matching operations on all ORB feature points of the keyframes, only supplementing the initial ORB feature points for matching. Based on the matching advantages of ORB features themselves, the success rate of ORB feature matching in weak texture environments is improved by more than 30%, while reducing the computational cost of matching operations.

[0040] S50. If the current frame is determined to be a key frame, perform local inertial bundle adjustment optimization based on the ORB map points, IMU data and camera pose data in the local map including the current frame, output the target ORB map points and target camera pose data in the local map, and correct the camera pose data of the next frame based on the target camera pose data of the current frame.

[0041] For example, based on determining the current frame as a keyframe, keyframe-level local inertial bundle adjustment optimization is performed using ORB map points, IMU data, and camera pose data within the local map range including the current frame. The optimized ORB map points and camera pose data are then output, representing the target ORB map points and target camera pose data within the local map. The local map range includes the ORB map points of the current frame (i.e., the keyframe) and the target ORB map points contained in keyframes with a previous number of targets (e.g., 10). The IMU data consists of the IMU data corresponding to all keyframes included in the local map. Subsequently, the target camera pose data of the current frame (i.e., the keyframe) is used to correct the pose of the stereo camera in the next frame, reducing pose drift during long-term operation.

[0042] The optimization problem constraints for Local Inertial Bundle Adjustment (LIBA) mainly include visual reprojection error, IMU pre-integration residual, and prior relative pose residual.

[0043] Reference Figure 2C and Figure 2D , Figure 2C This is a top-down view of the pose trajectory of the binocular camera and the target ORB map points. Figure 2D This is a plan view of the pose trajectory of the stereo camera and the target ORB map points. Green frames represent normal frames, blue frames represent keyframes, keyframes associated with purple lines are keyframes corresponding to the local map, red 3D points are map points corresponding to the local map, and other colored 3D points are non-local map points.

[0044] This invention introduces camera pose data from keyframes as prior constraints, which improves the convergence speed and accuracy of the optimization process.

[0045] Optionally, the total residual expression for the optimization problem corresponding to the local inertial bundle adjustment optimization is:

[0046] in: The set of second state parameters to be optimized; The optimal set of second state parameters obtained by solving; This represents the maximum index of the ORB map points. Indicates the maximum index of the keyframe; For the first The motion state of each keyframe, where... The first The rotation and translation components of the camera pose data for each keyframe. For the first The speed of each keyframe For the first Accelerometer zero bias in keyframes For the first Zero gyroscope bias in each keyframe; For the first The motion state of each keyframe; It is the first World coordinates of an ORB map point; For the reprojection residuals of ORB map points, For the first The first keyframe Pixel observations of ORB map points This is the corresponding covariance matrix; This is a set of ORB feature observation pairs within a local map area, where each observation pair includes an observation frame index. and feature point index .

[0047] For IMU pre-integration residuals, For the first From the first keyframe to the... IMU data for each keyframe. This is the corresponding covariance matrix; For the prior relative pose residual, For the first Camera pose data corresponding to each keyframe This is the corresponding covariance matrix; .

[0048] Optionally, the expression for the prior relative pose residual is:

[0049] in: and These are the rotation and translation components of the camera pose data, respectively. Let SO(3) be the logarithmic mapping from the Lie group SO(3) to the Lie algebra SO(3).

[0050] Optionally, the method of the present invention further includes: updating the target map database based on the target ORB map points of the current frame; the historical target ORB map points are obtained from the target map database.

[0051] For example, the target ORB map points of the current frame (i.e., the keyframe) are updated to the target map database for use in subsequent tracking.

[0052] The present invention provides a visual-inertial SLAM method based on optical flow sliding window and ORB fusion, which has the following advantages compared with the prior art: 1. Strong environmental adaptability: Combining the advantages of optical flow matching and ORB feature matching, it can maintain stable feature tracking capabilities in complex environments such as weak texture, low light, abrupt changes in lighting, multi-scale, rotation, and dynamic object occlusion, greatly improving the system's robustness.

[0053] 2. Low computational power consumption: ORB features are extracted only in key frames, and only lightweight optical flow tracing is performed in ordinary frames. The overall computational power consumption is significantly reduced compared to the pure ORB solution, and it can run in real time on ARM (Advanced RISC Machine) embedded platforms.

[0054] 3. High positioning accuracy: This invention performs long-term ORB feature association and constructs LIBA optimization under long-term feature constraints. Combined with prior pose constraints, the pose drift during long-term operation is less than 0.5%, which can achieve positioning accuracy comparable to the pure ORB scheme.

[0055] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0056] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0057] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0058] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A visual-inertial SLAM method based on optical flow sliding window and ORB fusion, characterized in that, include: Optical flow feature matching is performed based on the matching optical flow feature points and matching ORB feature points in the previous frame's binocular image, as well as the historical target ORB map points of the region contained in the current frame's binocular image obtained in advance, to obtain the matching optical flow feature points and matching ORB feature points of the current frame. Based on the matched optical flow feature points of the current frame, generate optical flow map points for the current frame; Based on the optical flow map points of historical frames and the current frame, the historical target ORB map points corresponding to the matched ORB feature points, and the pre-acquired IMU data, visual-inertial joint sliding window optimization is performed to obtain the camera pose data of the current frame. If the current frame is determined to be a keyframe, then based on the stereo image of the current frame, the ORB feature points of the current frame are obtained; ORB feature matching is performed on the ORB feature points of the current frame to generate the ORB map points of the current frame. If the current frame is determined to be a keyframe, local inertial bundle adjustment optimization is performed based on the ORB map points, IMU data, and camera pose data in the local map including the current frame. The target ORB map points and target camera pose data in the local map are output, and the camera pose data of the next frame is corrected based on the target camera pose data of the current frame.

2. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 1, characterized in that, The optical flow feature matching based on the matched optical flow feature points and matched ORB feature points in the previous frame's stereo image, as well as the historical target ORB map points of the region contained in the current frame's stereo image (pre-acquired), yields the matched optical flow feature points and matched ORB feature points for the current frame, including: Based on the matching optical flow feature points and matching ORB feature points in the previous frame's binocular image, optical flow tracing between frames is performed to obtain the first matching optical flow feature point and the first matching ORB feature point in the current frame. Corner detection is performed on the unmatched regions in the current frame's binocular image to obtain the optical flow feature points to be matched in the current frame; Based on the historical target ORB map points of the region contained in the stereo image of the current frame, the unmatched region in the stereo image of the current frame is projected to obtain the ORB feature points to be matched. Based on the optical flow feature points to be matched and the ORB feature points to be matched, left and right optical flow feature matching is performed to obtain the second matching optical flow feature points and the second matching ORB feature points of the current frame; the first matching optical flow feature points, the first matching ORB feature points, the second matching optical flow feature points, and the second matching ORB feature points of the current frame constitute the matching optical flow feature points and the matching ORB feature points of the current frame.

3. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 1, characterized in that, The step of obtaining ORB feature points of the current frame based on the stereo image of the current frame includes: Extract initial ORB feature points from the stereo image of the current frame; The initial ORB feature points and the matched ORB feature points are merged to obtain the ORB feature points of the current frame; The step of performing ORB feature matching on the ORB feature points of the current frame to generate ORB map points for the current frame includes: When performing ORB feature matching on the ORB feature points of the current frame, the matching operation is performed only on the initial ORB feature points, and the matching relationship of the matched ORB feature points is reused to obtain the ORB map points of the current frame.

4. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 1, characterized in that, The method further includes: The target map database is updated based on the target ORB map points in the current frame; the historical target ORB map points are obtained from the target map database.

5. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 1, characterized in that, The method further includes: If the time interval between the current frame and the previous keyframe is greater than a preset time interval, or the rotation angle of the current frame relative to the previous keyframe is greater than a preset angle, or the translation distance of the current frame relative to the previous keyframe is greater than a preset distance, or the number of feature interior points of the current frame is less than a preset multiple of the number of interior points of the previous keyframe, then the current frame is determined to be a keyframe.

6. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 1, characterized in that, The total residual expression for the optimization problem corresponding to the visual-inertial joint sliding window optimization is: in: It is the set of first state parameters to be optimized; The optimal set of first state parameters obtained by solving; The maximum frame index of the stereo image. To match the maximum index of the optical flow map points corresponding to the optical flow feature points, To match the maximum index of historical target ORB map points corresponding to ORB feature points; For the sliding window The motion state corresponding to the frame, where, The first The rotation and translation components of the camera pose data for a frame. For the first Frame speed, For the first The frame's accelerometer has zero bias. For the first The gyroscope in the frame has zero bias; For the sliding window The motion state corresponding to the frame; For the first The inverse depth of optical flow map points corresponding to each matched optical flow feature point; This is a set of optical flow feature observation pairs within a sliding window, where each observation pair includes a reference frame index. Observation Frame Index And matching optical flow feature point index ; For the first The inverse depth of the historical target ORB map points corresponding to each matched ORB feature point; This is the set of ORB feature observation pairs within the sliding window, where each observation pair includes an observation frame index. Matching ORB feature point index ; To match the reprojection residuals of optical flow feature points, and The first Frame number Pixel observations of the matching optical flow feature points and the first Frame number Pixel observations of matching optical flow feature points This is the corresponding covariance matrix; To match the reprojection residuals of ORB feature points, This is the first Frame number Pixel observations matching ORB feature points This is the corresponding covariance matrix; For IMU pre-integration residuals, For the first Frame to the IMU data of the frame This is the corresponding covariance matrix; , , These represent the Mahalanobis distances of the corresponding terms.

7. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 6, characterized in that, The expression for the reprojection residual of the matched ORB feature points is: in: For camera projection function; , The extrinsic parameters of rotation and translation from the IMU to the stereo camera are respectively indicated by superscript. Indicates transpose; It is the first The unit direction vector of the historical target ORB map point corresponding to each matched ORB feature point in the world coordinate system.

8. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 1, characterized in that, The total residual expression for the optimization problem corresponding to the local inertial beam adjustment optimization is: in: The set of second state parameters to be optimized; The optimal set of second state parameters obtained by solving; This represents the maximum index of the ORB map points. Indicates the maximum index of the keyframe; For the first The motion state of each keyframe, where... The first The rotation and translation components of the camera pose data for each keyframe. For the first The speed of each keyframe For the first Accelerometer zero bias in keyframes For the first Zero gyroscope bias in each keyframe; For the first The motion state of each keyframe; It is the first World coordinates of an ORB map point; For the reprojection residuals of ORB map points, For the first The first keyframe Pixel observations of ORB map points This is the corresponding covariance matrix; This is a set of ORB feature observation pairs within a local map area, where each observation pair includes an observation frame index. and feature point index ; For IMU pre-integration residuals, For the first From the first keyframe to the second IMU data for each keyframe, This is the corresponding covariance matrix; For the prior relative pose residual, For the first Camera pose data corresponding to each keyframe This is the corresponding covariance matrix; , , These represent the Mahalanobis distances of the corresponding terms.

9. The visual-inertial SLAM method based on optical flow sliding window and ORB fusion according to claim 8, characterized in that, The expression for the prior relative pose residual is: in: and These are the rotation and translation components of the camera pose data, respectively. Let SO(3) be the logarithmic mapping from the Lie group SO(3) to the Lie algebra SO(3).

Images