A binocular inertial navigation SLAM positioning method integrated with deep learning

Abstract

This invention provides a binocular inertial navigation SLAM localization method integrating deep learning, specifically including: the system receives images from a binocular camera and IMU data, extracts image feature points and feature descriptors using a Superpoint network; performs feature matching and 3D reconstruction using previous and next frames and left and right eye images, and detects dynamic objects using a YOLOv8-seg network for dynamic point removal; calculates camera pose based on IMU pre-integration, uses a sliding window mechanism to filter keyframes and performs BA optimization to complete binocular inertial navigation initialization; trains a DBoW3 bag-of-words model using binary encoding of feature descriptors for loop closure detection; and applies a LightGlue network to match the current keyframe with candidate keyframes, optimizing the pose graph using RANSAC to optimize the matching results. The method proposed in this invention can effectively improve the robustness of the SLAM front-end in low-light and weak-texture environments, reduce dynamic scene interference, improve the accuracy of pose graph optimization, and ensure high precision and stability of real-time localization.

Description

Technical Field

[0001] This invention belongs to the field of spatial positioning and mapping technology, specifically relating to a binocular inertial navigation SLAM positioning method that integrates deep learning. Background Technology

[0002] With the advancement of the aerospace manufacturing industry, the assembly precision of complex components is becoming increasingly important for product quality and safety. Traditional manual inspection methods are inefficient and prone to errors in complex environments, making it difficult to meet the high-precision assembly inspection requirements of modern manufacturing. Machine vision technology, with its advantages of non-contact operation and high accuracy, is widely used in industrial inspection. However, when inspecting large structural components, traditional vision inspection methods require sectional inspection and result stitching. The inspection equipment cannot accurately determine its own current position and orientation, resulting in a lack of flexibility and accuracy in the inspection process.

[0003] Visual SLAM (Simultaneous Localization and Mapping) technology, as an advanced automated localization and mapping method, has shown great application potential. Traditional visual SLAM frameworks include front-end visual odometry and back-end optimization. The front-end processes sensor data, tracks camera pose, and generates local maps, while the back-end improves the accuracy of the global map through simultaneous optimization. However, complex environments present numerous challenges to traditional visual SLAM. Sparse textures and poor lighting conditions in the environment make it difficult to stably extract and match feature points, easily leading to lost camera pose tracking. Furthermore, frequent dynamic disturbances, such as moving objects and human activity, further affect the stability and accuracy of the system. These factors combined severely impact the accuracy and stability of visual SLAM, posing a significant challenge to its practical applications. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a binocular inertial navigation SLAM localization method that integrates deep learning. By using a deep learning network for feature extraction, matching, dynamic point removal, and pose optimization, the localization accuracy and stability are significantly improved, making it suitable for practical application scenarios.

[0005] To achieve the above technical objectives, the present invention provides the following technical solution:

[0006] A binocular inertial navigation SLAM localization method integrating deep learning specifically includes the following steps:

[0007] S1. Receive frame-by-frame images of the video stream through a binocular camera and perform preprocessing and image analysis, while simultaneously receiving inertial navigation information;

[0008] S2, the front end of the SLAM system extracts feature points and feature descriptors of the image through the Superpoint network; the feature points are input as a point set into the YoloV8-seg network for dynamic point removal, and the descriptors are binary encoded to train the DBoW3 bag-of-words model.

[0009] S3. Use the YOLOV8-seg network for object detection and motion / static classification of images; including:

[0010] The system acquires the detection bounding box of the dynamic object and performs semantic segmentation on the image within the box, obtaining the set of coordinate points of the region where the dynamic object is located. It determines whether the feature points are within the detection bounding box of the dynamic object by comparing the coordinates of each feature point within the box with the coordinates of the dynamic region. Feature points within 2 pixels around the dynamic target mask (mask M) are marked as dynamic points and removed. The filtered point set is used to perform forward and reverse optical flow tracking of the feature points in the two frames before and after the left eye image as a reference, and the camera pose change is calculated. Forward and reverse optical flow matching is performed on the left and right eye images to perform 3D reconstruction of the feature points.

[0011] S4. Acquire IMU data between the previous frame and the current image frame, including the measurement data of the accelerometer and gyroscope of the IMU sensor, i.e., acceleration and angular velocity; take the average value of the IMU acceleration in the current frame, align it with gravity, and obtain the initial IMU attitude; acquire the image pose of the previous frame, apply the IMU pre-integration result during this period, and obtain the initial pose of the current frame.

[0012] S5. Select key frames from ordinary image frames to fill the sliding window, use the sliding window mechanism to perform BA optimization, and complete the initialization of the binocular inertial navigation.

[0013] S6. Encode the extracted feature descriptors into binary form to train the DBoW3 bag-of-words model and build an offline dictionary for loop closure detection; perform loop closure detection on keyframes, use the offline dictionary to perform feature statistics on each new keyframe, and search for candidate keyframes corresponding to the current keyframe.

[0014] S7. The LightGlue network is used to match the feature points of the current keyframe with the feature points of the candidate keyframes. The RANSAC algorithm is used to optimize the feature matching results and the PnP method is used to calculate the relative pose between the current keyframe and the candidate keyframes. The candidate object frame index and relative pose are stored and added to the pose graph as additional constraints for pose optimization.

[0015] Furthermore, the extraction of feature points from the image in step S2 specifically includes:

[0016] S21. An encoder based on the VGG architecture is used to progressively reduce the size of the image and extract features; the encoder consists of convolutional layers, max-pooling layers, and non-linear activation function layers; through three consecutive max-pooling layers, the image size is reduced to 1 / 8 of the original image;

[0017] S22. Perform feature point extraction; reduce the dimensionality of the 128-dimensional feature map to 65-dimensional; the 65-dimensional feature map represents a potential feature point of each 8×8 local region of the original image and a Dustbin channel for outputting no feature points.

[0018] S23. The probability is normalized by Softmax operation, then slice processing is performed, and finally the output feature point position is determined by Non-maximum Suppression (NMS).

[0019] More specifically, during feature point extraction in step S2, feature descriptors are also extracted simultaneously for training the DBoW3 word band model in step S6, specifically including:

[0020] The 128-dimensional feature map is upgraded to 256-dimensional and normalized. Based on the location information of the feature points, the corresponding feature descriptors are extracted from the feature map using the bilinear interpolation method.

[0021] Furthermore, step S3 specifically includes:

[0022] S31. Use the YOLOV8-seg network to perform object detection and semantic segmentation on the current frame image, and obtain the bounding box B = {b1, b2, ..., b} for object detection. n} and the semantic segmentation mask M = {m1, m2, ..., m n};

[0023] S32. Based on the bounding box B, store the feature points of each dynamic region in the point set P = {p1, p2, ..., p...} n In this context, any point within two pixels of the dynamic bounding box is considered to be inside the bounding box; a single feature point p i The position coordinates are (x i ,y i );

[0024] S33. For each feature point p i Calculate its confidence score C based on its position in the mask M. i The formula is expressed as:

[0025] C i =α·D j +β·M j (x i ,yi );

[0026] Among them, D j M is the confidence score for object detection. j (x i ,y i Let α be the confidence value of the semantic segmentation mask at the feature point location, and β be the weight coefficients, satisfying α + β = 1; set a confidence threshold T, and select feature points p with confidence values ​​less than the threshold T. i By removing points from set P, we obtain the point set P' after removing dynamic feature points, i.e.: P' = {p i ∈P|C i ≥T}.

[0027] Furthermore, step S5 specifically includes:

[0028] S51. Using ordinary frame image frames with good feature point tracking as keyframes and filling the sliding window, calculate the standard deviation of IMU acceleration within the sliding window to determine the speed of movement; the sliding window refers to a container containing N keyframes constructed during the optimization process.

[0029] S52. Find a frame with a sufficiently large disparity to the current frame and an accurate match in the sliding window, calculate the relative pose change, and use binocular triangulation to triangulate all feature points.

[0030] S53. Correct the gyroscope bias, initialize gravity, gravity direction and scale; using a frame with sufficient parallax and accurate matching as the reference frame, perform BA optimization on the visual and IMU redundant terms within the window to complete the binocular inertial navigation initialization.

[0031] More specifically, in step S52:

[0032] The criterion for determining sufficiently large disparity is: the difference in the x-coordinate pixels of the same matching point in different frames is greater than the disparity threshold T. d1 This is considered to indicate a sufficiently large parallax;

[0033] The criterion for judging accurate matching is: for each feature point, the projection error calculated using the matched feature points is less than the matching threshold T. d2 If so, the match is considered accurate.

[0034] Furthermore, step S6 specifically includes:

[0035] S61. Extract each feature descriptor d through the Superpoint network. i Perform binary encoding to generate binary feature descriptors bi i ;

[0036] S62. Construct an initial vocabulary V = {v1, v2, ..., v...}K}, where K is the size of the predetermined vocabulary, and the K-means clustering algorithm is used to cluster all binary feature descriptors {bi1,bi2,...,bi}. n Clustering is performed to obtain K cluster centers {ct1, ct2, ..., ct3}. K}, where each cluster center ct j To represent a word, we treat it as a word item in vocabulary V;

[0037] S63. For each binary feature descriptor bi i Calculate its relationship with each word v j Hamming distance H(bi) i ,v j The formula is expressed as:

[0038]

[0039] Where L is the length of the binary feature descriptor. Indicates a bitwise XOR operation;

[0040] When calculating the Hamming distance, bi i and v j It is indexed according to its respective feature points and cluster k; the feature points p i binary feature descriptor bi i Assign the word closest to Hamming's distance (v) j That is, it is represented as

[0041] S64, Words v in statistical images j The number of feature points assigned forms the word frequency vector w = [w1, w2, ..., w K ], any w indexed by j j The formula is expressed as:

[0042]

[0043] The value of δ(a,b) is 1 when a = b, and 0 otherwise. The word frequency vector w is normalized to obtain the final bag-of-words representation.

[0044] Furthermore, the optimization of the feature matching results using the RANSAC algorithm in step S7 specifically includes:

[0045] S71. Input the images of the current keyframe and candidate keyframes, and use the LightGlue network to perform preliminary matching to obtain the point pair set I0 = {(f c1 ,f r1 ),(f c2 ,fr2 ),...,(f cn ,f rn )}; Each pair of matching points (f ci ,f rj ) Contains matching feature points f in the current keyframe ci Matching feature points f in candidate keyframes rj Their coordinates are (x ci ,y ci ) and (x rj ,y rj );

[0046] S72. Randomly select 4 pairs of points from the initially matched point pair set I0, and use the selected 4 pairs of points to estimate the geometric transformation matrix H between the two frames of images, expressed by the formula:

[0047]

[0048] in, and The homogeneous coordinates of the matching points;

[0049] Calculate all matching point pairs (f ci ,f rj The projection error ∈ ij =||(x ci ,y ci )-H(x rj ,y rj )||;

[0050] S73, Set the interior point error threshold ∈ th Points with projection errors less than a threshold are considered interior points, forming an interior point set I. This process is repeated multiple times until the largest interior point set becomes the final set of matched point pairs I. max .

[0051] Based on the above technical solution, the method proposed in this invention has the following beneficial effects:

[0052] The system receives images from a stereo camera and IMU data, and extracts image feature points and feature descriptors through the Superpoint network, which improves the robustness of the SLAM system front-end in low-light and weak-texture environments. At the same time, it uses the YOLOV8-seg network to detect and remove dynamic objects, effectively reducing interference in dynamic scenes.

[0053] The SLAM system backend uses binary-encoded Superpoint feature descriptors to train a DBoW3 bag-of-words model for loop closure detection, and uses a LightGlue network to match the current keyframe with candidate keyframes. RANSAC is used to optimize the matching results, improve the accuracy of the pose graph optimization code, and ensure high precision and strong robustness of real-time localization. Attached Figure Description

[0054] Figure 1 This is a flowchart of the overall process of a binocular inertial navigation SLAM localization method that integrates deep learning proposed in this invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0057] like Figure 1 As shown, a binocular inertial navigation SLAM localization method integrating deep learning proposed in this invention is presented, specifically including the following steps:

[0058] S1. Receive frame-by-frame images of the video stream through a binocular camera and perform preprocessing and image analysis, while simultaneously receiving inertial navigation information;

[0059] S2. The SLAM system front-end extracts feature points and feature descriptors from the image using the Superpoint network; as a preferred implementation, step S2 specifically includes:

[0060] S21. An encoder based on the VGG architecture is used to progressively reduce the size of the image and extract features; the encoder consists of convolutional layers, max-pooling layers, and non-linear activation function layers; through three consecutive max-pooling layers, the image size is reduced to 1 / 8 of the original image;

[0061] S22. Perform feature point extraction; reduce the dimensionality of the 128-dimensional feature map to 65-dimensional; the 65-dimensional feature map represents a potential feature point of each 8×8 local region of the original image and a Dustbin channel for outputting no feature points; the introduction of the Dustbin channel avoids misjudgment of no feature point regions;

[0062] S23. The probability is normalized by Softmax operation, then slice processing is performed, and finally the output feature point position is determined by Non-maximum Suppression (NMS).

[0063] Simultaneously with feature point extraction, feature descriptors are extracted for use in training the DBoW3 bag-of-words model in step S6. The specific process is as follows:

[0064] The 128-dimensional feature map is upscaled to 256 dimensions, where 256 represents the dimension of the features to be output. The upscaled feature map is then normalized. Based on the location information of the feature points, the corresponding feature descriptors are extracted from the feature map using the bilinear interpolation method.

[0065] The loss function formula based on feature points and feature descriptors is as follows:

[0066]

[0067] In the above formula, For feature point correlation loss, λ represents the loss related to the feature descriptors, and λ is the weight of the loss function balancing the loss function for feature points and the loss function for feature descriptors; Y、 Y' is a feature point; is the feature descriptor; S represents the set of all correctly matched pairs. This loss function is used to evaluate the entire training process of the network.

[0068] S3. Use the YOLOV8-seg network for object detection and motion / static classification of images; including:

[0069] The system acquires the detection bounding box of the dynamic object and performs semantic segmentation on the image within the box, obtaining the set of coordinate points of the region where the dynamic object is located. It determines whether the feature points are within the detection bounding box of the dynamic object by comparing the coordinates of each feature point within the box with the coordinates of the dynamic region. Feature points within 2 pixels around the dynamic target mask (mask M) are marked as dynamic points and removed. The filtered point set is used to perform forward and reverse optical flow tracking of the feature points in the two frames before and after the left eye image as a reference, and the camera pose change is calculated. Forward and reverse optical flow matching is performed on the left and right eye images to perform 3D reconstruction of the feature points.

[0070] As a preferred embodiment, step S3 specifically includes:

[0071] S31. Use the YOLOV8-seg network to perform object detection and semantic segmentation on the current frame image, and obtain the bounding box B = {b1, b2, ..., b} for object detection. n} and the semantic segmentation mask M = {m1, m2, ..., mn};

[0072] S32. Based on the bounding box B, store the feature points of each dynamic region in the point set P = {p1, p2, ..., p...} n In this context, any point within two pixels of the dynamic bounding box is considered to be inside the bounding box; a single feature point p i The position coordinates are (x i ,y i );

[0073] S33. For each feature point p i Calculate its confidence score C based on its position in the mask M. i The formula is expressed as:

[0074] C i =α·D j +β·M j (x i ,y i );

[0075] Among them, D j M represents the confidence score for object detection. j (x i ,y i Let α be the confidence value of the semantic segmentation mask at the feature point location, and β be the weight coefficients, satisfying α + β = 1; set a confidence threshold T, and select feature points p with confidence values ​​less than the threshold T. i By removing points from set P, we obtain the point set P' after removing dynamic feature points, i.e.: P' = {p i ∈P|C i ≥T}.

[0076] S4. Acquire IMU data between the previous frame and the current image frame, including the measurement data of the accelerometer and gyroscope of the IMU sensor, i.e., acceleration and angular velocity; take the average value of the IMU acceleration in the current frame, align it with gravity, and obtain the initial IMU attitude; acquire the image pose of the previous frame, apply the IMU pre-integration result during this period, and obtain the initial pose of the current frame.

[0077] In this embodiment, step S4 specifically calculates the pose by constructing IMU pre-integration and residuals, as follows:

[0078] Acquire measurement data from the accelerometer and gyroscope of the IMU sensor, respectively, for acceleration a. t and angular velocity ω tThe data is sampled at a fixed frequency, and the median pre-integration of the IMU is calculated in discrete time. The changes in position and velocity are obtained by calculating the pre-integration, and the pose residual during this time is derived from the pre-integration. The pose estimation is optimized by minimizing the residual. The pose change between two frames can be obtained by optimizing the IMU residual and the image frame residual.

[0079] S5. Select key frames from ordinary image frames to fill the sliding window, use the sliding window mechanism to perform BA optimization, and complete the initialization of the binocular inertial navigation.

[0080] In a preferred embodiment, step S5 specifically includes:

[0081] S51. Use ordinary frame image frames with good feature point tracking as keyframes and fill the sliding window. Calculate the standard deviation of IMU acceleration within the sliding window to determine the speed of movement.

[0082] S52. Find a frame with a sufficiently large disparity to the current frame and a relatively accurate match in the sliding window, calculate the relative pose change, and use binocular triangulation to triangulate all feature points.

[0083] The criterion for determining sufficiently large disparity is: the difference in the x-coordinate pixels of the same matching point in different frames is greater than the disparity threshold T. d1 This is considered to indicate a sufficiently large parallax;

[0084] The criterion for judging accurate matching is: for each feature point, the projection error calculated using the matched feature points is less than the matching threshold T. d2 Then the match is considered accurate;

[0085] In this embodiment, T is taken as... d1 =10, T d2 =2;

[0086] S53. Correct the gyroscope bias, initialize gravity, gravity direction and scale; using a frame with a sufficiently large parallax and relatively accurate matching as the reference frame, perform BA optimization on the visual and IMU redundant terms within the window to complete the binocular inertial navigation initialization.

[0087] S6. Encode the extracted feature descriptors into binary form to train the DBoW3 bag-of-words model and build an offline dictionary for loop closure detection; perform loop closure detection on keyframes, use the offline dictionary to perform feature statistics on each new keyframe, and search for candidate keyframes corresponding to the current keyframe.

[0088] In a preferred embodiment, step S6 specifically includes:

[0089] S61. Extract each feature descriptor d through the Superpoint network. iPerform binary encoding to generate binary feature descriptors bi i ;

[0090] S62. Construct an initial vocabulary V = {v1, v2, ..., v...} K}, where K is the size of the predetermined vocabulary, and the K-means clustering algorithm is used to cluster all binary feature descriptors {bi1,bi2,...,bi}. n Clustering is performed to obtain K cluster centers {ct1, ct2, ..., ct3}. K}, where each cluster center ct j To represent a word, we treat it as a word item in vocabulary V;

[0091] S63. For each binary feature descriptor bi i Calculate its relationship with each word v j Hamming distance H(bi) i ,v j The formula is expressed as:

[0092]

[0093] Where L is the length of the binary feature descriptor. Indicates a bitwise XOR operation;

[0094] When calculating the Hamming distance, bi i and v j It is indexed according to its respective feature points and cluster k; the feature points p i binary feature descriptor bi i Assign the word closest to Hamming's distance (v) j That is, it is represented as

[0095] S64, Words v in statistical images j The number of feature points assigned forms the word frequency vector w = [w1, w2, ..., w K ], any w indexed by j j The formula is expressed as:

[0096]

[0097] The value of δ(a,b) is 1 when a = b, and 0 otherwise. The word frequency vector w is normalized to obtain the final bag-of-words representation.

[0098] S7. The LightGlue network is used to match the feature points of the current keyframe with the feature points of the candidate keyframes. The RANSAC algorithm is used to optimize the feature matching results and the PnP method is used to calculate the relative pose between the current keyframe and the candidate keyframes. The candidate object frame index and relative pose are stored and added to the pose graph as additional constraints for pose optimization.

[0099] In this embodiment, the LightGlue network is used to match feature points of the current keyframe with feature points of candidate keyframes. A pair of points is considered to have a correspondence only if both points are predicted to be matching points and their similarity is higher than that of other points in the image. The LightGlue network describes how to calculate the loss during training through a loss function, so as to encourage the model to predict the correct correspondence as early as possible. The loss function of the LightGlue network is shown below:

[0100]

[0101] The loss function consists of three parts, corresponding to the correctly matched points and the mismatched points in Figures A and B, respectively. Here, γ is the log-likelihood loss for calculating the prediction of the correctly matched point (i,j)∈M. l P ij The elements in the time prediction assignment matrix; α is the log-likelihood loss used to calculate the predicted matching scores of points marked as mismatches in graph A. Let be the matchability score of point i; β is the log-likelihood loss used to calculate the predicted match scores of points marked as mismatches belonging to graph B. Let be the matching score for point j;

[0102] For a correct matching point (i,j)∈M, calculate the predicted correspondence. l P ij The log-likelihood loss; for those marked as unmatchable. Calculate the predicted matchability score The log-likelihood loss; for those marked as unmatchable. Calculate the predicted matchability score The log-likelihood loss is used; the loss function is optimized based on their matching scores, encouraging the network to learn higher matching confidence.

[0103] Furthermore, the optimization of the feature matching results using the RANSAC algorithm in step S7 includes:

[0104] S71. Input the images of the current keyframe and candidate keyframes, and use the LightGlue network to perform preliminary matching to obtain the point pair set I0 = {(f c1 ,f r1 ),(f c2,f r2 ),...,(f cn ,f rn )}; where each pair of matching points (f ci ,f rj ) contains feature points f in the current image ci and feature points f in candidate images rj The coordinates of the feature points are (x, y, y) ci ,y ci ) and (x rj ,y rj );

[0105] S72. Randomly select 4 pairs of points from the initially matched point pair set I0, and use the selected point pairs to estimate the geometric transformation matrix H between the two frames of images. The formula is expressed as follows:

[0106]

[0107] in, and The homogeneous coordinates of the matching points;

[0108] Calculate all matching point pairs (f ci ,f rj The projection error ∈ ij =||(x ci ,y ci )-H(x rj ,y rj )||;

[0109] S73, Set the interior point error threshold ∈ th Points with projection errors less than a threshold are considered interior points, forming an interior point set I. This process is repeated multiple times until the largest interior point set becomes the final set of matched point pairs I. max .

[0110] In summary, this invention proposes a binocular inertial navigation SLAM localization method that integrates deep learning. At the system front end, a Superpoint network and a YoloV8-seg network extract image feature points and feature descriptors from the images received from the binocular camera and IMU data. At the system back end, a DBoW3 bag-of-words model is trained using binary-encoded Superpoint feature descriptors for loop closure detection. A LightGlue network is used to match the current keyframe with candidate keyframes, and RANSAC is used to optimize the matching results. This improves robustness in low-light and weak-texture environments, reduces interference from dynamic scenes, and enhances the accuracy of pose graph optimization, ensuring high precision and stability in real-time localization.

[0111] It should be noted that, for those skilled in the art, it is obvious that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0112] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A binocular inertial navigation SLAM localization method integrating deep learning, characterized in that, Specifically include: S1. Receive frame-by-frame images of the video stream through a binocular camera and perform preprocessing and image analysis, while simultaneously receiving inertial navigation information; S2, the SLAM system front-end extracts feature points and feature descriptors from the image through the Superpoint network; the feature points are input as a point set into the YOLOV8-seg network for dynamic point removal; the feature descriptors are binary encoded to train the DBoW3 bag-of-words model; S3. Use the YOLOV8-seg network for object detection and motion / static classification of images; including: The system acquires the detection bounding box of the dynamic object and performs semantic segmentation on the image within the box, obtaining the set of coordinate points of the region where the dynamic object is located. It determines whether the feature points are within the detection bounding box of the dynamic object by comparing the coordinates of each feature point within the box with the coordinates of the dynamic region. Feature points within 2 pixels around the dynamic target mask (mask M) are marked as dynamic points and removed. The filtered point set is used to perform forward and reverse optical flow tracking of the feature points in the two frames before and after the left eye image as a reference, and the camera pose change is calculated. Forward and reverse optical flow matching is performed on the left and right eye images to perform 3D reconstruction of the feature points. S4. Acquire IMU data between the previous frame and the current image frame, including the measurement data of the accelerometer and gyroscope of the IMU sensor, i.e., acceleration and angular velocity; take the average value of the IMU acceleration in the current frame, align it with gravity, and obtain the initial IMU attitude; acquire the image pose of the previous frame, apply the IMU pre-integration result during this period, and obtain the initial pose of the current frame. S5. Select key frames from ordinary image frames to fill the sliding window, use the sliding window mechanism to perform BA optimization, and complete the initialization of the binocular inertial navigation. S6. Encode the extracted feature descriptors into binary form to train the DBoW3 bag-of-words model and build an offline dictionary for loop closure detection; perform loop closure detection on keyframes, use the offline dictionary to perform feature statistics on each new keyframe, and search for candidate keyframes corresponding to the current keyframe. S7. The LightGlue network is used to match the feature points of the current keyframe with the feature points of the candidate keyframes. The RANSAC algorithm is used to optimize the feature matching results and the PnP method is used to calculate the relative pose between the current keyframe and the candidate keyframes. The candidate object frame index and relative pose are stored and added to the pose graph as additional constraints for pose optimization.

2. The binocular inertial navigation SLAM localization method fused with deep learning according to claim 1, characterized in that, The extraction of feature points from the image in step S2 specifically includes: S21. An encoder based on the VGG architecture is used to progressively reduce the size of the image and extract features; the encoder consists of convolutional layers, max-pooling layers, and non-linear activation function layers; through three consecutive max-pooling layers, the image size is reduced to 1 / 8 of the original image; S22. Perform feature point extraction; reduce the dimensionality of the 128-dimensional feature map to 65-dimensional; the 65-dimensional feature map represents a potential feature point of each 8×8 local region of the original image and a Dustbin channel for outputting no feature points. S23. The probability is normalized by Softmax operation, then slice processing is performed, and finally the output feature point position is determined by Non-maximum Suppression (NMS).

3. The binocular inertial navigation SLAM localization method integrating deep learning according to claim 2, characterized in that, During feature point extraction in step S2, feature descriptors are also extracted simultaneously for training the DBoW3 bag-of-words model in step S6, specifically including: The 128-dimensional feature map is upgraded to 256-dimensional and normalized. Based on the location information of the feature points, the corresponding feature descriptors are extracted from the feature map using the bilinear interpolation method.

4. The binocular inertial navigation SLAM localization method fused with deep learning according to claim 1, characterized in that, Step S3 specifically includes: S31. Use the YOLOV8-seg network to perform object detection and semantic segmentation on the current frame image, and obtain the bounding box B = {b1, b2, ..., b} for object detection. n } and the semantic segmentation mask M = {m1, m2, ..., m n }; S32. Based on the bounding box B, store the feature points of each dynamic region in the point set P = {p1, p2, ..., p...} n In this context, any point within two pixels of the dynamic bounding box is considered to be inside the bounding box; a single feature point p i The position coordinates are (x i ,y i ); S33. For each feature point p i Calculate its confidence score C based on its position in the mask M. i The formula is expressed as: C i =α·D j +β·M j (x i ,y i ); Among them, D j M represents the confidence score for object detection. j (x i ,y i Let α be the confidence value of the semantic segmentation mask at the feature point location, and β be the weight coefficients, satisfying α + β = 1; set a confidence threshold T, and select feature points p with confidence values ​​less than the threshold T. i By removing points from set P, we obtain the point set P' after removing dynamic feature points, i.e.: P' = {p i ∈P|C i ≥T}.

5. The binocular inertial navigation SLAM localization method fused with deep learning according to claim 1, characterized in that, Step S5 specifically includes: S51. Using ordinary frame image frames with good feature point tracking as keyframes and filling the sliding window, calculate the standard deviation of IMU acceleration within the sliding window to determine the speed of movement; the sliding window refers to a container containing N keyframes constructed during the optimization process. S52. Find a frame with a sufficiently large disparity to the current frame and an accurate match in the sliding window, calculate the relative pose change, and use binocular triangulation to triangulate all feature points. S53. Correct the gyroscope bias, initialize gravity, gravity direction and scale; using a frame with sufficient parallax and accurate matching as the reference frame, perform BA optimization on the visual and IMU redundant terms within the window to complete the binocular inertial navigation initialization.

6. The binocular inertial navigation SLAM localization method fused with deep learning according to claim 5, characterized in that, In step S52: The criterion for determining sufficiently large disparity is: the difference in the x-coordinate pixels of the same matching point in different frames is greater than the disparity threshold T. d1 This is considered to indicate a sufficiently large parallax; The criterion for judging accurate matching is: for each feature point, the projection error calculated using the matched feature points is less than the matching threshold T. d2 If so, the match is considered accurate.

7. The binocular inertial navigation SLAM localization method fused with deep learning according to claim 1, characterized in that, Step S6 specifically includes: S61. Extract each feature descriptor d through the Superpoint network. i Perform binary encoding to generate binary feature descriptors bi i ; S62. Construct an initial vocabulary V = {v1, v2, ..., v...} K }, where K is the size of the predetermined vocabulary, and the K-means clustering algorithm is used to cluster all binary feature descriptors {bi1,bi2,...,bi}. n Clustering is performed to obtain K cluster centers {ct1, ct2, ..., ct3}. K }, where each cluster center ct j To represent a word, we treat it as a word item in vocabulary V; S63. For each binary feature descriptor bi i Calculate its relationship with each word v j Hamming distance H(bi) i ,v j The formula is expressed as: Where L is the length of the binary feature descriptor. Indicates a bitwise XOR operation; When calculating the Hamming distance, bi i and v j It is indexed according to its respective feature points and cluster k; the feature points p i binary feature descriptor bi i Assign the word closest to Hamming's distance (v) j That is, it is represented as S64, Words v in statistical images j The number of feature points assigned forms the word frequency vector w = [w1, w2, ..., w K ], any w indexed by j j The formula is expressed as: The value of δ(a,b) is 1 when a = b, and 0 otherwise. The word frequency vector w is normalized to obtain the final bag-of-words representation.

8. The binocular inertial navigation SLAM localization method fused with deep learning according to claim 7, characterized in that, In step S7, optimizing the feature matching results using the RANSAC algorithm specifically includes: S71. Input the images of the current keyframe and candidate keyframes, and use the LightGlue network to perform preliminary matching to obtain the point pair set I0 = {(f c1 ,f r1 ),(f c2 ,f r2 ),...,(f cn ,f rn )}; Each pair of matching points (f ci ,f rj ) Contains matching feature points f in the current keyframe ci Matching feature points f in candidate keyframes rj Their coordinates are (x ci ,y ci ) and (x rj ,y rj ); S72. Randomly select 4 pairs of points from the initially matched point pair set I0, and use the selected 4 pairs of points to estimate the geometric transformation matrix H between the two frames of images, expressed by the formula: in, and The homogeneous coordinates of the matching points; Calculate all matching point pairs (f ci ,f rj The projection error ∈ ij =||(x ci ,y ci )-H(x rj ,y rj )||; S73, Set the interior point error threshold ∈ th Points with projection errors less than a threshold are considered interior points, forming an interior point set I. This process is repeated multiple times until the largest interior point set becomes the final set of matched point pairs I. max .

Images