Binocular inertial navigation SLAM positioning method fused with deep learning

Abstract

The invention provides a binocular inertial navigation SLAM positioning method fused with deep learning, and the method specifically comprises the steps: receiving an image of a binocular camera and IMU data through a system, and extracting image feature points and feature descriptors through a Superpoint network; feature matching and three-dimensional reconstruction are carried out through front and back frames and left and right eye images, and a dynamic object is detected through a YoV8-seg network to carry out dynamic point elimination; calculating a camera pose based on IMU pre-integration, screening key frames by using a sliding window mechanism, and performing BA optimization to complete binocular inertial navigation initialization; carrying out loopback detection on the feature descriptor binary coding training DBoW3 bag-of-word model; and a LightGlue network is applied to match the current key frame and the candidate key frame, and pose map optimization is carried out through an RANSAC optimization matching result. According to the method provided by the invention, the robustness of the SLAM front end in a dark and weak texture environment can be effectively improved, dynamic scene interference is reduced, the accuracy of pose map optimization is improved, and high precision and stability of real-time positioning are ensured.

Description

Technical Field

[0001] The present invention belongs to the field of spatial positioning and mapping technology, and in particular relates to a binocular inertial navigation SLAM positioning method integrating deep learning. Background Art

[0002] With the advancement of the aviation manufacturing industry, the assembly accuracy of complex parts is becoming increasingly important to product quality and safety. Traditional manual inspection methods are inefficient and have large errors in complex environments, making it difficult to meet the needs of modern manufacturing for high-precision assembly inspection. Machine vision technology is widely used in industrial inspection due to its non-contact and high-precision advantages. However, when facing the inspection of large structural parts, traditional visual inspection methods require partition inspection and result splicing, and the inspection equipment cannot accurately determine its current position and posture, resulting in a lack of flexibility and accuracy in the inspection process.

[0003] As an advanced automated positioning and mapping method, visual SLAM (Simultaneous Localization and Mapping) technology shows great application potential. The traditional visual SLAM framework includes a front-end visual odometer and back-end optimization. The front-end is responsible for processing sensor data, tracking camera poses and generating local maps, while the back-end improves the accuracy of the global map through synchronous optimization. However, complex environments bring many challenges to traditional visual SLAM. The sparse texture and poor lighting conditions in the environment make it difficult to stably extract and match feature points, which easily leads to the loss of camera pose tracking; in addition, frequent dynamic interference, such as moving objects and personnel activities, further affects the stability and accuracy of the system. These factors work together to seriously affect the accuracy and stability of visual SLAM, making visual SLAM technology face severe tests in practical applications. Summary of the invention

[0004] In response to the problems existing in the above-mentioned prior art, the present invention proposes a binocular inertial navigation SLAM positioning method integrated with deep learning, which performs feature extraction, matching, dynamic point elimination and pose optimization through a deep learning network, significantly improving the positioning accuracy and stability, making it suitable for practical application scenarios.

[0005] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0006] A binocular inertial navigation SLAM positioning method integrating deep learning specifically comprises the following steps:

[0007] S1, receiving video stream frame by frame through binocular camera and performing preprocessing and image analysis, while receiving inertial navigation information;

[0008] S2, the front end of the SLAM system extracts the feature points and feature descriptors of the image through the Superpoint network; the feature points are input into the YoloV8-seg network as a point set for dynamic point removal, and the descriptors are binary encoded to train the DBoW3 bag-of-words model;

[0009] S3. Use the YoloV8-seg network to detect objects and classify them into static and dynamic categories. This includes:

[0010] Get the detection frame of the dynamic object and perform semantic segmentation on the image in the frame to obtain the coordinate point set of the area where the dynamic object is located; determine whether the feature point is in the dynamic object detection frame, compare the coordinates of the feature points in the frame with the coordinates of the dynamic area one by one, mark the dynamic target Mask, that is, the feature points within 2 pixels around the mask M as dynamic points and remove them; for the selected point set, use the left eye image as the reference to perform forward and reverse optical flow tracking of the feature points of the previous and next two frames, and calculate the change in camera posture; match the forward and reverse optical flow of the left and right eye images, and perform 3D reconstruction of the feature points;

[0011] S4, obtaining the IMU data between the previous frame and the current image frame, including the measurement data of the accelerometer and gyroscope of the IMU sensor, that is, acceleration and angular velocity; taking the average of the IMU acceleration of the current frame, aligning the gravity, and obtaining the IMU initial posture; obtaining the image posture of the previous frame, and applying the IMU pre-integration result during this period to obtain the initial posture of the current frame;

[0012] S5, select key frames from ordinary image frames to fill the sliding window, use the sliding window mechanism to perform BA optimization, and complete the initialization of the binocular inertial navigation;

[0013] S6. Binary encode the extracted feature descriptors to train the DBoW3 bag-of-words model and build an offline dictionary for loop detection; perform loop detection on the key frames, use the offline dictionary to perform feature statistics on each new key frame, and search for candidate key frames corresponding to the current key frame;

[0014] S7. Apply the LightGlue network to match the feature points of the current key frame with the feature points of the candidate key frames, use the RANSAC algorithm to optimize the feature matching results, and use the PnP method to calculate the relative pose between the current key frame and the candidate key frames, store the candidate object frame index and relative pose, and add them to the pose graph as additional constraints for pose optimization.

[0015] Furthermore, the step of extracting feature points of the image in step S2 specifically includes:

[0016] S21, using an encoder based on the VGG architecture to gradually reduce the size of the image and extract features; the encoder is composed of a convolutional layer, a maximum pooling layer Max-Pooling and a non-linear activation function layer; through three consecutive maximum pooling layers, the image size is reduced to 1 / 8 of the original image;

[0017] S22, extracting feature points; reducing the 128-dimensional feature map to 65 dimensions; the 65 dimensions represent a potential feature point in each 8×8 local area of ​​the original image and a Dustbin channel for outputting no feature points;

[0018] S23. Ensure probability normalization through Softmax operation, then perform Slice processing, and finally determine the output feature point position through non-maximum suppression NMS.

[0019] More specifically, when extracting feature points in step S2, feature descriptors are extracted at the same time to train the DBoW3 word band model in step S6, which specifically includes:

[0020] The 128-dimensional feature map is upgraded to 256 dimensions and normalized. According to the position information of the feature points, the bilinear interpolation method is used to extract the corresponding feature descriptor from the feature map according to the position of the feature points.

[0021] Furthermore, step S3 specifically includes:

[0022] S31, use YoloV8-seg network to perform target detection and semantic segmentation on the current frame image, and obtain the target detection bounding box B = {b1, b2, ..., b n} and the semantic segmentation mask M = {m1,m2,...,m n};

[0023] S32, based on the bounding box B, store the feature points in each dynamic area in the point set P = {p1, p2, ..., p n}, points within two 2 pixels around the dynamic bounding box are considered to be within the bounding box; a single feature point p i The position coordinates are (x i ,y i );

[0024] S33. For each feature point p i , calculate its confidence score C according to its position in the mask M i , the formula is:

[0025] C i =α·D j +β·M j (x i ,yi );

[0026] Among them, D j is the confidence score of target detection, M j (x i ,y i ) is the confidence value of the semantic segmentation mask at the feature point position, α and β are weight coefficients, satisfying α+β=1; set the confidence threshold T, and the feature point p with a confidence value less than the threshold T is i Eliminate from the set P, and obtain the point set P' after dynamic feature points are eliminated, that is: P'={p i ∈P|C i ≥T}.

[0027] Furthermore, step S5 specifically includes:

[0028] S51, taking a normal frame image frame with good feature point tracking as a key frame and filling a sliding window, calculating the standard deviation of the IMU acceleration in the sliding window to determine the moving speed; the sliding window refers to a container containing N key frames constructed during the optimization process;

[0029] S52, find a frame in the sliding window that has a sufficiently large disparity with the current frame and matches accurately, calculate the relative pose change, and use binocular triangulation to triangulate all feature points;

[0030] S53, correct the gyroscope zero bias, initialize the gravity, gravity direction and scale; use a frame with sufficiently large parallax and accurate matching as the reference system, perform BA optimization on the visual residual items and IMU residual items in the window, and complete the initialization of the binocular inertial navigation.

[0031] More specifically, in step S52:

[0032] The criterion for determining whether there is a sufficiently large parallax is: the horizontal coordinate pixel difference of the same matching point in different frames is greater than the parallax threshold T d1 It is considered that there is a sufficiently large parallax;

[0033] The criterion for accurate matching is: for each feature point, the projection error calculated using the matched feature points is less than the matching threshold T d2 The match is considered accurate.

[0034] Furthermore, step S6 specifically includes:

[0035] S61, each feature descriptor d extracted by the Superpoint network i Perform binary encoding to generate binary feature descriptor bi i ;

[0036] S62, construct an initial vocabulary V = {v1, v2, ..., vK}, where K is the size of the predetermined vocabulary, and the K-means clustering algorithm is used to cluster all binary feature descriptors {bi1,bi2,...,bi n} to cluster and obtain K cluster centers {ct1, ct2, ..., ct K}, where each cluster center ct j Represents a word as a vocabulary item in the vocabulary table V;

[0037] S63, for each binary feature descriptor bi i , calculate its correlation with each vocabulary v j The Hamming distance H(bi i ,v j ), the formula is:

[0038]

[0039] Where L is the length of the binary feature descriptor, Represents a bitwise exclusive OR operation;

[0040] When calculating the Hamming distance, bi i and v j are indexed according to their respective feature points and cluster k; the feature point p i The binary feature descriptor bi i Assign to the word v with the closest Hamming distance j , which is expressed as

[0041] S64. Count the words v in the image j The number of feature points assigned to form the word frequency vector w = [w1,w2,...,w K ], any w with j as index j The formula is:

[0042]

[0043] Among them, the value of δ(a,b) is, when a=b, δ is 1, otherwise it is 0; the word frequency vector w is normalized to obtain the final bag-of-words representation.

[0044] Furthermore, the optimization of the feature matching result using the RANSAC algorithm in step S7 specifically includes:

[0045] S71, input the current key frame and the candidate key frame images and perform preliminary matching using the LightGlue network to obtain the point pair set I0 = {(f c1 ,f r1 ),(f c2 ,fr2 ),...,(f cn ,f rn )}; Each pair of matching points (f ci ,f rj ) contains the matching feature points f in the current keyframe ci and the matching feature points f in the candidate keyframes rj , whose coordinates are (x ci ,y ci ) and (x rj ,y rj );

[0046] S72, randomly select 4 pairs of points from the preliminary matching point pair set I0, and use the selected 4 pairs of points to estimate the geometric transformation matrix H between the two frames of images. The formula is expressed as

[0047]

[0048] in, and is the homogeneous coordinates of the matching points;

[0049] Calculate all matching point pairs (f ci ,f rj )’s projection error∈ ij =||(x ci ,y ci )-H(x rj ,y rj )||;

[0050] S73, set the internal point error threshold ∈ th , the projection error is less than the threshold as the internal point, forming the internal point set I; repeat the above steps many times until the largest internal point set is the final matching point pair set I max .

[0051] Based on the above technical solution, the method proposed by the present invention has the following beneficial effects:

[0052] The system receives images and IMU data from the binocular camera, and extracts image feature points and feature descriptors through the Superpoint network, which improves the robustness of the SLAM system front end in dim light and weak texture environments. At the same time, the YoloV8-seg network is used to detect and eliminate dynamic objects, effectively reducing interference in dynamic scenes.

[0053] The backend of the SLAM system uses binary-coded Superpoint feature descriptors to train the DBoW3 bag-of-words model for loop detection, matches the current keyframe with the candidate keyframes through the LightGlue network, and uses RANSAC to optimize the matching results, thereby improving the accuracy of the pose graph optimization code and ensuring high precision and strong robustness of real-time positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is an overall flow chart of a binocular inertial navigation SLAM positioning method integrated with deep learning proposed in the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0057] like Figure 1 As shown, a binocular inertial navigation SLAM positioning method integrating deep learning proposed by the present invention is given, which specifically includes the following steps:

[0058] S1, receiving video stream frame by frame through binocular camera and performing preprocessing and image analysis, while receiving inertial navigation information;

[0059] S2, the SLAM system front end extracts the feature points and feature descriptors of the image through the Superpoint network; as a preferred implementation, step S2 specifically includes:

[0060] S21, using an encoder based on the VGG architecture to gradually reduce the size of the image and extract features; the encoder is composed of a convolutional layer, a maximum pooling layer Max-Pooling and a non-linear activation function layer; through three consecutive maximum pooling layers, the image size is reduced to 1 / 8 of the original image;

[0061] S22, extracting feature points; reducing the 128-dimensional feature map to 65 dimensions; the 65 dimensions represent a potential feature point in each 8×8 local area of ​​the original image and a Dustbin channel for outputting no feature points; the introduction of the Dustbin channel avoids misjudgment of regions without feature points;

[0062] S23. Ensure probability normalization through Softmax operation, then perform Slice processing, and finally determine the output feature point position through non-maximum suppression NMS.

[0063] While extracting feature points, feature descriptors are extracted to train the DBoW3 bag-of-words model in step S6. The specific process is as follows:

[0064] The 128-dimensional feature map is upgraded to 256 dimensions, where 256 dimensions represent the dimensions of the features to be output; the upgraded feature map is normalized, and the corresponding feature descriptors are extracted from the feature map according to the position information of the feature points using the bilinear interpolation method. Corresponding feature descriptors;

[0065] The loss function formula based on feature points and feature descriptors is as follows:

[0066]

[0067] In the above formula, is the loss associated with the feature points, is the loss related to the feature descriptor, λ is the weight for balancing the loss function of the feature point and the loss function of the feature descriptor; Y. Y' is the characteristic point; is a feature descriptor; S represents the set of all correct matching pairs. This loss function is used to evaluate the training process of the entire network.

[0068] S3. Use the YoloV8-seg network to detect objects and classify them into static and dynamic categories. This includes:

[0069] Get the detection frame of the dynamic object and perform semantic segmentation on the image in the frame to obtain the coordinate point set of the area where the dynamic object is located; determine whether the feature point is in the dynamic object detection frame, compare the coordinates of the feature points in the frame with the coordinates of the dynamic area one by one, mark the dynamic target Mask, that is, the feature points within 2 pixels around the mask M as dynamic points and remove them; for the selected point set, use the left eye image as the reference to perform forward and reverse optical flow tracking of the feature points of the previous and next two frames, and calculate the change in camera posture; match the forward and reverse optical flow of the left and right eye images, and perform 3D reconstruction of the feature points;

[0070] As a preferred implementation mode, step S3 specifically includes:

[0071] S31, use YoloV8-seg network to perform target detection and semantic segmentation on the current frame image, and obtain the target detection bounding box B = {b1, b2, ..., b n} and the semantic segmentation mask M = {m1,m2,...,mn};

[0072] S32, based on the bounding box B, store the feature points in each dynamic area in the point set P = {p1, p2, ..., p n}, points within two 2 pixels around the dynamic bounding box are considered to be within the bounding box; a single feature point p i The position coordinates are (x i ,y i );

[0073] S33. For each feature point p i , calculate its confidence score C according to its position in the mask M i , the formula is:

[0074] C i =α·D j +β·M j (x i ,y i );

[0075] Among them, D j is the confidence score of target detection, M j (x i ,y i ) is the confidence value of the semantic segmentation mask at the feature point position, α and β are weight coefficients, satisfying α+β=1; set the confidence threshold T, and the feature point p with a confidence value less than the threshold T is i Eliminate from the set P, and obtain the point set P' after dynamic feature points are eliminated, that is: P'={p i ∈P|C i ≥T}.

[0076] S4, obtaining the IMU data between the previous frame and the current image frame, including the measurement data of the accelerometer and gyroscope of the IMU sensor, that is, acceleration and angular velocity; taking the average of the IMU acceleration of the current frame, aligning the gravity, and obtaining the IMU initial posture; obtaining the image posture of the previous frame, and applying the IMU pre-integration result during this period to obtain the initial posture of the current frame;

[0077] In this embodiment, step S4 specifically calculates the pose by constructing IMU pre-integration and residual, specifically:

[0078] Get the measurement data of the accelerometer and gyroscope of the IMU sensor, which are acceleration a t and angular velocity ω tThe data is sampled at a fixed frequency, and the IMU median pre-integral under discrete time is calculated; the changes in position and velocity are obtained by calculating the pre-integral, and then the pose residual during this period is deduced based on the pre-integration, and the pose estimation is optimized by minimizing the residual. The IMU residual and the image frame residual are optimized to obtain the pose change between the two frames.

[0079] S5, select key frames from ordinary image frames to fill the sliding window, use the sliding window mechanism to perform BA optimization, and complete the initialization of the binocular inertial navigation;

[0080] As a preferred implementation, step S5 specifically includes:

[0081] S51, taking a normal frame image frame with good feature point tracking as a key frame and filling the sliding window, calculating the standard deviation of the IMU acceleration in the sliding window to determine the moving speed;

[0082] S52, find a frame in the sliding window that has a sufficiently large disparity with the current frame and matches more accurately, calculate the relative pose change, and use binocular triangulation to triangulate all feature points;

[0083] The criterion for determining whether there is a sufficiently large parallax is: the horizontal coordinate pixel difference of the same matching point in different frames is greater than the parallax threshold T d1 It is considered that there is a sufficiently large parallax;

[0084] The criterion for accurate matching is: for each feature point, the projection error calculated using the matched feature points is less than the matching threshold T d2 The match is considered accurate;

[0085] In this embodiment, T d1 =10, T d2 =2;

[0086] S53, correct the gyroscope zero bias, initialize the gravity, gravity direction and scale; use a frame with sufficiently large parallax and relatively accurate matching as the reference system, perform BA optimization on the visual residual items and IMU residual items in the window, and complete the initialization of the binocular inertial navigation.

[0087] S6. Binary encode the extracted feature descriptors to train the DBoW3 bag-of-words model and build an offline dictionary for loop detection; perform loop detection on the key frames, use the offline dictionary to perform feature statistics on each new key frame, and search for candidate key frames corresponding to the current key frame;

[0088] As a preferred implementation, step S6 specifically includes:

[0089] S61, each feature descriptor d extracted by the Superpoint network iPerform binary encoding to generate binary feature descriptor bi i ;

[0090] S62, construct an initial vocabulary V = {v1, v2, ..., v K}, where K is the size of the predetermined vocabulary, and the K-means clustering algorithm is used to cluster all binary feature descriptors {bi1,bi2,...,bi n} to cluster and obtain K cluster centers {ct1, ct2, ..., ct K}, where each cluster center ct j Represents a word as a vocabulary item in the vocabulary table V;

[0091] S63, for each binary feature descriptor bi i , calculate its correlation with each vocabulary v j The Hamming distance H(bi i ,v j ), the formula is:

[0092]

[0093] Where L is the length of the binary feature descriptor, Represents a bitwise exclusive OR operation;

[0094] When calculating the Hamming distance, bi i and v j are indexed according to their respective feature points and cluster k; the feature point p i The binary feature descriptor bi i Assign to the word v with the closest Hamming distance j , which is expressed as

[0095] S64. Count the words v in the image j The number of feature points assigned to form the word frequency vector w = [w1,w2,...,w K ], any w with j as index j The formula is:

[0096]

[0097] Among them, the value of δ(a,b) is, when a=b, δ is 1, otherwise it is 0; the word frequency vector w is normalized to obtain the final bag-of-words representation.

[0098] S7. Apply the LightGlue network to match the feature points of the current key frame with the feature points of the candidate key frames, use the RANSAC algorithm to optimize the feature matching results, and use the PnP method to calculate the relative pose between the current key frame and the candidate key frames, store the candidate object frame index and relative pose, and add them to the pose graph as additional constraints for pose optimization.

[0099] In this embodiment, the LightGlue network is used to match the feature points of the current key frame with the feature points of the candidate key frame; a pair of points is considered to have a corresponding relationship only when both points are predicted as matching points and their similarity is higher than other points in the image; the LightGlue network describes how to calculate the loss during the training process through the loss function, so as to promote the model to predict the correct corresponding relationship as early as possible; the loss function of the LightGlue network is as follows:

[0100]

[0101] The loss function consists of three parts, corresponding to the correct matching points and the unmatched points in Figures A and B, where γ is the log-likelihood loss corresponding to the prediction of the correct matching point (i, j)∈M. l P ij is the element in the prediction assignment matrix; α is the log-likelihood loss of the predicted matching score for the points marked as mismatches in the computation graph A, is the matchability score of point i; β is the log-likelihood loss for calculating the predicted match scores of points marked as mismatches belonging to graph B, is the matchability score of point j;

[0102] For the correct matching point (i, j)∈M, calculate the predicted correspondence l P ij The log-likelihood loss of Calculating the predicted matchability score The log-likelihood loss of Calculating the predicted matchability score The loss function is optimized based on their matching scores, encouraging the network to learn higher matching confidence.

[0103] In addition, the optimization of the feature matching result using the RANSAC algorithm in step S7 includes:

[0104] S71, input the current key frame and the candidate key frame images and perform preliminary matching using the LightGlue network to obtain the point pair set I0 = {(f c1 ,f r1 ),(f c2,f r2 ),...,(f cn ,f rn )}; where each pair of matching points (f ci ,f rj ) contains the feature points f in the current image ci and the feature points f in the candidate image rj The coordinates of the feature points are (x ci ,y ci ) and (x rj ,y rj );

[0105] S72, randomly select 4 pairs of points from the preliminary matching point pair set I0, and use the selected point pairs to estimate the geometric transformation matrix H between the two frames of images. The formula is expressed as

[0106]

[0107] in, and is the homogeneous coordinates of the matching points;

[0108] Calculate all matching point pairs (f ci ,f rj )’s projection error∈ ij =||(x ci ,y ci )-H(x rj ,y rj )||;

[0109] S73, set the internal point error threshold ∈ th , the projection error is less than the threshold as the internal point, forming the internal point set I; repeat the above steps many times until the largest internal point set is the final matching point pair set I max .

[0110] In summary, the present invention proposes a binocular inertial navigation SLAM positioning method integrating deep learning. The Superpoint network and YoloV8-seg network at the front end of the system extract image feature points and feature descriptors from the images and IMU data of the binocular camera received by the system. The binary-coded Superpoint feature descriptor is used to train the DBoW3 bag-of-words model at the back end of the system for loop detection. The current key frame is matched with the candidate key frame through the LightGlue network, and the matching result is optimized by RANSAC. The robustness in dim light and weak texture environment is improved, the interference of dynamic scenes is reduced, and the accuracy of pose graph optimization is improved, so as to ensure the high precision and stability of real-time positioning.

[0111] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-restrictive from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.

[0112] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A binocular inertial navigation SLAM positioning method integrating deep learning, characterized in that: Specifically include: S1, receiving video stream frame by frame through binocular camera and performing preprocessing and image analysis, while receiving inertial navigation information; S2, the SLAM system front end extracts the feature points and feature descriptors of the image through the Superpoint network; the feature points are input as point sets into the YoloV8-seg network for dynamic point removal; the feature descriptors are binary encoded to train the DBoW3 bag-of-words model; S3. Use the YoloV8-seg network to detect objects and classify them into static and dynamic categories. This includes: Get the detection frame of the dynamic object and perform semantic segmentation on the image in the frame to obtain the coordinate point set of the area where the dynamic object is located; determine whether the feature point is in the dynamic object detection frame, compare the coordinates of the feature points in the frame with the coordinates of the dynamic area one by one, mark the dynamic target Mask, that is, the feature points within 2 pixels around the mask M as dynamic points and remove them; for the selected point set, use the left eye image as the reference to perform forward and reverse optical flow tracking of the feature points of the previous and next two frames, and calculate the change in camera posture; match the forward and reverse optical flow of the left and right eye images, and perform 3D reconstruction of the feature points; S4, obtaining the IMU data between the previous frame and the current image frame, including the measurement data of the accelerometer and gyroscope of the IMU sensor, that is, acceleration and angular velocity; taking the average of the IMU acceleration of the current frame, aligning the gravity, and obtaining the IMU initial posture; obtaining the image posture of the previous frame, and applying the IMU pre-integration result during this period to obtain the initial posture of the current frame; S5, select key frames from ordinary image frames to fill the sliding window, use the sliding window mechanism to perform BA optimization, and complete the initialization of the binocular inertial navigation; S6. Binary encode the extracted feature descriptors to train the DBoW3 bag-of-words model and build an offline dictionary for loop detection; perform loop detection on the key frames, use the offline dictionary to perform feature statistics on each new key frame, and search for candidate key frames corresponding to the current key frame; S7. Apply the LightGlue network to match the feature points of the current key frame with the feature points of the candidate key frames, use the RANSAC algorithm to optimize the feature matching results, and use the PnP method to calculate the relative pose between the current key frame and the candidate key frames, store the candidate object frame index and relative pose, and add them to the pose graph as additional constraints for pose optimization.

2. A binocular inertial navigation SLAM positioning method integrating deep learning according to claim 1, characterized in that: The step S2 of extracting the feature points of the image specifically includes: S21, using an encoder based on the VGG architecture to gradually reduce the size of the image and extract features; the encoder is composed of a convolutional layer, a maximum pooling layer Max-Pooling and a non-linear activation function layer; through three consecutive maximum pooling layers, the image size is reduced to 1 / 8 of the original image; S22, extracting feature points; reducing the 128-dimensional feature map to 65 dimensions; the 65 dimensions represent a potential feature point in each 8×8 local area of ​​the original image and a Dustbin channel for outputting no feature points; S23. Ensure probability normalization through Softmax operation, then perform Slice processing, and finally determine the output feature point position through non-maximum suppression NMS.

3. A binocular inertial navigation SLAM positioning method integrating deep learning according to claim 2, characterized in that: When extracting feature points in step S2, feature descriptors are extracted at the same time to train the DBoW3 word band model in step S6, which specifically includes: The 128-dimensional feature map is upgraded to 256 dimensions and normalized. According to the position information of the feature points, the bilinear interpolation method is used to extract the corresponding feature descriptor from the feature map according to the position of the feature points.

4. A binocular inertial navigation SLAM positioning method integrating deep learning according to claim 1, characterized in that: Step S3 specifically includes: S31, use YoloV8-seg network to perform target detection and semantic segmentation on the current frame image, and obtain the target detection bounding box B = {b1, b2, ..., b n } and the semantic segmentation mask M = {m1,m2,...,m n }; S32, based on the bounding box B, store the feature points in each dynamic area in the point set P = {p1, p2, ..., p n }, points within two 2 pixels around the dynamic bounding box are considered to be within the bounding box; a single feature point p i The position coordinates are (x i ,y i ); S33. For each feature point p i , calculate its confidence score C according to its position in the mask M i , the formula is: C i =α·D j +β·M j (x i ,y i ); Among them, D j is the confidence score of target detection, M j (x i ,y i ) is the confidence value of the semantic segmentation mask at the feature point position, α and β are weight coefficients, satisfying α+β=1; set the confidence threshold T, and the feature point p with a confidence value less than the threshold T is i Eliminate from the set P, and obtain the point set P' after dynamic feature points are eliminated, that is: P'={p i ∈P|C i ≥T}.

5. The binocular inertial navigation SLAM positioning method integrating deep learning according to claim 1, characterized in that: Step S5 specifically includes: S51, taking a normal frame image frame with good feature point tracking as a key frame and filling a sliding window, calculating the standard deviation of the IMU acceleration in the sliding window to determine the moving speed; the sliding window refers to a container containing N key frames constructed during the optimization process; S52, find a frame in the sliding window that has a sufficiently large disparity with the current frame and matches accurately, calculate the relative pose change, and use binocular triangulation to triangulate all feature points; S53, correct the gyroscope zero bias, initialize the gravity, gravity direction and scale; use a frame with sufficiently large parallax and accurate matching as the reference system, perform BA optimization on the visual residual items and IMU residual items in the window, and complete the initialization of the binocular inertial navigation.

6. The binocular inertial navigation SLAM positioning method integrating deep learning according to claim 5, characterized in that: In step S52: The criterion for determining whether there is a sufficiently large parallax is: the horizontal coordinate pixel difference of the same matching point in different frames is greater than the parallax threshold T d1 It is considered that there is a sufficiently large parallax; The criterion for accurate matching is: for each feature point, the projection error calculated using the matched feature points is less than the matching threshold T d2 The match is considered accurate.

7. The binocular inertial navigation SLAM positioning method integrating deep learning according to claim 1, characterized in that: Step S6 specifically includes: S61, each feature descriptor d extracted by the Superpoint network i Perform binary encoding to generate binary feature descriptor bi i ; S62, construct an initial vocabulary V = {v1, v2, ..., v K }, where K is the size of the predetermined vocabulary, and the K-means clustering algorithm is used to cluster all binary feature descriptors {bi1,bi2,...,bi n } to cluster and obtain K cluster centers {ct1, ct2, ..., ct K }, where each cluster center ct j Represents a word as a vocabulary item in the vocabulary table V; S63, for each binary feature descriptor bi i , calculate its correlation with each vocabulary v j The Hamming distance H(bi i ,v j ), the formula is: Where L is the length of the binary feature descriptor, Represents a bitwise exclusive OR operation; When calculating the Hamming distance, bi i and v j are indexed according to their respective feature points and cluster k; the feature point p i The binary feature descriptor bi i Assign to the word v with the closest Hamming distance j , which is expressed as S64. Count the words v in the image j The number of feature points assigned to form the word frequency vector w = [w1,w2,...,w K ], any w with j as index j The formula is expressed as: Among them, the value of δ(a,b) is, when a=b, δ is 1, otherwise it is 0; the word frequency vector w is normalized to obtain the final bag-of-words representation.

8. The binocular inertial navigation SLAM positioning method integrating deep learning according to claim 7, characterized in that: In step S7, the optimization of the feature matching result using the RANSAC algorithm specifically includes: S71, input the current key frame and the candidate key frame images and perform preliminary matching using the LightGlue network to obtain the point pair set I0 = {(f c1 ,f r1 ),(f c2 ,f r2 ),...,(f cn ,f rn )}; Each pair of matching points (f ci ,f rj ) contains the matching feature points f in the current keyframe ci and the matching feature points f in the candidate keyframe rj , whose coordinates are (x ci ,y ci ) and (x rj ,y rj ); S72, randomly select 4 pairs of points from the preliminary matching point pair set I0, and use the selected 4 pairs of points to estimate the geometric transformation matrix H between the two frames of images. The formula is expressed as Among them, p ci =[x ci ,y ci ,1] T and p ri =[x rj ,y rj ,1] T is the homogeneous coordinates of the matching points; Calculate all matching point pairs (f ci ,f rj )’s projection error∈ ij =||(x ci ,y ci )-H(x rj ,y rj )||; S73, set the internal point error threshold ∈ th , the projection error is less than the threshold as the internal point, forming the internal point set I; repeat the above steps many times until the largest internal point set is the final matching point pair set I max .

Images