A visual slam method based on improved superpoint in dynamic environment

Abstract

The application belongs to the technical field of visual SLAM, and specifically provides a visual SLAM method based on an improved SuperPoint in a dynamic environment, to solve the problem of insufficient robustness of the existing visual SLAM method in a dynamic environment; the application firstly constructs a feature point and descriptor extraction network with an added residual weight branch on the basis of the SuperPoint network, performs weighted fusion on the network residual weight branch output and the feature point branch output, so that the constructed network has the function of eliminating dynamic feature points; then the constructed network is applied to the tracking thread of ORB-SLAM2, to replace the ORB method to extract image feature points and descriptors; in combination with the local mapping and loop detection threads of ORB-SLAM2, a robust visual SLAM system in a dynamic environment is realized, and the accuracy and robustness of the visual SLAM system in the dynamic environment are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual SLAM, and specifically provides a visual SLAM method based on improved SuperPoint in a dynamic environment. Background Art

[0002] Simultaneous Localization and Mapping (SLAM), a hot topic in robotics and computer vision, is widely used in augmented reality, indoor service robots, and autonomous driving. Visual SLAM uses camera images as input and simultaneously estimates the camera's pose and constructs a map of the surrounding area based on changes between images. Most current visual SLAM methods rely on the strong assumption that the observed environment is static. The presence of dynamic objects in real-world environments leads to unstable data associations, resulting in accumulated drift and even loss.

[0003] In current visual SLAM systems for dynamic environments, semantic-based methods obtain pixel-level masks or bounding boxes of prior dynamic objects through target detection or semantic segmentation, treat feature points on moving objects in the scene as "bad points", and eliminate feature points on dynamic objects to reduce the impact of dynamic objects on pose estimation, thereby improving positioning accuracy in dynamic environments. At the same time, current mainstream visual SLAM systems mainly rely on manually designed point features for matching and tracking between images, which can easily lead to tracking failures in environments with varying viewing angles and large changes in lighting environments. In recent years, networks based on deep learning feature points and descriptor extraction have performed well in scenes with drastic changes in lighting and viewing angles. Therefore, the present invention provides a visual SLAM method based on improved SuperPoint in dynamic environments. Summary of the Invention

[0004] The purpose of the present invention is to address the problem of insufficient robustness of existing visual SLAM methods in dynamic environments and propose a visual SLAM method based on improved SuperPoint in dynamic environments. The present invention adds a residual weight map on the basis of the feature point and descriptor extraction network SuperPoint, utilizes semantic information to eliminate dynamic feature points, and guides SuperPoint to detect stable static feature points, thereby significantly improving the accuracy and robustness of the visual SLAM method in dynamic scenes.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A visual SLAM method based on improved SuperPoint in a dynamic environment, characterized by comprising the following steps:

[0007] Step 1: Build a labeled dataset;

[0008] Take the unlabeled real image as input, use the end-to-end visual relocalization method PixLoc to generate the residual weight map of the input image as the residual weight label, and use the feature point self-labeling method to obtain the feature point pseudo label;

[0009] Step 2: Build an image static feature point and descriptor extraction network and complete offline training on a labeled dataset;

[0010] The image static feature point and descriptor extraction network includes: a shared encoder, a residual weight branch, a feature point branch and a descriptor branch. The input image is downsampled by the shared encoder to obtain a preliminary feature map. The residual weight branch, the feature point branch and the descriptor branch respectively decode the preliminary feature map to obtain a residual weight map, a feature point probability map and a descriptor feature map. The residual weight map and the feature point probability map are further weighted and fused to obtain a feature point confidence map. The confidence threshold is then set, and feature points are obtained by a fast approximate non-maximum suppression algorithm. The descriptor feature map is bicubic interpolated and then normalized to unit length by L2 to obtain a dense descriptor vector.

[0011] Step 3: Use the trained image static feature point and descriptor extraction network in the feature point and descriptor extraction process of ORB-SLAM2 to implement the visual SLAM method based on the improved SuperPoint.

[0012] Furthermore, the shared encoder obtains a preliminary feature map from the input image, which is specifically composed of Conv3×3×1×64 / s2, Conv3×3×64×64, Conv3×3×64×128 / s2, Conv3×3×128×128, Conv3×3×128×256 / s2, and Conv3×3×256×256 connected in sequence, and a nonlinear function ReLU layer is set after each convolution layer.

[0013] Furthermore, the residual weight branch decodes the preliminary feature map through the residual weight decoder, and uses the PixelShuffle operation to upsample the residual weight map during the decoding process to restore it to the original size of the input image. Finally, the output is mapped to the [0,1] range through the Sigmoid function to obtain the residual weight map; the residual weight decoder is composed of Conv3×3×256×256, Conv3×3×64×64, Conv3×3×16×16, and Conv1×1×4×1 connected in sequence, and the nonlinear function Relu layer is set after the convolution layers Conv3×3×256×256, Conv3×3×64×64, and Conv3×3×16×16.

[0014] Furthermore, in the feature point extraction branch, the preliminary feature map is decoded by a feature point decoder to obtain an output, and then the recovery channel is discarded through a Softmax operation, and finally the dimension is switched using a Reshape operation to obtain a feature point probability map; the feature point decoder is composed of Conv 3×3×256×256 and Conv1×1×256×65, and a nonlinear function Relu layer is set after Conv 3×3×256×256.

[0015] Furthermore, in the descriptor detection branch, the preliminary feature map is decoded by the descriptor decoder to obtain a descriptor feature map with a dimension of (H / 8)×(W / 8)×256, and then bicubic linear interpolation and L2 normalization are performed to obtain a dense descriptor vector of unit length (dimension is H×W×256); the descriptor decoder is composed of Conv 3×3×256×256 and Conv1×1×256×256, and a nonlinear function ReLU layer is set after Conv 3×3×256×256.

[0016] Furthermore, the offline training is specifically as follows:

[0017] Perform a random homography transformation on the real image in the labeled dataset to obtain a replica image. The real image and the replica image are combined into an image pair and input into the image static feature point and descriptor extraction network. A loss function is set and the image static feature point and descriptor extraction network is trained offline.

[0018] The loss function L adds residual weight loss to the SuperPoint loss function, that is, the loss function is composed of feature point loss L p , descriptor loss L d and the residual weight loss L w It consists of three parts, specifically:

[0019] L(X p ,X′ p ,X w ,X′ w ,D,D′;Y p ,Y′ p ,Y w ,Y′ w ,S)=L p (X p ,Y p )+L p (X′ p ,Y′ p )+λ1L w (X w ,Y w )+λ1L w(X′ w ,Y′ w )+λ2L d (D,D′,S)

[0020] Among them, X p With Y p The predicted output and label value of the feature points of the real image, X′ p and Y′ p The feature point prediction output and label value for the copy image, X w With Y w Then it is the residual weight prediction output and label value of the real image, X′ w and Y′ w is the residual weight prediction output and label value of the replica image, D and D′ are the descriptor feature maps obtained after the real image and the replica image are input into the network, S represents the positional relationship between D and D′; λ1 and λ2 are hyperparameters used to balance the loss function, specifically 1.2 and 0.001;

[0021] L p The loss function for feature point prediction uses full convolution cross entropy loss, specifically:

[0022]

[0023] Among them, H c =H / 8,W c =W / 8, H and W are the height and width of the image; (h, w) represents the position coordinates of the feature point probability map, x hwk Represents the predicted value of the kth channel (h, w) position in the feature point probability map (dimension is (H / 8)×(W / 8)×65), Represents the feature point probability map at the feature point label value y hw The predicted value of the corresponding channel (h,w) position;

[0024] L w is the residual weight prediction loss function, using pixel-by-pixel L1 loss, specifically:

[0025]

[0026] Among them, x hw is the predicted value of the position (h,w) in the residual weight map, y hw is the label value of the (h,w) position in the residual weight label map;

[0027] L d To describe the loss function of the feature points, hinge loss is used, specifically:

[0028]

[0029] Among them, d hw Describes the descriptor unit at the (h,w) position in the descriptor feature map D, d′ h′w′ Descriptor unit representing the position (h′, w′) in the descriptor feature map D′;

[0030] s hwh′w′ To characterize d hw and d′ h′w′ Is the location close to:

[0031]

[0032] p hw Describes the subunit d hw The coordinate position on the real image, Indicates that the subunit position p will be described hw The position of the descriptor unit after the corresponding homography transformation (consistent with the transformation between the real image and the copy image); p h′w′ Describes the subunit d h′w′ At the coordinate position of the copy image;

[0033] l d For: l d (d, d′; s) = λ d ×s×max(0,m p -d T d′)+(1-s)×max(0,d T d′-m n ), m p With m n is the positive margin and negative margin, λ d is a hyperparameter used to balance the positive and negative example losses in the descriptor loss.

[0034] Furthermore, in step 1, a feature point self-labeling method is used to obtain feature point pseudo labels, specifically: the unlabeled real image is used as the input of the MagicPoint network, the input image is first subjected to N random homography transformation operations to obtain an image copy, the MagicPoint network is used to extract feature points on the transformed image copy, the image copy containing the feature points is restored through the corresponding inverse homography change, and then merged with the feature points of the real image as feature point pseudo labels.

[0035] Furthermore, the step 3 is specifically as follows: in the Tracking thread of ORB-SLAM2, the trained image static feature point and descriptor extraction network is used instead of the ORB method to extract feature points and descriptors from the image captured by the camera.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention provides a visual SLAM method based on an improved SuperPoint in a dynamic environment. First, a residual weight branch is added to a SuperPoint network, and feature point labels and residual weight map labels are generated to perform offline training on the constructed network. The output of the residual weight branch is weightedly fused with the output of the feature point branch, so that the network has the function of eliminating dynamic feature points. Then, the improved network is applied to the feature point and descriptor extraction process of ORB-SLAM2, and combined with the local mapping and loop detection threads of ORB-SLAM2, a robust visual SLAM system in a dynamic environment is realized, thereby improving the robustness of the visual SLAM system in pose estimation in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of the visual SLAM method based on the improved SuperPoint in a dynamic environment in the present invention.

[0039] Figure 2 Schematic diagram of the structure of the image static feature point and descriptor extraction network combined with residual weights in the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0041] This embodiment provides a visual SLAM method based on improved SuperPoint in a dynamic environment. The process is as follows: Figure 1 As shown, the specific steps include:

[0042] Step 1: Build a labeled dataset;

[0043] Take the unlabeled real image as input, use the end-to-end visual relocalization method PixLoc to generate the residual weight map of the input image as the residual weight label, and use the feature point self-labeling method to obtain the feature point pseudo label;

[0044] Specifically, the unlabeled real image is used as the input of the PixLoc algorithm to generate the residual weight label map, the residual weight ω h,w ∈[0,1]; PixLoc is an end-to-end visual relocalization method that uses the residual weights generated by the uncertainty map to reflect the importance of different features for positioning. It pays more attention to road signs, contours or prominent structures. For dynamic objects such as cars and fallen leaves, the closer the residual weight is to 0;

[0045] Unlabeled real images are used as input, and feature point self-labeling methods are used to obtain feature point pseudo labels. Specifically, a SuperPoint network ignoring descriptor branches is trained on a virtual dataset composed of cubes, polygons, stars, chessboards, and straight lines to enable it to have corner point extraction capabilities. This network is called MagicPoint. Unlabeled real images are used as input to the MagicPoint network. In order to detect more potential feature points, the input image is first subjected to N random homography transformation operations to obtain an image copy. The MagicPoint network is used to extract feature points on the transformed image copy. The image copy containing the feature points is restored through the corresponding inverse homography change and then merged with the feature points of the original image (input image) as pseudo labels for subsequent work.

[0046] Step 2: Build an image static feature point and descriptor extraction network and perform offline training on a labeled dataset; specifically:

[0047] The image static feature point and descriptor extraction network is as follows Figure 2 As shown, it is possible to achieve simultaneous extraction of feature points and descriptors, and at the same time, the feature point probability map is weighted in combination with the generated residual weight prediction to achieve the effect of removing dynamic feature points; specifically, it includes: a shared encoder, a residual weight branch, a feature point branch, and a descriptor branch. The input image is downsampled through the shared encoder to obtain a preliminary feature map. The residual weight branch, the feature point branch, and the descriptor branch respectively decode the preliminary feature map to obtain a residual weight map, a feature point probability map, and a descriptor feature map. Among them, the residual weight map and the feature point probability map are further weighted and fused to obtain a feature point confidence map, and then the confidence threshold is set, and the feature points are obtained by a fast approximate non-maximum suppression algorithm; the obtained descriptor feature map is subjected to bicubic linear interpolation, and then normalized to unit length through L2 to finally obtain a dense descriptor vector;

[0048] The shared encoder consists of a 3×3 convolutional layer and a nonlinear Relu function layer. After downsampling through three 3×3 convolutional layers with a stride of 2, a preliminary feature map of (H / 8)×(W / 8) size is obtained from the H×W input image. Specifically, it is composed of Conv3×3×1×64 / s2, Conv3×3×64×64, Conv3×3×64×128 / s2, Conv3×3×128×128, Conv3×3×128×256 / s2, and Conv3×3×256×256 connected in sequence, and a nonlinear function Relu layer is set after each convolutional layer.

[0049] In the residual weight branch, the preliminary feature map is decoded by the residual weight decoder. During the decoding process, the residual weight map is upsampled three times with a sampling scale of 2 using the PixelShuffle operation to return it to the original size H×W of the input image. Finally, the output is mapped to the range of [0,1] by the Sigmoid function to obtain the residual weight map; the residual weight map can guide the extraction of interest points through visual priors, so that the network learns to ignore dynamic objects such as cars and fallen leaves; the residual weight decoder is composed of Conv3×3×256×256, Conv3×3×64×64, Conv3×3×16×16, and Conv1×1×4×1 connected in sequence, and a nonlinear function ReLU layer is set after the convolution layers Conv3×3×256×256, Conv3×3×64×64, and Conv3×3×16×16;

[0050] In the feature point extraction branch, the preliminary feature map is decoded by a feature point decoder to obtain an output of dimension (H / 8)×(W / 8)×65, where the number of channels 65 corresponds to a non-overlapping local 8×8 pixel block area in the input image and one recycling channel representing no feature points; the recycled channels are then discarded through a Softmax operation, and finally a Reshape operation is used to convert the output from the (H / 8)×(W / 8)×64 dimension back to the H×W×1 dimension to obtain a feature point probability map; the feature point decoder is composed of Conv 3×3×256×256 and Conv1×1×256×65, and a nonlinear function ReLU layer is set after Conv 3×3×256×256;

[0051] In the descriptor detection branch, the preliminary feature map is decoded by the descriptor decoder to obtain a descriptor feature map with a dimension of (H / 8)×(W / 8)×256, and then bicubic linear interpolation and L2 normalization are performed to obtain a dense descriptor vector of unit length (dimension is H×W×256); the descriptor decoder is composed of Conv3×3×256×256 and Conv1×1×256×256, and a nonlinear function ReLU layer is set after Conv 3×3×256×256;

[0052] The fast non-maximum suppression algorithm is specifically as follows: first, a grid of size H×W is created, and the coordinates of the feature points obtained by filtering with a confidence threshold are used as input; the values ​​of the corresponding coordinates in the grid are set to 1, and the values ​​of the remaining coordinates are set to 0; then, the positions with a value of 1 in the grid are traversed, and the corresponding values ​​of the traversed grids are set to -1, and the values ​​of the coordinates in the traversed position area are set to 0; after the traversal is completed, all the coordinates with a value of 1 in the grid are obtained to achieve fast non-maximum suppression of the feature points;

[0053] To be more precise, all the convolutional layers in the above network are shown in the following table, where "a×a×b×c / s2" represents the convolution kernel size, "×b" represents the number of input channels, "×c" represents the number of output channels, and " / s" represents the stride. Taking "3×3×1×64 / s2" as an example, it specifically represents a convolutional layer with a convolution kernel size of 3×3, 1 input channel, 64 output channels, and a stride of 2. In addition, a nonlinear function ReLU layer is set after each 3×3 convolution layer. The default stride of other convolutional layers without a stride is set to 1.

[0054] Shared encoder Feature point decoder Descriptor decoder Residual Weight Decoder 3×3×1×64 / s2 3×3×256×256 3×3×256×256 3×3×256×256 3×3×64×64 1×1×256×65 1×1×256×256 3×3×64×64 3×3×64×128 / s2 3×3×16×16 3×3×128×128 1×1×4×1 3×3×128×256 / s2 3×3×256×256

[0055] The offline training is specifically as follows:

[0056] Perform a random homography transformation on the real image in the labeled dataset to obtain a replica image. The real image and the replica image are combined into an image pair and input into the image static feature point and descriptor extraction network. A loss function is set and the image static feature point and descriptor extraction network is trained offline.

[0057] The loss function L adds residual weight loss to the SuperPoint loss function, that is, the loss function is composed of feature point loss L p , descriptor loss L d and the residual weight loss L w It consists of three parts, specifically:

[0058] L(X p ,X′ p ,X w ,X′ w ,D,D′;Y p ,Y p ′,Y w ,Y w ′,S)=

[0059] L p (X p ,Y p )+L p (X′ p ,Y p ′)+λ1L w (X w ,Y w )+λ1L w (X′ w ,Y w ′)+λ2L d (D,D′,S)

[0060] Among them, X p With Y p For the feature point prediction output and label value of the real image, Xp ′ and Y p ′ is the feature point prediction output and label value of the copy image, X w With Y w Then it is the residual weight prediction output and label value of the real image, X′ w With Y w ′ is the residual weight prediction output and label value of the replica image, D and D′ are the descriptor feature maps obtained after the real image and the replica image are input into the network, S represents the positional relationship between D and D′; λ1 and λ2 are hyperparameters used to balance the loss function, specifically 1.2 and 0.001;

[0061] L p The loss function for feature point prediction uses full convolution cross entropy loss, specifically:

[0062]

[0063] Among them, H c =H / 8,W c =W / 8, H and W are the height and width of the image; (h, w) represents the position coordinates of the feature point probability map, x hwk Represents the predicted value of the kth channel (h, w) position in the feature point probability map (dimension is (H / 8)×(W / 8)×65), Represents the feature point probability map at the feature point label value y hw The predicted value of the corresponding channel (h,w) position;

[0064] L w is the residual weight prediction loss function, using pixel-by-pixel L1 loss, specifically:

[0065]

[0066] Among them, x hw is the predicted value of the position (h,w) in the residual weight map, y hw is the label value of the (h,w) position in each residual weight label map;

[0067] L d To describe the loss function of the feature points, hinge loss is used, specifically:

[0068]

[0069] Among them, d hw Describes the descriptor unit at the (h,w) position in the descriptor feature map D, d′ h′w′ Descriptor unit representing the position (h′, w′) in the descriptor feature map D′;

[0070] s hwh′w′ To characterize dhw and d′ h′w′ Is the location close to:

[0071]

[0072] p hw Describes the subunit d hw The coordinate position on the real image, Indicates that the subunit position p will be described hw The position of the descriptor unit after the corresponding homography transformation (consistent with the transformation between the real image and the copy image); p h′w′ Describes the subunit d h′w′ At the coordinate position of the copy image;

[0073] l d For: l d (d, d′; s) = λ d ×s×max(0,m p -d T d′)+(1-s)×max(0,d T d′-m n ), m p With m n is the positive margin and negative margin, λ d It is a hyperparameter used to balance the positive and negative example losses in the descriptor loss.

[0074] Step 3: Use the trained image static feature point and descriptor extraction network in the feature point and descriptor extraction process of ORB-SLAM2 to implement the visual SLAM method based on the improved SuperPoint; specifically:

[0075] The image captured by the camera is used as input, and the constructed static feature point and descriptor extraction network is used to extract feature points from the image. That is, in the Tracking thread of ORB-SLAM2, the visual front-end uses the constructed static feature point and descriptor extraction network instead of ORB to extract feature points and descriptors, and realizes feature point tracking of adjacent image frames through the nearest neighbor matching descriptor; combined with the Local Mapping and Loop Closing threads of ORB-SLAM2, a robust visual SLAM system in dynamic environments is realized.

[0076] In summary, the present invention builds on the SuperPoint network, combines residual weights to establish an image static feature point extraction and descriptor network structure, and performs offline training. The improved network is then used to obtain and track image feature points, and a complete SLAM system is constructed by combining back-end optimization, loop detection, and map construction. This invention uses deep learning to extract feature points in the SLAM visual front-end while effectively eliminating dynamic feature points. This can reduce the impact of dynamic objects on SLAM positioning and improve the robustness of the system in scenes with varying light levels.

[0077] The above description is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A visual SLAM method based on improved SuperPoint in a dynamic environment, characterized in that: The following steps are involved: Step 1: Build a labeled dataset; Take the unlabeled real image as input, use the end-to-end visual relocalization method PixLoc to generate the residual weight map of the input image as the residual weight label, and use the feature point self-labeling method to obtain the feature point pseudo label; Step 2: Build an image static feature point and descriptor extraction network and complete offline training on a labeled dataset; The image static feature point and descriptor extraction network includes: a shared encoder, a residual weight branch, a feature point branch and a descriptor branch. The input image is downsampled by the shared encoder to obtain a preliminary feature map. The residual weight branch, the feature point branch and the descriptor branch respectively decode the preliminary feature map to obtain a residual weight map, a feature point probability map and a descriptor feature map. The residual weight map and the feature point probability map are further weighted and fused to obtain a feature point confidence map. The confidence threshold is then set, and feature points are obtained by a fast approximate non-maximum suppression algorithm. The descriptor feature map is bicubic interpolated and then normalized to unit length by L2 to obtain a dense descriptor vector. Step 3: Use the trained image static feature point and descriptor extraction network in the feature point and descriptor extraction process of ORB-SLAM2 to implement the visual SLAM method based on the improved SuperPoint.

2. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1, characterized in that: The shared encoder obtains a preliminary feature map from the input image, which is specifically composed of Conv3×3×1×64 / s2, Conv3×3×64×64, Conv3×3×64×128 / s2, Conv3×3×128×128, Conv3×3×128×256 / s2, and Conv3×3×256×256 connected in sequence, and a nonlinear function ReLU layer is set after each convolution layer.

3. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1 is characterized in that: In the residual weight branch, the preliminary feature map is decoded by the residual weight decoder. During the decoding process, the residual weight map is upsampled by the PixelShuffle operation to restore it to the original size of the input image. Finally, the output is mapped to the [0,1] range by the Sigmoid function to obtain the residual weight map; the residual weight decoder is composed of Conv3×3×256×256, Conv3×3×64×64, Conv3×3×16×16, and Conv1×1×4×1 connected in sequence, and a nonlinear function ReLU layer is set after the convolution layers Conv3×3×256×256, Conv3×3×64×64, and Conv3×3×16×16.

4. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1 is characterized in that: In the feature point extraction branch, the preliminary feature map is decoded by the feature point decoder to obtain an output, and then the recycling channel is discarded through the Softmax operation. Finally, the dimension is switched using the Reshape operation to obtain a feature point probability map; the feature point decoder is composed of Conv3×3×256×256 and Conv1×1×256×65, and a nonlinear function ReLU layer is set after Conv3×3×256×256.

5. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1, characterized in that: In the descriptor detection branch, the preliminary feature map is decoded by the descriptor decoder to obtain a descriptor feature map with a dimension of (H / 8)×(W / 8)×256, and then bicubic linear interpolation and L2 normalization are performed to obtain a dense descriptor vector of unit length; the descriptor decoder is composed of Conv3×3×256×256 and Conv1×1×256×256, and a nonlinear function ReLU layer is set after Conv3×3×256×256.

6. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1, characterized in that: The offline training is specifically as follows: Perform a random homography transformation on the real image in the labeled dataset to obtain a replica image. The real image and the replica image are combined into an image pair and input into the image static feature point and descriptor extraction network. A loss function is set and the image static feature point and descriptor extraction network is trained offline. The loss function L adds residual weight loss to the SuperPoint loss function, that is, the loss function is composed of feature point loss L p , descriptor loss L d and the residual weight loss L w It consists of three parts, specifically: L(X p ,X′ p ,X w ,X′ w ,D,D′;Y p ,Y′ p ,Y w ,Y′ w ,S)=L p (X p ,Y p )+L p (X′ p ,Y′ p )+λ1L w (X w ,Y w )+λ1L w (X′ w ,Y′ w )+λ2L d (D,D′,S) Among them, X p With Y p The predicted output and label value of the feature points of the real image, X′ p and Y′ p The feature point prediction output and label value for the copy image, X w With Y w Then it is the residual weight prediction output and label value of the real image, X′ w and Y′ w is the residual weight prediction output and label value of the replica image, D and D′ are the descriptor feature maps obtained after the real image and the replica image are input into the network, S represents the positional relationship between D and D′, and λ1 and λ2 are hyperparameters; L p The loss function for feature point prediction uses full convolution cross entropy loss, specifically: Among them, H c =H / 8,W c =W / 8, H and W are the height and width of the image; (h, w) represents the position coordinates, x hwk Represents the predicted value of the kth channel (h, w) position in the feature point probability map (dimension is (H / 8)×(W / 8)×65), Represents the feature point probability map at the feature point label value y hw The predicted value of the corresponding channel (h,w) position; L w is the residual weight prediction loss function, using pixel-by-pixel L1 loss, specifically: Among them, x hw is the predicted value of the position (h,w) in the residual weight map, y hw is the label value of the (h,w) position in the residual weight label map; L d To describe the loss function of the feature points, hinge loss is used, specifically: Among them, d hw Describes the descriptor unit at the (h,w) position in the descriptor feature map D, d′ h′w′ Descriptor unit representing the position (h′, w′) in the descriptor feature map D′; s hwh′w′ To characterize d hw and d′ h′w′ Is the location close to: p hw Describes the subunit d hw At the coordinate position of the real image, Indicates that the subunit position p will be described hw The position of the descriptor unit after the corresponding homography transformation; p h′w′ Describes the subunit d h′w′ At the coordinate position of the copy image; l d For: l d (d, d′; s) = λ d ×s×max(0,m p -d T d′)+(1-s)×max(0,d T d′-m n ), m p With m n is the positive margin and negative margin, λ d is a hyperparameter.

7. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1, characterized in that: In step 1, a feature point self-labeling method is used to obtain feature point pseudo labels, specifically: an unlabeled real image is used as the input of the MagicPoint network, the input image is first subjected to N random homography transformation operations to obtain an image copy, the MagicPoint network is used to extract feature points on the transformed image copy, the image copy containing the feature points is restored through the corresponding inverse homography change, and then merged with the feature points of the real image as feature point pseudo labels.

8. The visual SLAM method based on improved SuperPoint in a dynamic environment according to claim 1, characterized in that: The step 3 is specifically as follows: in the Tracking thread of ORB-SLAM2, the trained image static feature point and descriptor extraction network is used instead of the ORB method to extract feature points and descriptors from the image captured by the camera.

Images