A visual SLAM method based on RGB-D camera point and line feature fusion

Abstract

The present invention discloses a visual SLAM method based on the fusion of RGB-D camera point and line features. The scenarios in which mainstream SLAM systems are applicable are strictly limited by their assumptions about static environments. In addition, in low-texture or weak-texture areas, ordinary feature point extraction methods cannot detect stable image features. At the same time, weak-texture areas generally have weak gradient field distribution characteristics, and feature descriptors in these areas have low discrimination, which easily leads to mismatching problems. Targeting the described problem scenarios, the present invention combines the attention mechanism and graph neural network to train a neural network for point and line feature extraction and matching, and proposes a visual SLAM method based on the fusion of RGB-D camera point and line features. The feature extraction and matching network of the present invention has strong reusability and also has application scenarios in the field of visual matching.

Description

Technical field:

[0001] The present invention relates to the field of simultaneous positioning and mapping (SLAM), and in particular to a visual SLAM method based on RGB-D camera point and line feature fusion. Background technology:

[0002] Simultaneous Localization and Mapping (SLAM) is a crucial method for robots to determine their own position and perceive their surroundings in unfamiliar environments. This technology primarily addresses the problem of how robots can use sensor data to calculate a map of their surroundings and simultaneously determine their position within that map.

[0003] In most cases, mainstream SLAM systems are able to operate robustly. However, their applicable scenarios are strictly limited by their assumptions about static environments. Methods based on the static world assumption are affected or even fail due to moving objects appearing in the field of view. In slightly dynamic environments, only a small portion of the FOV is covered by moving objects. Traditional robust estimation methods such as Random Sample Consensus (RANSAC) and robust weighting functions can eliminate the influence of most moving objects. Conversely, if most of the FOV is occupied by moving objects, the observations of moving objects are more numerous than those of static scenes, causing the robust estimation methods to fail. In addition, in low-texture or weak-texture areas, ordinary feature point extraction methods cannot detect stable image features. At the same time, weak gradient field distribution characteristics are common in weak-texture areas. Feature descriptors in these areas have low discrimination and are prone to mismatching.

[0004] With the rapid development of deep learning, the combination of deep learning and Simultaneous Localization and Mapping (SLAM) has shown great potential in the fields of robotics and computer vision. Deep learning uses neural network models to learn complex feature representations, improving the understanding of image and sensor data. Simultaneously, SLAM focuses on building and updating environmental maps in real time and locating the robot's position within that map. Combining deep learning with SLAM allows deep learning networks to analyze sensor data in real time, extract key features, and improve map construction accuracy. This fusion approach enables robots to more intelligently perceive and understand their environment, improving autonomous navigation and positioning.

[0005] The self-attention mechanism is a key technology in deep learning, initially applied to natural language processing but later finding further application in vision. Its unique feature is the network's ability to dynamically assign different attention weights to different positions in the input sequence. By learning the correlations between elements within the input sequence, the model can more flexibly capture long-range dependencies. Graph Neural Networks (GNNs) are a type of deep learning model specifically designed for processing graph-structured data. Graph-structured data consists of nodes and the edges connecting them. GNNs aim to learn and reason about graph data by learning the relationships between nodes.

[0006] In order to explore how to improve the robustness of vSLAM in dynamic environments, this method trains a neural network for point and line feature extraction and matching based on the attention mechanism and graph neural network, and proposes a visual SLAM method based on RGB-D camera point and line feature fusion. Summary of the invention:

[0007] The purpose of the present invention is to solve the problems of vSLAM failure due to highly dynamic environment in complex environment, slow network speed and low accuracy due to depth reasons of some vSLAM methods using neural networks, and provide a visual SLAM method based on RGB-D camera point and line feature fusion.

[0008] A visual SLAM method based on RGB-D camera point and line feature fusion includes the following steps:

[0009] (1) Construct and train the neural network DPLAGNNs, obtain two consecutive frames in the video stream as pictures A and B, input pictures A and B into the neural network DPLAGNNs, and obtain the matching matrix of pictures A and B; the matching matrix contains the position information and matching information of feature points and feature lines in pictures A and B;

[0010] The neural network DPLAGNNs includes an input layer and a module consisting of L AGNNs layers;

[0011] The input layer includes: a feature extraction layer Extractor and a merging point mechanism Merge Point;

[0012] The AGNNs layer includes: two multi-layer perceptron networks (MLPs), a self-attention layer (Node Self-Attention), a line passing layer (Line Passing), a cross-attention layer (Node Cross-Attention), a discriminator (Confidence) layer, and a softmax matching layer;

[0013] (2) Key frame initialization:

[0014] Set a threshold; obtain a matching number based on the matching information. When the matching number between picture A and picture B exceeds the set threshold, set picture A as the initial key frame;

[0015] First, use the feature points to estimate the initial pose: construct an estimate of the basic matrix and solve it to obtain the rotation matrix and translation vector as the estimate of the initial pose.

[0016] Then, the reprojection error of feature points and feature lines is used to construct a cost function, and the estimated value of the initial pose is further optimized to obtain the initial pose by minimizing this cost function.

[0017] (3) Based on the initial pose, local mapping and loop detection are performed.

[0018] Preferably, in step (1), the process of obtaining the matching matrix of picture A and picture B includes the following steps:

[0019] Input images A and B into the input layer to obtain a set of key points V and a set of line segments E. The elements in set V are recorded as key points v; the elements in set E are line segments e, and each e is defined as the connection between two key points v.

[0020] The key point set V and the line segment set E are input into the AGNNs layer for processing and converted into a graph. After passing through two multi-layer perceptron networks MLP, each key point and line segment obtains an initial feature descriptor The initial feature descriptor The feature descriptor x is obtained by sequentially processing the self-attention layer Node Self-Attention, the line passing layer Line Passing, and the cross-attention layer NodeCross-Attention. i .

[0021] The feature descriptor x i Input to the discriminator Confidence layer, dynamically adjust the number of neural network layers to reduce the computational burden; actively filter and exclude those points that are judged as irrelevant by the model with high confidence, thereby further accelerating the reasoning process. Calculate its prediction distribution confidence c iThis confidence is calculated to determine whether to exit the network early. If yes, it exits the network early and inputs it into the Softmax matching layer. If not, it enters the Delte mechanism to determine the points that need to be deleted.

[0022] The feature descriptor x i After inputting into the Softmax matching layer, we get the point matching matrix P and line matching matrix P l , as matching information.

[0023] Preferably, the training process of the neural network DPLAGNNs includes the following steps: in the first stage, the Softmax matching network of L different AGNNs layers is trained to predict their corresponding relationships, that is, to guide the point matching matrix P and the line matching matrix P by using the precise labels obtained by dual view conversion. l In the second stage, the multi-layer perceptron MLP in the confidence calculation formula in the Confidence layer is trained.

[0024] Preferably, in step (1), obtaining the key point set V and the line segment set E comprises the following specific steps:

[0025] First, the SuperPoint algorithm is used to extract SP key points and key point descriptors in Figures A and B. The universal detector LSD is then used to identify line segments in Figures A and B. A distance threshold d is set, and line segments whose endpoints are less than the distance threshold d are merged by merging endpoints. The specific formula is as follows:

[0026]

[0027] The visual descriptors of the line segment endpoints are then generated by interpolating the SP features.

[0028] Finally, the key points and line endpoints in Figures A and B are converted into a graph. The converted graph contains the key point set V. Each element in the key point set V contains three attributes: coordinates (x p ,y p ,u p ), confidence score s p and visual descriptor d vis ∈R D The transformed graph also contains a set of line segments E, each of which contains a pair of key point coordinates (x p ,y p ,u p ) and (x′ p ,y′ p ,u′ p ), and has a line segment fraction s l .

[0029] Preferably, the two multi-layer perceptron networks MLP in the AGNNs layer in step (1) are point encoders PE p and Line Encoder PE e The formulas of the two multi-layer perceptron networks MLP are as follows:

[0030]

[0031] Point Encoder PE p and Line Encoder PE e Generate a spatial description d for each keypoint p , and generate a line descriptor d for each line connected to the key point e ;

[0032] The initial feature descriptor This node descriptor is then progressively enriched and optimized with its context in L iterations of self-attention, line-attention, and cross-attention layers.

[0033] Preferably, in step (1), the calculation of the predicted distribution confidence c i The steps include:

[0034] First, use the discriminator Confidence layer to infer the confidence of the predicted assignment of each point according to the following formula:

[0035] c i =Sigmoid(MLP(x i ))∈[0,1]

[0036] A higher value means that the feature representation of node i has a high degree of confidence and certainty—it can be confidently matched or confirmed as unmatched.

[0037] Setting thresholds At a specific network layer In the example, if the confidence level c of a point i is i More than layer Threshold Then the point is considered to be sufficiently reliable. When all the points judged to be reliable in image sets A and B exceed the ratio α, the entire network will terminate at this layer:

[0038]

[0039] Where α is the set ratio value, N and M are the number of nodes in image A and image B respectively.

[0040] Secondly, when the termination condition is not met, those points that are judged to be confident but failed to match are often located in areas of the image that are obviously invisible or have unclear features. These points are eliminated and only the points that meet the termination condition are passed to the next layer.

[0041] Preferably, the step (3) includes the following steps: constructing and maintaining a co-viewing map, wherein the co-viewing map includes co-viewing key frames, and the method of constructing the co-viewing map includes the following steps:

[0042] Calculate the geometric mean between the key points of the current frame and other key frames If the geometric mean exceeds a preset threshold, a co-viewing relationship is determined to exist. The current frame with the co-viewing relationship is added to the co-viewing graph. The co-viewing graph is a graph structure in which nodes represent key frames and edges represent the co-viewing relationships between them.

[0043] Local BA optimization: Use the reprojection error of feature points and feature lines to construct the cost function; And all the points that can be observed by these keyframes and line All other keyframes Not in keyframe But it can be observed and The points and lines in the co-visual map are also added to the cost function without changing their pose. Then, a local map is obtained based on the key frames in the co-visual map, and this local map is used for real-time positioning and mapping.

[0044] In addition, a database is established for the feature points and feature lines in the key frame; the distribution matrix S is calculated based on the feature points and feature lines. p And calculate its geometric mean To determine whether a closed loop has formed between the two images, if a closed loop occurs, a global BA optimization is performed to adjust the camera pose and the positions of the landmarks. The optimization strategy of the global BA is as follows: when a closed loop occurs, a separate thread is used to optimize the reprojection error between all keyframes, allowing the system to create a map and detect loops. If a new loop is detected during the optimization run, the optimization is aborted and the loop closing process is continued, which will restart the global BA optimization. When the global BA is completed, the updated subset of keyframes and keypoints optimized by BA is merged with the non-updated keyframes and points inserted during the optimization run.

[0045] Beneficial effects of the present invention:

[0046] This invention combines a deep learning model with existing SLAM to enhance the rotational invariance of feature points and the fusion of line features, thereby strengthening scene structure features and performing well even in low-texture scenes. The simultaneously trained feature extraction network avoids extracting features of dynamic objects in highly dynamic scenes, thereby achieving strong robustness. Furthermore, the network speed is increased due to the incorporation of a pruning mechanism. The feature extraction and matching network of this invention is highly reusable and has application scenarios in the field of visual matching. Description of the drawings:

[0047] Figure 1 This is the DPLAGNNs neural network structure diagram of the present invention;

[0048] Figure 2 This is a structural diagram of the SLAM method of the present invention;

[0049] Figure 3 This is an experimental comparison diagram of the SLAM method of the present invention; Specific implementation method:

[0050] In order to allow relevant personnel to have a clearer understanding of the technical content of the present invention, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. The present invention discloses a visual SLAM method based on the fusion of RGB-D camera point and line features. Feature points and lines are extracted through a neural network and fused and matched, making the feature matching between frames more robust. At the same time, the rotational invariance of feature points and fusion line features are enhanced, and the scene structure features are enhanced. Even in low-texture scenes, the performance is still good, and the feature extraction network avoids extracting the features of dynamic objects in large dynamic scenes. The feature extraction and matching network of the present invention is highly reusable and also has application scenarios in the field of image matching.

[0051] The SLAM method of the present invention comprises the following steps:

[0052] (1) Construct and train the neural network DPLAGNNs, obtain two consecutive frames in the video stream as pictures A and B, input pictures A and B into the neural network DPLAGNNs, and obtain the matching matrix of pictures A and B; the matching matrix contains the position information and matching information of feature points and feature lines in pictures A and B;

[0053] The neural network DPLAGNNs includes an input layer and a module consisting of L AGNNs layers;

[0054] The input layer includes: a feature extraction layer Extractor and a merging point mechanism Merge Point;

[0055] The AGNNs layer includes: two multi-layer perceptron networks (MLPs), a self-attention layer (Node Self-Attention), a line passing layer (Line Passing), a cross-attention layer (Node Cross-Attention), a discriminator (Confidence) layer, and a softmax matching layer;

[0056] The network uses the attention mechanism to integrate visual and spatial information through the input layer and AGNNs layer. First, the Node Self-Attention layer connects all nodes in the same image, allowing the network to capture the interdependence between nodes in a local range, thereby strengthening the integration of spatial and visual information within the image. Secondly, the Node Cross-Attention layer connects nodes from different images. This connection mechanism is crucial for achieving cross-image feature matching and spatial reasoning. It allows the network to align and compare features from different perspectives on a global scale. Finally, the Line Passing layer connects the nodes at both ends of the same line segment. This connection method enables the network to learn the geometric and visual relationship between the two end points of the line segment, thereby constructing a powerful line segment structure representation in the image.

[0057] The network utilizes the Delte mechanism in the Confidence layer to improve computational efficiency and reduce unnecessary processing. First, the number of neural network layers is dynamically adjusted based on the complexity of the input image pair, reducing the computational burden. Second, points that the model has determined with high confidence to be irrelevant are actively filtered and excluded, further accelerating the inference process. These measures collectively improve the overall system performance while significantly reducing processing time.

[0058] 1) Input layer: Use the SuperPoint algorithm to extract key points and key point descriptors, use the universal detector LSD to detect line segments, and merge the SP key points whose distance from the line endpoint is less than the set threshold d with the line endpoint. The mathematical expression is as follows:

[0059]

[0060] Where M is the number of SP keypoints whose distance is less than a threshold d. This process lifts the unstructured line cloud into interconnected wireframes. The visual descriptors of the line segment endpoints are then generated by interpolating the SP features.

[0061] In addition, the network receives two images A and B as input. After processing in the input layer, each image is converted into a set of nodes p, which contains coordinates (x p ,y p ,u p ), confidence score s pand visual descriptor d vis ∈R D ; and a set of line segments E, defined as a pair of nodes (x p ,y p ,u p ) and (x′ p ,y′ p ,u′ p ), and has a line segment fraction s l The segment score is the length of the segment returned by the segment detector, so that longer segments are given more emphasis. p From the keypoint detector, equal to s when it is a line segment endpoint l . The coordinates (x p ,y p ,u p ),x p and y p is the coordinate of the node on the image, u p is the virtual coordinate obtained according to the depth d of the RGB-D camera,

[0062]

[0063] where f x is the horizontal focal length, and b is the baseline between the structured light projector and the infrared camera.

[0064] 2) AGNNs layer: In order to deeply encode the features of each node, a method is adopted to learn spatial features and visual descriptors simultaneously. A multi-layer perceptron (MLP) is used to learn the position encoder (PE p and PE e ), the two encoders generate a spatial descriptor d for each node p , and generates an edge descriptor d for each line segment starting from this node e For example, a node with a connectivity of 3 will be assigned a d p and 3D e (One for each outgoing segment.) The position encoding of an edge also contains the offset to its other endpoint as additional information, making it possible to access the angle and length of the segment:

[0065]

[0066] Space descriptor d p Used to initialize node features, and edge descriptors d e Used for line messaging.

[0067] In AGNN, three types of undirected edges are used. Self-attention edge ε in the self-attention layer selfConnect all nodes of the same image. Line passing layer Line passing middle edge ε line Connect the nodes that are the endpoints of the same line segment. Cross-Attention Edge ε in Cross-Attention Layer cross Connect the nodes of one image with the nodes of another image. Node i is assigned an initial feature descriptor that fuses its spatial and visual information: This node descriptor is then progressively enriched and optimized with its context in L iterations of self-attention, line-attention, and cross-attention layers. Finally, the features of each node are linearly projected to obtain the output features.

[0068] In the self-attention edge, the feature state is updated through a multi-layer perceptron (MLP). For the mth feature, its updated state is defined as follows:

[0069]

[0070] The symbol [·|·] represents the vector concatenation operation, and m i It is obtained by taking a weighted average of all nodes j in the image, with the weights determined by the attention scores:

[0071]

[0072] Where W is the matrix used for feature projection, a ij Represents the attention score between node i and node j. For each image I, perform the following steps: First, the current state x of each image node i i is decomposed into key vectors k through different linear projections i and query vector q i Next, define the attention score between node i and node j as:

[0073]

[0074] function It is a rotation code used to encode the relative position relationship between points.

[0075]

[0076] In the cross-attention layer ε cross In

[15] , a slightly different definition from the self-attention layer is used, as shown below:

[0077]

[0078] In this formula, a m (x i; ε) represents the multi-head attention mechanism between node i and point set ε. In image I, each point will pay attention to all points in another image S:

[0079]

[0080] Similarly, the key vector k j , query vector q i Sum value vector v j They are node features x i and x j The linear projection of . D represents the dimension of the vector, and the point set ε represents the node set in another image S.

[0081] Through the line edge ε line Line Message Transfer (LMP), the i-th node uses local edge connectivity to communicate with the set of neighboring nodes The mth LMP update is implemented by aggregating the two endpoint features. (m) x i and (m) x j and the corresponding endpoint position codes Information contained in:

[0082]

[0083] in, is the number of neighbors of node i.

[0084] 3) Softmax Matching Layer: After the three different types of undirected edges, a matching layer is introduced. Its task is to predict the matching relationship between points and lines based on the updated state of each layer. Specifically, the distribution matrix between image pairs is first obtained based on the distribution score. Its calculation formula is defined as follows:

[0085]

[0086] Where Linear(·) represents a learned linear transformation including a bias term. For each point, we also calculate a matchability score, which is expressed as:

[0087] σ i =Sigmoid(Linear(x i ))∈[0,1]

[0088] Compatibility score σ i It reflects the probability that node i has a corresponding point in another image.

[0089] The corresponding relationship is constructed, and the similarity score and the matchability score are integrated to form a flexible partial allocation matrix P. The specific calculation method is as follows:

[0090]

[0091] In order to select the corresponding point pairs, select those whose matrix elements P ij The nearest neighbor point pairs that are greater than the set threshold τ are regarded as valid matching correspondences.

[0092] On the other hand, line segments are matched in a similar way, except that each line segment is represented by the features of its two endpoints, which are and To make the matching independent of the order of endpoints, take the maximum of two configurations in the assignment matrix:

[0093]

[0094] 4) Confidence Layer and Delete Mechanism: To improve computational efficiency and reduce unnecessary processing, the network adopts two optimization strategies: First, the number of neural network layers is dynamically adjusted based on the complexity of the input image pairs to reduce the computational burden; second, points that the model determines as irrelevant with high confidence are actively screened and excluded, further accelerating the inference process.

[0095] The Confidence layer is introduced, which is responsible for evaluating the prediction confidence of each key point. The Confidence layer infers the confidence of the prediction assignment of each point as follows:

[0096] c i =Sigmoid(MLP(x i ))∈[0,1]

[0097] A higher value means that the feature representation of node i has a high degree of confidence and certainty - it can be confidently matched or confirmed as unmatched. In the example, if the confidence level c of a point i is i More than layer Threshold Then the point is considered to be sufficiently reliable. When all the points judged to be reliable in image sets A and B exceed a certain proportion, the entire network will terminate at this layer:

[0098]

[0099] Where α is the set ratio, and N and M are the number of nodes in images A and B, respectively. When the exit criteria are not met, points that are considered confident but failed matches are often located in areas of the image that are clearly invisible or lack distinct features. Based on this consideration, the network filters and eliminates these points at each layer, passing only those that meet the criteria to the next layer.

[0100] (2) Key frame initialization:

[0101] Set a threshold; obtain a matching number based on the matching information. When the matching number between picture A and picture B exceeds the set threshold, set picture A as the initial key frame;

[0102] First, use the feature points to estimate the initial pose: construct an estimate of the basic matrix and solve it to obtain the rotation matrix and translation vector as the estimate of the initial pose.

[0103] Then, the reprojection error of feature points and feature lines is used to construct a cost function, and the estimated value of the initial pose is further optimized to obtain the initial pose by minimizing this cost function.

[0104] Using point features for initial pose estimation involves calculating the fundamental matrix. The fundamental matrix contains information about the relative motion between the two views. By matching the point features in the two images, an estimate of the fundamental matrix is ​​constructed. The solution formula can be expressed as follows:

[0105]

[0106] in, is the estimate of the fundamental matrix, (x i1 ,y i1 ) and (x i2 ,y i2 ) are the homogeneous coordinates of the matching points in the two views, and then the basic matrix is ​​decomposed into the rotation matrix R and the translation vector t by singular value decomposition. The cost function is then constructed using the reprojection error of the feature points and feature lines. The specific construction method is as follows: minimize the 3D point in the world coordinate system With key point p i The reprojection error between The projection line l (using Planck coordinates) j and the line feature endpoint (x p ,y p ) and (x′ p ,y′ p ), is the set of all point matches, is the set of all line matches:

[0107]

[0108] where ρ is the elastic Huber cost function, Σ p is the covariance matrix associated with the keypoint scale, Σ l is the covariance matrix of the key line correlation. The projection function π (·) is defined as follows:

[0109]

[0110] Where (f x ,f y ) is the focal length, (c x ,c y ) is the principal point and b is the baseline, both of which are known from the calibration.

[0111] (3) Based on the initial pose, local mapping and loop closure detection are performed. Step (3) is implemented as follows: a graph of common view key frames is maintained, where the common view graph is generated by checking the geometric mean between the key points of the current frame and other key frames. If the co-view relationship exceeds a certain threshold, it can be determined that there is a co-view relationship. If the current frame has a sufficient co-view relationship with a key frame, the current frame is added to the co-view. The co-view is a graph structure, where nodes represent key frames and edges represent the co-view relationship between them. Local BA optimization: The set of co-views contains a set of key frames. And all the points observed in these keyframes and line All other keyframes Not present But it can be observed and The points and lines in are also added to the cost function without changing their pose. The cost function is the same as step (2). After that, a local map is formed in the key frames in the co-visual map, and this local map is used for real-time positioning and mapping.

[0112] In addition, a database is established for the feature points and lines in the key frame; the distribution matrix S is calculated based on the feature points and feature lines. p And calculate its geometric mean To determine whether a closed loop is formed between the two images, if a closed loop occurs, a global BA optimization is performed to adjust the camera pose and the position of the landmarks. The optimization strategy for global BA is as follows: when a closed loop occurs, a separate thread is used to optimize the reprojection error between all keyframes, allowing the system to create a map and detect loops. If a new loop is detected while the optimization is running, the optimization will be aborted and the loop will continue to be closed, which will restart the global BA optimization. When global BA is completed, the updated subset of keyframes and points optimized by BA needs to be merged with the non-updated keyframes and points inserted during the optimization run.

[0113] The DPLAGNNs neural network structure in the present invention is as follows Figure 1 As shown in the figure, the network uses the attention mechanism to integrate visual and spatial information through the input layer and AGNNs layer. Through the Confidence layer and Delte mechanism, the number of network layers is dynamically adjusted to eliminate the confident points that fail to match, reduce redundant calculations, and improve the computational efficiency of the system. The matching relationship between points and lines is predicted based on the updated state of each layer through Softmax matching. The network training process includes two key stages: In the first stage, the Softmax matching network of L different AGNNs layers is trained to predict their corresponding relationships, that is, to guide the point matching matrix P and the line matching matrix P by using the precise labels obtained by dual-view conversion. l In the second stage, the multi-layer perceptron MLP in the confidence calculation formula in the Confidence layer is trained.

[0114] The model was pre-trained on warped image pairs using homographies from the LSIR dataset, gradually increasing the difficulty of the homographies to accelerate convergence. The model was then fine-tuned on MegaDepth, a dataset of 195 outdoor landmark scenes, resizing each image to 640×640 px. The wireframe threshold d for node merging was set to 3 pixels, and the AGNNs contained L=9 layers, with a feature size of d=256 in the network. The network was optimized using Adam, with a homography pre-training learning rate of 10. -4 , MegaDepth learning rate is 10 -5 To limit the computational cost during training, a maximum of 1000 keypoints and 250 line segments are set for each image during training.

[0115] A quality assessment of the proposed neural network's ability to perform planar scene estimation on the HPatches dataset was performed. The precision and recall of the true match were measured based on the predictions having an error of no more than 3 pixels during reprojection. In addition, two algorithms were used to evaluate the accuracy of the homography matrix inferred from the matching points: the highly robust RANSAC algorithm and the weighted direct linear transform (DLT) algorithm. For each pair of images, the average reprojection error of the four corner points was calculated, and the area under the cumulative error curve was calculated at 1 pixel and 5 pixel thresholds.

[0116] The results are shown in the following table:

[0117]

[0118] Results show that DPLAGNNs produces correspondences with similar recall to LightGlue and GlueStick. This results in more accurate estimates than other matchers when estimating homographies using DLT. Even at a coarse threshold of 5px, DPLAGNNs is more accurate than LoFTR, despite being limited by sparse keypoints.

[0119] The effectiveness of the two pruning strategies, the Confidence layer and the Delte mechanism, are verified on MegaDepth image pairs. The results are shown in the following table.

[0120]

[0121] When processing simple samples, such as continuous video frames, the network is able to converge quickly and complete the calculation after a few layers, achieving a speed increase of approximately 1.91 times. When faced with scenes with low visual overlap, such as closed-loop scenes, although the network requires more layers to converge, it can early eliminate points that have high confidence but cannot be matched, reducing unnecessary computational burden. The introduced line message passing mechanism makes the average number of stopping layers of the network significantly less than LightGlue. Overall, the adaptive depth and width pruning reduces the running time by an average of 36%, and is particularly efficient when processing simple scenes.

[0122] The SLAM method structure proposed in the present invention is as follows Figure 2 As shown in the figure, it is divided into several parts: data processing, tracking thread, local mapping thread, loop detection thread, global pose graph optimization and map library.

[0123] The data processing phase first extracts image data from the RGB-D camera, including RGB images and depth information. This data is then fed into the DPLAGNNs network for processing. The network is responsible for calculating the positions p = (x, y, u) of point and line features in the image and generating the corresponding matching matrix P. The tracking thread is responsible for processing real-time incoming image and IMU data to accurately calculate the camera's position and orientation in the map. It includes responsibilities such as map initialization, keyframe tracking, and new keyframe decision making. The local mapping module is responsible for adding new keyframes and map points to the currently active map and improving map accuracy through local optimization. The loop detection and map fusion module monitors loops between new keyframes and historical keyframes, identifies cases where they return to previous positions, and performs loop correction to eliminate accumulated errors. The map management module maintains the atlas, manages active and inactive maps, and ensures that the system can effectively utilize and update map data.

[0124] To verify the effectiveness of this SLAM, the TUM dataset was used to simulate and test the proposed algorithm in dynamic scenes. Based on the previous evaluation results of the DPLAGNNs network, the experiment chose a configuration with 7 layers of AGNNs. The reason for this is that the SLAM system mainly processes close-range matching between key frames. Only when there is a large amount of movement or a large number of obstructions, more layers of DPLAGNNs are needed. The comparison results are shown in Figure 2. Figure 3 The color band on the right side of the figure indicates the error size, with blue indicating a smaller error and red indicating a larger error. It can be seen that the algorithm proposed in this study consistently maintains a low level of positioning error. Compared with the other two algorithms, ORB-SLAM3 and DynaSLAM, its error range is smaller, and the pose estimation results are closer to the actual camera pose. This shows that the algorithm has significant advantages in positioning accuracy and system performance, and can maintain global consistency.

[0125] It should be understood that any portion not elaborated in detail in this specification belongs to the prior art. Those skilled in the art should understand that the above embodiments are intended solely to help readers understand the principles and implementation methods of the present invention, and the scope of protection of the present invention is not limited to such embodiments. All equivalent substitutions made based on the present invention are within the scope of protection of the present invention.

Claims

1. A visual SLAM method based on RGB-D camera point and line feature fusion, characterized by , including the following steps: (1) Construct and train the neural network DPLAGNNs, obtain two consecutive frames in the video stream as pictures A and B, input pictures A and B into the neural network DPLAGNNs, and obtain the matching matrix of pictures A and B; the matching matrix contains the position information and matching information of feature points and feature lines in pictures A and B; The neural network DPLAGNNs includes an input layer and a module consisting of L AGNNs layers; The input layer includes: a feature extraction layer Extractor and a merging point mechanism Merge Point; The AGNNs layer includes: two multi-layer perceptron networks (MLPs), a self-attention layer (Node Self-Attention), a line passing layer (Line Passing), a cross-attention layer (Node Cross-Attention), a discriminator (Confidence) layer, and a softmax matching layer; (2) Key frame initialization: Set a threshold; obtain a matching number based on the matching information. When the matching number between picture A and picture B exceeds the set threshold, set picture A as the initial key frame; First, use the feature points to estimate the initial pose: construct an estimate of the basic matrix and solve it to obtain the rotation matrix and translation vector as the estimated value of the initial pose; Then, the cost function is constructed using the reprojection error of the feature points and feature lines. The estimated value of the initial pose is further optimized to obtain the initial pose by minimizing this cost function. (3) Based on the initial pose, local mapping and loop detection are performed; Step (3) includes the following steps: constructing and maintaining a co-viewing map, wherein the co-viewing map includes co-viewing key frames, and the method of constructing the co-viewing map includes the following steps: Calculate the geometric mean between the key points of the current frame and other key frames If the geometric mean exceeds a preset threshold, it is determined that a co-viewing relationship exists; the current frame with the co-viewing relationship is added to the co-viewing graph; the co-viewing graph is a graph structure, in which nodes represent key frames and edges represent the co-viewing relationship between them; Local BA optimization: Use the reprojection error of feature points and feature lines to construct the cost function; And all the points that can be observed by these keyframes and line All other keyframes Not in keyframe But it can be observed and The points and lines in the co-visual map are also added to the cost function without changing their pose. Then, a local map is obtained based on the key frames in the co-visual map, and this local map is used for real-time positioning and mapping. Establish a database for the feature points and feature lines in the key frame; calculate the distribution matrix S based on the feature points and feature lines p And calculate its geometric mean to determine whether a closed loop is formed between the two images. If a closed loop occurs, global BA optimization is performed to adjust the camera pose and the position of the landmark points; the optimization strategy of the global BA is as follows: when a closed loop occurs, a separate thread is used to optimize the reprojection error between all key frames, allowing the system to create a map and detect loops; if a new loop is detected during the optimization run, the optimization will be aborted and the loop will continue to be closed, which will restart the global BA optimization; when the global BA is completed, the updated subset of key frames and key points optimized by BA will be merged with the non-updated key frames and points inserted during the optimization run.

2. A visual SLAM method based on RGB-D camera point and line feature fusion according to claim 1, characterized in that, In step (1), the process of obtaining the matching matrix of picture A and picture B includes the following steps: Input images A and B into the input layer to obtain a set of key points V and a set of line segments E. The elements in set V are recorded as key points v; the elements in set E are line segments e, and each e is defined as the connection between two key points v. The key point set V and the line segment set E are input into the AGNNs layer for processing and converted into a graph. After passing through two multi-layer perceptron networks MLP, each key point and line segment obtains an initial feature descriptor The initial feature descriptor The feature descriptor x is obtained by sequentially processing the self-attention layer Node Self-Attention, the line passing layer Line Passing, and the cross-attention layer Node Cross-Attention. i ; The feature descriptor x i Input into the discriminator Confidence layer to calculate its prediction distribution confidence c i , this confidence is calculated to determine whether to exit the network early. If yes, it exits the network early and inputs it into the Softmax matching layer. If not, it enters the Delte mechanism to determine the points that need to be deleted; The feature descriptor x i After inputting into the Softmax matching layer, we get the point matching matrix P and line matching matrix P l , as matching information.

3. A visual SLAM method based on RGB-D camera point and line feature fusion according to claim 2, characterized in that, The training process of the neural network DPLAGNNs includes the following steps: In the first stage, the softmax matching network of L different AGNNs layers is trained to predict their corresponding relationships, that is, to guide the point matching matrix P and the line matching matrix P by using the precise labels obtained by dual-view conversion. l In the second stage, the multi-layer perceptron MLP in the confidence calculation formula in the Confidence layer is trained.

4. A visual SLAM method based on RGB-D camera point and line feature fusion according to claim 2, characterized in that, In step (1), obtaining the key point set V and the line segment set E includes the following specific steps: First, the SuperPoint algorithm is used to extract SP key points and key point descriptors in Figures A and B. The universal detector LSD is then used to identify line segments in Figures A and B. A distance threshold d is set, and line segments whose endpoints are less than the distance threshold d are merged by merging endpoints. The specific formula is as follows: Then, the visual descriptors of the line segment endpoints are generated by interpolating the SP features; Finally, the key points and line endpoints in Figures A and B are converted into a graph. The converted graph contains the key point set V. Each element in the key point set V contains three attributes: coordinates (x p ,y p ,u p ), confidence score s p and visual descriptor d vis ∈R D The transformed graph also contains a set of line segments E, each of which contains a pair of key point coordinates (x p ,y p ,u p ) and (x′ p ,y′ p ,u′ p ), and has a line segment fraction s l .

5. A visual SLAM method based on RGB-D camera point-line feature fusion according to claim 2, characterized in that The two multi-layer perceptron networks MLP in the AGNNs layer in step (1) are point encoders PE and p and Line Encoder PE e ; The formulas of the two multi-layer perceptron networks MLP are as follows: Point Encoder PE p and Line Encoder PE e Generate a spatial description d for each keypoint p , and generate a line descriptor d for each line connected to the key point e ; The initial feature descriptor 6. A visual SLAM method based on RGB-D camera point and line feature fusion according to claim 2, characterized in that In the step (1), the calculation of the prediction distribution confidence c i The steps include: First, use the discriminator Confidence layer to infer the confidence of the predicted assignment of each point according to the following formula: c i =Sigmoid(MLP(x i ))∈[0,1] Setting thresholds At a specific network layer In the example, if the confidence level c of a point i is i More than layer Threshold Then the point is considered to be sufficiently reliable; when all the points judged to be reliable in image sets A and B exceed the ratio α, the entire network will terminate at this layer: Where α is the set ratio value, N and M are the number of nodes in image A and image B respectively; Secondly, when the termination condition is not met, the points that are judged to be confident but failed to match are eliminated, and only the points that meet the termination condition are passed to the next layer.

Images