A visual odometry method and system fusing semantic and point-line features

Abstract

The application relates to a visual odometer method and system fusing semantics and point-line features, and the method comprises the following steps: inputting two continuous frames, performing pretreatment, processing the pretreated image tensor through a GlueStick mixed deep learning network and a semantic segmentation network YOLOv8, respectively obtaining point-line features and their descriptors and matching relationships of the image and semantic segmentation prediction results, inputting the point-line features and their descriptors and the matching relationships and the semantic segmentation prediction results into a feature processing module to perform feature filtering and fusion, obtaining a matching relationship matrix of the two images, calculating a rotation matrix and a translation vector of the two images through the matching relationship matrix, and realizing pose evaluation of the visual odometer through the rotation matrix and the translation vector. Compared with the prior art, the application has the advantages of solving the problems of low monocular VO estimation accuracy, weak dynamic robustness and low scene understanding capability in a complex dynamic urban road scene.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and in particular relates to a visual odometry method and system that integrates semantics and point and line features. Background Technology

[0002] Visual odometry plays a crucial role in the perception and control of intelligent agents. The design of visual odometry systems is based on the assumption of "stationary environment, moving camera." However, due to the complex lighting conditions of urban roads and the uncertainties of many dynamic objects, the pose estimation accuracy of conventional visual odometry still needs improvement.

[0003] The accuracy and reliability of feature matching directly affect pose estimation performance. In outdoor environments with uneven feature distribution and drastic changes in lighting conditions, point features struggle to achieve good results. Line features, due to their stronger geometric constraints, can effectively mitigate the influence of external lighting. By combining the advantages of line and point features, the performance of visual odometry under low-light conditions can be improved. However, current methods do not effectively combine line features with semantic information in the scene, resulting in a lack of comprehensive scene understanding in the VO system.

[0004] To improve the robustness of visual odometry in complex environments, some existing methods attempt to remove dynamic objects from the camera's field of view. These methods ignore static feature categories with stable properties within the field of view, while introducing additional risks such as inaccurate dynamic judgments and feature loss. Some methods integrate semantic information into visual odometry systems to guide feature point selection and tracking. However, most semantic visual odometry focuses on specific feature categories in specific scenes, requiring customized training for dynamic categories in urban road scenes. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a visual odometry method and system that integrates semantics and point and line features.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] This invention provides a visual odometry method that integrates semantics and point / line features, comprising the following steps: acquiring two consecutive frames of images; preprocessing the images to obtain image tensors of the two frames; inputting the image tensors of the two frames into a feature extraction module for processing to obtain feature prediction results of feature detection matching of the two frames; inputting the image tensors of the two frames into a semantic segmentation module for processing to obtain segmentation prediction results of the two frames; inputting the feature prediction results and segmentation prediction results into a feature processing module for processing to obtain a matching relation matrix; inputting the matching relation matrix into a pose estimation module for processing to obtain a rotation matrix and a translation vector; and evaluating and visualizing the results using the rotation matrix and translation vector.

[0008] Furthermore, the two consecutive frames are captured by a monocular vision sensor and are two RGB images.

[0009] Furthermore, the image preprocessing includes: reading the image file using the OpenCV image processing library, adjusting the image to the input size required by the network, normalizing and scaling the pixel values ​​from the 0-255 range to the 0-1 range, and converting the image to tensor format using the PyTorch deep learning framework.

[0010] Furthermore, the feature extraction module is built based on the GlueStick hybrid deep learning network, including a front-end and a graph neural network.

[0011] Furthermore, the front end is based on the SuperPoint network and the LSD line detector, and the graph neural network is an attention graph neural network.

[0012] Furthermore, the image tensors of the two frames are input into the feature extraction module for processing. The processing includes: after inputting the image tensors of the two frames, the front end extracts point features and their descriptors through the SuperPoint network, detects line segment endpoints through LSD, and fuses the key points and line segment endpoints to generate nodes of the structure frame. The line segments also constitute edges connecting several nodes in the structure frame. The nodes and edges are combined to form a single wireframe. The two wireframes are input into the attention graph neural network. The multilayer perceptron of the attention graph neural network generates a spatial descriptor for each node. Each edge of a point generates an edge descriptor, which is used to encode the relationship between nodes and edges. Self-attention and cross-attention mechanisms are used to process the encoded nodes and edges. The node spatial descriptor and the edge descriptor of each edge are iteratively updated through an L-layer attention graph neural network. The similarity of the updated node and edge descriptors is examined, and a point and line matching assignment matrix between the two graphs is jointly generated. All rows and columns of the point and line assignment matrix are processed by double softmax to generate the point and line matching relationship between the two graphs, and the feature prediction results of the feature detection matching of the two frames of images are obtained.

[0013] Furthermore, the feature prediction results of the feature detection and matching between the two frames of images include point and line features and their descriptions, the original matching relationship of feature points and the original feature line matching relationship.

[0014] Furthermore, the semantic segmentation network is constructed based on the YOLOv8x semantic segmentation network. The segmentation prediction result includes a polygon composed of ordered coordinate vertices, a classification index, and a confidence value. The segmentation prediction result is denoted as Pred. s : in It is an array consisting of a set of segmented polygon vertices and their corresponding semantic categories with the same index, i.e. in C is a predefined integer representing various categories during the training phase. C is divided into three categories, including man-made static objects, natural static objects, and movable objects.

[0015] Furthermore, the feature prediction results and segmentation prediction results are input into the feature processing module for processing. The processing includes: for the polygon of the movable object, connecting its vertices to generate a binary mask; deleting the matching point features located in the movable object region; performing stability filtering on the edges of artificial static objects and matching them with the lines nearby; discarding cluttered, low-texture, and statically unstable features in natural static objects; filtering and merging line features through length and semantics; and connecting the coordinates of the filtered points and the merged line matches to form a corresponding matching relationship matrix between the two frames of images.

[0016] This invention also provides a visual odometry system that integrates semantic and point / line features, including a preprocessing module, a feature extraction module, a semantic segmentation module, a feature processing module, and a pose estimation module. The preprocessing module converts RGB images into image tensors. The feature extraction module is responsible for processing the input image tensors to obtain the point / line features of the image and their matching relationships. The semantic segmentation module is built based on the YOLOv8x semantic segmentation network and is responsible for processing the input image tensors to obtain segmentation prediction results. The feature processing module is responsible for point / line filtering and edge fusion to obtain the corresponding matching relationship matrix of two frames of images. The pose estimation module obtains the rotation matrix and translation vector by processing the corresponding matching relationship matrix of two frames of images.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] (1) This invention provides an indirect point-line feature visual odometry method with semantic information, which solves the problems of low accuracy, weak dynamic robustness and low scene understanding ability of monocular VO estimation in complex dynamic urban road scenes.

[0019] (2) This invention improves scene understanding and camera motion estimation accuracy by combining two neural networks and utilizing powerful geometric constraints and semantically perceptible data.

[0020] (3) This invention utilizes the GlueStick network to extract and match point and line features in images to overcome the generalization limitations of traditional handwritten point features. Focusing on stationary objects in the scene, it uses the YOLOv8x semantic segmentation network to provide semantic information, and integrates prior semantic information to identify and enhance feature robustness, thereby improving the accuracy of the visual odometry system in complex dynamic environments and under different lighting conditions. Attached Figure Description

[0021] Figure 1 This is a system structure diagram of the present invention;

[0022] Figure 2 This is a network structure diagram of the feature extraction module;

[0023] Figure 3 This is a schematic diagram illustrating the fusion of semantic segmentation results and line segments;

[0024] Figure 4 This is a visualization of the results of running the present invention on the KITTI dataset. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] Example 1:

[0027] This invention proposes a visual odometry method and system that integrates semantic and point / line features. The system consists of five main components, such as... Figure 1 As shown, the system includes a preprocessing module, a feature extraction module, a semantic segmentation module, a feature processing module, and a pose estimation module. The system uses a monocular vision sensor to capture two consecutive frames I... t-1 ,I t As input, given a known camera intrinsic parameter matrix K, the system outputs a rotation matrix R and a translation vector t representing the camera pose after each estimation.

[0028] First, the system preprocesses two frames of RGB images I t-1 ,I tThe system creates tensors that meet the input requirements of two neural networks. It uses the OpenCV image processing library to read image files, resizing them to the required input size for the networks; simultaneously, it normalizes and scales pixel values ​​from the 0-255 range to the 0-1 range. The PyTorch deep learning framework is then used to convert the images to tensor format. These tensors are then input into the feature extraction module, generating Pred, a feature prediction result of feature detection matching between the two frames. f Pred f This includes points, lines, their descriptors, and matching relationships.

[0029] Feature extraction and matching modules: Feature extraction is based on a GlueStick hybrid deep learning network, and its structure is as follows: Figure 2 As shown, the GlueStick network consists of two parts: a front-end and a graph neural network. The front-end is based on the SuperPoint network and the LSD line detector. After receiving an image tensor, the front-end extracts point features and their descriptors through the SuperPoint network, detects line segment endpoints through LSD, and fuses keypoints and line segment endpoints to generate nodes for the structure frame. In addition, line segments also form edges connecting several nodes in the structure frame, and nodes and edges are combined to form a single wireframe graph.

[0030] Subsequently, the two wireframes are input into a set of attention map neural networks. First, a multilayer perceptron opportunistic process generates a spatial descriptor d for each node. p Generate an edge descriptor d for each edge originating from this node. e First, it encodes the relationships between nodes and edges. Second, this GNN network uses self-attention and cross-attention mechanisms to process the encoded nodes and edges. Self-attention edges connect a node to all nodes in the image, while cross-attention edges connect nodes in one image to nodes in another image, thus establishing and transmitting the relationships between each node and edge. The node descriptor d... p Through iterative processing of the L-layer network, the appearance and spatial information within and between images are enhanced and updated. Edge processing is similar to node processing; its descriptor aggregates the feature codes of the two endpoints of the edge, also containing the edge's appearance and position information. Finally, by examining the similarity of the updated node and edge descriptors, a joint assignment matrix for point and line matching between the two graphs is generated. All rows and columns of the point and line assignment matrix are processed using double softmax, ultimately producing the point and line matching relationships between the two graphs.

[0031] The point and line features generated by feature extraction and their descriptors are represented as p. i =(x i ,y i ,d i ),l j =(x j1 ,xj2 ,y j1 ,y 22 ,d j1 ,d j2 ), where d is the vector descriptor of the keypoint or line endpoint. Furthermore, Pred f It also includes the original matching relationship M of the feature points. p =(p u ,p v ), Original feature line matching relationship M l =(l p ,l q ),

[0032] The semantic segmentation network and the feature extraction network operate simultaneously. The semantic segmentation network is built upon the YOLOv8x semantic segmentation network, using I... t-1 and I t The input is a tensor, and the segmentation prediction result includes a polygon composed of ordered coordinate vertices, a classification index, and a confidence value. The prediction of the segmentation result is denoted as Pred. s : in It is an array consisting of a set of segmented polygon vertices and their corresponding semantic categories with the same index, i.e. in is a predefined integer representing various categories during the training phase. In the pre-training phase, the training ground values ​​C are initially classified into three categories based on static stability. Specifically:

[0033] (1)C A (Man-made static objects): Created or manufactured by humans, with clear and regular edges, easy to detect, and able to provide accurate and stable constraints for pose estimation;

[0034] (2)C N (Natural static objects): Naturally existing static objects with rough, irregular, or fragmented edges, making them difficult to integrate and utilize;

[0035] (3)C M (Movable objects): Objects capable of spatial movement, which can easily lead to incorrect attitude estimation.

[0036] The system focuses solely on identifying and processing reliable, absolutely static objects in urban road scenes, such as building facades, streetlights, and signs. a Artificial static objects. Specifically for the CityScapes dataset, the semantic classification, corresponding index values, and classification results are shown in the table below.

[0037] Table 1 shows three categories of truth values ​​based on the CityScapes semantic segmentation dataset.

[0038]

[0039]

[0040] The point and line geometric features generated by feature extraction and the semantic segmentation prediction results generated by semantic segmentation will be jointly processed in the feature processing section. The polygons detected by the segmentation network will be used for feature classification. Specifically, for movable objects... A polygon, whose vertices are connected to generate a binary mask, located at C M Matching point features within the movable region will be directly deleted. A The edges of category objects (such as building walls, utility poles, and fences) will be matched with nearby lines for stability filtering. Other C objects in the field of view that are cluttered, have low texture, and lack static stability will be filtered out. N Features such as tree shadows and sky will be discarded.

[0041] For line features, the system uses two types of filtering conditions: length and semantics. The system filters line matches using a predefined length threshold to exclude short line segments and retain more reliable long line segments. Secondly, to identify lines located at the edges of man-made static objects, this invention designs a detection method based on the semantic segmentation result C. A Filter and match line segment pairs with edges.

[0042] like Figure 3 As shown, A1, A2, ..., A5 are the sorted vertices of the static object polygons generated by the segmentation network. and Given unit vectors A1A5 and A1A2 respectively, calculate the angle bisector vector. To expand the polygon outwards, it is necessary to... The vertex A′1 corresponding to A1 in the opposite direction, and |A1A′1| is the same as... They have the same predefined modulus. Repeating this process yields the vertices of the expanded contour, and vice versa, the extracted polygons. The neighborhood of a static object is defined as follows: S near =(S big ∪S small )-(S small ∩S big ), S big S represents an extended polygon. small S represents a shrunken polygon. near This indicates the adjacent area that needs to be identified, i.e., the gray band in the diagram above. The endpoint falls on C. ALine segment l within the gray band area of ​​a static object will be preserved to provide the most reliable static features.

[0043] The above steps establish two sets of matching relationships. The coordinates of the filtered points and the merged lines will be connected to form a matching relationship matrix M between the two frames. combine This allows for pose estimation to be performed on the CPU. According to epipolar geometry theory, the rotation matrix R and translation vector t can be calculated using at least eight matching point pairs. The system is based on the `findEssentialMatrix` function from the OpenCV library, and also incorporates a random sample consensus algorithm to remove outliers, using a matching relation matrix M. combine The essential matrix E is calculated. After obtaining the E matrix, non-singular decomposition (SVD) is performed to obtain the rotation matrix R and the translation vector t. To avoid the scale uncertainty problem inherent in monocular cameras, the system adds a scale recovery mechanism for the ground truth of the KITTI dataset, using the ground truth scale information of the previous frame to calibrate the inherent scale uncertainty. Finally, the processed pose data is visualized in real time and its accuracy is evaluated using OpenCV's plotting function and the EVO evaluation tool.

[0044] Example 2:

[0045] The parts not mentioned in this embodiment are the same as in Embodiment 1.

[0046] To compare the impact of various features in this method on improving VO accuracy, a pose estimation feature ablation experiment was conducted on the KITTI sequence 08 using points, lines, and semantics based on the system. The results are detailed in the table below.

[0047] Table 2 Comparison of Multi-Feature Fusion Results Based on KITTI Dataset 08 Sequences

[0048]

[0049] First, when only point features are input, the estimation quality obtained by the VO system is below average. Adding line features significantly improves the estimation results, with the absolute pose error (APE) decreasing by 29.5%. It is worth noting that directly fusing semantic segmentation on feature points leads to a slight performance decrease. For the KITTI dataset, most images do not contain moving objects in the field of view; using semantic segmentation alone removes a large portion of static feature points, resulting in a decrease in the effectiveness of direct fusion. With the combined effect of point and line features and semantics, the VO system achieves the best evaluation results on Sequence 08, surpassing the performance of using any single or dual features in both APE and RPE metrics. Feature ablation experiments demonstrate that fusing multiple features can enhance the pose estimation capability of the VO system, enabling the agent to perceive its surrounding environment more comprehensively.

[0050] To evaluate the performance superiority, effectiveness, and interpretability of the proposed VO system, we compared it with various representative indirect VO algorithms, including handwritten ORB features, learned feature SuperPoint, and the end-to-end deep learning direct method TartanVO. Sequences 01, 05, 07, and 08 from the KITTI odometry dataset were selected for evaluation, each containing over 1000 frames across diverse scenes. For a fair comparison, only the front-end components such as feature extraction and matching were replaced; additional performance enhancements such as mapping and loop closure detection found in mainstream SLAM frameworks were not used. The comparison results are shown in the table below.

[0051] Table 3. Comparison of APE and RPE based on KITTI dataset sequences 01, 05, 07, and 08

[0052]

[0053] The results show that, for sequences 01, 05, and 08, the system outperforms other methods in terms of the APE metric. Similarly, for the RPE metric, the system outperforms other systems in sequences 05, 07, and 08. Overall, the test data for the four sequences demonstrate the system's superior ability to reduce overall error accumulation and improve single-shot estimation accuracy, outperforming traditional VO techniques in dynamic environments and exhibiting greater robustness and accuracy.

[0054] The system software's visual operation results are as follows: Figure 4 As shown, the first row of images represents the results from the monitoring window during the VO run, displaying the distribution of processed point and line features within the view. Dynamic object regions, such as bicycles and cars, are highlighted by removing point masks. Conversely, masked regions in the semantic segmentation results represent man-made static objects, such as parts of building facades, fences, road signs, lampposts, etc. The second row shows a comparison between the final running trajectories of sequences 05, 07, and 08 and the ground truth trajectories.

[0055] The system was deployed on Ubuntu 20.04, using Python 3.9 and an NVIDIA GeForce GTX 4090 for model training and simulation. For the feature extraction model GlueStick, a pre-trained model with invariant parameters was used. For the segmentation model YOLO, YOLOv8x was chosen for VO system integration considering time complexity and model performance, and then trained on the Cityscapes dataset. Only the left-hand RGB images from the dataset were used, with a batch size of 16 images. During training preprocessing, the images were resized to 640 pixels * 640 pixels, and pixel values ​​were normalized to ensure they were within the range of 0 to 1.

[0056] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual odometry method integrating semantic and point / line features, characterized in that, Includes the following steps: Two frames of images are acquired and preprocessed to obtain image tensors for the two frames. These image tensors are then input into the feature extraction module for processing to obtain feature prediction results for feature detection and matching. The image tensors are then input into the semantic segmentation module for processing to obtain segmentation prediction results for the two frames. The feature prediction results and segmentation prediction results are then input into the feature processing module for processing to obtain a matching relation matrix. This matching relation matrix is ​​then input into the pose estimation module for processing to obtain a rotation matrix and a translation vector. The rotation matrix and translation vector are then used for evaluation and visualization. The process involves inputting the image tensors of two frames into a feature extraction module for processing. The process includes: after inputting the image tensors of the two frames, the front end extracts point features and their descriptors using a SuperPoint network, detects line segment endpoints using LSD, and fuses keypoints and line segment endpoints to generate nodes for the structure frame. Line segments simultaneously form edges connecting several nodes in the structure frame. Nodes and edges are combined to form a single wireframe. The two wireframes are then input into an attention graph neural network. The multilayer perceptron of the attention graph neural network generates a spatial descriptor for each node and an edge descriptor for each edge originating from that node. These descriptors are used to encode the relationship between nodes and edges. Self-attention and cross-attention mechanisms are used to process the encoded nodes and edges. The node spatial descriptors and the edge descriptors of each edge are then processed by the attention graph neural network. L The iterative update of the layer network examines the similarity of the descriptors of the updated nodes and edges, and jointly generates the assignment matrix for point and line matching between the two graphs. All rows and columns of the point and line assignment matrix are processed by double softmax to generate the point and line matching relationship between the two graphs, and obtain the prediction result of feature detection matching between the two frames of images.

2. The visual odometry method fusing semantics and point / line features according to claim 1, characterized in that, The two consecutive frames are captured by a monocular vision sensor and are two RGB images.

3. The visual odometry method fusing semantics and point / line features according to claim 1, characterized in that, The image preprocessing includes: reading the image file using the OpenCV image processing library, adjusting the image to the input size required by the network, normalizing and scaling the pixel values ​​from the 0-255 range to the 0-1 range, and converting the image to tensor format using the PyTorch deep learning framework.

4. The visual odometry method fusing semantics and point / line features according to claim 1, characterized in that, The feature extraction module is built on a GlueStick hybrid deep learning network, including a front-end and a graph neural network.

5. The visual odometry method fusing semantics and point / line features according to claim 4, characterized in that, The front end is based on the SuperPoint network and the LSD line detector, and the graph neural network is an attention graph neural network.

6. The visual odometry method fusing semantics and point / line features according to claim 1, characterized in that, The feature prediction results of the feature detection and matching between the two frames of images include point and line features and their descriptions, the original matching relationship of feature points and the original feature line matching relationship.

7. The visual odometry method fusing semantic and point / line features according to claim 1, characterized in that, The semantic segmentation network is built based on the YOLOv8x semantic segmentation network. The segmentation prediction result includes a polygon composed of ordered coordinate vertices, a classification index, and a confidence value. The segmentation prediction result is denoted as... Pred s : ,in It is an array consisting of a set of segmented polygon vertices and their corresponding semantic categories with the same index, i.e. ,in These are predefined integers representing various categories during the training phase. They are divided into three categories: man-made static objects, natural static objects, and movable objects.

8. The visual odometry method fusing semantics and point / line features according to claim 1, characterized in that, The feature prediction results and segmentation prediction results are input into the feature processing module for processing. The processing process includes: for the polygon of the movable object, connecting its vertices to generate a binary mask, deleting the matching point features located in the movable object region, performing stability filtering on the edges of artificial static objects and matching them with the lines nearby, discarding cluttered, low-texture, and statically unstable features in natural static objects, filtering and merging line features through length and semantics, and connecting the coordinates of the filtered points and the merged line matches to form a corresponding matching relationship matrix between the two frames of images.

9. A system for the visual odometry method fusing semantics and point / line features as described in any one of claims 1 to 8, characterized in that, The system includes a preprocessing module, a feature extraction module, a semantic segmentation module, a feature processing module, and a pose estimation module. The preprocessing module converts RGB images into image tensors. The feature extraction module processes the input image tensors to obtain the point and line features of the image and their matching relationships. The semantic segmentation module is built based on the YOLOv8x semantic segmentation network and is responsible for processing the input image tensors to obtain segmentation prediction results. The feature processing module is responsible for point and line filtering and edge fusion to obtain the corresponding matching relationship matrix of two frames. The pose estimation module obtains the rotation matrix and translation vector by processing the corresponding matching relationship matrix of two frames.

Images