Visual odometer method and system fusing semantics and point-line features
Through the visual odometry method that combines semantics and dot-line features, combined with GlueStick and YOLOv8x networks, the problems of low visual odometry accuracy and insufficient dynamic robustness in complex environments are solved, and higher pose estimation accuracy and scenario understanding are achieved ability.
Patent Information
- Application Number
- CN202411181492.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-27
AI Technical Summary
In complex dynamic urban road environments, the visual odometer system has low pose estimation accuracy, is not dynamically robust, and lacks the ability to fully understand the scene. The existing methods have shortcomings in combining line features and semantic information.
Using a visual odometry method that combines semantics and dot-line features, point-line features are extracted through GlueStick hybrid deep learning network, and combined with YOLOv8x semantic segmentation network to provide semantic information, filter and fusion features, generate matching relationship matrix for pose estimation, use attention Strive to process feature descriptors with neural networks and self-attention mechanisms to enhance the robustness and accuracy of the system.
It improves the accuracy of visual odometers in complex dynamic environments and different lighting conditions, enhances scene understanding ability, improves camera motion estimation accuracy, reduces posture errors, and shows higher robustness and accuracy.
Smart Images

Figure CN119090916A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing, and in particular relates to a visual odometer method and system integrating semantics with point and line features. Background Art
[0002] Visual odometry plays a vital role in the perception and control of intelligent agents. The design of the visual odometry system is based on the assumption that "the environment is stationary, but the camera is moving". However, due to the complex lighting conditions of urban roads and the uncertainty of many dynamic objects, the accuracy of pose estimation of ordinary visual odometry still needs to be improved.
[0003] The accuracy and reliability of feature matching directly affect the performance of pose estimation. In outdoor environments with uneven feature distribution and drastic changes in lighting conditions, point features are difficult to achieve good results. Line features can effectively weaken the impact of external lighting due to their stronger geometric constraints. By combining the advantages of line features and point features, the performance of visual odometry in low-light conditions can be improved. However, in current methods, line features and semantic information in the scene are not effectively combined, and the VO system lacks the ability to fully understand the scene.
[0004] In order to improve the robustness of visual odometry in complex environments, some existing methods attempt to remove dynamic objects from the camera's field of view. Such methods ignore static feature categories with stable properties in the field of view, while introducing additional risks such as dynamic judgment accuracy and feature loss. Some methods integrate semantic information into the visual odometry system to guide the selection and tracking of feature points. However, most semantic visual odometry focuses on specific feature categories in specific scenes and requires customized training for dynamic categories in urban road scenes. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a visual odometer method and system that integrates semantic and point-line features.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] The present invention provides a visual odometer method for fusing semantics with point and line features, comprising the following steps: acquiring two frames of images, preprocessing the images to obtain image tensors of the two frames of images, inputting the image tensors of the two frames of images into a feature extraction module for processing to obtain feature prediction results of feature detection and matching of the two frames of images, inputting the image tensors of the two frames of images into a semantic segmentation module for processing to obtain segmentation prediction results of the two frames of images, inputting the feature prediction results and the segmentation prediction results into a feature processing module for processing to obtain a matching relationship matrix, inputting the matching relationship matrix into a posture estimation module for processing to obtain a rotation matrix and a translation vector, and performing evaluation and visualization through the rotation matrix and the translation vector.
[0008] Furthermore, the two frames of images are two consecutive frames, which are captured by a monocular vision sensor and are two RGB images.
[0009] Furthermore, the image preprocessing includes: reading the image file using the OpenCV image processing library, adjusting the image to the input size required by the network, normalizing and scaling the pixel value from the range of 0-255 to the range of 0-1, and converting the image to a tensor format using the PyTorch deep learning framework.
[0010] Furthermore, the feature extraction module is built based on the GlueStick hybrid deep learning network, including the front end and the graph neural network.
[0011] Furthermore, the front end is based on the SuperPoint network and the LSD line detector, and the graph neural network is an attention graph neural network.
[0012] Furthermore, the image tensors of the two frames of images are input into the feature extraction module for processing. The processing process includes: after the two frames of image tensors are input, the front end extracts point features and their descriptors through the SuperPoint network, detects the line segment endpoints through LSD, and generates nodes of the structure frame after fusing the key points and the line segment endpoints. The line segments also constitute edges connecting several nodes in the structure frame. The nodes and edges are combined into a single wireframe diagram. The two wireframe diagrams are input into the attention graph neural network. The multi-layer perceptron of the attention graph neural network generates a spatial descriptor for each node, and the nodes from the node are descriptors. An edge descriptor is generated for each edge of the point, and the descriptor is used to encode the relationship between nodes and edges. The self-attention mechanism and the cross-attention mechanism are used to process the encoded nodes and edges. The node space descriptor and the edge descriptor of each edge are iteratively updated through an L-layer network of the attention graph neural network. The similarity of the descriptors of the updated nodes and edges is examined, and a distribution matrix for point and line matching between the two graphs is jointly generated. All rows and columns of the point and line distribution matrix are processed through double softmax to generate point and line matching relationships between the two graphs, and obtain feature prediction results for feature detection matching of the two frames of images.
[0013] Furthermore, the feature prediction results of the feature detection and matching of the two frames of images include point and line features and their descriptions, original matching relationships of feature points and original matching relationships of feature lines.
[0014] Furthermore, the semantic segmentation network semantic segmentation is constructed based on the YOLOv8x semantic segmentation network, and the segmentation prediction result includes a polygon composed of ordered coordinate vertices, a classification index and a confidence value, and the segmentation prediction result is recorded as Pred s : in is an array consisting of a set of segmented polygon vertices and their corresponding semantic categories with the same index, that is, in is a predefined integer representing various categories in the training stage. The C is divided into three categories, including artificial static objects, natural static objects and movable objects.
[0015] Furthermore, the feature prediction results and the segmentation prediction results are input into the feature processing module for processing, and the processing process includes: for the polygons of the movable object, connecting their vertices to generate a binary mask, deleting the matching point features located in the movable object area, matching the edges of artificial static objects with the lines near them for stability filtering operations, discarding the cluttered, low-texture, and lack of static stability features in natural static objects, filtering and merging the line features by length and semantics, and connecting the coordinates of the filtered points and merged line matches into a corresponding matching relationship matrix of the two frames of images.
[0016] The present invention also provides a visual odometer system that integrates semantics and point and line features, including a preprocessing module, a feature extraction module, a semantic segmentation module, a feature processing module and a pose estimation module. The preprocessing module converts an RGB image into an image tensor. The feature extraction module is responsible for processing the input image tensor to obtain the point and line features of the image and their matching relationships. The semantic segmentation module is constructed based on the YOLOv8x semantic segmentation network and is responsible for processing the input image tensor to obtain a segmentation prediction result. The feature processing module is responsible for performing point and line filtering and edge fusion to obtain a corresponding matching relationship matrix between two frames of images. The pose estimation module obtains a rotation matrix and a translation vector by processing the corresponding matching relationship matrix between the two frames of images.
[0017] Compared with the prior art, the present invention has the following advantages:
[0018] (1) The present invention provides an indirect point-line feature visual odometry method with semantic information, which solves the problems of low monocular VO estimation accuracy, weak dynamic robustness, and low scene understanding ability in complex dynamic urban road scenes.
[0019] (2) The present invention combines two neural networks and utilizes powerful geometric constraints and semantic perception data to improve scene understanding and camera motion estimation accuracy.
[0020] (3) The present invention uses the GlueStick network to extract and match point and line features in the image to overcome the generalization limitations of traditional handwritten point features. Focusing on static objects in the scene, the YOLOv8x semantic segmentation network is used to provide semantic information, and the feature robustness is enhanced by integrating prior semantic information to identify and enhance feature robustness, thereby improving the accuracy of the visual odometer system in complex dynamic environments and under different lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a system structure diagram of the present invention;
[0022] Figure 2 This is the network structure diagram of the feature extraction module;
[0023] Figure 3 Schematic diagram of the fusion of semantic segmentation results and line segments;
[0024] Figure 4 This is a visualization of the results of the present invention running on the KITTI dataset. DETAILED DESCRIPTION
[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0026] Embodiment 1:
[0027] This paper proposes a visual odometry method and system that integrates semantic and point-line features. The system is divided into five main components, such as Figure 1 As shown in the figure, it includes a preprocessing module, a feature extraction module, a semantic segmentation module, a feature processing module and a pose estimation module. The system uses two consecutive frames captured by a monocular vision sensor. t-1 ,I t As input, given the known camera intrinsic parameter matrix K, the system outputs the rotation matrix R and translation vector t representing the camera pose after each estimation.
[0028] First, the system preprocesses two frames of RGB images I t-1 ,I tTo create a tensor that meets the input requirements of the two neural networks. The system uses the OpenCV image processing library to read the image file and resize the image to the input size required by the network; at the same time, the pixel value is normalized and scaled from the range of 0-255 to the range of 0-1. The image is converted into a tensor format using the PyTorch deep learning framework. After being input into the feature extraction module, the feature prediction result Pred of the feature detection matching of the two frames of images is generated. f . Pred f Includes points, lines, their descriptors and matching relationships.
[0029] Feature extraction and matching module Feature extraction is based on GlueStick hybrid deep learning network. Figure 2 As shown in the figure. The GlueStick network consists of two parts: the front end and the graph neural network. The front end is based on the SuperPoint network and the LSD line detector. After inputting an image tensor, the front end extracts point features and their descriptors through the SuperPoint network, detects line segment endpoints through LSD, and fuses key points and line segment endpoints to generate nodes of the structure frame. In addition, the line segments also constitute edges connecting several nodes in the structure frame, and the nodes and edges are combined into a single wireframe diagram.
[0030] The two wireframes are then fed into a set of attention graph neural networks. First, a multi-layer perceptron generates a spatial descriptor d for each node. p , generate edge descriptor d for each edge originating from this node e , used to encode the relationship between nodes and edges. Secondly, the GNN network uses self-attention mechanism and cross-attention mechanism to process the encoded nodes and edges. Among them, the self-attention edge connects a node with all the nodes of the image, and the cross-attention edge connects the nodes of one image with the nodes of another image, so as to establish and transmit the connection between each node and edge. The descriptor d of the node p Through the iteration of the L-layer network, the appearance and spatial information within and between images is enhanced and updated. The processing of edges is similar to that of nodes. Its descriptor aggregates the feature encodings of the two endpoints of the edge and also contains the appearance and position information of the edge. Finally, by examining the similarity of the descriptors of the updated nodes and edges, the assignment matrix of the point and line matching between the two images is jointly generated. All rows and columns of the point and line assignment matrix are processed by double softmax, and finally the point and line matching relationship between the two images is generated.
[0031] The point and line features generated by feature extraction and their descriptors are represented as p i =(x i ,y i ,d i ),l j =(x j1 ,xj2 ,y j1 ,y 22 ,d j1 ,d j2 ), where d is the vector descriptor of a keypoint or line endpoint. In addition, Pred f It also includes the original matching relationship M of the feature points p =(p u ,p v ), Original feature line matching relationship M l =(l p ,l q ),
[0032] The semantic segmentation network semantic segmentation and feature extraction network feature extraction run simultaneously. The semantic segmentation network is built based on the YOLOv8x semantic segmentation network. t-1 and I t The tensor is input, and the segmentation prediction result includes a polygon composed of ordered coordinate vertices, a classification index, and a confidence value. The prediction of the segmentation result is denoted as Pred s : in is an array consisting of a set of segmented polygon vertices and their corresponding semantic categories with the same index, that is, in is a predefined integer representing various categories in the training phase. In the pre-training phase, the training truth classification C is initially divided into three categories based on static stability. Specifically:
[0033] (1)C A (Artificial static objects): created or manufactured by humans, with clear and regular edges, easy to detect, and able to provide accurate and stable constraints for pose estimation;
[0034] (2)C N (Natural static objects): Static objects that exist naturally have rough, irregular or fragmented edges and are difficult to integrate and utilize;
[0035] (3)C M (Movable objects): Objects that can move in space are prone to pose estimation errors.
[0036] The system only focuses on identifying and processing reliable and absolutely static objects in urban road scenes, such as building facades, street lights, signs, etc. a Man-made static objects. Especially for the CityScapes dataset, the semantic classification, corresponding index value and classification are shown in the following table.
[0037] Table 1 Three classifications based on the true value of the CityScapes semantic segmentation dataset
[0038]
[0039]
[0040] The point and line geometric features generated by feature extraction and the semantic segmentation prediction results generated by semantic segmentation will be jointly operated in the feature processing part. The polygons detected by the segmentation network will be used for feature classification processing. Specifically, for movable objects The vertices of a polygon are connected to generate a binary mask, located at C M The matching point features in the movable area will be deleted directly. A The edges of categorical objects (such as building walls, utility poles, and fences) are matched with their nearby lines for stability filtering. Other C objects in the field of view that are cluttered, low-textured, and lack static stability are N Class features such as tree shadows, sky, etc. will be discarded.
[0041] For line features, the system will use two types of filtering conditions: length and semantics. The system uses a predefined length threshold to filter line matching to exclude short line segments and retain more reliable long line segments. Secondly, in order to identify lines located at the edge of man-made static objects, the present invention designs a detection method based on the semantic segmentation results C A Filter the line segment pairs that match the edges.
[0042] like Figure 3 As shown, A1, A2, …, A5 are the sorted vertices of the static object polygons generated by the segmentation network. and are the unit vectors of A1A5 and A1A2 respectively, and calculate the angle bisector vector To expand the polygon outward, you need to The vertex A′1 corresponding to A1 in the opposite direction, and |A1A′1| have the same predefined modulus. Repeating this process can find the vertices of the extended contour, and vice versa, to obtain the extracted polygon. The vicinity of a static object is defined as follows: S near =(S big ∪S small )-(S small ∩S big ), S big represents the extended polygon, S small Represents a shrinking polygon, S near Indicates the adjacent area to be determined, which is the gray band area in the above figure. The endpoint falls on C AThe line segments l within the gray band area of static objects will be retained to provide the most reliable static features.
[0043] Through the above steps, two sets of matching relationships can be established. The coordinates of the filtered points and the merged line matches will be connected into the corresponding matching relationship matrix M of the two frames. combine , so that the pose estimation process can be performed on the CPU. According to the epipolar geometry theory, the rotation matrix R and the translation vector t can be calculated using at least 8 matching point pairs. The system is based on the findEssentialMatrix function of the OpenCV library, and at the same time introduces a random sample consistency algorithm to remove outliers, using the matching relationship matrix M combine Calculate the essential matrix E. After obtaining the E matrix, perform non-singular decomposition SVD on it to obtain the rotation matrix R and translation vector t. In order to avoid the scale uncertainty problem of the monocular camera, the system adds a scale recovery mechanism for the true value of the KITTI dataset, using the true value scale information of the previous frame to calibrate the inherent scale uncertainty. The final processed pose data is visualized and evaluated in real time through the OpenCV drawing function and evo evaluation tool.
[0044] Embodiment 2:
[0045] The parts not mentioned in this embodiment are the same as those in Embodiment 1.
[0046] In order to compare the effects of various features in this method on improving VO accuracy, a pose estimation feature ablation experiment was performed on KITTI sequence 08 using points, lines, and semantics based on the system. The results are detailed in the table below.
[0047] Table 2 Comparison of multi-feature fusion effects based on KITTI dataset 08 sequence
[0048]
[0049] First, when only point features are input, the estimation quality obtained by the VO system is at a medium-low level. After adding line features, the estimation results are significantly improved, and the absolute pose error APE is reduced by 29.5%. It is worth noting that directly fusing semantic segmentation on feature points will lead to a slight decrease in performance. For the KITTI dataset, most images do not contain moving objects in the field of view. Using semantic segmentation alone will remove a large part of static feature points, resulting in a decrease in the effect of direct fusion. Under the combined effect of point-line features and semantics, the VO system obtained the best evaluation results on sequence 08, surpassing the effect of using any single or dual features in terms of absolute pose error APE and relative pose error RPE. The results of feature ablation experiments show that fusing multiple features can enhance the pose estimation ability of the VO system, enabling the intelligent agent to perceive the surrounding environment more comprehensively.
[0050] In order to evaluate the performance superiority of the VO system, its effectiveness in feature utilization, and its interpretability, we compared it with various representative indirect VO algorithms, including handwritten ORB features, learning features SuperPoint, and end-to-end deep learning direct methods TartanVO. Sequences 01, 05, 07, and 08 were selected from the KITTI odometer dataset for evaluation, each containing more than 1,000 frames with different scenes. For a fair comparison, only the front-end parts such as feature extraction and matching were replaced, and additional performance enhancements such as mapping and loop detection in the mainstream SLAM framework were not used. The comparison results are shown in the following table.
[0051] Table 3 APE and RPE comparison table based on KITTI dataset 01, 05, 07, 08 sequences
[0052]
[0053] The results show that the system outperforms other methods in terms of the APE metric in sequences 01, 05, and 08. Similarly, for the RPE metric, the system outperforms other systems in sequences 05, 07, and 08. Overall, the test data of the four sequences demonstrate that the system has an excellent ability to reduce overall error accumulation and improve single-shot estimation accuracy, outperforming traditional VO techniques in dynamic environments and showing higher robustness and accuracy.
[0054] The visual operation results of the system software are as follows: Figure 4 As shown, the first row of images is the result of the monitoring window during VO operation, showing the distribution of processed point features and line features within the view. Dynamic object areas such as bicycles and cars are highlighted by eliminating the mask of the points. In contrast, the masked area in the semantic segmentation result represents man-made static objects, such as part of the building facade, fences, road signs, lamp posts, etc. The second row shows the comparison results of the final running trajectories of the three sequences 05, 07 and 08 with the true value trajectory.
[0055] The system is deployed on an Ubuntu 20.04 system, using Python 3.9 and NVIDIA GeForce GTX 4090 for model training and simulation. For the feature extraction model GlueStick, a pre-trained model with unchanged parameters is used. For the segmentation model YOLO, YOLOv8x is selected for VO system integration considering the time complexity and model performance, and then trained on the Cityscapes dataset. Only the left RGB image in the dataset is used, and the dataset batch size is set to 16 images. During training preprocessing, the image size is resized to 640 pixels * 640 pixels, and the pixel values are normalized to ensure that their values are in the range of 0 to 1.
[0056] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A visual odometry method that integrates semantic and point-line features. It is characterized in that The following steps are involved: Acquire the front and back two frames of images, preprocess the images to obtain image tensors of the two frames of images, input the image tensors of the two frames of images into the feature extraction module for processing, obtain the feature prediction results of feature detection and matching of the two frames of images, input the image tensors of the two frames of images into the semantic segmentation module for processing, obtain the segmentation prediction results of the two frames of images, input the feature prediction results and the segmentation prediction results into the feature processing module for processing to obtain the matching relationship matrix, input the matching relationship matrix into the pose estimation module for processing to obtain the rotation matrix and translation vector, and perform evaluation and visualization through the rotation matrix and translation vector.
2. According to the visual odometer method of claim 1, It is characterized in that The two frames of images are two consecutive frames, which are captured by a monocular vision sensor and are two RGB images.
3. According to the visual odometer method integrating semantic features with point and line features in claim 1, It is characterized in that The image preprocessing includes: reading the image file using the OpenCV image processing library, adjusting the image to the input size required by the network, normalizing and scaling the pixel value from the range of 0-255 to the range of 0-1, and converting the image into a tensor format using the PyTorch deep learning framework.
4. According to the visual odometer method integrating semantic features with point and line features in claim 1, It is characterized in that The feature extraction module is built based on the GlueStick hybrid deep learning network, including the front end and the graph neural network.
5. According to claim 4, a visual odometer method integrating semantic features with point and line features, It is characterized in that The front end is based on the SuperPoint network and the LSD line detector, and the graph neural network is an attention graph neural network.
6. According to the visual odometer method of claim 1, It is characterized in that The image tensors of the two frames are input into the feature extraction module for processing. The processing process includes: after the two frames of image tensors are input, the front end extracts point features and their descriptors through the SuperPoint network, detects the line segment endpoints through LSD, and generates nodes of the structure frame after fusing the key points and the line segment endpoints. The line segments also constitute edges connecting several nodes in the structure frame. The nodes and edges are combined into a single wireframe diagram. The two wireframe diagrams are input into the attention graph neural network. The multi-layer perceptron of the attention graph neural network generates a spatial descriptor for each node, and generates a spatial descriptor for the nodes originating from the node. An edge descriptor is generated for each edge of the graph, and the descriptor is used to encode the node and edge relationship. The self-attention mechanism and the cross-attention mechanism are used to process the encoded nodes and edges. The node space descriptor and the edge descriptor of each edge are iteratively updated through an attention graph neural network L-layer network, and the similarity of the updated node and edge descriptors is examined. The distribution matrix of point and line matching between the two graphs is jointly generated, and all rows and columns of the point and line distribution matrix are processed through double softmax to generate point and line matching relationships between the two graphs, and obtain the prediction results of feature detection matching of the two frames of images.
7. The visual odometer method for integrating semantic features with point and line features according to claim 6, It is characterized in that The feature prediction results of the feature detection and matching of the two frames of images include point and line features and their descriptions, original matching relationships of feature points and original matching relationships of feature lines.
8. The visual odometer method for integrating semantic features with point and line features according to claim 1, It is characterized in that The semantic segmentation network is constructed based on the YOLOv8x semantic segmentation network. The segmentation prediction result includes a polygon composed of ordered coordinate vertices, a classification index and a confidence value. The segmentation prediction result is recorded as Pred s : in is an array consisting of a set of segmented polygon vertices and their corresponding semantic categories with the same index, that is, in is a predefined integer representing various categories in the training stage. The C is divided into three categories, including artificial static objects, natural static objects and movable objects.
9. The visual odometer method for integrating semantic features with point and line features according to claim 1, It is characterized in that The feature prediction results and the segmentation prediction results are input into the feature processing module for processing, and the processing process includes: for the polygons of the movable object, its vertices are connected to generate a binary mask, the matching point features located in the movable object area are deleted, the edges of the artificial static objects are matched with the lines near them for stability filtering operations, the cluttered, low-textured, and statically instable features of the natural static objects are discarded, the line features are filtered and merged by length and semantics, and the coordinates of the filtered points and the merged line matches are connected to form a corresponding matching relationship matrix of the two frames of images.
10. A visual odometry system integrating semantic and point-line features. It is characterized in that It includes a preprocessing module, a feature extraction module, a semantic segmentation module, a feature processing module and a pose estimation module. The preprocessing module converts the RGB image into an image tensor. The feature extraction module is responsible for processing the input image tensor to obtain the point and line features of the image and their matching relationships. The semantic segmentation module is built based on the YOLOv8x semantic segmentation network and is responsible for processing the input image tensor to obtain the segmentation prediction result. The feature processing module is responsible for performing point and line filtering and edge fusion to obtain the corresponding matching relationship matrix of the two frames of images. The pose estimation module obtains the rotation matrix and translation vector by processing the corresponding matching relationship matrix of the two frames of images.
Citation Information
Patent Citations
Automatic driving infrared semantic odometer implementation method and system based on dynamic scene
CN116597401A
Dot-line feature SLAM (Simultaneous Localization and Mapping) method for fusing depth information in low-texture scene
CN117011660A
Visual laser inertia SLAM positioning method fusing point features and line features
CN117824616A
Positioning method and device based on multi-sensor fusion
EP3940421A1
Cited By
Virtual scene texture generation system and method
CN120472074A
