A visual relocalization method based on bird's-eye view
By using a bird's-eye view-based visual relocalization method, local semantic voxel reconstruction is generated using 2D and 3D convolutional neural networks and transformed into a semantic topology map. This solves the problem of scene structure information loss in existing visual relocalization methods and achieves better localization results and generalization ability.
Patent Information
- Application Number
- CN202310291330.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-23
- Publication Date
- 2025-12-23
- Estimated Expiration
- 2043-03-23
AI Technical Summary
Existing visual relocalization methods lose the structural information of the scene and cannot effectively utilize sequence information like humans. Their generalization ability decreases when there is a large viewpoint shift or scene change.
A visual relocalization method based on a bird's-eye view is adopted. Local semantic voxel reconstruction is generated by 2D and 3D convolutional neural networks. Distinguishing target objects are extracted as nodes of the scene topology map and transformed into a semantic topology map. The affinity matrix is solved by combining graph neural networks to achieve localization.
It enables more complete utilization of sensor information, improves the robustness and generalization of positioning, and can better cope with viewpoint shifts and scene changes.
Smart Images
Figure CN116309842B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of robot technology, and in particular to a visual relocalization method based on bird's eye view. BACKGROUND
[0002] Mobile robot technology is developing rapidly and is being applied more and more widely. In an outdoor environment, a mobile robot can obtain its own position information through a global navigation satellite system (GNSS). However, as the application scenarios expand, in a densely populated urban environment, an indoor environment disturbed by buildings, and even a battlefield environment vulnerable to attack, satellite signal denial makes it impossible to obtain accurate positioning. Visual relocalization technology uses visual information to identify previously visited locations. Due to the simple deployment, low cost, and easy-to-obtain image information of camera sensors, visual relocalization technology is attracting more and more attention in the field of mobile robots. Accurate relocalization that can cope with situations such as perception aliasing, appearance, lighting, and changes in viewing angle is crucial for subsequent loop closure detection in simultaneous localization and mapping (SLAM) to correct positioning drift and mobile robot navigation tasks. Visual relocalization methods initially use various types of hand-designed descriptors to extract feature points to encode image information, such as sift and orb features. With the development of deep learning in recent years, data-driven learning algorithms use neural networks to replace traditional hand-designed descriptors to encode image information, making the obtained feature information more robust. However, there are still some problems in this field that need to be solved. Directly encoding the entire image information with the powerful learning ability of neural networks can lead to a high degree of coupling with training data, and the robustness can decrease when migrating to different scenarios, and the interpretability is poor. The feature point extraction-based method focuses on low-level visual information such as image edges and corner points, but does not process high-level semantic information such as objects and scene structures like humans, so the robustness can also decrease when there are changes in appearance, lighting, and viewing angle.
[0003] Therefore, the skilled in the art is committed to developing a visual relocalization method based on bird's eye view. The method realizes more complete utilization of sensor information, better positioning effect, and improved generalization. SUMMARY
[0004] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present application is that the existing visual relocalization method loses the structural information of the scene, making it difficult to effectively utilize the sequence information like humans, making it difficult to cope with large viewing angle shifts, and reducing the generalization when there are large changes in the scene.
[0005] To achieve the above-mentioned purpose, the present application provides a visual relocalization method based on bird's eye view, comprising the following steps:
[0006] Step 1, accumulate mobile robot camera sensor images, generate robot-centric local semantic voxel reconstruction through 2D convolutional neural network and 3D convolutional neural network;
[0007] Step 2, extract distinguishable objects from the local semantic voxel reconstruction as nodes of the scene topology graph in the bird's eye view perspective;
[0008] Step 3, store the category and appearance information of the objects into the corresponding node attributes;
[0009] Step 4, connect the objects two by two as edges of the scene topology graph;
[0010] Step 5, store the length of the edges and the proportion of the background area they span into the corresponding edge attributes;
[0011] Step 6, calculate the similarity between the current scene topology graph and the scene topology graph in the map database according to the attributes of the nodes and edges, and construct an affinity matrix;
[0012] Step 7, solve the quadratic assignment problem converted from the affinity matrix through the graph neural network to obtain the positioning result.
[0013] Further, in step 1, the images collected by the camera sensor during the operation of the mobile robot are input into the convolutional neural network, and the spatial information hidden in the images is extracted through the 2D UNet network structure and the 3D UNet network structure to obtain the local semantic voxel reconstruction centered on the robot.
[0014] Further, the output of the convolutional neural network divides the local three-dimensional space centered on the robot into voxels of equal size, and each voxel stores semantic category information at the corresponding position.
[0015] Further, the parameters of the convolutional neural network are obtained through end-to-end training on the public autonomous driving data set semantic kitti.
[0016] Further, in step 2, the object semantic categories with high distinguishability and easy positioning are pre-defined, and a density-based clustering algorithm DBSCAN is used for clustering. Each cluster result is regarded as an object with the corresponding semantic category label, and is recorded as a node in the scene topology graph.
[0017] Further, the object semantic categories with high distinguishability and easy positioning include road signs, traffic lights, and buildings.
[0018] Further, in step 5, the background semantic categories are pre-defined, which refer to categories that are not suitable for clustering into a single object but can provide effective information for the layout of the scene.
[0019] Further, the background semantic category includes a vehicle passable area, a lawn, and a sidewalk.
[0020] Further, the topological graph represents objects with distinguishing degrees as topological nodes with semantic labels, and represents background areas between the topological nodes as topological edges, abstracts high-level semantic information and geometric structure information of the scene, and meanwhile, preserves low-level feature point information as appearance attributes of objects corresponding to the topological nodes, and combines multi-level information.
[0021] Further, in step 6, similarity of the topological graph nodes and edges is calculated in the form of a predefined rule, and then an affinity matrix is obtained, a pure optimization problem of secondary distribution is converted, and a graph convolution network is used for solving.
[0022] In the preferred embodiment of the present application, for the existing visual repositioning method, a single frame front view collected by a camera is mostly encoded as a whole, and the structural information of the scene is lost and it is difficult to effectively use sequence information like a human being. The front view is converted to a bird's eye view, so that the sequence image has a unified coordinate system, and data fusion is more naturally realized, and information in a larger time window and a spatial window can be used to represent the scene at the same time. More complete sensor information is used, so that better positioning effect is realized.
[0023] Due to the loss of the structural information of the scene, when a large visual angle offset occurs, the existing method is difficult to cope with (extreme case is reverse loop, observing the same area from two opposite visual angles). Reverse loop is very natural for a human being and can be completed. From the perspective of brain, the present method further abstracts the generated scene bird's eye view representation into a semantic topological graph, represents objects with distinguishing degrees as topological nodes with semantic labels, and represents background areas between the topological nodes as topological edge information, abstracts high-level semantic information and geometric structure information of the scene, and meanwhile, preserves low-level feature point information as appearance attributes of objects corresponding to the topological nodes, combines multi-level information, improves stability and robustness of the scene representation, and realizes joint use of multi-level information by describing the structural information of the scene through the semantic topological graph and taking high-level semantic information and low-level feature point information as node attributes.
[0024] The existing method based on traditional feature extraction has poor effect, and the feature extraction method based on deep learning needs to be trained, and therefore is more seriously coupled with the data set, and when the scene changes greatly, the generalization problem of the network will decrease. The similarity of the nodes and edges of the topological graph is calculated in the form of a predefined rule, and then an affinity matrix is obtained, the two graph matching is converted into a quadratic assignment optimization problem, and then a graph convolution network is used for solving, which avoids the problem of high coupling with data and generalization decrease caused by directly training the graph convolution network to realize the embedding of the topological graph into a feature vector. The calculation of the similarity of the two graphs is converted into a pure optimization problem, and the trained graph convolution network is used to solve the problem, and in this way, the input data of the network is in the form of a small size affinity matrix, which can avoid the generalization decrease caused by the too large difference between the input pictures. The coupling degree of the trained network and the data is reduced, and therefore the generalization is improved.
[0025] Compared with the prior art, the present application has the following obvious essential features and significant advantages:
[0026] 1. More complete sensor information utilization, thereby realizing better positioning effect. The front view is converted to the bird's eye view, so that the sequence image has a unified coordinate system, and data fusion is realized more naturally, and the information in a large time window and a large space window is used to represent the scene.
[0027] 2. More rich scene representation, thereby realizing more robust positioning effect. The structure information of the scene is described by the semantic topological graph, and the high-level semantics and low-level feature point information are used as node attributes, thereby realizing the joint utilization of multi-level information.
[0028] 3. The coupling degree of the trained network and the data is reduced, thereby improving the generalization. The similarity calculation is converted into a pure optimization problem, the trained graph convolution network is used to solve the problem, and in this way, the input data of the network is in the form of a small size affinity matrix, thereby avoiding the generalization decrease caused by the too large difference between the input pictures.
[0029] The concept, specific structure and technical effects of the present application will be further described below with reference to the accompanying drawings, so as to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS
[0030] Figure 1 is the implementation process of a preferred embodiment of the present application;
[0031] Figure 2 is a scene topological graph extracted from the bird's eye view according to the three-dimensional semantic voxel reconstruction of a preferred embodiment of the present application;
[0032] Figure 3An affinity matrix diagram of a two-scene topology graph of a preferred embodiment of the present application;
[0033] Figure 4 A node correspondence matrix diagram of a two-scene topology graph solved by a graph convolution network of a preferred embodiment of the present application;
[0034] Figure 5 A matching diagram of a two-scene topology graph drawn according to node correspondence of a preferred embodiment of the present application. DETAILED DESCRIPTION
[0035] The technical content of the present application will be more clearly and conveniently understood by introducing the preferred embodiments of the present application with reference to the accompanying drawings of the specification. The present application can be embodied in many different forms, and the scope of protection of the present application is not limited to the embodiments mentioned herein.
[0036] In the drawings, components of the same structure are denoted by the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.
[0037] As shown in Figure 1 , the present application is a visual repositioning method, system and medium based on an aerial view angle.
[0038] Step 1: Accumulate multi-frame mobile robot camera sensor images to generate a local semantic voxel reconstruction centered on the robot by a 2D / 3D convolutional neural network;
[0039] Step 2: Extract discriminative target objects from the local semantic voxel reconstruction as nodes of the scene topology graph in the aerial view angle;
[0040] Step 3: Store the category, appearance, etc. of the target object into the corresponding node attribute;
[0041] Step 4: Connect the target objects two by two as edges of the scene topology graph;
[0042] Step 5: Store the length of the edge and the proportion of the background area (such as road surface, greenery, etc.) crossed into the corresponding edge attribute;
[0043] Step 6: Calculate the similarity between the current scene topology graph and the scene topology graph of the map database according to the attributes of the nodes and edges, and construct an affinity matrix;
[0044] Step 7: Solve the quadratic assignment problem converted by the affinity matrix by a graph neural network to obtain the final positioning result;
[0045] The present application is characterized in that:
[0046] The specific implementation process of step 1 is as follows:
[0047] Multiple frames of images captured by the camera sensor during the operation of the mobile robot are input into a convolutional neural network. The images are then processed through a 2D UNet network structure and a 3D UNet network structure to extract the implicit spatial information from the images, resulting in a local semantic voxel reconstruction centered on the robot. The output of the neural network segments the local 3D space centered on the robot into voxels of equal size, with each voxel storing the semantic category information at that location. The parameters of this neural network were obtained through end-to-end training on the publicly available autonomous driving dataset Semantic Kitti.
[0048] The specific implementation process of step 2 is as follows:
[0049] Based on the semantic voxel reconstruction results from step 1, a series of highly discriminative and easily localized object semantic categories are predefined, including road signs, traffic lights, buildings, etc. For voxels of these corresponding categories in the local reconstruction, the density-based clustering algorithm DBSCAN is used for clustering. Each cluster is considered as an object with a corresponding semantic category label, denoted as a node v in the scene topology graph G. i .
[0050] The specific implementation process of step 3 is as follows:
[0051] Step 3 assigns attributes to the nodes in the topology. For each node v... i The corresponding clustering semantic category is used as its class attribute. i The feature point descriptor vectors extracted by the superpoint feature point extraction method are used as the appearance attributes of the node. i The position coordinates of the cluster center point relative to the mobile robot are used as its position attribute "center". i This constitutes the complete node information for each node.
[0052] The specific implementation process of step 4 is as follows:
[0053] Take topology node v i and v j Where i≠j, forming a topological edge e ij =(v i v j Here, the topological edges are undirected edges, i.e., e ij =(v i v j ) = e ji =(v j v i); traverse all the nodes generated in step 3, connect two by two to get all the topological edges.
[0054] The specific implementation process of step 5 is as follows:
[0055] Step 5 gives the edge in the topology a property. First, the background semantic voxel category is predefined, which is different from the semantic classification result of the object semantic category, and is not suitable for clustering into a single object, but can provide effective information for the layout of the scene, such as vehicle accessible area, grassland, sidewalk, etc. For each topological edge e ij , according to the position information center i , center j stored in the node, the discrete straight line connecting the two nodes is obtained using the computer graphics bresenham algorithm, the total length of the discrete straight line is taken as the length attribute len ij of the edge e ij , and the distribution of the proportion of each background semantic category voxel on the straight line is taken as its semantic attribute semantic ij , thereby forming complete edge information. Combined with the node information, the complete scene semantic topology graph G=(v, e) is obtained. Figure 2 A set of query point scene topology graphs and a historical scene topology graph are given. For the convenience of visualization, buildings and vehicles are taken as nodes. The upper scene topology graph has 14 nodes from 0 to 13, and the lower scene topology graph has 15 nodes from 0 to 14. The two scenes have a certain translation and a large scale rotation relationship.
[0056] The specific implementation process of step 6 is as follows:
[0057] Repeat steps 1 to 5 to process the historical sensor information, divide the historical explored map area into several overlapping groups, construct the scene topology graph respectively, and store it in the map database. When there is a need for repositioning in the new operation of the mobile robot, the query point scene topology graph G q =(v q , e q ) is constructed according to the current collected sensor information. Then a historical scene topology graph G db =(v db , ed b ) is taken out from the map database. Then the similarity is calculated according to the attributes of the nodes and edges, and the similarity between the i-th node v q，i in G db and the j-th node v db,j in G q,i is the product of the category similarity and the appearance similarity, i.e. sim(v db,j , v i )=sim(classi , class j ) x sim(feat i , feat j ), where the class similarity sim(class i , class j ) is 1 when the semantic class of two nodes is the same and 0 otherwise, and the appearance similarity sim(feat i , feat j ) is computed by the feature point matching method superglue; the similarity of the edge e q in G q，ij and the edge e db in G db,mn is the product of the length similarity and the semantic similarity, i.e., sim(e q，ijx e db,mn ) = sim(len q，ij , len db,mn ) + sim(semantic q,ij , semantic db,mn ), where the length similarity is computed by and the semantic similarity is computed by the Wasserstein distance. The similarity of all node pairs (v q,i , v db,j ) and edge pairs (e q,ij , e db,mn ) in the two topological graphs are computed in turn, and the final affinity matrix K is constructed. Let G q have N nodes and G db have M nodes, then the dimension of K is MN x MN, where the vertical coordinates represent v q,1 v db,1 , v q,1 v db,2 , v q,1 v db,3 ... v q,1 v db,M , v q,2 v db,1 , v q,2 v db,1 ,..., v q,N v db,M respectively, and the horizontal coordinates have the same meaning as the vertical coordinates, i.e., the final affinity matrix K is obtained, where the physical meaning of the elements on the diagonal is the node similarity sim(v q,i , v db,j ), and the physical meaning of other elements is the edge similarity sim(e q，ij , e db,mn ) = sim((v q,i vq,j ), (v db, m v db,n Repeat the above process to obtain the affinity matrix between the query point scene topology map and all historical scene topology maps. For example... Figure 3 The result shows the final affinity matrix K. The two topological graphs have 14 and 15 nodes respectively, so the size of the affinity matrix is (14*15)*(14*15).
[0058] The specific implementation process of step 7 is as follows:
[0059] From step 6, query the scene topology graph G. q =(v q e q ) and a certain historical scene topology graph G db =(v db e db Let the affinity matrix be K, and the correspondence between nodes be X∈R. M×N Where the ordinate of X represents G q M nodes, where the x-coordinate of X represents G. db In a set of N nodes, if the element at position (m, n) is 1, then G represents a given set of N nodes. q Middle node m and G db The node n in the diagram represents the correspondence, such as... Figure 4 As shown, the element value at the dark position is 1, meaning that there is a corresponding relationship between the nodes in the two graphs at that location. The similarity between the query point and the queried point is represented by the similarity J(X) between the current scene topology graph and the historical scene topology graph corresponding to the queried location. A suitable node correspondence X∈R is then found. M×N Maximizing j(X) transforms the problem into solving an optimization problem of the following form:
[0060] J(X)=vec(X) T Kvec(X)
[0061] stX∈{0,1} N×M X1 M =1 N X T 1 N ≤1 M
[0062] The affinity matrix K can be viewed as an adjacency matrix of a topological graph. The optimization problem can then be solved using a graph convolutional neural network (GNM) to obtain the matching matrix X. Finally, the RANSAC algorithm is used to filter out incorrect matches based on homography transformation, resulting in the final matching result X′ and the corresponding similarity score J(X′) = vec(X′). T K vec(X′). For example... Figure 5As shown, the connection between the two topological graph nodes represents the correspondence between the nodes, where the thin lines are the false matches filtered out after RANSAC geometric correction, and the thick lines are the correct matches reserved for calculating the similarity score. For clearer visualization, the matching diagram without the edges of the original topological graph is also attached, as shown below. Figure 5 The similarity between the query point scene topological graph and all historical scene topological graphs in the map database is calculated according to the above process, and the pair with the highest similarity score is found. If the similarity score exceeds the set similarity threshold, it is considered that the matching is successful, that is, the mobile robot is repositioned to the position corresponding to the historical scene topological graph; otherwise, the matching fails, that is, the current position of the mobile robot has not been passed in the historical movement.
[0063] The preferred embodiments of the present application are described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the prior art according to the concept of the present application shall be within the protection scope defined by the claims.
Claims
1. A method of visual relocalization based on bird's eye view perspective, characterized in that, The method comprises the following steps: Step 1, accumulate mobile robot camera sensor image, generate local semantic voxel reconstruction centered on the robot through a 2D convolutional neural network and a 3D convolutional neural network; Step 2, extract distinguishable target objects from the local semantic voxel reconstruction as nodes of a scene topology graph in a bird's eye view perspective; Step 3, store the category and appearance information of the target objects in the corresponding node attributes; Step 4, connect the target objects two by two as edges of the scene topology graph; Step 5, store the length of the edges and the proportion of the background area crossed in the corresponding edge attributes; Step 6, calculate the similarity of the current scene topology graph and the scene topology graph in the map database according to the attributes of the nodes and edges, and construct an affinity matrix; Step 7, solve a quadratic assignment problem converted from the affinity matrix through a graph neural network to obtain a positioning result.
2. The bird's eye view based visual relocalization method of claim 1, wherein, In step 1, the images collected by the camera sensor during the operation of the mobile robot are input into the convolutional neural network, and the spatial information hidden in the images is extracted through a 2D UNet network structure and a 3D UNet network structure to obtain a local semantic voxel reconstruction centered on the robot.
3. The bird's eye view based visual relocalization method of claim 2, wherein, The output of the convolutional neural network divides the local three-dimensional space centered on the robot into voxels of equal size, and each voxel stores semantic category information at the corresponding position.
4. The bird's-eye view based visual relocalization method of claim 2, wherein, The parameters of the convolutional neural network are obtained through end-to-end training on the autonomous driving public dataset semantic kitti.
5. The bird's-eye view based visual relocalization method of claim 1, wherein, In step 2, high-distinguishability and easy-to-position object semantic categories are predefined, and a density-based clustering algorithm DBSCAN is used for clustering. Each cluster result is regarded as an object with a corresponding semantic category label, which is recorded as a node in the scene topology graph.
6. The bird's-eye view based visual relocalization method of claim 5, wherein, The high-distinguishability and easy-to-position object semantic categories include road signs, signal lights, and buildings.
7. The bird's-eye view based visual relocalization method of claim 1, wherein, In step 5, background semantic categories are predefined, which refer to categories that are not suitable for clustering into a single object but can provide effective information for the layout of the scene.
8. The bird's-eye view based visual relocalization method of claim 7, wherein, The background semantic categories include vehicle passable areas, grasslands, and sidewalks.
9. The bird's-eye view based visual relocalization method of claim 1, wherein, The topology graph represents the distinguishable objects as topology nodes with semantic labels, and represents the background areas between the topology nodes as topology edges, abstracts high-level semantic information and geometric structure information of the scene, and retains low-level feature point information as appearance attributes of the objects corresponding to the topology nodes, combining multi-level information.
10. The bird's-eye view based visual relocalization method of claim 1, wherein, In step 6, the similarity of the topology graph nodes and edges is calculated through a predefined rule, and then an affinity matrix is obtained, which is converted into a pure optimization problem of quadratic assignment, and then solved by using a graph convolution network.