An online vector map construction method based on graph structure modeling and optimization

By using graph structure modeling in vectorized map construction, and using attention mechanisms between and within sub-graphs, the structural representation of map elements is enhanced, the problem of insufficient modeling of map elements in the existing technology is solved, and the map is improved Construction accuracy, suitable for building online maps of autonomous driving.

CN118736154BActive Publication Date: 2024-11-12NORTHEASTERN UNIV CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411222915.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-11-12
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

The existing end-to-end vectorized map construction method modeles map elements as a set of points, and does not fully utilize the structural information between map elements, resulting in poor map construction accuracy in complex traffic scenarios.

Method used

Using the online vectorized map construction method based on graph structure modeling, the map elements are regarded as sub-graphs, and the structure of sub-graph nodes to the graph is enhanced through the attention mechanism between sub-graphs and the node and edge attention mechanism inside the sub-graph. Characterize and interact with BEV features through the deformation-capable cross attention module to improve the prediction accuracy of map element attributes.

Benefits of technology

It effectively improves the representation ability of sub-graph node embedding to the graph structure, improves the accuracy of vectorized map construction, and meets the real-time needs of online map construction of autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118736154B_ABST
    Figure CN118736154B_ABST
Patent Text Reader

Abstract

The present invention belongs to the technical field of computer vision and autonomous driving, and discloses an online vectorized map construction method based on graph structure modeling and optimization. The vectorized local map is regarded as a graph structure, the map elements are regarded as subgraphs, the key points in the map elements are regarded as subgraph nodes, the subgraph node query is initialized and input into the map decoding module based on the graph structure, the subgraph query feature interaction is performed through the attention between the subgraphs, the subgraph node query is enhanced through the subgraph internal node attention and edge attention to the subgraph global feature and local structural feature representation, and the interactive enhancement is performed through the deformable attention module and the bird's-eye view feature, and the enhanced node query is used for subgraph attribute prediction. The present invention further explores the structural attributes of the vectorized map, based on the graph structure of map element modeling and map decoding, to improve the comprehensive representation of the map element structure by the subgraph node query, improve the accuracy of vectorized map construction, and promote the research on the construction of online maps for autonomous driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision and autonomous driving technology, and in particular to an online vectorized map construction method based on graph structure modeling and optimization. Background Art

[0002] Maps play a vital role in autonomous driving systems, providing a reliable perception basis for tasks such as behavior prediction and path planning. Typically, these maps have centimeter-level accuracy and are generated through processes such as collecting data from vehicles equipped with expensive equipment, building through simultaneous localization and mapping (SLAM) algorithms, and manual proofreading. Such high-precision maps are expensive and have a long update cycle. Therefore, it is difficult to meet the real-time requirements of autonomous driving for maps. In order to meet the above challenges, people give priority to using online construction of local vectorized maps to guide autonomous driving vehicles. This online map construction scheme collects data from sensors installed on the vehicle and then inputs it into a trained neural network model to generate a vectorized local map around the vehicle. "MapTR: Structured Modeling and Learningfor Online Vectorized HD Map Construction, The Eleventh International Conference on Learning Representations (ICLR). 2023" first proposed an end-to-end online map framework based on Transformer. This method models map elements as point sets for processing. "Pivotnet: Vectorized pivot learning for end-to-end HD map construction. InProceedings of the IEEE / CVF International Conference on Computer Vision (pp.3672-3682)" uses pivot point-based modeling to model map elements, and uses a pivot point-based dynamic matching module to model the topology in a dynamic point sequence. The above-mentioned visual positioning algorithms all model map elements as a collection of points, and do not fully utilize the structural information between points, thereby limiting the quality of vector map construction, especially in complex traffic scenarios, where the accuracy of map construction is often poor. Therefore, it is necessary to explore a modeling model suitable for local vector maps and introduce map element structural information to improve the accuracy of vector map construction. Summary of the invention

[0003] In view of the fact that the existing end-to-end vector map construction method models map elements as point sets, resulting in insufficient expression of map element structural information, the present invention proposes an online vector map construction method based on graph structure modeling and optimization. This method regards the vectorized local map to be constructed as a graph structure, in which each map element is regarded as a subgraph, and the key points in the map elements are regarded as subgraph nodes, thereby giving each map element structural attributes; in the map decoding module based on the graph structure, the attention mechanism between subgraphs is used to enhance the understanding and representation of the entire graph model by subgraph queries, and the node attention mechanism and edge attention mechanism within the subgraph are used to enhance the global understanding and structural perception of the subgraph by the subgraph nodes, and the deformable cross-attention module is used to interact and enhance with the BEV features, and finally used for the prediction of map element attributes. This method effectively improves the structural representation of the graph by subgraph node embedding, thereby improving the accuracy of vector map construction.

[0004] The technical solution of the present invention is as follows: an online vector map construction method based on graph structure modeling and optimization, the specific steps are as follows:

[0005] Step 1: The multi-view perspective images of the vehicle’s surroundings collected by the multi-channel surround view cameras are recorded as ,in , is the number of multi-view perspective images collected at each moment; , Respectively represent the size of perspective images;

[0006] Step 2: Input the perspective images of multiple viewpoints into a shared general deep learning feature learning module to obtain multiple channel perspective image features recorded as: ; is the number of feature channels;

[0007] Step 3: The perspective image features of multiple channels are converted into a unified bird's-eye view BEV feature through the perspective conversion method, which is recorded as: , and The dimensions representing the features of the bird's-eye view;

[0008] Step 4: Input the BEV features into a raster map segmentation network consisting of multiple convolutional layers to generate a feature map containing the foreground priors of map elements: ;

[0009] Step 5: Initialize the subgraph node query and BEV features , feature map containing map element foreground prior , is input into the map decoding module based on graph structure modeling to extract and enhance the features of subgraph node queries; the specific process is as follows:

[0010] Step 5.1: Initialize the decoding layer count k=0;

[0011] Step 5.2: Enhance the understanding of the entire graph model by querying each subgraph node through multi-head self-attention between subgraphs;

[0012] Step 5.3: Enhance the representation of the subgraph global and local structure of the subgraph by subgraph node query through node attention and edge attention within the subgraph;

[0013] Step 5.4: The subgraph node query updated in step 5.3 is interacted and enhanced with the BEV features through the deformable cross-attention mechanism;

[0014] Step 5.5: Update the decoding layer count k=k+1, and update the subgraph node reference position in an incremental manner;

[0015] Step 5.6: Determine whether the termination condition is met. When k is less than the preset total number of decoding layers L, re-enter steps 5.2 to 5.5 to enhance the subgraph node query. Otherwise, output the enhanced subgraph node query.

[0016] Step 6: Use the enhanced sub-graph node query to perform vectorized map element category prediction and update the map element node position;

[0017] Step 7: Compare the final predicted map element category and map element node position coordinates with the corresponding true value, calculate the loss, and combine the grid map segmentation loss to iterate and optimize the image feature extraction, BEV feature generation, grid map segmentation, map decoding, category prediction, and position update.

[0018] The subgraph node query acquisition process initialized in step 5 is as follows:

[0019] The vectorized map is used as the entire graph model, where each map element is a subgraph and the key points of the map elements are subgraph nodes; the number of subgraphs is set to , the number of subgraph nodes in each subgraph is , initialize the subgraph node query as ,in, .

[0020] The step 5.2 uses multi-head self-attention between subgraphs to enhance the understanding of the entire graph model by each subgraph node query. The specific process is as follows:

[0021] First, the subgraph node queries belonging to the same subgraph are fused to generate a subgraph query, as shown in formula (1):

[0022] sq i = F fusion ([ q i ,1 , q i ,2 ,…, q i , N v ]) (1)

[0023] in, It is a feature fusion operation, which is a convolution operation or an averaging operation;

[0024] The feature interaction between sub-graph queries is performed through multi-head self-attention, as shown in formula (2):

[0025] (2)

[0026] in, are the feature mapping matrices, is the matrix composed of the fused subgraph queries;

[0027] Finally, the subgraph query enhanced by self-attention is added to the subgraph node query for enhancement, as shown in formula (3):

[0028] (3).

[0029] The specific process of step 5.3 to enhance the representation of the global and local structure of the subgraph by subgraph node query through node attention and edge attention within the subgraph is as follows:

[0030] First, multi-head self-attention calculations are performed between subgraph nodes within each subgraph. The attention between subgraph nodes within a subgraph is shown in formula (4):

[0031] (4)

[0032] in, Q i ' =[ q i,1 ' , q i,1 ' , ⋯, q i, N v ' ] For the A matrix consisting of all node queries in the subgraph;

[0033] Then, the local geometric attention score of the edge in the subgraph is calculated:

[0034] By The sub-graph The reference point positions generated by the node query in the bird's-eye view perception area are recorded as: , sampling is performed on the line connecting the two reference points to obtain the sampling points, as shown in formula (5):

[0035] (5)

[0036] in, is the total number of samples;

[0037] Prior feature map of foreground elements in the map On the top, take the features corresponding to the sampling point positions, splice them together, and obtain the edge attention score between the two nodes through the multi-layer perceptron network, as shown in formula (6):

[0038] S_ Attn i,mn e = Sigmoid ( F MLPs ([ S r P i , mn 1 || S r P i , mn 2 ||…|| S r P i , mn N s ])) (6)

[0039] in, stands for Multilayer Perceptron Network;

[0040] Finally, the node attention score is fused with the edge attention score and The operation is normalized as shown in formula (7):

[0041] (7)

[0042] Update the subgraph node query, as shown in formula (8):

[0043] (8).

[0044] The step 6 performs vectorized map element category prediction and map element node position update using the enhanced sub-graph node query. The specific calculation formulas are as shown in the following formulas (9) and (10):

[0045] (9)

[0046] (10)

[0047] in, and Represent the predicted subgraph category and subgraph node position respectively. and They represent the subgraph classification branch and the subgraph node position regression branch, respectively, and are composed of multiple linear layers and nonlinear mappings.

[0048] The network loss calculation method in step 7 is as follows:

[0049] (11)

[0050] in, , , They represent the Focal form of cross entropy loss for supervising subgraph category prediction, the L1 norm regression loss for supervising subgraph node positions, and the binary cross entropy loss for supervising map segmentation. , , Respectively represent the predicted map element subgraph category, subgraph node position and map segmentation probability output, , , They respectively represent the true value of the map element subgraph category, the true value of the node position, and the true value of the map segmentation binary classification.

[0051] Beneficial effects of the present invention: The present invention proposes an online vectorized map construction method based on graph structure modeling and optimization, which uses graph structure to model local vectorized maps, gives map elements subgraph structural attributes, and uses attention between subgraphs to perform information interaction between subgraphs. The global attention and edge attention of graph nodes work together within the subgraphs to enhance the subgraph nodes' understanding and representation of the global relationship and local structure of the subgraphs, thereby improving the accuracy of vectorized map construction, promoting research on online map construction for autonomous driving, and contributing to the development of research work such as path planning and behavior prediction based on vectorized maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of the execution flow of the vectorized map construction method based on graph structure modeling and optimization of the present invention;

[0053] Figure 2 This is a flow chart of the algorithm of the vector map decoding module based on graph structure modeling of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the examples of the present invention to clearly and completely describe the technical methods in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] See also Figure 1 to Figure 2 ,A vectorized map construction method based on graph structure modeling and optimization, includes the following steps.

[0056] Step 1: Take the self-driving vehicle with 6 surround-view camera sensors as an example, and collect multi-view perspective images around the vehicle through 6 surround-view cameras, denoted as ,in ;

[0057] Step 2: Input the perspective images of the six viewing angles into a shared universal deep learning feature learning module, such as ResNet50, EfficientNet, etc., for feature extraction. The features of the extracted perspective images are recorded as: , in is the number of feature channels, for example D=256;

[0058] Step 3: Through perspective conversion methods such as geometric transformation or neural network, the features of multiple perspective images are transformed into features in a unified bird's-eye view (BEV) coordinate system, which is called BEV feature: , and Represents the size of the bird's eye view feature.

[0059] Step 4: Pass the BEV features through the grid map segmentation network to generate a priori probability feature map of map elements: ,

[0060] Map elements refer to road signs on the ground, such as lane lines, sidewalks, road boundaries, etc.

[0061] The segmentation network is composed of multiple The convolutional layer consists of:

[0062] (1)

[0063] Step 5: The local map refers to the map of a certain area around the vehicle under the bird's-eye view. The local map is regarded as a graph model, in which each map element is regarded as a subgraph, and the key points of the map elements are regarded as subgraph nodes.

[0064] Assume that the maximum number of sub-images detected is , the number of nodes in each subgraph is , initialize the subgraph node query as ;

[0065] The initialized subgraph node query is input into the graph-based map decoding module for feature extraction and enhancement, as follows:

[0066] Step 5.1: Initialize the decoding layer count k=0;

[0067] Step 5.2: Through the multi-head self-attention mechanism between sub-graphs, enhance the understanding of the entire local map by embedding each sub-graph:

[0068] First, the subgraph internal node queries are fused through convolution operations to generate subgraph queries:

[0069] s q i = F Conv 1 d ([ q i ,1 , q i ,2 ,…, q i , N v ]) (2)

[0070] Then, the multi-head self-attention mechanism is used to interact the features between sub-graph queries:

[0071] (3)

[0072] in, , is the matrix generated by all fused subgraph queries.

[0073] Finally, the subgraph query enhanced by self-attention is added to each node query inside the subgraph for enhancement:

[0074] (4)

[0075] Step 5.3: Enhance the representation of the subgraph global and local structure by subgraph node embedding through node attention and edge attention within the subgraph. Take a subgraph as an example:

[0076] First, multi-head self-attention calculation is performed between subgraph nodes within the subgraph. The attention between nodes is divided into:

[0077] (5)

[0078] in, Q i ' =[ q i,1 ' , q i,1 ' , ⋯, q i, N v ' ] For the The matrix formed by all node queries in the subgraph.

[0079] Then, the local geometric attention scores of the edges in each subgraph are calculated:

[0080] By The sub-graph The reference point position generated by the node query on the bird's-eye view perception area is recorded as: . Sampling is performed on the line connecting the two reference points to obtain the sampling points:

[0081] (6)

[0082] in, is the total number of samples.

[0083] Prior feature map of foreground elements in the map On the top, take the features corresponding to the sampling point positions and concatenate them together to generate the edge attention score between the two nodes through the multi-layer perceptron network:

[0084] S_ Attn i,mn e = Sigmoid ( F MLPs ([ S r P i , mn 1 || S r P i , mn 2 ||…|| S r P i , mn N s ])) (7)

[0085] Finally, the node attention score is fused with the edge attention score and The operation is normalized and the subgraph node query embedding is updated:

[0086] (8)

[0087] (9)

[0088] Step 5.4: The subgraph node query is interacted and enhanced with the BEV feature through the deformable cross attention mechanism;

[0089] Step 5.5: Update the decoding layer count k=k+1,

[0090] Update the subgraph node reference position incrementally:

[0091] (10)

[0092] Step 5.6: Determine whether the termination condition k=L is met, where L is the total number of pre-set decoding layers, for example, L=6. If k is less than L, then re-enter steps 5.2-5.5 to enhance the subgraph node query.

[0093] Otherwise, output the enhanced subgraph node query;

[0094] Step 6: Predict map attributes,

[0095] The enhanced subgraph node query is transformed linearly to predict the vectorized map element category:

[0096] (11)

[0097] The enhanced subgraph node query is used to incrementally update the map element subgraph node reference position as the final predicted node position:

[0098] (12)

[0099] Step 7: Calculate the network loss and iteratively optimize the network:

[0100] (13)

[0101] in, , , They represent the Focal form of cross entropy loss for supervising subgraph category prediction, the L1 norm regression loss for supervising subgraph node positions, and the binary cross entropy loss for supervising map segmentation. , , Respectively represent the predicted map element subgraph category, subgraph node position and map segmentation probability output, , , They respectively represent the true value of the map element subgraph category, the true value of the node position, and the true value of the map segmentation binary classification.

Claims

1. An online vector map construction method based on graph structure modeling and optimization, characterized in that: The specific steps are as follows: Step 1: The multi-view perspective images of the vehicle’s surroundings collected by the multi-channel surround view cameras are recorded as ,in , is the number of multi-view perspective images collected at each moment; , Respectively represent the size of perspective images; Step 2: Input the perspective images of multiple viewpoints into a shared general deep learning feature learning module to obtain multiple channel perspective image features recorded as: ; is the number of feature channels; Step 3: The perspective image features of multiple channels are converted into a unified bird's-eye view BEV feature through the perspective conversion method, which is recorded as: , and The dimensions representing the perspective features of the bird's-eye view; Step 4: Input the BEV features into a raster map segmentation network consisting of multiple convolutional layers to generate a feature map containing the foreground priors of map elements: ; Step 5: Initialize the subgraph node query and BEV features , feature map containing map element foreground prior , which is input into the map decoding module based on graph structure modeling to extract and enhance the features of subgraph node queries; The specific process is: Step 5.1: Initialize the decoding layer count k=0; Step 5.2: Enhance the understanding of the entire graph model by querying each subgraph node through multi-head self-attention between subgraphs; Step 5.3: Enhance the representation of the subgraph global and local structure of the subgraph by subgraph node query through node attention and edge attention within the subgraph; Step 5.4: The subgraph node query updated in step 5.3 is interacted and enhanced with the BEV features through the deformable cross-attention mechanism; Step 5.5: Update the decoding layer count k=k+1, and update the subgraph node reference position in an incremental manner; Step 5.6: Determine whether the termination condition is met. When k is less than the preset total number of decoding layers L, re-enter steps 5.2 to 5.5 to enhance the subgraph node query. Otherwise, output the enhanced subgraph node query. Step 6: Use the enhanced sub-graph node query to perform vectorized map element category prediction and update the map element node position; Step 7: Compare the final predicted map element category and map element node position coordinates with the corresponding true value, calculate the loss, and combine the grid map segmentation loss to iterate and optimize the image feature extraction, BEV feature generation, grid map segmentation, map decoding, category prediction, and position update.

2. The online vector map construction method based on graph structure modeling and optimization according to claim 1 is characterized in that: The subgraph node query acquisition process initialized in step 5 is as follows: The vectorized map is used as the entire graph model, where each map element is a subgraph and the key points of the map elements are subgraph nodes; the number of subgraphs is set to , the number of subgraph nodes in each subgraph is , initialize the subgraph node query as ,in, .

3. The online vector map construction method based on graph structure modeling and optimization according to claim 1 or 2, characterized in that: The step 5.2 uses multi-head self-attention between subgraphs to enhance the understanding of the entire graph model by each subgraph node query. The specific process is as follows: First, the subgraph node queries belonging to the same subgraph are fused to generate a subgraph query, as shown in formula (1): (1) in, It is a feature fusion operation, which is a convolution operation or an averaging operation; The feature interaction between sub-graph queries is performed through multi-head self-attention, as shown in formula (2): (2) in, are the feature mapping matrices, is the matrix composed of the fused subgraph queries; Finally, the subgraph query enhanced by self-attention is added to the subgraph node query for enhancement, as shown in formula (3): (3)。 4. The online vector map construction method based on graph structure modeling and optimization according to claim 3 is characterized in that: The specific process of step 5.3 to enhance the representation of the global and local structure of the subgraph by subgraph node query through node attention and edge attention within the subgraph is as follows: First, multi-head self-attention calculations are performed between subgraph nodes within each subgraph. The attention between subgraph nodes within a subgraph is shown in formula (4): (4) in, For the A matrix consisting of all node queries in the subgraph; Then, the local geometric attention score of the edge in the subgraph is calculated: By The sub-graph The reference point positions generated by the node query in the bird's-eye view perception area are recorded as: , sample on the line connecting the two reference points to obtain the sampling points, as shown in formula (5): (5) in, is the total number of samples; Prior feature map of foreground elements in the map On the top, take the features corresponding to the sampling point positions, splice them together, and obtain the edge attention score between the two nodes through the multi-layer perceptron network, as shown in formula (6): (6) in, stands for Multilayer Perceptron Network; Finally, the node attention score is fused with the edge attention score and The operation is normalized as shown in formula (7): (7) Update the subgraph node query, as shown in formula (8): (8)。 5. The online vector map construction method based on graph structure modeling and optimization according to claim 1 is characterized in that: The step 6 performs vectorized map element category prediction and map element node position update on the enhanced map element query. The specific calculation formulas are as shown in the following formulas (9) and (10): (9) (10) in, and Represent the predicted subgraph category and subgraph node position respectively. and They represent the subgraph classification branch and the subgraph node position regression branch, respectively, and are composed of multiple linear layers and nonlinear mappings.

6. The online vector map construction method based on graph structure modeling and optimization according to claim 1 is characterized in that: The network loss calculation method in step 7 is as follows: (11) in, , , They represent the cross entropy loss in the Focal form for supervising subgraph category prediction, the regression loss of the L1 norm for supervising subgraph node positions, and the binary cross entropy loss for supervising map segmentation. , , Respectively represent the predicted map element subgraph category, subgraph node position and map segmentation probability output, , , They respectively represent the true value of the map element subgraph category, the true value of the node position, and the true value of the map segmentation binary classification.