Multi-granularity representation map reconstruction method

Abstract

The invention relates to a multi-granularity representation map reconstruction method. The method comprises the following steps: S1, extracting visual features of a surround view camera; s2, inputting the visual features into a multi-scale encoder of a map reconstruction model to obtain unified multi-scale aerial view features; s3, inputting the unified multi-scale aerial view features into a decoder of the map reconstruction model, and outputting instance granularity query and point granularity query; and S4, inputting the instance granularity query and the point granularity query into a regression perception head and a classification perception head, and outputting a point position prediction result and a prediction result of a map element category to obtain a map reconstruction result. Compared with the prior art, the method has the advantages that coarse-grained instance-level query and fine-grained point-level query are combined in the same framework, and then end-to-end construction of the high-precision vectorized map is achieved.

Description

Technical Field

[0001] The present invention relates to the field of vector map reconstruction for autonomous driving perception, and in particular to a map reconstruction method with multi-granularity representation. Background Art

[0002] In recent years, with the rapid development of autonomous driving technology, researchers have increasingly used on-board sensors to build high-precision (HD) maps. Early research work focused on projecting and lifting map elements detected on the perspective view (PV) plane into three-dimensional space to achieve map reconstruction. However, in order to better integrate multiple sensors such as panoramic cameras and laser radar (LiDAR), online HD map construction methods have gradually turned to bird's eye view (BEV) representation.

[0003] At present, HD map construction methods can be roughly divided into two categories: rasterized maps and vectorized maps. The rasterized method performs semantic segmentation through BEV features, and then obtains vectorized map instances through post-processing steps. Similarly, the model outputs a confidence score for each grid. Although rasterized maps can provide detailed road information, their reliance on post-processing steps limits their scope of application.

[0004] With the emergence of end-to-end vectorization methods based on the DETR architecture, the need for post-processing steps has been weakened, making HD map construction more concise and efficient. The latest scheme samples a unified map element modeling method, which solves the ambiguity problem in existing methods and ensures the stability of the training process. However, existing methods usually only use one of point-level queries or instance-level queries, failing to fully utilize the complementary advantages of the two in granularity. They often rely solely on point-level or instance-level queries, resulting in deficiencies in the comprehensive representation of fine-grained location and coarse-grained classification information. Therefore, existing methods have certain deficiencies in the comprehensive representation of geometric details and category information, and it is difficult to capture local fine location and global category information at the same time.

[0005] In summary, existing HD map construction methods still have shortcomings in integrating multi-granularity information to improve the comprehensiveness and accuracy of map element representation. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of insufficient performance of single-dependence point-level or instance-level query representation, and to provide a multi-granularity representation map reconstruction method by combining coarse-grained instance-level query and fine-grained point-level query in the same framework to achieve end-to-end construction of high-precision vectorized maps.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A multi-granularity representation map reconstruction method, the method comprising the following steps:

[0009] Extract visual features from surround view cameras;

[0010] The visual features are fed into the multi-scale encoder of the map reconstruction model to obtain a unified multi-scale bird's-eye view feature;

[0011] The unified multi-scale bird's-eye view feature is input into a decoder of a map reconstruction model, wherein the decoder is composed of L layers, each layer including self-attention, multi-granularity attention and a feedforward neural network, wherein the multi-granularity attention includes a multi-granularity aggregator and a point-instance interaction module, wherein the multi-granularity attention interacts with the multi-scale bird's-eye view feature based on an instance-level query, and in the point-instance interaction module, the instance-level query and the point-level query output by the multi-granularity attention are subjected to information interaction, and an instance-granularity query and a point-granularity query are output;

[0012] The instance granularity query and point granularity query are input into the regression perception head and the classification perception head, and the point position prediction results and the map element category prediction results are output to obtain the map reconstruction results.

[0013] Furthermore, the specific steps of S3 are:

[0014] A1. The unified multi-scale bird's-eye view features are input into the decoder of the map reconstruction model to obtain the initial instance-level query as the previous instance-level query;

[0015] A2. The previous instance-level query is used as the input of the multi-granularity aggregator in the first layer to obtain the previous reference point and generate the corresponding location information encoding according to the previous reference point. Assign multiple sampling points to each current layer reference point, and the sampling points are offset ΔS relative to the reference point. l and weight W l By encoding the current instance-level query and the corresponding location information Generate, based on the position shift ΔS l Get the current layer sampling position S l , based on the current layer sampling position S l Compute instance-level queries for the current layer and current layer point level query Generate the current layer reference point based on the previous layer reference point, use the current layer instance level query, the current layer point level query and the current layer reference point as the previous layer instance level query, the previous layer point level query and the previous layer reference point, the number of layers is l+1, repeat A2 until the L-th layer multi-granularity aggregator outputs the L-th layer instance level query and the L-th layer point level query;

[0016] A3, the point instance interaction module samples the current layer position S of the l-th layer multi-granularity aggregatorl And the attention weight and Flatten and concatenate to calculate P2P attention encoding position information and P2I attention encoding position information The P2P attention module of the point instance interaction module encodes position information based on P2P attention and P2I attention encoding position information Compute the new current layer point-level query

[0017] A4. The P2I attention module of the point-instance interaction module is based on the new current layer point-level query Calculation point granularity query And query the point granularity Aggregate to get instance granularity query

[0018] Furthermore, the current layer reference point RF l for:

[0019]

[0020] Among them, Q l ins represents the instance-level query at layer l, Q l pts represents the point-level query at layer l, RF l-1 Represents the previous layer reference point Sigmoid and sigmoid -1 They represent the Sigmoid activation function and its inverse function respectively, and MLP represents the multi-layer perceptron layer.

[0021] Furthermore, the position shift ΔS l for:

[0022]

[0023] Current layer sampling position S l for:

[0024] S l =(RF l-1 +ΔS l ).

[0025] Furthermore, the current layer instance level query and current layer point level query for:

[0026]

[0027] Among them, j represents the N pThe index of the point, k represents the N points assigned to the reference point rep The index of the sampling points, and Respectively, in N p ×N rep and N rep The weights normalized by softmax in the range, F ms_bev Represents the multi-scale bird's-eye view features used. Sampling is the process of sampling from the bird's-eye view features. N rep The number of sampling points assigned to each reference point, N p is the total number of points contained in each instance.

[0028] Furthermore, P2P attention encodes position information and P2I attention encoding position information for:

[0029]

[0030] in, and They are multi-layer perceptrons for instance-level queries and point-level queries respectively.

[0031] Furthermore, the new current layer point level query for:

[0032]

[0033] in, Indicates the previous point-level query.

[0034] Furthermore, point-granularity query and instance-granularity queries for:

[0035]

[0036] Among them, j represents the Nth p The index in the point-level query, Represents a multi-layer perceptron.

[0037] Furthermore, the training process of the map reconstruction model is:

[0038] Updated instance-level query output from the map reconstruction network Perform dot product operation with multi-scale bird's-eye view features to obtain instance segmentation mask M pred , and then use the positive sample index to extract the corresponding prediction mask and the true mask Computing instance segmentation loss Based on instance segmentation loss Train the map reconstruction network.

[0039] Furthermore, the instance segmentation loss for:

[0040]

[0041] Among them, N pos represents the total number of positive sample queries, L dice represents the Dice loss, L bce represents the binary cross entropy loss.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention proposes a robust multi-granularity representation method, which combines coarse-grained instance-level queries with fine-grained point-level queries in the same framework, takes into account both category information and geometric details, improves the comprehensiveness and accuracy of map reconstruction, and realizes the end-to-end construction of high-precision vectorized maps. At the same time, the multi-granularity aggregator of the present invention generates point-level queries through the interaction of instance-level queries and multi-scale BEV features. Specifically, the improved multi-head deformable attention mechanism introduces multiple reference points for each query, and aggregates long-distance features from multi-scale BEV features. Through the multi-granularity aggregator, instance-level queries and point-level queries are generated from multi-scale BEV features at the same time, achieving a balance between global information capture and local detail retention.

[0044] (2) The present invention achieves efficient interaction between point-level query and instance-level query, and effectively completes the exchange of category information and geometric information. This design fully utilizes the global category perception capability of instance-level query and the local geometric detail capture capability of point-level query, and improves the comprehensive performance of the model in map element detection and reconstruction tasks.

[0045] (3) The present invention generates corresponding rasterization results by fusing instance query with BEV features, and optimizes the instance segmentation loss based on the results, thereby improving the segmentation accuracy and robustness of the model and reducing the errors caused by jitter points.

[0046] (4) The present invention introduces strategy optimization during the training process, so that the proposed MGMapNet achieves the latest performance in the single-frame map reconstruction task on the autonomous driving map reconstruction dataset. These optimization strategies fully improve the perceptual accuracy and stability of the model, thereby showing excellent performance advantages in map element detection and reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a structural schematic diagram of the present invention;

[0048] Figure 2A model framework for map reconstruction based on multi-granularity representation;

[0049] Figure 3 It is a multi-granularity attention unit;

[0050] Figure 4 It is a schematic diagram of the point instance interaction module;

[0051] Figure 5 Perception instances for map reconstruction. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0053] The present invention proposes a map reconstruction method with multi-granularity representation. In autonomous driving applications, perception and understanding of road map elements are crucial to ensure driving safety. High-precision (HD) maps enable autonomous vehicles to maintain lane positions, predict intersections, and plan optimal routes to reduce potential risks by providing categories and geometric information of road elements. This method combines coarse-grained instance-level queries and fine-grained point-level queries to achieve a comprehensive representation of map elements. Specifically, the present invention first proposes a multi-granularity query representation. In each decoder layer, a multi-granularity aggregator is used to simultaneously calculate point-level queries and instance-level queries, and information of different granularities is obtained by querying multi-scale bird's-eye view (BEV) features. Secondly, in order to improve the consistency of the two queries, the present invention proposes a point-instance interaction module, designs a point-to-point attention and a point-to-instance attention mechanism, enhances the intrinsic relationship between point-level queries and instance-level queries, and realizes the effective exchange of category information and geometric information. In addition, two granularity queries are used respectively during detection. The point granularity query is used to locate the coordinates of the point to achieve an accurate geometric description of the map element, and the instance granularity query is used to determine the category of the map element to achieve a comprehensive representation of the category information. In addition, multiple strategies are used to optimize the training process and improve the perceptual accuracy and stability of the model on single-frame data. Compared with existing advanced methods, this method uses coarse-grained and fine-grained queries at the same time, taking into account category information and geometric details, improving the comprehensiveness and accuracy of map reconstruction, and enhancing the relationship between queries through the point-instance interaction module, improving the positioning accuracy of map elements and the accuracy of category recognition. The optimal level of single-frame performance is achieved on mainstream map reconstruction datasets, proving the effectiveness and leading edge of the method. The present invention adopts a multi-granularity query representation method, calculates point-level and instance-level queries in the decoder through a multi-granularity aggregator, and extracts information of different granularities in combination with multi-scale bird's-eye view features. At the same time, in order to enhance the consistency of the two queries, a point-instance interaction module is designed, including point-to-point attention and point-to-instance attention, to achieve efficient exchange of category information and geometric information. In the detection stage, point-granularity queries are used to accurately locate the geometry of map elements, and instance-granularity queries are used to comprehensively characterize category information. In the multi-granularity aggregator, instance-level queries interact with multi-scale BEV features to generate point-level queries. To this end, the improved multi-head deformable attention mechanism introduces multiple reference points to aggregate long-range features from multi-scale BEV features. The instance-level query features are further enhanced. Through the multi-granularity aggregator, point-level queries and instance-level queries capture both global category information and local geometric details. The point-instance interaction module promotes information interaction between point-level queries and instance-level queries through point-to-point (P2P) attention and point-to-instance (P2I) attention. P2P attention combines the point-level queries of the current layer and the previous layer to refine the geometric information. The first layer performs self-attention operations only based on the current point-level query, while the subsequent layers enhance the expression ability of local geometric information by fusing point-level queries at different levels.The collaborative design of the multi-granularity aggregator and the point-instance interaction module enables the model to capture multi-granularity information more efficiently and comprehensively improve the map reconstruction performance in complex road scenarios.

[0054] The method of the present invention specifically comprises the following steps: extracting visual features of a surround view camera;

[0055] The visual features are fed into the multi-scale encoder of the map reconstruction model to obtain a unified multi-scale bird's-eye view feature;

[0056] The unified multi-scale bird's-eye view feature is input into a decoder of a map reconstruction model, wherein the decoder is composed of L layers, each layer including self-attention, multi-granularity attention and a feedforward neural network, wherein the multi-granularity attention includes a multi-granularity aggregator and a point-instance interaction module, wherein the multi-granularity attention interacts with the multi-scale bird's-eye view feature based on an instance-level query, and in the point-instance interaction module, the instance-level query and the point-level query output by the multi-granularity attention are subjected to information interaction, and an instance-granularity query and a point-granularity query are output;

[0057] The instance granularity query and point granularity query are input into the regression perception head and the classification perception head, and the point position prediction results and the map element category prediction results are output to obtain the map reconstruction results.

[0058] The map reconstruction model framework is as follows Figure 2 As shown in Figure 3 As shown; the schematic diagram of the point instance interaction module is as follows Figure 4 shown.

[0059] The flow chart of the present invention can be shown as Figure 1 As shown, the above steps can be divided into:

[0060] S1. Extract visual features of the surround view camera:

[0061] The surround view camera for autonomous driving is a set of cameras installed around the vehicle, providing a 360° panoramic view, used to capture environmental information around the vehicle, such as roads, pedestrians, obstacles, and traffic signs, and supports environmental perception, close-range object detection, and local navigation. In order to make full use of the image information collected by the surround view camera, this method uses ResNet50 as a feature extraction network to extract multi-level visual features from multi-view images. Through its deep residual structure, ResNet50 extracts shallow edge and texture information and deep global semantic information layer by layer to generate multi-scale feature representations. Shallow features retain local details, and deep features capture global semantics. High-quality input features are generated through multi-scale integration for subsequent encoders to use, providing accurate feature support for tasks such as target detection and map reconstruction.

[0062] S2. Construct an encoder to obtain unified bird's-eye view features.

[0063] Based on the LSS (Lift-Splat-Shoot) framework, multi-view features are extracted from the surround camera images. Since map elements have different lengths, relying solely on single-scale BEV features cannot meet the needs of detecting elements of different lengths. Therefore, we halve the spatial resolution of BEV features through a downsampling module to generate a lower resolution feature representation. Although more scales may bring additional benefits, experiments show that dual-scale features are effective enough. Subsequently, BEV features of both scales are used in the decoding stage. Figure 2 The map reconstruction model framework of multi-granularity representation is shown.

[0064] S3, multi-granularity aggregator obtains queries of two granularities.

[0065] 31) In the multi-granularity aggregator, instance-level queries interact with multi-scale BEV features to generate point-level queries. Figure 3 As shown in Figure 3, we improve the multi-head deformable attention mechanism by introducing multiple reference points for each query to aggregate long-range features from multi-scale BEV features.

[0066] More specifically, the multi-granularity aggregator queries at the instance level in the first layer. as input, while subsequent layers also receive point-level queries and reference point As input. q is the total number of instance-level queries, N p is the total number of points contained in each instance.

[0067]

[0068] Among them, l represents the current lth layer, sigmoid and sigmoid -1 They represent the Sigmoid activation function and its inverse function respectively, and MLP represents the multi-layer perceptron layer.

[0069] 32) Since an instance is represented as a point sequence, position encoding is added in instance-level query. According to the position RF of the reference point, we use RF to generate the position encoding PE ref :

[0070]

[0071] 33) We assign N to each reference point rep sampling points from which features are aggregated to enhance the features of the reference point. The position offset ΔS of the sampling point relative to the reference point and the associated weight W are obtained by combining the instance-level query Q ins and position encoding PE refThe calculations are as follows:

[0072]

[0073]

[0074] Among them, RF l-1 is expanded accordingly to match ΔS l By using the sampling bias ΔS l and reference point RF l-1 , sampling position S l By l-1 and ΔS l Add to update.

[0075] 34) Then, Q ins and Q pts It is generated by the weighted sum of the sampling features, and the specific calculation is as follows:

[0076]

[0077] Among them, j represents the N p The index of the point, k represents the N points assigned to the reference point rep The index of the sampling points, and Respectively, in N p ×N rep and N rep The weights normalized by softmax in the range, F ms_bev Represents the multi-scale BEV features used, and sampling is the process of sampling from bev features.

[0078] 35) Through multi-granularity aggregators, Q ins and Q pts Generated from multi-scale BEV features, it is able to capture global and local information for each map element. Our method introduces point-level queries directly from multi-scale BEV features by sampling points, thereby improving the accuracy of geometric point prediction. In addition, our model updates instance-level queries by sampling point features, effectively capturing the overall category and shape information of road elements.

[0079] S4. Construct a point-instance interaction module in the decoder to interact with queries of two granularities.

[0080] 41) The Point Instance Interaction module is designed to enhance the interaction of location information and category information between queries of different granularities. Figure 4As shown in the figure, the point-instance interaction module contains two different attention operations: P2P (Point-to-Point) attention and P2I (Point-to-Instance) attention.

[0081] At the same time, the sampling position S obtained from the multi-granularity aggregator of the lth layer l And the attention weight and are flattened and concatenated to encode position information in P2P attention and P2I attention:

[0082]

[0083] in, and They are multi-layer perceptrons (MLPs) for instance-level queries and point-level queries, respectively. and is the corresponding generated position information embedding.

[0084] 42) Since the coordinates of map elements are fine-tuned based on the point-level queries in the previous layer of multi-granular attention, these point-level queries play a key role in predicting the coordinates in the current layer. Therefore, the P2P attention module is designed to include both the point-level queries of the current layer l and the previous layer l-1 as inputs to the attention layer. The formal expression is as follows:

[0085]

[0086] It should be noted that since the first layer of multi-granular attention does not have the point-level query of the previous decoding layer, the self-attention operation in P2P attention is only performed on the current point-level query. In the subsequent multi-granular attention layers, the P2P attention module will add the point-level query of the previous layer before calculation. With the currently generated point-level query mix.

[0087] 43) After P2P attention, P2I attention operation realizes information interaction between different granularities. Point-level query exchanges geometric information with instance-level query through cross attention, as follows:

[0088]

[0089] Finally, point-level queries belonging to the same instance-level query are aggregated to update the corresponding instance-level query as follows:

[0090]

[0091] Where j represents the Nth p Indexes in point-level queries.

[0092] 44) Finally, the point-granularity query is used for point location prediction through a multi-layer perceptron (MLP) as a regression head, while the instance-granularity query is used for map element category prediction through another MLP. In summary, through the multi-granularity aggregator and point-instance interaction module, multi-granularity queries can be generated and updated, and the geometric information and category information of each map element can be effectively perceived.

[0093] S5. Auxiliary supervision optimizes the reconstruction results.

[0094] 51) In order to reduce the error caused by jitter points, the corresponding rasterization result is generated by fusion of instance query and BEV features, and the instance segmentation loss is optimized based on the result, thereby improving the segmentation accuracy and robustness of the model.

[0095] Instance segmentation loss It is not only used to segment BEV features, but also helps to extract more precise instance localization information for each instance-level query. First, by adding the updated instance-level query With BEV feature F∈R H×W×C Perform a dot product operation between them to calculate the instance segmentation mask (Where N q is the total number of instance-level queries).

[0096] 52) Then, using the positive sample index, extract the corresponding prediction mask and the true mask For each positive instance mask (Where N pos is the total number of positive sample queries), respectively through BinaryCross-Entropy loss and Dice loss Calculate the segmentation loss. Generate M pred And the process formula for calculating loss is:

[0097]

[0098] where · represents the dot product operation.

[0099] S6. Comprehensive performance evaluation of the multi-granularity representation map reconstruction model.

[0100] In order to verify the performance of the method of this application, the following experiments were designed.

[0101] We use the mean average precision (mAP) based on Chamfer distance, a widely recognized metric that is commonly used in high-precision map construction tasks. Specifically, APped ,AP div and AP bou Represents the average accuracy of the sidewalk, divider, and curb, respectively. In Experiment 1, Table 1 uses the nuScenes dataset, which is a widely used dataset in the field of autonomous driving research. It contains 1,000 scenes, each lasting 20 seconds and captured by six synchronized RGB cameras, while providing accurate pose information. In Experiment 2, Table 2 uses the Argoverse2 dataset, which comes from 1,000 sequences of multimodal data, including high-resolution images from seven ring cameras and two stereo cameras, LiDAR point clouds, and 6-degrees-of-freedom (6-DoF) pose data aligned with the map. This embodiment is compared with four other advanced vector map perception models. The specific details are as follows:

[0102] (1) VectorMapNet, a model that post-processes high-definition maps; (2) MapTRv2, a model that uses point queries for perception; (3) MGMap, a model that uses rasterization to optimize point queries; (4) MapQR, a model that uses instance queries to incorporate point granularity information;

[0103] The specific experimental results are shown in Table 1 and Table 2.

[0104] Table 1 Performance (%) of different map reconstruction models on the nuScenes dataset.

[0105]

[0106] Table 2 Performance of different map reconstruction models on the Argoverse2 dataset (%).

[0107]

[0108] From Table 1 and Table 2 and Figure 5 It can be seen that the method of the present invention obtains better results than other map reconstruction models.

[0109] This invention proposes an efficient high-precision map reconstruction method (MGMapNet) with multi-granularity query representation. The overall network architecture of MGMapNet is similar to other end-to-end HD map construction models like DETR. MGMapNet consists of two main parts: BEV feature encoder and transformer decoder. The BEV feature encoder is responsible for extracting multi-scale bird's-eye view (BEV) features from perspective images. The transformer decoder stacks multiple layers of multi-granularity attention mechanisms to generate prediction results of map elements through these attention layers. The prediction of each layer not only contains the category information of the road element, but also covers its geometric position, ensuring a comprehensive description of the map element within the perception range.

[0110] The method first obtains a unified BEV feature through an encoder. The model takes the RGB image of the surrounding view as input and converts it into a unified bird's-eye view (BEV) feature representation using the lift-splat-shoot (LSS) encoding method. The LSS method effectively integrates multi-view information by "lifting" and "flattening" points in three-dimensional space to generate high-quality BEV features. Given the diversity of map element lengths, BEV features of a single scale are difficult to fully detect elements of different lengths. Therefore, the model introduces a downsampling module to halve the spatial resolution of the BEV feature and generate BEV features of another scale. Although adding more scales may have potential advantages, experiments have shown that dual scales are sufficient to meet the needs. Subsequently, the BEV features of the original scale and the downsampled scale are spliced ​​to form multi-scale BEV features, which are comprehensively utilized in the transformer decoder. Experimental results show that the introduction of multi-scale BEV features significantly improves the overall performance and enhances the model's ability to detect map elements in various complex road scenes.

[0111] The decoder part in the present invention is as follows:

[0112] The decoder consists of L layers, each of which contains self-attention, multi-granularity attention, and a feed-forward neural network. Multi-granularity attention consists of two parts: a multi-granularity aggregator and a point-instance interaction module. Instance-level queries are initialized with learnable parameters and updated by queries on BEV features; point-level queries are dynamically generated by aggregating BEV features. Subsequently, the point-instance interaction module is used to interact between local geometric information and global category information to enhance the comprehensive representation of map elements.

[0113] The multi-granularity aggregator in the present invention is specifically:

[0114] In the multi-granularity aggregator, instance-level queries interact with multi-scale BEV features to generate point-level queries. Specifically, the multi-granularity aggregator introduces multiple reference points for each query by improving the multi-head deformable attention mechanism to aggregate long-range features from multi-scale BEV features.

[0115] In the first layer of the present invention, the multi-granularity aggregator receives the instance-level query Q ins The reference points in the first layer are generated by instance-level query prediction, and the reference points in the subsequent layers are updated by the reference points in the previous layer. Since the instance is represented by a point sequence, position encoding is added to the instance-level query. According to the position of the reference point, the corresponding position information encoding PE is generated. ref , and assign multiple sampling points to each reference point, by calculating the position offset ΔS and weight W, combining instance-level query and position information encoding, thereby enhancing the characteristics of the reference point.

[0116] In the present invention, through the multi-granularity aggregator, the instance-level query Q ins and point-level query Q pts Generated from multi-scale BEV features, it captures global information while retaining local details. Compared with the multi-point attention mechanism, this method improves the accuracy of geometric point prediction by directly sampling and generating point-level queries from multi-scale BEV features. In addition, compared with methods that only rely on point-level representations such as MapTR and MapTRv2, this model can effectively capture the overall category and shape information of road elements by updating instance-level queries through sampling point features.

[0117] The point instance interaction module in the present invention is specifically:

[0118] The point-instance interaction module aims to enhance the interaction of location information and category information between point-level queries and instance-level queries. The point-instance interaction module contains two different attention mechanisms: point-to-point (P2P) attention and point-to-instance (P2I) attention.

[0119] In the Lth layer, the sampled positions and attention weights generated by the multi-granularity aggregator are flattened and concatenated to encode the position information in P2P attention and P2I attention. With this design, point-level queries can effectively capture local geometric information, while instance-level queries can integrate global category information, thereby improving the comprehensive representation ability of map elements.

[0120] Through P2P and P2I attention mechanisms, the interaction of location information and category information between point-level and instance-level queries is promoted. The sampling positions and attention weights generated by the multi-granularity aggregator are used to encode and fuse the location information in the multi-scale BEV features. The point-instance interaction module enables the model to capture local details and global category information at the same time, improving the detection accuracy and comprehensive representation ability of map elements.

[0121] The point-to-point attention module in the present invention is specifically:

[0122] In the previous layer of multi-granularity attention, the coordinates of the map elements are refined based on point-level queries, which play a key role in the coordinate prediction of the current layer. Therefore, the P2P attention module is designed to include point-level queries of the current Lth layer and the previous L-1th layer as inputs to the attention layer. It should be noted that due to the lack of the previous decoding layer, the self-attention operation of the first layer of multi-granularity attention is only performed in the current point-level query. In the subsequent layers of P2P attention, the point-level query of the previous layer and the currently generated point-level query will be fused before the P2P attention. Through the above design, the P2P attention module effectively combines point-level queries at different levels, realizes the mutual enhancement of local geometric information and global category information, and thus improves the detection accuracy and overall representation ability of map elements.

[0123] The point-to-instance attention module in the present invention is specifically:

[0124] After completing the point-to-point attention operation, the point-to-instance attention mechanism realizes information interaction between queries of different granularities. Specifically, point-level queries exchange geometric information with instance-level queries through cross-attention. Subsequently, point-level queries belonging to the same instance-level query are aggregated to update the corresponding instance-level query. Through the point-to-instance attention mechanism, MGMapNet can integrate local geometric information and global category information at different granularity levels, further improving the detection accuracy and representation ability of map elements.

[0125] The final output of the present invention is:

[0126] Point-granularity queries are used for point location prediction through a multi-layer perceptron (MLP) as a regression head, while instance-granularity queries are used to predict the category of map elements through another MLP. By utilizing the multi-granularity aggregator and point-instance interaction module, multi-granularity queries are generated and updated, achieving effective perception of the geometric location and category information of each map element.

[0127] In order to reduce the error caused by jitter points, the present invention generates a corresponding rasterization result by fusing instance query with BEV features, and optimizes the instance segmentation loss based on the result, thereby improving the segmentation accuracy and the robustness of the model.

[0128] Compared with the prior art, the present invention has the following beneficial effects:

[0129] First, we propose a robust multi-granularity representation method, which achieves end-to-end construction of high-precision vectorized maps by combining coarse-grained instance-level queries with fine-grained point-level queries in the same framework.

[0130] Second, the multi-granularity aggregator generates point-level queries through the interaction of instance-level queries and multi-scale BEV features. Specifically, the improved multi-head deformable attention mechanism introduces multiple reference points for each query to aggregate long-range features from multi-scale BEV features. Through the multi-granularity aggregator, instance-level queries and point-level queries are generated from multi-scale BEV features at the same time, achieving a balance between capturing global information and preserving local details.

[0131] 3. Point-instance interaction module, which realizes efficient interaction between point-level query and instance-level query, and effectively completes the exchange of category information and geometric information. This design makes full use of the global category perception ability of instance-level query and the local geometric detail capture ability of point-level query, and improves the comprehensive performance of the model in map element detection and reconstruction tasks.

[0132] 4. Furthermore, in order to reduce the errors caused by jittering points, the corresponding rasterization results are generated by fusing instance query with BEV features, and the instance segmentation loss is optimized based on the results, thereby improving the segmentation accuracy and robustness of the model.

[0133] 5. We introduced strategy optimization during the training process, which enabled the proposed MGMapNet to achieve the latest performance in the single-frame map reconstruction task on the autonomous driving map reconstruction dataset. These optimization strategies fully improved the perceptual accuracy and stability of the model, thus showing excellent performance advantages in map element detection and reconstruction.

[0134] In the present invention, the construction and training process of the encoder and decoder models of multi-granularity representation includes the following steps:

[0135] 1) Visual feature extraction: Collect images from the vehicle's surround view camera and extract their visual features.

[0136] 2) Encoder construction: Based on the LSS framework, the encoder is constructed using image features to generate unified bird’s-eye view (BEV) features to provide input for the decoder.

[0137] 3) Decoder construction and verification: The proposed multi-granularity Attention is used to construct a decoder in a multi-granularity representation manner and is verified on the autonomous driving map reconstruction perception dataset.

[0138] The visual feature extraction process specifically includes:

[0139] ResNet50 is used as the feature extraction network to extract multi-level visual features from the images collected by the surround view camera and generate high-quality input feature representation for the subsequent encoder.

[0140] The encoder construction process specifically includes:

[0141] Based on the LSS (Lift-Splat-Shoot) framework, the multi-view features extracted from the surround camera images are spatially projected, and a unified bird's-eye view feature representation is generated through feature fusion. The bird's-eye view feature effectively integrates multi-view information and provides high-quality input features for the decoder for subsequent map reconstruction tasks.

[0142] The decoder construction and verification process specifically includes:

[0143] A multi-granularity query representation method is adopted. The multi-granularity aggregator is used to simultaneously calculate point-level queries and instance-level queries in the decoder, and the multi-scale bird's-eye view (BEV) features are combined to extract information of different granularities. A multi-granularity Attention mechanism is introduced to enhance the interaction and information fusion between point-level queries and instance-level queries to achieve a comprehensive representation of category information and geometric details. A decoder is constructed based on the multi-granularity representation method, and experimental verification is carried out on the autonomous driving map reconstruction perception dataset to ensure the effectiveness and performance superiority of the model.

[0144] The multi-granularity aggregator in multi-granularity Attention generates point-level queries through the interaction of instance-level queries and multi-scale BEV features. Specifically, the multi-granularity aggregator introduces multiple reference points into each query based on the improved multi-head deformable attention mechanism, so that long-distance information can be aggregated from multi-scale BEV features, achieving accurate representation of point-level features and efficient information extraction.

[0145] The point-instance interaction module in multi-granularity Attention achieves more efficient feature expression by enhancing the interaction of location information and category information between point-level queries and instance-level queries. The point-instance interaction module contains two attention mechanisms: point-to-point (P2P) attention and point-to-instance (P2I) attention. Fine-grained location information interaction is established between point-level queries to capture local detail features, and category information and geometric information interaction is established between point-level queries and instance-level queries to enhance global feature expression. The point-instance interaction module can effectively capture local geometric details and overall category information, significantly improving the detection accuracy and comprehensive representation capabilities of map elements.

[0146] In order to reduce the error caused by jittering points, the corresponding rasterization result is generated by fusing instance query and BEV features, and the instance segmentation loss is optimized based on the result to improve the segmentation accuracy and robustness of the model.

[0147] A comprehensive model evaluation of the map reconstruction method is conducted on a map reconstruction dataset for autonomous driving perception to verify its comprehensive performance in terms of map element detection accuracy and geometric reconstruction capability.

[0148] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A multi-granularity representation map reconstruction method, characterized in that: The method comprises the following steps: Extract visual features from surround view cameras; The visual features are fed into the multi-scale encoder of the map reconstruction model to obtain a unified multi-scale bird's-eye view feature; The unified multi-scale bird's-eye view feature is input into a decoder of a map reconstruction model, wherein the decoder is composed of L layers, each layer including self-attention, multi-granularity attention and a feedforward neural network, wherein the multi-granularity attention includes a multi-granularity aggregator and a point-instance interaction module, wherein the multi-granularity attention interacts with the multi-scale bird's-eye view feature based on an instance-level query, and in the point-instance interaction module, the instance-level query and the point-level query output by the multi-granularity attention are subjected to information interaction, and an instance-granularity query and a point-granularity query are output; The instance granularity query and point granularity query are input into the regression perception head and the classification perception head, and the point position prediction results and the map element category prediction results are output to obtain the map reconstruction results.

2. A multi-granularity representation map reconstruction method according to claim 1, characterized in that: The specific steps of S3 are: A1. The unified multi-scale bird's-eye view features are input into the decoder of the map reconstruction model to obtain the initial instance-level query as the previous instance-level query; A2. The previous instance-level query is used as the input of the multi-granularity aggregator in the first layer to obtain the previous reference point and generate the corresponding location information encoding according to the previous reference point. Assign multiple sampling points to each current layer reference point, and the sampling points are offset ΔS relative to the reference point. l and weight W l By encoding the current instance-level query and the corresponding location information Generate, based on the position shift ΔS l Get the current layer sampling position S l , based on the current layer sampling position S l Compute instance-level queries for the current layer and current layer point level query Generate the current layer reference point based on the previous layer reference point, use the current layer instance level query, the current layer point level query and the current layer reference point as the previous layer instance level query, the previous layer point level query and the previous layer reference point, the number of layers is l+1, repeat A2 until the L-th layer multi-granularity aggregator outputs the L-th layer instance level query and the L-th layer point level query; A3, the point instance interaction module samples the current layer position S of the l-th layer multi-granularity aggregator l And the attention weight and Flatten and concatenate to calculate P2P attention encoding position information and P2I attention encoding position information The P2P attention module of the point instance interaction module encodes position information based on P2P attention and P2I attention encoding position information Compute the new current layer point-level query A4. The P2I attention module of the point-instance interaction module is based on the new current layer point-level query Calculation point granularity query And query the point granularity Aggregate to get instance granularity query 3. A multi-granularity representation map reconstruction method according to claim 2, characterized in that: Current layer reference point RF l for: Among them, Q l ins represents the instance-level query at layer l, Q l pts represents the point-level query at layer l, RF l-1 Represents the previous layer reference point Sigmoid and Sigmoid -1 They represent the Sigmoid activation function and its inverse function respectively, and MLP represents the multi-layer perceptron layer.

4. A multi-granularity representation map reconstruction method according to claim 3, characterized in that: Position deviation ΔS l for: Current layer sampling position S l for: S l =(RF l-1 +ΔS l )。 5. A multi-granularity representation map reconstruction method according to claim 4, characterized in that: Current layer instance level query and current layer point level query for: Among them, j represents the N p The index of the point, k represents the N points assigned to the reference point rep The index of the sampling points, and Respectively, in N p ×N rep and N rep The weights normalized by softmax in the range, F ms_bev Represents the multi-scale bird's-eye view features used. Sampling is the process of sampling from the bird's-eye view features. N rep The number of sampling points assigned to each reference point, N p is the total number of points contained in each instance.

6. A multi-granularity representation map reconstruction method according to claim 4, characterized in that: P2P attention encodes position information and P2I attention encoding position information for: in, and They are multi-layer perceptrons for instance-level queries and point-level queries respectively.

7. A multi-granularity representation map reconstruction method according to claim 6, characterized in that: The new current layer point level query for: in, Indicates the previous point-level query.

8. The method for reconstructing a map with multiple granularity representations according to claim 7, characterized in that: Point granularity query and instance-granularity queries for: Among them, j represents the Nth p The index in the point-level query, Represents a multi-layer perceptron.

9. The method for reconstructing a map with multiple granularity representations according to claim 1, characterized in that: The training process of the map reconstruction model is: Updated instance-level query output from the map reconstruction network Perform dot product operation with multi-scale bird's-eye view features to obtain instance segmentation mask M pred , and then use the positive sample index to extract the corresponding prediction mask and the true mask Computing instance segmentation loss Based on instance segmentation loss Train the map reconstruction network.

10. The method for reconstructing a map with multiple granularity representations according to claim 3, characterized in that: Instance segmentation loss for: Among them, N pos represents the total number of positive sample queries, represents the Dice loss, represents the binary cross entropy loss.

Images