Online vector map construction method, system and device and storage medium

Abstract

The invention belongs to the field of automatic driving, and relates to an online vector map construction method, system and device and a storage medium, and the method comprises the steps: 1) carrying out the fine adjustment of a stable diffusion model through an image in a source image set and a corresponding prompt text; 2) generating a prompt template by using a large language model, wherein the prompt template describes a scene containing an object, weather and time; 3) based on the prompt template, generating an image sample by using the fine-tuned stable diffusion model; 4) based on the images in the source image set and the generated image samples, generating mask perception query and BEV segmentation masks through an instance level mask prediction network; and 5) based on mask perception query and BEV segmentation mask, generating an instance classification score and a normalized BEV coordinate through a mask-driven mapping prediction network, and realizing construction of a vector map. Through a data enhancement strategy combining segmentation prior, a large language model and a stable diffusion model, the accuracy and details of vector map construction are improved.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving and relates to an online vector map construction method, system, device and storage medium, and in particular to an online vector map construction method, system, device and storage medium based on a large language model and diffusion model data enhancement strategy using segmentation priors. Background Art

[0002] High-precision (HD) mapping is a key technology in the field of autonomous driving. By providing detailed location and semantic information about map instances, HD maps enable accurate path planning and obstacle avoidance for autonomous vehicles. However, traditional HD maps are typically generated using offline simultaneous localization and mapping (SLAM) methods, which are complex, costly, and unable to reflect dynamic road conditions.

[0003] In recent years, online HD map construction has become a research hotspot. This approach leverages sensor data from vehicles to generate local maps in real time. This online approach, based on a bird's-eye view (BEV) representation, eliminates the reliance on static HD maps and directly provides map instances that reflect real-time traffic scenarios. However, current approaches still struggle to capture sufficient semantic and geometric information when dealing with complex scenes.

[0004] To address these challenges, Mask2Map proposes an online vector map construction method based on mask-aware querying. Through multi-scale BEV feature extraction, geometric feature encoding, and a cross-attention mechanism, it extracts instance-level semantic and geometric information, significantly improving the accuracy and applicability of vector maps. However, this method still has room for improvement in cross-network consistency training and the efficiency of mask generation.

[0005] Meanwhile, in semantic segmentation tasks, WeatherDG leverages a stable diffusion model and a large language model (LLM) to address domain generalization issues and generate diverse, weather-specific driving scene images. By fine-tuning the diffusion model to align the content and layout of the source data, and using an LLM-based hint generation method, WeatherDG generates high-quality synthetic data even in adverse weather conditions. This data significantly improves the generalization ability of semantic segmentation models in unseen domains and has been validated using unsupervised domain adaptation (UDA) methods.

[0006] In summary, although existing HD map construction methods and data augmentation strategies have made progress in some aspects, there is still room for optimization in combining segmentation priors, geometric feature extraction, and scene diversity generation.

[0007] Therefore, in order to address the defects in the above-mentioned prior art, it is necessary to develop a new online vector map construction method. Summary of the Invention

[0008] To overcome the shortcomings of the existing technology, the present invention proposes an efficient and accurate online vector map construction method. By combining a large language model with a diffusion model, the method uses segmentation priors and multi-scale feature extraction technology to construct an efficient and accurate vector map online, providing an innovative solution for the real-time perception system of autonomous vehicles.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] An online vector map construction method, characterized by comprising the following steps:

[0011] 1) Fine-tune the stable diffusion model using the images in the source image set and the corresponding prompt texts to obtain a fine-tuned stable diffusion model;

[0012] 2) Generate prompt templates using a large language model, which describe scenes including objects, weather, and time;

[0013] 3) generating image samples based on the hint template using the fine-tuned stable diffusion model;

[0014] 4) generating mask-aware queries and BEV segmentation masks based on the images in the source image set and the generated image samples through an instance-level mask prediction network;

[0015] 5) Based on the mask-aware query and the BEV segmentation mask, a mask-driven mapping prediction network is used to generate instance classification scores and normalized BEV coordinates, and vectorization processing is performed to fit the coordinate set corresponding to each instance into line segments or polygonal primitives and generate structured vector map data to achieve vector map construction.

[0016] Preferably, in step 1), when fine-tuning the stable diffusion model using images in the source image set and corresponding prompt texts, the prompt text is linked to a single image selected from the source image set using a unique identifier to achieve correspondence between the prompt text and the selected single image.

[0017] Preferably, the step 2) specifically includes:

[0018] 21) using an instance sampler to calculate the sampling probability of each object category according to the object category distribution in the severe weather dataset, and selecting an object category based on the sampling probability of each object category;

[0019] 22) Based on the object category selected by the instance sampler and the weather condition and time condition, using the scene composer to compose a prompt containing specific weather and time for the selected object category;

[0020] 23) creating a scene description using a scene describer based on the prompts written by the scene writer;

[0021] 24) The selected object category, the prompt including the specific weather and time, and the scene description are input into the large language model, and the large language model generates the prompt template.

[0022] Preferably, the step 3) further includes: using the DAFormer or HRDA network structure as the architecture of the segmentation model and using the Cityscape dataset as the source domain, the generated image samples as the pseudo target domain, and training the segmentation model through the UDA method; using the trained segmentation model for an unlabeled target domain and evaluating the trained segmentation model on the unlabeled target domain to evaluate the generalization ability of the fine-tuned stable diffusion model when generating image samples using the prompt template.

[0023] Preferably, the step 4) specifically includes:

[0024] 41) Inputting the images in the source image set and the generated image samples into the feature extraction network, obtaining the image features and mapping them into the BEV space to obtain the corresponding BEV features, then connecting the BEV features and fusing them through a convolutional layer to form a unified BEV feature representation;

[0025] 42) using a BEV encoder to generate multi-scale BEV features based on the unified BEV feature representation, and using a deformable encoder to jointly encode the multi-scale BEV features to encode the relationship between the multi-scale BEV features;

[0026] 43) The mask-aware query generator uses the mask converter to extract mask-aware queries from the multi-scale BEV features and initialize the mask-aware queries;

[0027] 44) Multi-layer decoding is performed through the multi-layer decoding layers of the Transformer decoder to obtain the final mask-aware query and BEV segmentation mask.

[0028] Preferably, the step 5) of generating instance classification scores and normalized BEV coordinates by a mask-driven mapping prediction network based on the mask-aware query and the BEV segmentation mask specifically includes:

[0029] 51) deriving a sparsified BEV mask from the BEV segmentation mask by a location query generator, obtaining a location query based on the sparsified BEV mask, and concatenating the location query with the mask-aware query mask to generate a combined location query;

[0030] 52) a location query generator generates a query feature using the combined location query;

[0031] 53) The geometric feature extractor obtains geometric feature extractor features from the BEV segmentation mask;

[0032] 54) The query features and geometric feature extractor features are fused into mask-aware hybrid queries through a cross-attention mechanism;

[0033] 55) The deformable Transformer decoder obtains instance classification scores and normalized BEV coordinates based on the mask-aware hybrid query prediction.

[0034] In addition, the present invention also provides an online vector map construction system, which is characterized by comprising:

[0035] a fine-tuning module, configured to fine-tune the stable diffusion model using images in a source image set and corresponding prompt texts to obtain a fine-tuned stable diffusion model;

[0036] a prompt template generation module, configured to generate a prompt template using a large language model, wherein the prompt template describes a scene including an object, weather, and time;

[0037] An image sample generation module, configured to generate image samples based on the prompt template using a fine-tuned stable diffusion model;

[0038] a mask generation module for generating mask-aware queries and BEV segmentation masks based on the images in the source image set and the generated image samples through an instance-level mask prediction network;

[0039] A vector map construction module is used to generate instance classification scores and normalized BEV coordinates based on the mask-perceived query and BEV segmentation mask through a mask-driven mapping prediction network, and perform vectorization processing to fit the coordinate set corresponding to each instance into line segments or polygonal primitives and generate structured vector map data to achieve vector map construction.

[0040] Furthermore, the present invention also provides an online vector map construction device, characterized in that it includes:

[0041] one or more processors;

[0042] a memory for storing one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the online vector map construction method as described above.

[0044] Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is characterized in that when the program is executed by a processor, the steps of the online vector map construction method described above are implemented.

[0045] Compared with the prior art, the online vector map construction method, system, device, and storage medium of the present invention have one or more of the following beneficial technical effects:

[0046] 1. The present invention utilizes segmentation priors combined with diffusion model fine-tuning to generate diverse image samples that are consistent with the source data, ensuring the efficiency of vector map construction and the reliability of generated data.

[0047] 2. This invention generates diversified scene descriptions by designing prompt templates and combining them with a large language model, effectively enriching the scene diversity and reality simulation capabilities in the data enhancement process.

[0048] 3. The present invention generates mask-aware queries through instance-level mask prediction, and combines the location query generator with the geometric feature extractor to optimize the query results from both the geometric and semantic feature levels, further improving the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the online vector map construction method of the present invention;

[0050] Figure 2 It is a schematic diagram of the composition of the online vector map construction system of the present invention. DETAILED DESCRIPTION

[0051] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components set forth in the following description or illustrated in the following figures. The present invention is capable of other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used herein are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and their variations herein is intended to cover the items and their equivalents set forth below and additional items. Unless otherwise specified or limited, the terms "mounted", "connected", "supported" and "coupled" and their variations are used broadly and cover direct mounting and indirect mounting, connection, support and coupling. In addition, "connected" and "coupled" are not limited to physical or mechanical connections or couplings.

[0052] Furthermore, on the first hand, in the disclosure of the present invention, the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second hand, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0053] While existing HD map construction methods and data augmentation strategies have achieved some progress, there is still room for improvement in integrating segmentation priors, geometric feature extraction, and scene diversity generation. Therefore, this paper aims to propose an efficient and accurate online vector map construction method by combining a large language model with a diffusion model, leveraging segmentation priors and multi-scale feature extraction techniques, providing an innovative solution for the real-time perception system of autonomous vehicles.

[0054] Figure 1 FIG. 4 is a flow chart showing the online vector map construction method of the present invention. Figure 1 As shown, the online vector map construction method of the present invention includes the following steps:

[0055] 1. Fine-tuning of the stable diffusion model.

[0056] In the present invention, the first step is to fine-tune the stable diffusion model. When fine-tuning the stable diffusion model, images from a source image set (i.e., an existing source domain dataset) and corresponding prompt text are used to fine-tune the stable diffusion model to obtain a fine-tuned stable diffusion model. Through fine-tuning, the fine-tuned stable diffusion model learns to generate images that maintain the style and structure of the source domain under different prompt conditions. In other words, the image samples generated by the fine-tuned stable diffusion model are consistent in style and structure with the images in the source image set.

[0057] Specifically, when fine-tuning the stable diffusion model, the input is a paired data consisting of a single image selected from the source image set and the corresponding prompt text, and a unique identifier V is used. * The prompt text is linked to a single image selected from the source image set. This fine-tuning of the stable diffusion model ensures that the fine-tuned stable diffusion model retains the content and layout of the images in the source image set when generating images, ensuring that the output diverse image samples are consistent with the images in the source image set and are close to the images in the source image set in terms of content and structure.

[0058] 2. Prompt template generation.

[0059] In order to enable the fine-tuned stable diffusion model to generate image samples, it is necessary to provide a prompt template for it. In the present invention, the prompt template is generated using a large language model. The format of the prompt template is "a photo of <cls> , <weather> , <time>". Among them, CLS is the object category, that is, which objects are included in the generated image; weather is weather. Since the present invention mainly generates image samples under severe weather conditions, it includes three common weather conditions, namely snow, rain and fog; TIME is time, which includes two time periods of a day, namely day and night.

[0060] In the present invention, generating the prompt template using a large language model specifically includes:

[0061] 1. Use the instance sampler (EIS) to calculate the sampling probability of each object category according to the object category distribution in the severe weather dataset (e.g., the existing Adverse Conditions Dataset for Car Driving), and select the object category based on the sampling probability of each object category.

[0062] Specifically, first, the semantic label distribution ratio of the i-th object in the severe weather dataset is calculated Among them, E i Represents the distribution ratio of semantic labels of the i-th type of object, D i represents the number of times the i-th type of object appears in the severe weather dataset, D thing Represents the total number of times all classes of objects (all identifiable instances of objects) appear in the severe weather dataset. Then, the sampling probability is calculated based on the distribution ratio of semantic labels. Among them, P i represents the probability of the i-th class object being sampled, n is the total number of objects of all classes, is the indicator function, when E i The value is 1 when it exists, otherwise it is 0. Finally, one or more object categories are selected based on the sampling probability. For example, if "car" has the highest sampling probability, "car" is selected as the object in the prompt template.

[0063] 2. Based on the object category selected by the instance sampler and the weather conditions (i.e., snow, rain, and fog) and time conditions (i.e., day and night), use the scene composer (ESC) to write prompts containing specific weather and time for the selected object category.

[0064] For example, after the instance sampler (EIS) selects "car" as the object in the prompt template, the scene composer (ESC) can select snow as the weather and day as the time, or select snow as the weather and night as the time, or select rain as the weather and night as the time, etc., thereby forming various types of prompts containing specific weather and time.

[0065] 3. Create a scene description using the scene describer (ESD) based on the prompts written by the scene writer.

[0066] That is, the scene descriptor (ESD) creates a detailed description of the weather and lighting conditions according to the weather and time selected by the scene composer (ESC). For example, if the scene composer (ESC) selects snow as the weather and night as the time, the scene descriptor (ESD) creates a detailed description of snow and a detailed description of dark lighting.

[0067] 4. Input the selected object category, the prompt including the specific weather and time, and the scene description into the large language model, and the large language model generates the prompt template.

[0068] The selected object category, prompts including specific weather and time, and scene description are input into the large language model, and the large language model generates the prompt template based on these conditions. The prompt template describes the detailed scene including the object, weather, and time.

[0069] The present invention generates diversified scene descriptions by designing prompt templates and combining them with a large language model, effectively enriching the scene diversity and reality simulation capabilities in the data enhancement process.

[0070] 3. Image sample generation.

[0071] That is, the prompt template generated by the large language model is input into the fine-tuned stable diffusion model, and the fine-tuned stable diffusion model generates image samples. By generating image samples, the number of usable images is increased.

[0072] In the present invention, the generalization ability of the fine-tuned stable diffusion model when generating image samples using the hint template is also evaluated, which specifically includes:

[0073] 1. Use the DAFormer or HRDA (High-Resolution Domain-Adaptive Semantic Segmentation) network structure as the segmentation model and the Cityscape dataset as the source domain. The image samples generated by the fine-tuned stable diffusion model are used as the pseudo target domain. The segmentation model is trained by the UDA (unsupervised domain adaptation) method.

[0074] 2. Applying the trained segmentation model to an unlabeled target domain and evaluating the trained segmentation model on the unlabeled target domain to evaluate the generalization ability of the fine-tuned stable diffusion model when generating image samples using the hint template.

[0075] In the present invention, the pixel accuracy can be used to evaluate the trained segmentation model on a real dataset of an unlabeled target domain. Where n represents the total number of categories; TP i is the true positive of the i-th class, that is, the number of pixels correctly predicted to be of this class; FP i is the false positive of the i-th class, that is, the number of pixels that are incorrectly predicted to be of this class; FN i is the false negative for class i, i.e., the number of pixels that were actually of that class but were incorrectly predicted. Evaluation revealed that the trained segmentation model achieved high pixel accuracy on a real-world dataset from an unlabeled target domain, demonstrating the strong generalization capability of the fine-tuned stable diffusion model when generating image samples using the hint template.

[0076] The present invention utilizes segmentation priors combined with diffusion model fine-tuning to generate diverse image samples that are consistent with the source data, ensuring the efficiency of vector map construction and the reliability of generated data.

[0077] 4. Mask generation.

[0078] In the present invention, a multi-scale BEV feature is constructed from the generated image samples and the images in the source image set through an instance-level mask prediction network, and a mask-aware query and a BEV segmentation mask are generated to capture the semantic features of the instance of the vector map from a global perspective, which specifically includes:

[0079] 1. Input the images in the source image set and the generated image samples into the feature extraction network, obtain the image features and map them to the BEV space to obtain the corresponding BEV features, then connect the BEV features and fuse them through the convolution layer to form a unified BEV feature representation.

[0080] Specifically, the generated image samples and the images in the source image set are respectively input into the feature extraction network. After obtaining the image features, they are mapped to the BEV space to obtain the corresponding BEV features. Then, the BEV features corresponding to the generated image samples and the images in the source image set are connected and fused through the convolutional layer to form a unified BEV feature representation for subsequent encoding.

[0081] 2. Using a BEV encoder to generate multi-scale BEV features based on the unified BEV feature representation, and using a deformable encoder to jointly encode the multi-scale BEV features to encode the relationship between the multi-scale BEV features.

[0082] Specifically, the BEV encoder generates BEV features of multiple scales through the downsampling layer. Finally, these multi-scale features are jointly encoded using a deformable Transformer encoder to encode the relationship between multi-scale BEV features.

[0083] 3. The mask-aware query generator uses the mask converter to extract mask-aware queries from multi-scale BEV features and initialize the mask-aware queries.

[0084] Among them, mask-aware queries can be initialized using learnable vectors.

[0085] 4. Multi-layer decoding is performed through the multi-layer decoding layers of the Transformer decoder to obtain the final mask-aware query and BEV segmentation mask.

[0086] Specifically, the multi-scale BEV feature F is obtained from the (m-1)th decoding layer through the M-th decoding layer of the Transformer decoder. BEV and BEV segmentation mask And the mask-aware query Updated to:

[0087]

[0088] Where I represents the scale index of BEV features, ranging from 1≤I≤S, S represents the total number of scales of BEV features, and N I is the number of instances, is the segmentation mask of the i-th instance, τ M represents the threshold parameter of the BEV segmentation mask, Q m , K m and V m are the query matrix, key matrix and value matrix obtained by linear transformation of BEV features, W Q 、W K 、W V is the learnable weight matrix.

[0089] By characterizing the BEV at the largest scale and mask-aware queries Perform dot product along the channel axis to obtain the BEV segmentation mask Apply sigmoid function to normalize the BEV segmentation mask, and feed these BEV segmentation masks to the next decoding layer for further refinement. After M layers of decoding, the instance-level mask prediction is used as the final mask-aware query. and BEV segmentation mask M BEV =M M Finally, they are passed to the subsequent mask-driven map prediction network.

[0090] 5. Vector map construction.

[0091] Based on the mask-aware query and the BEV segmentation mask, a mask-driven mapping prediction network is used to generate instance classification scores and normalized BEV coordinates, and vectorization processing is performed. The coordinate set corresponding to each instance is fitted into line segments or polygonal primitives and structured vector map data is generated to realize the construction of a vector map.

[0092] In this paper, based on the mask-aware query provided by the instance-level mask prediction network, the mask-driven mapping prediction network optimizes the query results from both the geometric and semantic features through a position query generator and a geometric feature extractor, further improving the prediction accuracy of the lightweight unimodal network. Specifically,

[0093] 1. A sparse BEV mask is derived from the BEV segmentation mask by a location query generator, a location query is obtained based on the sparse BEV mask, and the location query is concatenated with the mask-aware query mask to generate a combined location query.

[0094] Specifically, the location query generator first extracts the segmentation mask from the BEV Derived sparse BEV mask At the same time, average pooling is applied in the x-domain and y-domain to obtain the location query f i pos ,Right now in, is the number of non-zero pixels in the sparse BEV mask, H and W are the height and width of the sparse BEV mask, and PE(x,y) is generated by a sine function. Finally, the position query f i pos With mask-aware query Mask join to generate combined location queries

[0095] 2. A location query generator generates query features using the combined location query.

[0096] Specifically, the location query generator uses a combined location query Generate N for the i-th mapping instance p Point-level features. Location query generator replicates N p The second and compare them with N p P can learn point queries q1,…,qN p , generate query features By qN p The learnable query q1 makes the mask-aware query understand the order of points generated for the i-th mapping instance, and the generated query features is sent to the mask guidance map decoder.

[0097] 3. Geometric feature extractor obtains geometric feature extractor features from BEV segmentation mask.

[0098] Specifically, the geometric feature extractor uses the threshold τ of the BEV segmentation mask G Segmentation mask from BEV Generate sparse BEV mask And from the sparse BEV mask Sampling N S key pixels to generate point-level geometric features; the maximum survival method (Max Survival) is used to select the strongest pixel from a non-overlapping window of size G×G, while setting the rest of the pixels to zero; next, the farthest point sampling (FPS) algorithm is applied to iteratively select the output of the maximum survival method and identify N S key points; finally, according to this N S The location of the key points, from the BEV features The maximum scale N S features, and use a multi-layer perceptron (MLP) to S The (x, y) coordinates of the key points are encoded and the two features are connected to obtain the geometric feature extractor features, which are expressed as

[0099] 4. The query features and geometric feature extractor features are fused into mask-aware hybrid queries through the cross-attention mechanism.

[0100] By As the query vector, f i GFE As the key-value vector, the cross-attention module generates a mask-aware hybrid query, denoted as

[0101] 5. The deformable Transformer decoder predicts instance classification scores and normalized BEV coordinates based on the mask-aware hybrid query.

[0102] Mask-aware hybrid query It is then decoded by the deformable Transformer decoder and uses the multi-scale BEV features F BEV As a value vector, finally, the decoder predicts the instance classification score and normalized BEV coordinates, completing the classification and regression tasks for each map instance.

[0103] 6. Based on the instance classification scores and the normalized BEV coordinates, vectorization processing is performed to fit the coordinate set corresponding to each instance into line segments or polygonal primitives, and structured vector map data is generated to achieve complete vector map construction.

[0104] Figure 2 The figure shows the schematic diagram of the online vector map construction system of the present invention. Figure 2 As shown, the online vector map construction system of the present invention includes:

[0105] 1. Fine-tune the module.

[0106] The fine-tuning module is used to fine-tune the stable diffusion model using the images in the source image set and the corresponding prompt texts to obtain a fine-tuned stable diffusion model.

[0107] 2. Prompt template generation module.

[0108] The prompt template generation module is used to generate a prompt template using a large language model, where the prompt template describes a scene including an object, weather, and time.

[0109] 3. Image sample generation module.

[0110] The image sample generation module is used to generate image samples based on the prompt template using the fine-tuned stable diffusion model.

[0111] 4. Mask generation module.

[0112] The mask generation module is used to generate mask-aware queries and BEV segmentation masks through an instance-level mask prediction network based on the images in the source image set and the generated image samples.

[0113] 5. Vector map construction module.

[0114] The vector map construction module is used to generate instance classification scores and normalized BEV coordinates based on the mask-perceived query and BEV segmentation mask through a mask-driven mapping prediction network, and perform vectorization processing to fit the coordinate set corresponding to each instance into line segments or polygonal primitives and generate structured vector map data to achieve vector map construction.

[0115] At the same time, in the present invention, the online vector map construction system can be trained to improve its ability to construct vector maps. During training, the vector map constructed by the online vector map construction system can be compared with the real vector map, and a loss value can be calculated. This loss value is used as the loss to optimize the online vector map construction system, resulting in a lightweight single-modal online vector map construction system that approaches the accuracy of complex multimodal models. The optimized online vector map construction system can then be used to construct vector maps.

[0116] In addition, the present invention also provides an online vector map construction device, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the online vector map construction method as described above.

[0117] Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is characterized in that when the program is executed by a processor, the steps of the online vector map construction method described above are implemented.

[0118] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art may, based on the principles of the present invention, modify or replace the technical solutions of the present invention with equivalents without departing from the essence and scope of the technical solutions of the present invention.< / time> < / weather> < / cls>

Claims

1. A method for constructing an online vector map, characterized in that: The following steps are involved: 1) Fine-tune the stable diffusion model using the images in the source image set and the corresponding prompt texts to obtain a fine-tuned stable diffusion model; 2) Generate prompt templates using a large language model, which describe scenes including objects, weather, and time; 3) generating image samples based on the hint template using the fine-tuned stable diffusion model; 4) generating mask-aware queries and BEV segmentation masks based on the images in the source image set and the generated image samples through an instance-level mask prediction network; 5) Based on the mask-aware query and the BEV segmentation mask, a mask-driven mapping prediction network is used to generate instance classification scores and normalized BEV coordinates, and vectorization processing is performed to fit the coordinate set corresponding to each instance into line segments or polygonal primitives and generate structured vector map data to achieve vector map construction.

2. The online vector map construction method according to claim 1, characterized in that: In step 1), when fine-tuning the stable diffusion model using images in the source image set and corresponding prompt texts, the prompt text is linked to a single image selected from the source image set using a unique identifier to achieve correspondence between the prompt text and the selected single image.

3. The online vector map construction method according to claim 1, characterized in that: The step 2) specifically includes: 21) using an instance sampler to calculate the sampling probability of each object category according to the object category distribution in the severe weather dataset, and selecting an object category based on the sampling probability of each object category; 22) Based on the object category selected by the instance sampler and the weather condition and time condition, using the scene composer to compose a prompt containing specific weather and time for the selected object category; 23) creating a scene description using a scene describer based on the prompts written by the scene writer; 24) The selected object category, the prompt including the specific weather and time, and the scene description are input into the large language model, and the large language model generates the prompt template.

4. The online vector map construction method according to claim 1, characterized in that: The step 3) also includes: using the DAFormer or HRDA network structure as a segmentation model and the Cityscape dataset as a source domain, and the generated image samples as a pseudo target domain, and training the segmentation model through the UDA method; using the trained segmentation model for an unlabeled target domain and evaluating the trained segmentation model on the unlabeled target domain to evaluate the generalization ability of the fine-tuned stable diffusion model when generating image samples using the prompt template.

5. The online vector map construction method according to claim 1, characterized in that: The step 4) specifically includes: 41) Inputting the images in the source image set and the generated image samples into the feature extraction network, obtaining the image features and mapping them into the BEV space to obtain the corresponding BEV features, then connecting the BEV features and fusing them through a convolutional layer to form a unified BEV feature representation; 42) using a BEV encoder to generate multi-scale BEV features based on the unified BEV feature representation, and using a deformable encoder to jointly encode the multi-scale BEV features to encode the relationship between the multi-scale BEV features; 43) The mask-aware query generator uses the mask converter to extract mask-aware queries from the multi-scale BEV features and initialize the mask-aware queries; 44) Multi-layer decoding is performed through the multi-layer decoding layers of the Transformer decoder to obtain the final mask-aware query and BEV segmentation mask.

6. The online vector map construction method according to claim 1, characterized in that: The step 5) of generating instance classification scores and normalized BEV coordinates by using a mask-driven mapping prediction network based on the mask-aware query and the BEV segmentation mask specifically includes: 51) deriving a sparsified BEV mask from the BEV segmentation mask by a location query generator, obtaining a location query based on the sparsified BEV mask, and concatenating the location query with the mask-aware query mask to generate a combined location query; 52) a location query generator generates a query feature using the combined location query; 53) The geometric feature extractor obtains geometric feature extractor features from the BEV segmentation mask; 54) The query features and geometric feature extractor features are fused into mask-aware hybrid queries through a cross-attention mechanism; 55) The deformable Transformer decoder obtains instance classification scores and normalized BEV coordinates based on the mask-aware hybrid query prediction.

7. An online vector map construction system, characterized in that: include: a fine-tuning module, configured to fine-tune the stable diffusion model using images in a source image set and corresponding prompt texts to obtain a fine-tuned stable diffusion model; a prompt template generation module, configured to generate a prompt template using a large language model, wherein the prompt template describes a scene including an object, weather, and time; An image sample generation module, configured to generate image samples based on the prompt template using a fine-tuned stable diffusion model; a mask generation module for generating mask-aware queries and BEV segmentation masks based on the images in the source image set and the generated image samples through an instance-level mask prediction network; A vector map construction module is used to generate instance classification scores and normalized BEV coordinates based on the mask-perceived query and BEV segmentation mask through a mask-driven mapping prediction network, and perform vectorization processing to fit the coordinate set corresponding to each instance into line segments or polygonal primitives and generate structured vector map data to achieve vector map construction.

8. An online vector map construction device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the online vector map construction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the online vector map construction method according to any one of claims 1 to 6 are implemented.

Images