Map generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a map generation method and device, electronic equipment and storage medium, and relate to the technical field of map generation. The method comprises: obtaining a target image, processing the target image through an encoder to obtain target image features; determining corresponding target prior features from a neural map prior according to position information corresponding to the target image; inputting the target image features and the target prior features into a trained map feature generation model for feature fusion to obtain target fusion features; and inputting the target fusion features into a decoder to obtain a semantic map corresponding to the target image. Through the method of the embodiments of the present application, the inference performance of the map can be improved by fusing the current target image features with the neural map prior, so as to improve the prediction quality of the online semantic map.

Description

Technical Field

[0001] The present invention relates to the field of map generation technology, and in particular to a map generation method, apparatus, electronic device and storage medium. Background Technology

[0002] For high-definition semantic maps, most of them are currently offline maps built through expensive manual annotation. When road conditions change, they do not support timely updates, which leads to safety hazards caused by vehicles using outdated maps.

[0003] Based on this, related technologies have proposed online inference of semantic maps. These schemes typically use deep learning methods to infer semantic maps in real time, effectively addressing the problem of maps not being updated in a timely manner. However, the quality of currently inferred maps is usually far lower than that of pre-built offline maps. In other words, how to predict high-quality semantic maps in real time is a technical problem that this invention urgently needs to solve. Summary of the Invention

[0004] To address the aforementioned technical issues, embodiments of the present invention provide a map generation method, apparatus, electronic device, and storage medium to improve the prediction quality of semantic maps.

[0005] This invention provides a map generation method, the method comprising:

[0006] The target image is acquired, and the target image is processed by an encoder to obtain the target image features;

[0007] Based on the location information corresponding to the target image, the corresponding target prior features are determined from the neural map prior.

[0008] The target image features and the target prior features are input into a trained map feature generation model for feature fusion to obtain target fused features;

[0009] The target fusion features are input into the decoder to obtain the semantic map corresponding to the target image.

[0010] A second aspect of the present invention provides a map generation apparatus, the apparatus comprising:

[0011] The image feature determination module is used to acquire a target image and process the target image through an encoder to obtain target image features;

[0012] The prior feature determination module is used to determine the corresponding target prior features from the neural map prior based on the location information corresponding to the target image;

[0013] The feature fusion determination module is used to input the target image features and the target prior features into a trained map feature generation model for feature fusion to obtain target fused features;

[0014] The map determination module is used to input the target fusion features into the decoder to obtain the semantic map corresponding to the target image.

[0015] A third aspect of the present invention provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the map generation method of the first aspect of the present invention.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the map generation method of the first aspect of the present invention.

[0017] The map generation method of this invention involves acquiring a target image, processing the target image using an encoder to obtain target image features, determining corresponding target prior features from a neural map prior based on the location information of the target image, inputting the target image features and target prior features into a trained map feature generation model for feature fusion to obtain target fused features, and inputting the target fused features into a decoder to obtain a semantic map corresponding to the target image. This method proposes a neural map prior, which, during map inference, uses a trained map feature generation model to fuse current features (i.e., target image features) with previous features (corresponding target prior features in the neural map prior). This fusion of current target image features with the neural map prior improves map inference performance and enhances the prediction quality of online semantic maps. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a map generation method according to an embodiment of the present invention;

[0020] Figure 2 This is an example of an attention map after GRU fusion visualization, as shown in one embodiment of the present invention;

[0021] Figure 3This is a flowchart illustrating a map generation method according to an embodiment of the present invention;

[0022] Figure 4 This is a structural block diagram of a map generation device provided in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In fact, high-definition semantic maps have a wide range of applications and are of great significance for pedestrian and vehicle navigation. For example, high-definition (HD) semantic maps are a key component of autonomous driving on urban streets, as autonomous vehicles rely on them to predict future trajectories and navigate urban streets. Most autonomous vehicles use offline HD semantic maps, which are pre-annotated. These offline semantic maps are built using complex pipelines, including survey vehicles performing multiple LiDAR scans, global point cloud alignment, and manual annotation of map elements. While these offline mapping solutions achieve high accuracy, they are cumbersome and expensive, limiting their scalability.

[0026] As mentioned earlier, some solutions have proposed learning high-resolution semantic maps from observations by vehicle-mounted sensors. These solutions typically use deep learning methods to infer map elements in real time, effectively eliminating the map update problem. However, the high-resolution semantic map learning achieved by these methods depends on the sensor's sensing range and is easily affected by occlusion. In adverse weather conditions and under occlusion, the inferred map quality may further deteriorate, resulting in inferred map quality that is often lower than that of pre-built global offline maps, failing to meet the accuracy requirements of semantic maps.

[0027] Based on this, the present invention provides a map generation method. This method proposes a neural map prior, which is the neural network features of the map obtained through previous reasoning. A pre-trained map feature generation model is used to fuse the current image features (i.e., target image features) with the corresponding prior features in the neural map prior (i.e., target prior features) to obtain refined features (i.e., target fusion features). This is then used to further obtain the semantic map corresponding to the target object, thereby improving the online prediction quality of the semantic map. In other words, the present invention provides a hybrid mapping method that combines the best features from two timelines to further improve the learning performance of high-definition semantic maps through a pre-constructed and maintained neural map prior.

[0028] Reference Figure 1 , Figure 1 This is a flowchart illustrating a map generation method according to an embodiment of the present invention.

[0029] like Figure 1 As shown, the method includes the following steps:

[0030] Step S11: Acquire the target image and process the target image using an encoder to obtain target image features.

[0031] This embodiment first acquires a target image to infer a semantic map corresponding to the target image. The target image can be a road environment image or a street environment image; it can be a single-frame panoramic image, multiple frames of surrounding images, or an image stitched together from multiple surrounding images. Specifically, the target image can be acquired by a camera in the vehicle capable of capturing the surrounding environment, for example, images captured frame-by-frame by six onboard surround-view cameras. The vehicle can be an autonomous vehicle, an intelligent vehicle, or an ordinary vehicle. Further, this embodiment can acquire the target image in real-time and infer the semantic map corresponding to the target image using the method of this embodiment; alternatively, it can capture and save the target image, then acquire the captured target image again and infer the semantic map using the method of this embodiment. This embodiment does not impose any limitations on this approach.

[0032] This embodiment processes the acquired target image using an encoder to obtain the target image features corresponding to the target image. The encoder and decoder in this embodiment can be any encoder-decoder architecture used for map inference. For example, it could be the HDMapNet model (an online high-definition map construction and evaluation framework), the LSS model (Lift, Splat, Shoot model, which encodes images of arbitrary camera mounts through implicit non-projection to 3D), the BEVFormer model (a model that learns bird's-eye view representations from multi-lens images through spatiotemporal transformation), or the VectorMapNet model (an end-to-end vector high-definition map learning model), etc. These are all models used for online semantic map inference in related technologies. The map generation method provided in this embodiment can further improve the inference quality of semantic maps based on semantic map inference models in related technologies. In one embodiment, the target image features obtained by the encoder can be BEV features.

[0033] Step S12: Determine the corresponding target prior features from the neural map prior based on the location information corresponding to the target image.

[0034] In this embodiment, a Neural Map Prior (NMP) is pre-stored. This NMP is a neural representation of a global map, representing the neural network features of the map previously inferred using the method described in this embodiment. The NMP can be defined as an empty sparse map patch. That is, this embodiment initially creates a storage method for the sparse map patches, initially empty. Then, after obtaining the target fusion features using the method described in this embodiment, the obtained target fusion features are gradually stored as the NMP, thus obtaining the pre-stored NMP.

[0035] It is understood that the neural map prior in this embodiment is a global neural map prior, the target fusion feature is a local map feature, and the semantic map corresponding to the target image is a local semantic map. Here, "global" and "local" correspond to each other, meaning that the neural map prior in this embodiment (i.e., the global neural map prior) includes multiple "local" map features.

[0036] In this embodiment, the location information corresponding to the target image can be obtained. Then, based on the location information, the prior features corresponding to the location information (i.e., the fusion features previously inferred by this method) are determined from the neural map prior as the target prior features. In this embodiment, the neural map prior is stored using sparse map tiles. Each previously inferred fusion feature is stored in the corresponding map tile using the location information of its corresponding image as a geographic index. That is, each map tile corresponds to a location on the physical map. In other words, the image used for inference corresponds to location information. After the fusion features are obtained by inferring from the image using this method, the fusion features are stored in the form of map tiles corresponding to the location information, thus forming a neural map prior with multiple sparse map tiles.

[0037] In this embodiment, map tiles are used as the storage format for neural map priors. This is because in a city, buildings occupy most of the area, while road-related areas occupy only a small portion. To avoid increasing map storage as the physical size of the city expands, this embodiment designs a storage structure that divides the city into sparse map tiles, which are indexed by their physical coordinates. For example, in the nuScenes dataset, the top-left corner coordinates of the Boston area are (298m, 328m), and the bottom-right corner coordinates are (2527m, 1896m). Furthermore, Boston in the nuScenes dataset is a city area that is 2 km high and 1.5 km wide. If the feature dimension of the neural map prior is defined as 265 channels, and the resolution of the map prior features is 0.3m, then storing the data for Boston in the nuScenes dataset would require 38GB. Based on this, this embodiment does not store the neural map prior at every location in the city, but instead divides the city into 32x32 map tiles, each with a size of 69m x 49m. Typically, the map tiles in this embodiment are only slightly larger than the bird's-eye view (BEV) range, which is set to 60m x 30m. During onboard map inference, only relevant map tiles need to be extracted from the global map (i.e., neural map prior), especially those that overlap with the current perception range, thus eliminating the need to store map tiles that do not contain road-related information. After removing these map tiles, this embodiment only requires 12GB to store road information for an entire city, such as the Boston area.

[0038] Thus, the sparse map tile structure saves memory consumption, requiring only a small amount of disk memory per vehicle to utilize this embodiment's map tile structure. Vehicles do not need to store maps of the entire city; instead, they can download map tiles as needed. These map tiles are updated, integrated, and asynchronously uploaded to the cloud while maintaining a fixed training model. Over time, more and more trip data can be obtained, resulting in a broader and higher-quality map.

[0039] The location information corresponding to the target image can be obtained through a positioning system that provides precise positioning in the vehicle. For example, the vehicle is equipped with onboard sensors, which include cameras that capture the surrounding environment and positioning systems that provide precise positioning (such as GPS / IMU systems). Thus, the target image can be obtained through the camera, and the location information corresponding to the target image can be obtained through the positioning system.

[0040] Step S13: Input the target image features and the target prior features into the trained map feature generation model for feature fusion to obtain target fused features.

[0041] In this embodiment, after obtaining the target image features and target prior features, these features can be input into a pre-trained map feature generation model. The map feature generation model then fuses the target image features and target prior features to obtain the refined target fused features output by the map feature generation model. Specifically, the pre-trained map feature generation model in this embodiment is used to fuse the target image features and their corresponding neural map priors during local map prediction to further refine the target image features, thereby inferring a high-precision local semantic map.

[0042] Step S14: Input the target fusion features into the decoder to obtain the semantic map corresponding to the target image.

[0043] In this embodiment, after obtaining the target fusion features output by the map feature generation model, the target fusion features can be input into the decoder to obtain the local semantic map output by the decoder, that is, to obtain the semantic map corresponding to the target image, thereby completing the prediction of the online local map.

[0044] In this embodiment, a neural map prior is proposed for online semantic map prediction. This prior considers not only the acquired current features (i.e., target image features) but also previous features (corresponding target prior features in the neural map prior). A simple and efficient map feature generation model then fuses the current features with previous features for refined processing. The neural map prior, collected from different vehicles at different time points, is compatible with the target image features processed by the encoder in relevant map inference techniques, thereby improving map inference performance and enhancing the prediction quality of online semantic maps. Furthermore, this embodiment can utilize reliable information from the neural map prior to avoid the impact of severe weather on online local map inference, resulting in significantly improved map inference results under adverse weather conditions.

[0045] In conjunction with the above embodiments, in one implementation, the present invention also provides a map generation method, which may further include the step of: replacing the target prior features in the neural map prior with the target fusion features.

[0046] In this embodiment, after obtaining the target fusion feature, the target fusion feature can not only be processed by the decoder to obtain the semantic map corresponding to the target image, but it can also be used to update the neural map prior. Specifically, the size of the target image feature is the same as the size of the target prior feature, and the size of the fused target fusion feature is also the same as the size of the target prior feature. The neural map prior is updated after each acquisition of the target fusion feature. In particular, after obtaining the target fusion feature, the target prior feature in the neural map prior can be replaced with the target fusion feature, thereby updating the neural map prior.

[0047] In this embodiment, the target fusion features output by the map feature generation model can be used for updating the global neural map prior and reasoning the local map, so that the effect of local map reasoning and the expression quality of global neural map prior can be mutually improved. As a result, the quality of the predicted local semantic map is better and the global neural map prior is more complete and updated in a timely manner as the vehicle passes through more scenes.

[0048] In conjunction with the above embodiments, in one implementation, the present invention also provides a map generation method, which further includes a method for training a map feature generation model. Specifically, the training steps of the map feature generation model may include the following steps:

[0049] Step A: Acquire a sample image, and process the sample image using the encoder to obtain sample image features.

[0050] In this embodiment, sample images can be obtained from a sample library or database. For example, the database could be the nuScenes dataset, a large-scale autonomous driving dataset covering various weather conditions, traffic conditions, and different times of day. It includes multiple traversals, precise positioning, and high-definition map semantic labels with annotations. The dataset could also include external camera parameters and the transformation relationship between the autonomous vehicle and the global coordinate system.

[0051] After obtaining the sample images, image processing can be performed on them using an encoder, such as extracting BEV features, to obtain the sample image features corresponding to the sample images. These sample image features can be BEV features. In this embodiment, the encoder-decoder structure used during training is the same as the structure used in application. It can be any encoder-decoder architecture used for map inference, such as the HDMapNet model, the LSS model, the BEVFormer model, or the VectorMapNet model, etc.

[0052] Step B: Based on the location information corresponding to the sample image, determine the corresponding sample prior features from the neural map prior.

[0053] In this embodiment, prior features corresponding to the location information of the sample image can be determined from the neural map prior to serve as the sample prior features. The method for generating the neural map prior during training is the same as the method for generating it during application. It can be understood that during training, the neural map prior is generated from scratch and assists in the training of the fusion module (i.e., the initial model). During application, the neural map prior is generated from scratch and gradually filled and updated through the fusion module (i.e., the initial model) to help improve online map prediction.

[0054] Step C: Input the sample image features and the sample prior features into the initial model for feature fusion to obtain sample fusion features; the sample fusion features are used to determine the semantic map corresponding to the sample image and update the neural map prior.

[0055] This embodiment constructs an initial model for feature fusion of sample image features and sample prior features. Then, this embodiment outputs the sample image features and sample prior features to the initial model for feature fusion to obtain the sample fused features output by the initial model. These sample fused features can be used to determine the semantic map corresponding to the sample image and to update the neural map prior, i.e., to perform local map inference and update the global neural map prior. Specifically, the local map inference process can be run directly on the test vehicle, fusing onboard sensor observations and the global map prior for local map inference, or it can be performed using sample data from the acquired dataset. Conversely, local map inference can update the map prior through attention operations. These two processes form a cycle and can be improved by collecting large amounts of data (such as data collected daily from a large number of vehicles on driving roads).

[0056] Step D: Train the initial model based on the sample fusion features, and determine the trained initial model as the map feature generation model.

[0057] In this embodiment, after the initial model outputs sample fusion features, the initial model can be trained based on the sample fusion features, thereby determining the trained initial model as a map feature generation model.

[0058] For example, in one embodiment, a BEV encoder-decoder architecture is employed, with the BEV encoder referred to as f E The decoder is called f D The neural map prior (i.e., the global neural map prior) is denoted as Among them, H G and W G These represent the height and width of the city, respectively. First, a set of observations is derived from sample image I and its corresponding location information. (i.e., the vehicle position in the global coordinate system) is used, and in this embodiment, a set of observations is a set of training data. In this embodiment, G can be used. ego The local coordinates of each pixel of BEV ( Where H and W represent the dimensions of the BEV feature, this is converted to a fixed global coordinate system, i.e. First, we can obtain the features of the sample images (i.e., online BEV features). Where C represents the hidden embedding size of the network, and then the vehicle position P is used. ego Query global prior P g To obtain the sample prior features (i.e., local prior BEV features). Subsequently, the fusion function (i.e., the initial model) is applied to obtain the sample fusion features (i.e., the refined BEV features): denoted as F. refine =f Fusion (O, P) l ),in Finally, the refined BEV features are processed by the decoder f D Decode to the final map output, while using F refine Update global map prior P g The entire process runs continuously, integrating diverse and complementary observations from different periods over time to complete the training of the map feature generation model.

[0059] In this embodiment, a neural map prior is proposed for training the map feature generation model. This prior combines the maintenance of offline global neural map priors with online local map inference. The computational cost and memory consumption for local inference using the method in this embodiment are similar to those of single-frame systems in related technologies, but it achieves better technical results: improving the quality of local map inference.

[0060] In conjunction with the above embodiments, the present invention also provides a map generation method, in which the initial model includes: an attention mechanism module and a gated loop unit; and step C above may further include the following steps:

[0061] Step C1: Input the sample image features and the sample prior features into the attention mechanism module to obtain intermediate fusion features.

[0062] In this embodiment, the prior features in the proposed Neural Map Prior (NMP) provide strong prior information for online map inference. A fusion function is then needed to aggregate the prior and current features. Typically, join operations are widely used for feature aggregation; it is a symmetric function where prior and current features are considered equally important. However, in the map problem of this embodiment, road conditions may change significantly between traversals, meaning that current and prior features may have different importance. Therefore, this embodiment uses an asymmetric fusion function composed of an attention mechanism module and a variant of a gated recurrent unit to construct the initial model.

[0063] The attention mechanism module in the initial model of this embodiment is mainly used to dynamically capture the correlation between the current feature and the previous feature (i.e., prior feature) for feature fusion. Specifically, the attention mechanism module in this embodiment can be a current-to-previous cross-attention module (C2P attention). In this embodiment, sample image features and sample prior features are input to the attention mechanism module, which then performs feature fusion processing on the current and previous features to obtain the intermediate fused features output by the attention mechanism module.

[0064] Step C2: The intermediate fusion features and the sample prior features are fused through the gated loop unit to obtain the sample fusion features.

[0065] Regarding the updating of neural map priors, if the update speed is too fast, the neural map prior may be easily affected by some poor local observations; if the update speed is too slow, the neural map prior may fail to capture changes in road conditions in time. Based on this, this embodiment considers controlling the update speed of the neural map prior and uses a gated recurrent unit (GRU) to fuse the intermediate fused features output by the attention mechanism module with the sample prior features to balance the ratio between the newly generated intermediate fused features and the sample prior features (i.e., the old map prior), thereby obtaining the final sample fused features.

[0066] For example, a gated recurrent unit using a 2D convolutional variant could be used to balance the ratio of updates to forgetting. The intermediate fused features output by the attention mechanism module could be O′, the local map prior features updated at time t-1. (i.e., sample prior features or target prior features) are derived from the global neural map prior. Extracted from (i.e., neural map priors). The gated recurrent unit (GRU) compares O′ with the local prior features updated at time t-1. Fusion yields new prior features at time t. (i.e., sample fusion features or target fusion features), and predict the local semantic map through the decoder. Then, update the global neural map prior at the corresponding location by direct replacement. This means replacing the sample prior features or target prior features in the neural map prior with newly generated sample fusion features or target fusion features.

[0067] Specifically, the gated recurrent unit (GRU) can use the following operation to combine O′ with the previous feature. Fusion:

[0068]

[0069] Among them, z t Represented as an update gate, r t This is represented as the reset gate or forget gate, σ represents the Sigmoid function, and W* represents the weights of the 2D convolution (as shown in equation (1)). z W r W h (All are weights), and the operator ⊙ represents the Hadamard product. That is, the update gate z in GRU... t And the Gate of Oblivion t The determination is based on the previous traversal (i.e., previous features). The amount of information fused into the current BEV feature O′ and the amount of information fused into the global map prior features are considered. As a data-driven method, GRU in this embodiment serves as a selective attention mechanism, replacing some manually crafted linear update rules to achieve better results.

[0070] Furthermore, the final step in the GRU fusion process is as follows:

[0071]

[0072] Here, z can be understood as t It is a learnable parameter. Where H and W represent the height and width of the BEV feature, respectively. Local map prior features are called... The current feature is called It can be observed that when the prediction quality of the current frame is good, the network tends to learn a larger z-value. tThis gives greater weight to the current feature. When the prediction quality of the current frame is poor, typically at an intersection or far from the car, the network tends to learn a larger 1-z value. t This allows prior features to be given greater weight. Thus, the gated recurrent units in the initial model of this embodiment can learn to selectively combine features from the current and previous frames, thereby better controlling the update speed of the neural map prior.

[0073] like Figure 2 As shown, Figure 2 This is an example of a GRU fusion visualization attention map shown in one embodiment of the present invention. Figure 2 As shown, rows one through five represent: the ground truth map, the map inferred from the HDMapNet model, the map inferred from the BEVFormer model, and the map inferred from the BEVFormer model using the NMP neural map prior (i.e., the map generation method using NMP proposed in the above embodiments) and GRU weights, respectively. Figure 2 As can be seen from this, compared with the baseline method, the map generation method proposed in this embodiment can generate a more accurate and coherent semantic map by using the Neural Map Prior (NMP).

[0074] In conjunction with the above embodiments, in an optional embodiment, step C1 may specifically include the following steps:

[0075] Step C1-1: Divide the sample image features and the sample prior features into multiple blocks to obtain multiple sample image sub-features and multiple sample prior sub-features.

[0076] In this embodiment, after obtaining the sample image features and sample prior features, the sample image features and sample prior features can be divided into multiple small blocks, thereby obtaining multiple sample image sub-features and multiple sample prior sub-features. For example, 10x10 blocks can be used to represent a 3m x 3m region in BEV, thereby saving parameters (i.e., saving computational resources) while preserving local spatial information.

[0077] Step C1-2: After the multiple sample image sub-features and multiple sample prior sub-features enter the first linear layer, each of the sample image sub-features is labeled as a sample image sub-feature, and each of the sample prior sub-features is labeled as a sample prior sub-feature.

[0078] In this embodiment, after obtaining multiple sample image sub-features and multiple sample prior sub-features, these features enter the first linear layer of the attention mechanism module. This first linear layer can be a fully connected layer. After the multiple sample image sub-features and multiple sample prior sub-features enter the first linear layer, each block (i.e., each sub-feature) is treated as a label using the first linear layer. Specifically, each sample image sub-feature is treated as a sample image sub-feature label, and each sample prior sub-feature is treated as a sample prior sub-feature label.

[0079] Step C1-3: Use the sample image sub-feature label as the query, and the sample prior sub-feature label as the key and value. Perform calculations based on the query, key, and value to obtain the calculation result.

[0080] In this embodiment, the sample image sub-feature labels can be used as the query Q, and the sample prior sub-feature labels can be used as the key and value. Then, attention operations are performed based on the query, key, and value to obtain the result. Specifically, the query Q and the key can be operated on, and then multiplied by the value to obtain the final attention operation result.

[0081] Step C1-4: Input the calculation result into the second linear layer to obtain the intermediate fusion feature output by the second linear layer.

[0082] In this embodiment, after obtaining the calculation result, i.e., the attention-processed features, the calculation result (i.e., the attention-processed features) is input into the second linear layer, thereby obtaining the intermediate fused features output by the second linear layer, which is also the output of the entire attention mechanism module. The output features have the same size as the input features, that is, the size of the intermediate fused features is the same as the size of the sample image features and the size of the sample prior features, and the quality of the intermediate fused features (i.e., the optimized BEV features) is better than that of the prior features and the current features.

[0083] The second linear layer can also be a fully connected layer. In an optional embodiment, all linear layers in the attention mechanism module are embedded using a 256-dimensional feature dimension, meaning that both the first and second linear layers are fully connected layers with 256 filters.

[0084] In conjunction with the above embodiments, the present invention also provides a map generation method, wherein, prior to step C1 above, the training step of the map feature generation model may further include:

[0085] C0: Add corresponding positional codes to the sample image features and the sample prior features respectively to obtain intermediate sample image features and intermediate sample prior features.

[0086] Considering that the accuracy of the predicted map decreases as the location moves further away from the autonomous vehicle, in order to enable the initial model to recognize the influence of location and thus learn to trust current features where the location is closer to the vehicle, while trusting previous features where the location is farther away from the vehicle, this embodiment inputs sample image features and sample prior features into the initial model (e.g., f). Fusion Before that, the sample image features and sample prior features are preprocessed by position encoding.

[0087] In this embodiment, corresponding positional encodings are added to the sample image features and sample prior features, respectively, to obtain intermediate sample image features and intermediate sample prior features. The positional encoding is a learnable variable with a grid shape, used to allow the initial model to trust current features based on spatial location (e.g., closer to the car, trusting previous features if farther away). Specifically, a set of positional encodings (network-like learnable parameters) can be added to the sample prior features. Additionally, a set of location codes is added to the features of the sample images. Here, H and W represent the height and width of the BEV feature, respectively.

[0088] In this method, step C1 above may specifically include: inputting the intermediate sample image features and the intermediate sample prior features into the attention mechanism module to obtain the intermediate fusion features.

[0089] In this embodiment, after obtaining the intermediate sample image features and intermediate sample prior features, the intermediate sample image features and intermediate sample prior features can be input into the attention mechanism module for processing, thereby obtaining the intermediate fusion features output by the attention mechanism module.

[0090] In one embodiment, after adding positional encoding to the sample image features and sample prior features to obtain intermediate sample image features and intermediate sample prior features, if the attention mechanism module needs to divide the previous features and current features into multiple blocks, the intermediate sample image features and intermediate sample prior features are divided into multiple blocks respectively, thereby obtaining multiple sample image sub-features and multiple sample prior sub-features.

[0091] To illustrate the effectiveness of the position encoding, attention mechanism module, and gated recurrent unit in this embodiment, a simple fusion baseline, namely moving average (MA), is used for comparison. For MA, in a set of experiments, the moving average (MA) is used as the fusion function, i.e., the initial model, to replace the attention mechanism module and the gated recurrent unit. The update rule for MA can be:

[0092]

[0093] In Equation 3, α represents the ratio of manual searches. Let be the new prior features of the local map at time t. The local map prior features are updated at time t-1. As shown in Table 1 below, mIoU is the average intersection and union, Divider is the lane divider, Crossing is the pedestrian crossing, Boundary is the road boundary, and ALL is the total. The attention mechanism module C2P Attention, position encoding PE, and gated recurrent unit GRU proposed in this embodiment are all key to improving the performance of online map prediction. In particular, GRU and MA are used as update modules and have achieved similar performance improvements. This embodiment chooses to use GRU to avoid manual parameter search for MA. Comparing C to E and F to G in Table 1 below, local PE improves the mIoU of intersections by 2.67 and 2.72, respectively, indicating that local PE is beneficial for feature fusion, especially for intersections, which is also the most challenging category for single-frame model prediction. Adding local PE allows the model to extract more robust information from the neural map prior to supplement the information missing in the current observation. Comparing C to F and E to G in the table below, we can see that the mIoU of the lane segmentation lines increases by 1.83 and 2.05 respectively with CA (i.e., C2P Attention). This indicates that CA can better handle lane structure. Research shows that the three components proposed in this embodiment (position encoding, attention mechanism module, and gated recurrent unit) are effective for feature fusion and updating.

[0094]

[0095] Table 1

[0096] In another set of experiments, the method provided in this embodiment was applied to HDMapNet, LSS, BEVFormer, and VectorMapNet models to evaluate the effectiveness of the map generation method provided in this embodiment. During training, all modules before obtaining online BEV features were frozen, and only the C2P Attention module, local PE, GRU, and decoder were trained. During testing, all samples were sorted in chronological order. The experiments were conducted on eight NVIDIA 3090 GPUs with an input image size of 1600x900 and a batch size of 1. As shown in Tables 2 and 3 below, the results demonstrate that the NMP (i.e., the proposed map generation method) proposed in this embodiment consistently improves map segmentation and detection performance across all baseline models. In Tables 2 and 3, mIoU represents the average intersection and union, Divider represents lane dividers, Crossing represents pedestrian crossings, Boundary represents road boundaries, and ALL represents totals. These results indicate that the NMP provided in this embodiment is a general method that may be applied to other map learning frameworks.

[0097]

[0098] Table 2

[0099]

[0100] Table 3

[0101] In conjunction with the above embodiments, in an optional embodiment, the present invention also proposes a map generation method, in which the training step may further include:

[0102] Step E: Divide the dataset of the target region into training and test sets.

[0103] This embodiment takes into account the poor generalization of current online map inference. In order to train a map feature generation model that can still achieve good inference performance even when the training set and test set are distributed in different cities, this embodiment re-divides the dataset of the target area to obtain the training set and test set. The target area can be any region, such as any city, region, etc. For example, it can be a subset of the nuScenes dataset (such as the dataset of the Boston area) re-divides to obtain the training set and test set.

[0104] The improvement of neural map priors for online map inference is attributed to generating neural priors from other journeys. These priors provide closer observations and complementary perspectives, enabling the current observation to "see further" or bypass obstacles. Data lacking historical journey observations cannot benefit from the improvements of neural network priors. Therefore, this embodiment repartitions the dataset of the target region (e.g., a dataset of the Boston area) so that each training and test sample has a past journey, resulting in training and test sets. Specifically, the partitioned training set includes training sample images corresponding to neural map priors, and the test set includes test sample images corresponding to neural map priors, with the acquisition locations of the training and test sets being geographically disjoint.

[0105] Step F: Test the map feature generation model trained on the training set according to the test set, and obtain the test results.

[0106] In this embodiment, the initial model is trained using sample data from the training set to obtain a trained map feature generation model. Then, the map feature generation model trained on the training set is tested using data from the test set to obtain test results. The map feature generation model is then retrained based on the test results, and the test is repeated until a map feature generation model that meets the conditions is trained. In this way, the trained map feature generation model can achieve good results in other cities, avoiding geographical limitations as much as possible, thereby alleviating the problem of poor generalization of map learning to a certain extent.

[0107] In this embodiment, by re-dividing the dataset of the target region and accurately calculating the overlap between historical frames and the current frame, the historical journey corresponding to the current frame can be found more accurately. This embodiment uses the mean intersection and union (mIoU) to evaluate the quality of high-definition semantic learning to evaluate the following three static map factors: road boundaries, lane dividers, and pedestrian crossings. As shown in Table 4 below, Divider is the lane divider, Crossing is the pedestrian crossing, Boundary is the road boundary, ALL is the total, Boston Split is the segmentation of the dataset for the target region (e.g., segmentation of the dataset for the Boston area), Original Split is the ordinary split, i.e., the segmentation of the normal dataset in the original method, and NMP is the neural map prior, indicating that the method of this embodiment is used. It can be seen that the baseline result of the segmentation of the dataset for the target region proposed by this method is lower than that of the original split (i.e., ordinary split), and the improvement of the neural map prior in the segmentation of the dataset for the target region is greater than the improvement in the original split.

[0108]

[0109] Table 4

[0110] In one alternative implementation, for the NMP hyperparameters of this embodiment, the rasterized neural map prior can be defaulted to a quantization size of 0.3m, as shown in Table 5 below, where mIoU is the mean intersection and union, Divider is the lane divider, Crossing is the pedestrian crossing, Boundary is the road boundary, ALL is the total, NMP Grid Resolution is the NMP resolution, and Baseline is the baseline. The quantization size of the neural map prior was investigated, with a dimension of 256. Table 5 examines the impact of different resolutions of the global neural map prior, i.e., the neural map prior, on the effectiveness of online map learning. Intuitively, the processed road information can be considered as a small object, therefore, the prior information should be stored at a fine resolution to clearly indicate whether it is a road. Therefore, a smaller spatial quantization size is preferred. However, extremely small resolution means that the storage space required for the neural map prior increases quadratically and is susceptible to random errors in localization. Therefore, the choice of quantization size is a trade-off between smaller storage and higher accuracy. Research shows that optimal performance is achieved with an appropriate quantization size (0.3m).

[0111]

[0112] Table 5

[0113] In an alternative implementation, please refer to Figure 3 , Figure 3 This is a flowchart illustrating a map generation method according to an embodiment of the present invention. Figure 3 As shown, this can be represented as either the training process of the map feature generation module or the application process of the map feature generation module.

[0114] During the training process, it first passes through the encoder f E The sample image is processed to obtain the current BEV feature O (i.e., sample image feature); and the updated neural map prior (the selected map tile) at time t-1 is retrieved from the map tile memory. Then, based on the location information of the sample image, that is, the current vehicle position Pos ego from Medium-sampled map tiles form corresponding sample prior BEV features Then, the sample image features O and the sample prior features are used. The inputs are fed into the initial model (i.e., the fusion function in the diagram) for training. Specifically, these are the sample image features O and the sample prior features, respectively. Add positional encoding (PE) to the feature O of the sample image. c , for sample prior features Add location encoding PE p Then add position-coded PE c The feature O is divided into multiple small blocks to obtain multiple sample image sub-features, and position-encoded PE is added. p Prior features of the sample The image is divided into multiple small blocks to obtain multiple sample prior sub-features. Each sub-feature is converted into a label. The sample image sub-feature label is used as the query, and the sample prior sub-feature label is used as the key and value. Standard cross-attention (C2P Attention) processing is performed to obtain intermediate fused features. Then, the intermediate fused features are combined with the sample prior features. The initial model is processed using GRU to obtain its final output F. refine , that is Therefore, according to F refine via decoder f D Inferring semantic maps, and based on The neural map prior is replaced with the corresponding location, i.e., based on features. Update map tiles to achieve the update of neural map priors.

[0115] During the application process, it first passes through the encoder f E The target image is processed to obtain the current BEV feature O (i.e., target image feature); and the neural map prior updated at time t-1 (the selected map tile) is retrieved from the map tile memory. Then, based on the location information of the target image, that is, the current vehicle position Pos ego from Medium-sampled map tiles form corresponding target prior BEV features Then, the target image features O and the target prior features are used. The inputs are fed into the map feature generation model (i.e., the fusion function in the diagram) for processing. Specifically, this involves processing the target image features O and the target prior features. Add positional encoding (PE) to the target image feature O. c , as the prior features of the target Add location encoding PE p Then add position-coded PE c The feature O is divided into multiple small blocks to obtain multiple target image sub-features, and position-encoded PE is added. p Target prior features The image is divided into multiple small blocks to obtain multiple target prior sub-features. Each sub-feature is converted into a label. The target image sub-feature label is used as the query, and the target prior sub-feature is used as the key and value. Standard cross-attention (C2P Attention) processing is performed to obtain the target intermediate fusion feature. Then, the target intermediate fusion feature is combined with the target prior feature. The GRU is used for processing to obtain the final output F of the map feature generation model. refine , that is Therefore, according to F refine via decoder f D Inferring semantic maps, and based on The neural map prior is replaced with the corresponding location, i.e., based on features. Update map tiles to achieve the update of neural map priors.

[0116] Specifically, in Figure 3 In the top row, an online high-definition semantic map is learned, taking the image as input and the map segmentation result as output, through a BEV encoder and decoder. To better utilize the neural map prior, a custom fusion module, namely the map feature generation model or initial model (C2P Attention and GRU) in this embodiment, is added between the encoder and decoder—the fusion output is decoded by the decoder to obtain the final map output. In the bottom row, the neural prior map overlapping with the current BEV features is extracted from the storage disk, and then the prior features with the same position as the current frame are cropped. After updating, the previous neural prior map is put back into the extracted prior map.

[0117]

[0118] Table 6

[0119] Regarding the above embodiments, a study in one embodiment demonstrates that the neural map prior proposed in the above embodiments can help see further. Specifically, one of the traditional functions of maps is to provide road information beyond the line of sight (beyond the horizon), which is crucial for downstream navigation and planning and helps in making informed decisions. The neural map prior in the embodiments also supports achieving this fundamental purpose by allowing in-vehicle maps to infer further. As shown in Table 6 above, our proposed neural map prior method can uniformly improve the segmentation results of the baseline method map when the BEV range of the original baseline method is 60m×30m, 100m×100m, and 160m×100m.

[0120] In Table 6, mIoU represents the average intersection and union, Divider represents lane dividers, Crossing represents pedestrian crossings, Boundary represents road boundaries, ALL represents the total, and BEV Range represents the BEV range. Camera-based map segmentation and detection is often considered challenging in map portions furthest from the vehicle, as these portions occupy only a few pixels in the image. Therefore, incorporating historical priors of the scene is crucial for improving map segmentation and detection performance. As shown in the table, the method in the above embodiments has the advantage of enhancing long-range perception results, which is difficult to achieve with single-frame methods. Experimental results show that performance tends to decrease with decreasing distance, but our method still significantly improves the results.

[0121] Regarding the above embodiments, one study demonstrates that multi-trip fusion is better than intra-trip fusion in these embodiments. Table 7 below analyzes the importance of intra-trip information (intra-trip fusion) and inter-trip information (multi-trip fusion). In Table 7, mIoU represents the mean intersection and union, Divider represents lane dividers, Crossing represents pedestrian crossings, Boundary represents road boundaries, ALL represents totals, Intra-trip fusion represents intra-trip fusion, and Inter-trip fusion represents multi-trip fusion. Specifically, intra-trip information means that the available neural map prior is limited to a single current trip. Conversely, the inter-trip information model uses map priors generated from arbitrary trips from the same location. The results indicate that prior information from multiple trips is more important for map construction because the intra-trip model performs significantly worse than the inter-trip model.

[0122]

[0123] Table 7

[0124] Furthermore, the map generation method involving Neural Map Prior (NMP) provided in the above embodiments is more helpful for map inference under adverse weather conditions. This is because driving a car (whether autonomous, intelligent, or normal) inevitably faces challenges in adverse weather conditions, such as rain or nighttime driving, which can make it difficult for the vehicle to accurately identify road information. However, neural map priors obtained under better weather and lighting conditions can provide more reliable information, enabling the vehicle to perceive road information more accurately and drive safely in adverse weather conditions.

[0125] As shown in Table 8 below, using neural map priors on rainy days and at night achieves more significant improvements than under normal weather conditions. This indicates that the model in the above embodiments can effectively extract the necessary information from NMP to cope with severe weather scenarios. However, due to limited map prior information and a small sample size, the improvement is smaller on rainy days at night, as can be seen from the table.

[0126]

[0127] Table 8

[0128]

[0129] Table 9

[0130] In one embodiment, where the selected attention mechanism module is the C2P Attention (Current-to-prior attention) module, the C2P Attention module uses Crossattention technology, and the window size of the Crossattention module can be freely set. That is, for the design of C2P Attention, the road structure should maintain spatial continuity, which is also confirmed in Table 9 above.

[0131] As shown in Table 9, the performance of C2P Attention on dividing lines and boundaries improves with increasing window size. However, excessively large window sizes may introduce irrelevant information from adjacent lanes, and the number of parameters also increases significantly. Therefore, the choice of window size is a balance between capturing relevant road structure and excluding irrelevant spatial information. Based on this, this embodiment selected the optimal window size of 3m x 3m according to experimental performance.

[0132] In another implementation, research showed that the map generation method proposed in this embodiment only increased the model performance by 23ms per frame compared to the original model of the related technology, but significantly improved the model performance, including extracting and storing features from neural map priors and integrating other modules.

[0133] Building upon the above embodiments, one embodiment proposes a novel neural map prior system to aid in online high-definition semantic map learning. The key idea is to incrementally combine local map inference and global map prior updates frame-by-frame using C2PAttention and GRU. This design enables the neural map prior to output accurate and coherent global map priors, facilitating online semantic map learning. Furthermore, the neural map prior is compatible with state-of-the-art map segmentation / detection architectures, improves map prediction performance under adverse weather conditions, and enhances map prediction at a distance from the current location. The global map prior reconstructed by the fusion module can be directly used for downstream tasks such as planning and control. Based on this, through end-to-end joint training with downstream tasks, the neural map prior can unlock new possibilities for learning-based autonomous driving perception and recognition systems.

[0134] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0135] Based on the same inventive concept, one embodiment of the present invention provides a map generation device 400. (See reference...) Figure 4 , Figure 4 This is a structural block diagram of a map generation device provided in an embodiment of the present invention. Figure 4 As shown, the device 400 includes:

[0136] The image feature determination module 401 is used to acquire a target image and process the target image through an encoder to obtain target image features;

[0137] The prior feature determination module 402 is used to determine the corresponding target prior features from the neural map prior based on the location information corresponding to the target image;

[0138] The feature fusion determination module 403 is used to input the target image features and the target prior features into a trained map feature generation model to perform feature fusion and obtain target fusion features;

[0139] The map determination module 404 is used to input the target fusion features into the decoder to obtain the semantic map corresponding to the target image.

[0140] Optionally, the size of the target fusion feature is the same as the size of the target prior feature, and the neural map prior is updated after each acquisition of the target fusion feature; the device 400 further includes:

[0141] An update module is used to replace the target prior features in the neural map prior with the target fusion features.

[0142] Optionally, the map feature generation model is trained by a model generation module, which includes:

[0143] The sample image feature determination module is used to acquire sample images and process the sample images through the encoder to obtain sample image features;

[0144] The sample prior feature determination module is used to determine the corresponding sample prior features from the neural map prior based on the location information corresponding to the sample image;

[0145] The sample fusion feature determination module is used to input the sample image features and the sample prior features into the initial model for feature fusion to obtain sample fusion features; the sample fusion features are used to determine the semantic map corresponding to the sample image and update the neural map prior.

[0146] The model training module is used to train the initial model based on the sample fusion features, and to determine the trained initial model as the map feature generation model.

[0147] Optionally, the initial model includes: an attention mechanism module and a gated recurrent unit; the sample fusion feature determination module includes:

[0148] The first fusion module is used to input the sample image features and the sample prior features into the attention mechanism module to obtain intermediate fusion features;

[0149] The second fusion module is used to perform feature fusion between the intermediate fusion features and the sample prior features through the gated loop unit to obtain the sample fusion features.

[0150] Optionally, the model generation module further includes:

[0151] The position encoding module is used to add corresponding position encodings to the sample image features and the sample prior features respectively before inputting the sample image features and the sample prior features into the attention mechanism module to obtain intermediate fusion features, so as to obtain intermediate sample image features and intermediate sample prior features;

[0152] The first fusion module includes:

[0153] The first fusion submodule is used to input the intermediate sample image features and the intermediate sample prior features into the attention mechanism module to obtain the intermediate fusion features.

[0154] Optionally, the first fusion module includes:

[0155] The feature segmentation module is used to divide the sample image features and the sample prior features into multiple blocks respectively, to obtain multiple sample image sub-features and multiple sample prior sub-features;

[0156] The conversion module is used to, after the multiple sample image sub-features and multiple sample prior sub-features enter the first linear layer, label each of the sample image sub-features as a sample image sub-feature and label each of the sample prior sub-features as a sample prior sub-feature.

[0157] The calculation module is used to take the sample image sub-feature label as a query, the sample prior sub-feature label as a key and value, and perform calculations based on the query, key and value to obtain the calculation result;

[0158] The second fusion submodule is used to input the calculation result into the second linear layer to obtain the intermediate fusion feature output by the second linear layer.

[0159] Optionally, the model generation module further includes:

[0160] The dataset partitioning module is used to partition the dataset of the target region to obtain a training set and a test set; the training set includes training sample images, and the test set includes test sample images, wherein the collection locations of the training set and the collection locations of the test set do not intersect geographically.

[0161] The testing module is used to test the map feature generation model trained on the training set according to the test set, and obtain the test results.

[0162] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the map generation method as described in any of the above embodiments of the present invention.

[0163] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 500, such as... Figure 5 As shown. Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 502, a processor 501, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the map generation method described in any of the above embodiments of the present invention.

[0164] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0165] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0171] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0172] The above provides a detailed description of the map generation method, apparatus, electronic device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A map generation method, characterized in that, The method includes: The target image is acquired, and the target image is processed by an encoder to obtain the target image features; Based on the location information corresponding to the target image, the corresponding target prior features are determined from the neural map prior. The neural map prior is stored through sparse map tiles. Each previously inferred fusion feature is stored in the corresponding map tile using the location information of its corresponding image as a geographic index. The target image features and the target prior features are input into a trained map feature generation model for feature fusion to obtain target fused features. The map feature generation model includes an attention mechanism module and a gated loop unit. The attention mechanism module is used to dynamically capture the correlation between the current features and prior features for feature fusion. The gated loop unit is used to fuse the intermediate fused features output by the attention mechanism module with the prior features to balance the ratio between the newly generated intermediate fused features and the prior features. The target fusion features are input into the decoder to obtain the semantic map corresponding to the target image.

2. The map generation method according to claim 1, characterized in that, The size of the target fusion feature is the same as the size of the target prior feature, and the neural map prior is updated after each acquisition of the target fusion feature; the method further includes: Replace the target prior features in the neural map prior with the target fusion features.

3. The map generation method according to claim 1 or 2, characterized in that, The training steps of the map feature generation model include: A sample image is acquired, and the sample image is processed by the encoder to obtain sample image features; Based on the location information corresponding to the sample image, the corresponding sample prior features are determined from the neural map prior. The sample image features and the sample prior features are input into the initial model for feature fusion to obtain sample fusion features; the sample fusion features are used to determine the semantic map corresponding to the sample image and update the neural map prior. The initial model is trained based on the sample fusion features, and the trained initial model is determined as the map feature generation model.

4. The map generation method according to claim 3, characterized in that, The initial model includes an attention mechanism module and a gated recurrent unit; the step of inputting the sample image features and the sample prior features into the initial model for feature fusion to obtain sample fused features includes: The sample image features and the sample prior features are input into the attention mechanism module to obtain intermediate fused features; The intermediate fusion features and the sample prior features are fused by the gated loop unit to obtain the sample fusion features.

5. The map generation method according to claim 4, characterized in that, Before inputting the sample image features and the sample prior features into the attention mechanism module to obtain intermediate fused features, the training step further includes: Add corresponding positional codes to the sample image features and the sample prior features respectively to obtain intermediate sample image features and intermediate sample prior features; The step of inputting the sample image features and the sample prior features into the attention mechanism module to obtain intermediate fusion features includes: The intermediate sample image features and the intermediate sample prior features are input into the attention mechanism module to obtain the intermediate fusion features.

6. The map generation method according to claim 4, characterized in that, The step of inputting the sample image features and the sample prior features into the attention mechanism module to obtain intermediate fusion features includes: The sample image features and the sample prior features are divided into multiple blocks to obtain multiple sample image sub-features and multiple sample prior sub-features; After the multiple sample image sub-features and multiple sample prior sub-features enter the first linear layer, each of the sample image sub-features is labeled as a sample image sub-feature, and each of the sample prior sub-features is labeled as a sample prior sub-feature. The sample image sub-features are used as queries, and the sample prior sub-features are used as keys and values. Calculations are performed based on the queries, keys, and values ​​to obtain the calculation results. The calculation result is input into the second linear layer to obtain the intermediate fusion feature output by the second linear layer.

7. The map generation method according to claim 3, characterized in that, The training steps also include: The dataset of the target region is divided into a training set and a test set; the training set includes training sample images, and the test set includes test sample images, wherein the collection locations of the training set and the collection locations of the test set do not intersect geographically. The map feature generation model trained on the training set is tested based on the test set, and the test results are obtained.

8. A map generation device, characterized in that, The device includes: The image feature determination module is used to acquire a target image and process the target image through an encoder to obtain target image features; The prior feature determination module is used to determine the corresponding target prior features from the neural map prior based on the location information corresponding to the target image. The neural map prior is stored through sparse map tiles. Each previously inferred fusion feature is stored in the corresponding map tile as a geographic index through the location information of its corresponding image. The feature fusion determination module is used to input the target image features and the target prior features into a trained map feature generation model for feature fusion to obtain target fused features. The map feature generation model is obtained by training an initial model, which includes an attention mechanism module and a gated loop unit. The map determination module is used to input the target fusion features into the decoder to obtain the semantic map corresponding to the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the map generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the map generation method as described in any one of claims 1 to 7.

Images