Model training method, map data fusion method and related equipment

Abstract

The invention relates to a model training method, a map data fusion method and related equipment, and relates to the technical field of automatic driving. The model training method comprises the following steps: acquiring at least two local vector maps and global map data; extracting the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements; performing feature extraction on the category data and the coordinate data by using the initial map data fusion model to obtain vector map features; performing rasterization processing based on the coordinate data to obtain raster map features; performing feature fusion and feature extraction on the vector map features and the grid map features to obtain extracted map features; decoding and predicting the extracted map features to obtain global map prediction data; and training the initial map data fusion model based on the global map prediction data and the global map data to obtain a map data fusion model. According to the invention, an accurate fusion result can be obtained, and the fusion efficiency is high.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a model training method, a map data fusion method, and related equipment. Background Art

[0002] With the rapid development of autonomous driving technology, high-precision maps have become an indispensable part of the autonomous driving system.

[0003] In the related art, crowdsourced map data has become an important source of high-precision map data due to its advantages of fast update speed and wide coverage. However, due to the problems of diverse data sources and uneven data quality, the fused crowdsourced map data has low flexibility, poor accuracy, and low fusion calculation efficiency.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides a model training method, a map data fusion method and related equipment.

[0006] According to a first aspect of an embodiment of the present disclosure, a model training method is provided, including: Acquire a training data set of a plurality of map data fusion training samples; the map data fusion training samples include at least two local vector maps and global map data; the global map data is obtained by fusing the at least two local vector maps; Extracting the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps; and Using the initial map data fusion model, feature extraction is performed on the category data and the coordinate data to obtain vector map features; rasterization is performed based on the coordinate data to obtain raster map features; feature fusion and feature extraction are performed on the vector map features and the raster map features to obtain refined map features; decoding and prediction processing are performed on the refined map features to obtain global map prediction data; The initial map data fusion model is trained based on the global map prediction data and the global map data to obtain a map data fusion model.

[0007] In some embodiments of the present disclosure, feature extraction is performed on the category data and the coordinate data to obtain vector map features, including: Extracting features from the category data to obtain category features; Performing feature extraction on the coordinate data to obtain position features; The category feature and the location feature are concatenated to obtain the vector map feature.

[0008] In some embodiments of the present disclosure, the method further includes: calculating the center coordinates of each of the map elements based on the coordinate data to obtain 2D position information of the multiple map elements.

[0009] In some embodiments of the present disclosure, the vector map features and the raster map features are subjected to feature fusion and feature extraction to obtain the refined map features, including: Perform feature fusion based on the vector map features and the raster map features to obtain initial map features; Perform feature fusion based on the 2D position information and the grid map features to obtain a 2D position code corresponding to the initial map feature; Based on the 2D position coding, using a Transformer encoder to extract the initial map features to obtain the extracted map features; Wherein, the initial map data fusion model includes the Transformer encoder.

[0010] In some embodiments of the present disclosure, rasterization processing is performed based on the coordinate data to obtain a raster map feature, including: Obtain pixel coordinate data of a grid map based on the coordinate data conversion to obtain a grid map image; Extracting image features based on the grid map image to obtain grid map image features, and generating spatial position coordinate data corresponding to the grid map image features; The grid map features are obtained according to the grid map image features and the spatial position coordinate data.

[0011] In some embodiments of the present disclosure, decoding and prediction processing are performed on the refined map features to obtain global map prediction data, including: Decoding the extracted map features using a Transformer decoder to obtain prediction results of the multiple map elements; Using the prediction head to classify and regress the prediction results of the plurality of map elements to obtain the global map prediction data; The initial map data fusion model includes the Transformer decoder and the prediction head.

[0012] In some embodiments of the present disclosure, the refined map features are decoded using a Transformer decoder to obtain prediction results of the multiple map elements, including: Obtaining an initial content query vector; the initial content query vector is an all-zero vector; Based on the refined map features, generating core parameters of the attention mechanism of the mutual attention layer of the Transformer decoder; Based on the core parameters of the attention mechanism, the initial content query vector is refined to obtain a refined content query vector as a prediction result of the multiple map elements.

[0013] In some embodiments of the present disclosure, the global map prediction data includes: Category prediction data, confidence score prediction data, and coordinate prediction data corresponding to the map elements of the global map.

[0014] In some embodiments of the present disclosure, extracting the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps includes: Extracting the at least two local vector maps to obtain initial category data and initial coordinate data of a plurality of map elements in the at least two local vector maps; Dimensionally aligning the initial category data and the initial coordinate data to obtain category data and the coordinate data of a plurality of map elements in the at least two local vector maps.

[0015] According to a second aspect of an embodiment of the present disclosure, a map data fusion method is provided, comprising: Obtain multiple local vector maps to be fused; Extracting the multiple local vector maps to be fused, and obtaining category data and coordinate data of multiple map elements in the multiple local vector maps to be fused; Inputting category data and coordinate data of multiple map elements in the multiple local vector maps to be fused into a map data fusion model, and outputting global map data fused with the multiple local vector maps to be fused; Wherein, the map data fusion model is trained according to the model training method described in the first aspect above.

[0016] According to a third aspect of an embodiment of the present disclosure, a map data fusion model training device is provided, comprising: A sample acquisition unit, used to acquire a training data set of a plurality of map data fusion training samples; the map data fusion training samples include at least two local vector maps and global map data; the global map data is obtained by fusing the at least two local vector maps; A sample data extraction unit, used to extract the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps; The sample training unit is used to extract features from the category data and the coordinate data using an initial map data fusion model to obtain vector map features; perform rasterization processing based on the coordinate data to obtain raster map features; perform feature fusion and feature refinement on the vector map features and the raster map features to obtain refined map features; perform decoding and prediction processing on the refined map features to obtain global map prediction data; and train the initial map data fusion model based on the global map prediction data and the global map data to obtain a map data fusion model.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a map data fusion device is provided, comprising: A vector map acquisition unit, used to acquire multiple local vector maps to be fused; A map data extraction unit, used for extracting the plurality of local vector maps to be fused, and obtaining category data and coordinate data of a plurality of map elements in the plurality of local vector maps to be fused; A map fusion unit, used for inputting the category data and coordinate data of a plurality of map elements in the plurality of local vector maps to be fused into a map data fusion model, and outputting global map data fused with the plurality of local vector maps to be fused; Wherein, the map data fusion model is trained according to the model training method described in the first aspect above.

[0018] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: implement the model training method described in the first aspect above or implement the map data fusion method described in the second aspect above.

[0019] According to the sixth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When the instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to execute the model training method described in the first aspect or the map data fusion method described in the second aspect.

[0020] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: The present invention obtains the category data and coordinate data of multiple map elements by extracting at least two local vector maps of the training samples, extracts the features of the category data and coordinate data through the initial map data fusion model, and obtains the vector map features; performs rasterization processing based on the coordinate data to obtain the raster map features; performs feature fusion and feature extraction on the vector map features and the raster map features to obtain the refined map features; decodes and predicts the refined map features to obtain the global map prediction data; and trains the initial map data fusion model based on the global map prediction data and the global map data in the training samples to obtain the map data fusion model, and obtains the map data fusion model that can effectively fuse the vector map data. It only needs to extract the category data and coordinate data of the map elements from the vector map data to be fused collected by crowdsourcing, and input them into the trained map data fusion model, then an accurate fusion result can be obtained, and the fusion efficiency is high. Fusion can be performed by relying on the trained map data fusion model, even if the data sources are diverse or the quality is uneven, it can be accurately fused, which improves the flexibility of the fused map data.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0023] Figure 1 It is a flowchart of a model training method according to some embodiments of the present disclosure.

[0024] Figure 2 is a flowchart of the implementation process of step S104 according to some embodiments of the present disclosure.

[0025] Figure 3 The following is a flow chart of the implementation process of obtaining vector map features according to some embodiments of the present disclosure. Figure 1 .

[0026] Figure 4 The following is a flow chart of the implementation process of obtaining vector map features according to some embodiments of the present disclosure. Figure 2 .

[0027] Figure 5 It is a flowchart of the implementation process of obtaining grid map features according to some embodiments of the present disclosure.

[0028] Figure 6 It is a flowchart of the implementation process of obtaining refined map features according to some embodiments of the present disclosure.

[0029] Figure 7 It is a flowchart of the implementation process of obtaining global map prediction data according to some embodiments of the present disclosure.

[0030] Figure 8 is a flowchart of the implementation process of step S702 according to some embodiments of the present disclosure.

[0031] Fig. 9 It is a schematic diagram of the network structure of a map data fusion model provided by a specific example shown in some embodiments of the present disclosure.

[0032] Fig.10 The present invention is a flowchart of a map data fusion method according to some embodiments of the present invention.

[0033] Fig.11 It is a block diagram of a model training device according to some embodiments of the present disclosure.

[0034] Fig.12 It is a block diagram of a map data fusion device according to some embodiments of the present disclosure.

[0035] Fig.13 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Some embodiments of the present disclosure will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications and equivalents of the methods, devices and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as becomes apparent after understanding the present disclosure, except for operations that must be performed in a specific order. In addition, for clarity and brevity, descriptions of features known in the art may be omitted.

[0037] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0038] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0039] Figure 1 is a flowchart of a model training method according to some embodiments of the present disclosure. Figure 1 As shown, the model training method can be applied to various electronic devices, including but not limited to terminals such as smart phones, computers, smart tablets, vehicles, and independent physical servers, server clusters consisting of multiple physical servers or distributed systems, cloud servers and other types of servers.

[0040] Figure 1 The model training method shown includes the following steps.

[0041] In step S102, a training data set of a plurality of map data fusion training samples is obtained.

[0042] In some embodiments of the present disclosure, the map data fusion training sample includes at least two local vector maps and global map data, and the global map data is obtained by fusing at least two local vector maps. Each local vector map can be obtained by collecting map data provided by different users from a crowdsourcing platform or by multiple collections. Multiple data collections are performed on the same geographical area to obtain richer data. Different data sources can be fully utilized to improve the quality and coverage of map data.

[0043] It should be noted that crowd sourcing refers to a model in which tasks are distributed to a large number of individuals through the Internet. In the field of map data, crowdsourcing platforms allow users to upload and share map data of their areas.

[0044] It should be noted that a local vector map refers to a set of vectorized map elements within a specific geographic area. These elements may include: lane lines: indicating the boundaries of lanes on a road; lane center lines: indicating the center line of a lane; stop lines: indicating where a vehicle must stop; road signs: signs indicating directions or providing other traffic information; zebra crossings: safe passages for pedestrians to cross the street.

[0045] In step S104, at least two local vector maps are extracted to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps.

[0046] It should be noted that the category data of a map element refers to the category to which the map element belongs, which may include: lane lines, lane center lines, stop lines, road signs, and zebra crossings. Different categories may be identified by numbers, for example, 0-4 may be used to represent lane lines, lane center lines, stop lines, road signs, and zebra crossings, respectively. The coordinate data of a map element refers to the specific location information of each map element, which may be a 2D coordinate array of each map element.

[0047] In step S106, the initial map data fusion model is used to extract features from the category data and the coordinate data to obtain vector map features; rasterization is performed based on the coordinate data to obtain raster map features; the vector map features and the raster map features are feature fused and feature refined to obtain refined map features; the refined map features are decoded and predicted to obtain global map prediction data.

[0048] In some embodiments of the present disclosure, the initial map data fusion model is a deep learning model based on the attention mechanism (AttentionMechanism) deep learning architecture that can understand and process map data and better integrate vector map data. It can effectively integrate local vector map data from crowdsourcing platforms and multiple collections to generate high-precision and consistent global maps to meet the needs of autonomous driving systems and other applications. Among them, the attention mechanism is a computational model that simulates the human visual attention allocation process.

[0049] In some embodiments of the present disclosure, a pre-built initial map data fusion model is obtained, category data and coordinate data of multiple map elements in at least two local vector maps are used as input data of the initial map data fusion model, feature extraction is performed on the category data and coordinate data to obtain vector map features, rasterization is performed based on the coordinate data to obtain raster map features; feature fusion and feature refinement are performed on the vector map features and raster map features to obtain refined map features; the refined map features are then decoded and predicted to obtain global map prediction data that fuses at least two local vector maps.

[0050] In step S108, the initial map data fusion model is trained based on the global map prediction data and the global map data to obtain a map data fusion model.

[0051] In some embodiments of the present disclosure, after obtaining global map prediction data using an initial map data fusion model, the initial map data fusion model is trained based on the global map data in the acquired training data set, that is, the global map true value fused with at least two local vector maps. After the training is completed, a map data fusion model is obtained.

[0052] It can be seen from the above steps that the model training method provided by the embodiment of the present disclosure obtains the category data and coordinate data of multiple map elements by extracting at least two local vector maps of the training samples, extracts the features of the category data and coordinate data through the initial map data fusion model, and obtains the vector map features; performs rasterization processing based on the coordinate data to obtain the raster map features; performs feature fusion and feature extraction on the vector map features and the raster map features to obtain the refined map features; decodes and predicts the refined map features to obtain the global map prediction data; and trains the initial map data fusion model based on the global map prediction data and the global map data in the training samples to obtain the map data fusion model, and obtains the map data fusion model that can effectively fuse the vector map data. It is only necessary to extract the category data and coordinate data of the map elements from the vector map data to be fused collected by crowdsourcing, and input them into the trained map data fusion model, then an accurate fusion result can be obtained, and the fusion efficiency is high. Fusion can be performed by relying on the trained map data fusion model, even if the data sources are diverse or the quality is uneven, it can be accurately fused, which improves the flexibility of the fused map data.

[0053] In some exemplary embodiments of the present disclosure, Figure 2 As shown, it is a flowchart of the implementation process of step S104 provided for some exemplary embodiments of the present disclosure, including the following steps.

[0054] In step S202, at least two local vector maps are extracted to obtain initial category data and initial coordinate data of a plurality of map elements in the at least two local vector maps.

[0055] In some embodiments of the present disclosure, before extracting at least two local vector maps, the at least two local vector maps may be preprocessed, which may include: converting vector map data from different sources into a unified coordinate system; removing noise points and outliers to improve data quality; and converting the data into a unified format to facilitate subsequent processing.

[0056] It should be noted that different vector map data may be stored in different formats, such as GeoJSON, KML, Shapefile, etc., and corresponding libraries are needed to parse these files. Each map element in at least two local vector maps usually contains a category label indicating its type (such as lane line, lane center line, stop line, etc.). These labels can be extracted from the parsed data structure to obtain the initial category data. The geometric information of map elements is usually stored in geometric objects (such as points, lines, polygons). This geometric information can be extracted from the parsed data structure, and its coordinates can be obtained to obtain the initial coordinate data.

[0057] In step S204, dimension alignment is performed on the initial category data and the initial coordinate data to obtain category data and coordinate data of a plurality of map elements in at least two local vector maps.

[0058] In some exemplary embodiments of the present disclosure, since the number of map elements contained in each local vector map may be different, the array may not be aligned during subsequent processing, so the initial category data and the initial coordinate data need to be dimensionally aligned. After obtaining the initial category data and the initial coordinate data, the array may be padded with zeros at the end when forming a batch to perform dimension alignment for subsequent processing.

[0059] In some exemplary embodiments of the present disclosure, Figure 3 FIG. 1 is a flowchart of an implementation process for obtaining vector map features provided by some exemplary embodiments of the present disclosure. Figure 1 , comprising the following steps.

[0060] In step S302, feature extraction is performed on the category data to obtain category features.

[0061] In some embodiments of the present disclosure, the category data of multiple map elements in at least two local vector maps are input into an Embedding layer, the category features of a preset number of channels are extracted, and the category data is converted into a vector form suitable for subsequent neural network processing, wherein the preset number of channels is related to the number of categories of map elements. The embodiments of the present disclosure can effectively convert the category information in the vector map data into a rich feature representation through the Embedding layer, providing strong support for subsequent map fusion and feature extraction.

[0062] It should be noted that the Embedding layer is a technology used to map discrete categorical data (such as vocabulary, category labels, etc.) into continuous vector representations to achieve category embedding and semantic representation.

[0063] In step S304, feature extraction is performed on the coordinate data to obtain position features.

[0064] In some embodiments of the present disclosure, the coordinate data of multiple map elements in at least two local vector maps are input into the Subgraph module of VectorNet to aggregate the map elements in a local range, capture the relative positions and geometric relationships between them, and gradually capture the spatial dependencies in a larger range by constructing a hierarchical feature representation to extract position features. The embodiments of the present disclosure can effectively process complex road topological structures by using the Subgraph module, thereby improving the effect of map fusion tasks.

[0065] It should be noted that VectorNet is a neural network architecture specifically designed to process vectorized map data (such as lane lines, stop lines, etc.). A core component of VectorNet is its Subgraph module, which is used to aggregate information in a local range and capture the relative relationship between map elements.

[0066] In step S306, the category feature and the location feature are concatenated to obtain a vector map feature.

[0067] In some embodiments of the present disclosure, the category features and the location features are concatenated along the channel dimension to obtain vector map features. Specifically, the channel dimension can include a variety of different feature representations, such as category embedding (generated by the Embedding layer), location coordinates (generated by linear transformation or more complex encoding methods), etc.

[0068] It should be noted that the initial map data fusion model includes a vector map feature extraction module, which is used to implement the above steps S302 to S306 to obtain vector map features. Different types of map features can be effectively integrated to provide rich input data for further map fusion and feature extraction.

[0069] It will be understood by those skilled in the art that the above Figure 3 The step sequence shown is only an example and is not intended to limit the protection scope of the embodiments of the present disclosure. For example, step S302 may be implemented simultaneously with step S304, or may be implemented before or after step S304.

[0070] In some exemplary embodiments of the present disclosure, Figure 4 FIG. 1 is a flowchart of an implementation process for obtaining vector map features provided by some exemplary embodiments of the present disclosure. Figure 2 , Figure 4 Steps S402 to S406 in obtaining the vector map features are shown in FIG. Figure 3 Steps S302 to S306 in obtaining the vector map features are shown to correspond to each other and will not be repeated here.

[0071] exist Figure 3 On the basis of, the following steps may also be included.

[0072] In step S408, the center coordinates of each map element are calculated based on the coordinate data to obtain 2D position information of the multiple map elements.

[0073] In some embodiments of the present disclosure, a geometry type can be determined, such as a point, a line segment, or a polygon; the center coordinates are calculated based on the geometry type. For example, the center coordinates of a point are the coordinates of the point itself, the center coordinates of a line segment are the average of the coordinates of all endpoints, and the center coordinates of a polygon can be approximated by calculating the average of its vertex coordinates, or using a more complex centroid formula.

[0074] It should be noted that the vector map feature extraction module is used to implement the above steps S402 to S408, so as to determine the 2D position information of multiple map elements on the basis of obtaining the vector map features. The position information of the map elements can be effectively extracted and simplified to provide support for further map fusion and feature extraction.

[0075] It will be understood by those skilled in the art that the above Figure 4 The step sequence shown is only an example and is not intended to limit the protection scope of the embodiments of the present disclosure. For example, step S408 may be implemented simultaneously with steps S402 to S406, or may be implemented before or after them.

[0076] In some exemplary embodiments of the present disclosure, Figure 5 As shown, a flowchart of the implementation process of obtaining grid map features provided by some exemplary embodiments of the present disclosure includes the following steps.

[0077] In step S502, pixel coordinate data of the grid map is obtained based on the coordinate data conversion to obtain a grid map image.

[0078] In some embodiments of the present disclosure, a predefined raster map image is first determined, including the geographic range (i.e., minimum and maximum longitude and latitude) of the represented map area and the desired raster map resolution (the actual distance represented by each pixel), and the width and height of the raster map image can be calculated based on the geographic range and resolution. Then, the geographic coordinates (such as longitude and latitude) are converted into a plane coordinate system. Since the earth is a sphere, directly displaying the geographic coordinates on a two-dimensional plane will cause deformation. Therefore, a projection transformation (such as Web Mercator, etc.) is usually used to convert the geographic coordinates into plane coordinates. And the pixel coordinates corresponding to each geographic point are calculated based on the geographic range and resolution. Finally, based on the pixel coordinates corresponding to each geographic point, points or other graphic elements are drawn at the corresponding positions on the raster map to obtain the converted raster map image.

[0079] In step S504, image features are extracted based on the raster map image to obtain raster map image features, and spatial position coordinate data corresponding to the raster map image features are generated.

[0080] In some embodiments of the present disclosure, a suitable algorithm is selected to extract features from an image. For example, a grid map image can be input into a deep convolutional neural network to extract image features to adapt to feature extraction in complex scenes. The extracted features can also be flattened and permuted to obtain grid map image features. At the same time, the coordinate data of each spatial position corresponding to the grid map image features is calculated and generated.

[0081] It should be noted that before feature extraction, some preprocessing operations are usually required on the raster map image to improve the accuracy and efficiency of feature extraction. Common preprocessing steps include: resizing and resolution to ensure that the image size is suitable for subsequent processing. Denoising, using filters to remove noise from the image, such as Gaussian blur. Enhance contrast, using techniques such as histogram equalization to enhance image contrast.

[0082] In step S506, the grid map features are obtained according to the grid map image features and the spatial position coordinate data.

[0083] It should be noted that the raster map features include raster map image features and spatial position coordinate data. The initial map data fusion model includes a raster map feature extraction module, which is used to implement steps S502 to S506 to obtain raster map features for use by subsequent modules.

[0084] It should be noted that vector map features contain more overall information of map elements, while raster map features contain more local information of map elements. The two are highly complementary, so they are fused through splicing operations. Figure 6 As shown, a flowchart of the implementation process of obtaining refined map features provided for some exemplary embodiments of the present disclosure includes the following steps.

[0085] In step S602, feature fusion is performed based on the vector map features and the raster map features to obtain initial map features.

[0086] In some embodiments of the present disclosure, the vector map features and the raster map image features in the raster map features are fused through a splicing operation to obtain a fused initial map feature.

[0087] In step S604, feature fusion is performed based on the 2D position information and the grid map features to obtain the 2D position code corresponding to the initial map feature.

[0088] In some embodiments of the present disclosure, the 2D position information obtained in step S408 and the spatial position coordinate data in the raster map feature are fused through a splicing operation to obtain fused position information, and the fused position information is subjected to 2D sinusoidal position encoding to obtain a 2D position code corresponding to the initial map feature.

[0089] In step S606, based on the 2D position coding, the Transformer encoder is used to extract the initial map features to obtain the refined map features.

[0090] It should be noted that the initial map data fusion model includes a Transformer Encoder, which includes multiple Transformer Encoder Layers. Based on the 2D position encoding corresponding to the initial map features obtained in step S604, the initial map features are subjected to feature extraction consisting of self-attention processing, residual connection, and layer normalization processing using multiple Transformer Encoder Layers to obtain refined map features.

[0091] like Figure 7 As shown, it is a flowchart of the implementation process of obtaining global map prediction data provided by some exemplary embodiments of the present disclosure, including the following steps.

[0092] In step S702, the refined map features are decoded using a Transformer decoder to obtain prediction results of multiple map elements.

[0093] In step S704, the prediction head is used to classify and regress the prediction results of multiple map elements to obtain global map prediction data.

[0094] It should be noted that the initial map data fusion model includes a Transformer Decoder module and a prediction head.

[0095] In some embodiments of the present disclosure, after obtaining the refined map features using Transformer Encoder, the refined map features are decoded using Transformer Decoder to obtain prediction results of multiple map elements. Then, the prediction results of multiple map elements are classified and regressed using a prediction head to obtain global map prediction data. The initial map data fusion model may include multiple prediction heads, each of which receives the output of multiple Transformer DecoderLayers to predict map elements and obtain multiple groups of prediction results.

[0096] In some embodiments of the present disclosure, the global map prediction data includes: category prediction data, confidence score prediction data, and coordinate prediction data corresponding to the map elements of the global map. That is, the category prediction data, confidence score prediction data, and coordinate prediction data of each map element in the global map. The confidence score prediction data represents the credibility of what category each map element belongs to. For example, there are 5 category prediction data of a certain map element, which are numbered 0, 1, 2, 3, and 4. The confidence score prediction data contains 5 credibility when numbered 0, 1, 2, 3, and 4. The number with the highest credibility represents the category of this map element.

[0097] It should be noted that each group of prediction heads includes two classification heads and regression heads, both of which are composed of MLP (multi-layer perceptron). The category prediction data and confidence score prediction data are obtained by passing the output of each Transformer Decoder Layer through MLP and Softmax (activation function) layers in turn, and the coordinate prediction data is obtained by passing the output of each Transformer Decoder Layer through MLP and Sigmoid (activation function) layers in turn.

[0098] In some exemplary embodiments of the present disclosure, Figure 8 As shown, it is a flowchart of the implementation process of step S702 provided in some exemplary embodiments of the present disclosure, including the following steps.

[0099] In step S802, an initial content query vector is obtained.

[0100] In some embodiments of the present disclosure, the initial content query vector is an all-zero vector. The Transformer decoder includes multiple Transformer Encoder Layers. The initial content query vector is input into the first Transformer Encoder Layer, and after self-attention calculation, mutual attention calculation, and feedforward neural network calculation, the first-layer refined content query vector is output. The first-layer refined content query vector is input into the second Transformer Encoder Layer, and after the same process, the second-layer refined content query vector is output. And so on, until multiple Transformer Encoder Layers are processed.

[0101] In step S804, based on the refined map features, the core parameters of the attention mechanism of the mutual attention layer of the Transformer decoder are generated.

[0102] In some embodiments of the present disclosure, based on the refined map features obtained in step S606, the core parameters of the attention mechanism of the mutual attention layer of the Transformer decoder are generated, which may include: K (key), V (value).

[0103] In step S806, based on the core parameters of the attention mechanism, the initial content query vector is refined to obtain a refined content query vector as a prediction result of multiple map elements.

[0104] It should be noted that the refined content query vector includes the content query vector processed by each Transformer Encoder Layer.

[0105] In some exemplary embodiments of the present disclosure, when step S108 is implemented, the global map prediction data, including: category prediction data, confidence score prediction data, and coordinate prediction data of each map element in the global map, and the global map true value, including: category data and coordinate data of each real map element in the global map, can be Hungarian matched to determine positive and negative samples, and then calculate the loss function (loss). Among them, the loss design can adopt the loss function of Map TR, specifically including: classification loss, which is used to optimize the model's prediction of the map element category. Regression loss, which is used to optimize the model's prediction of the map element position coordinates. The total loss is usually a weighted sum of the classification loss and the regression loss to balance the impact of the two on the final loss, which helps the model improve the performance of classification and regression tasks simultaneously during the training process, thereby achieving more accurate map element detection and segmentation.

[0106] It should be noted that Map TR is a method for object detection and segmentation in Bird's-Eye View (BEV) maps.

[0107] It can be seen that the embodiment of the present disclosure designs and trains a map data fusion model based on a deep learning framework, and utilizes a neural network model to automatically fuse vector map data from multiple crowdsourcing platforms or collected multiple times, thereby generating high-precision and consistent global map data with the help of the trained map data fusion model.

[0108] In order to better illustrate the model training method of the map data fusion model provided by the embodiment of the present disclosure, a specific example is now provided for further explanation. This specific example provides a specific model architecture and training process of the model training method provided by the example of the present disclosure.

[0109] like Fig. 9 The network structure of the map data fusion model constructed in this specific example is shown in Figure 1. The network input is multiple local vector maps collected and generated through crowdsourcing. ,in is the number of map elements contained in each local vector map, is the category array of all map elements, is the 2D coordinate array of all map elements, represents a set of real numbers. Each map element is interpolated to There are 5 types of map element categories, as shown in Table 1. Different, so when forming a batch, it is necessary to pad the end of the array with zeros to align the dimensions. The data finally input to the network is the map element category and coordinates ( is the batch size, ).

[0110]

[0111] Fig. 9 The network output shown is the category of the fused map elements , confidence score and coordinates ,in Sets the maximum number of map elements to predict.

[0112] The network consists of five modules: vector map feature extraction module; raster map feature extraction module; Transformer Encoder; Transformer Decoder; prediction head.

[0113] Vector map feature extraction module: First, map element categories Send it to an Embedding layer, and the number of extracted channels is Category characteristics ,in Indicates the preset number of channels. Input to the Subgraph module of VectorNet to extract position features , and then concatenate the two along the channel dimension ( ), get the vector map features At the same time, the center coordinates of each map element are calculated as the 2D position information of the map element , the calculation formula of the entire module is shown in the following formula (1): (1) in, The representation specifies that the operation is performed on dimension 2 (the third dimension, which is the channel dimension). Representation mean.

[0114] Raster map feature extraction module: First, all map element coordinates are converted into map images according to formula (2) , where For map images The coordinate range of and The width and height of the grid map image are predefined, and the values ​​are set. Then the size is calculated according to formula (3): Map image of , the five map element categories correspond to map images Then the map image Input to the VGG19 network (a deep convolutional neural network) to extract the size The image features are finally obtained by flattening and permuting the size. ,in . Generate The feature map of represents the height of the feature map, The coordinates of each spatial position in the feature map , can be calculated by The coordinates of each point in each grid in the feature map are obtained for use in subsequent modules.

[0115] (2) (3) in, Represents the coordinates of each map element in a vector map . Representing map images The coordinates of each map element in , The category number that represents the map element, with a value of 0-4. The pixel coordinates representing the map element.

[0116] Transformer Encoder: Vector map features contain more overall information of map elements, while raster map features contain more local information of map elements. The two are highly complementary, so they are fused through splicing operations to obtain the fused initial map features. and location information , After 2D sinusoidal position encoding ( ) Get the 2D position code corresponding to the initial map feature , the calculation process is shown in formula (4).

[0117] (4) in, The representation specifies that the operation is performed on dimension 1 (the second dimension).

[0118] Fusion of map features The input is sent to the Encoder for feature extraction. Transformer Encoder includes Transformer Encoder Layer, for the Transformer Encoder Layer, input features First, through a Self-Attention module, the feature With position encoding Add together to get Q1 (query) and K1 (key), Directly (value), the output of the self-attention module is updated through the residual connection and LN (LayerNormalization) layer , Formula (5) shows the calculation process of the self-attention module.

[0119] (5) And features Then, the refined map features are obtained by passing through the FFN (Feed-Forward Neural Network) layer, residual connection and LN layer in sequence. , as shown in formula (6).

[0120] (6) in, The value range is 1 to .

[0121] go through After processing the Transformer Encoder Layer, the refined map features are obtained .

[0122] Transformer Decoder: Input to Transformer Decoder, at the same time, Transformer Decoder will also input a Content Query (content query) of all zeros and learnable Position Query .

[0123] Transformer Decoder includes Transformer Decoder Layer, each layer contains three modules: Self-Attention, Cross-Attention and FFN (Feedforward Neural Network).

[0124] For Transformer Decoder Layer, content query First, the self-attention is calculated through the Self-Attention module, where (query) and (Key) Pass Query with location Add them together and you get, for , the output of the self-attention module is then passed through the residual connection and LN layer to obtain the updated content query , the calculation process is shown in formula (7). The value range is 1 to .

[0125] (7) Updated content query Then it is input into the Cross-Attention module to interact with the map features. First, it is calculated according to formula (8): The corresponding position code ,if , Query by location It is predicted by MLP (Multi-Layer Perceptron), Sigmoid (activation function) and 2D sinusoidal position encoding in sequence. Otherwise, The coordinates of the map elements predicted by the previous Transformer Decoder Layer output The center point of is obtained through 2D sinusoidal position encoding.

[0126] (8) Cross-Attention module (Query)Query by content With position encoding Add them together and you get, (Key) The extracted map features 2D position encoding corresponding to the initial map features Add them together and you get, 3 (value) is the refined map feature , and It contains both content information and location information, which is conducive to the accurate positioning of map elements and the rapid convergence of the model. Similar to the self-attention module, the output of the mutual attention module will also pass through the residual connection and LN layer to generate an updated content query. , the calculation process is shown in formula (9).

[0127] (9) Content Query Then, the refined content query is obtained by passing through the FFN layer, residual connection and LN layer in sequence. , as shown in formula (10).

[0128] (10)

[0129] That is, for each Transformer Decoder Layer, the input After processing by formulas (7), (8), (9), and (10), the output is The prediction results of multiple map elements are .

[0130] Prediction head: The network architecture contains Group prediction header, receive The output of the TransformerDecoder Layer Perform map element prediction and obtain Group prediction results.

[0131] Each group of prediction heads includes two classification heads and regression heads, both of which are composed of MLP. Depend on After MLP and The position prediction result is obtained by Depend on It is obtained by passing through MLP and Sigmoid layers in sequence.

[0132] As shown in formula (11).

[0133] (11) During the model training phase, the model prediction results The map truth Perform Hungarian matching, determine positive and negative samples, and then calculate the loss, using the same loss function as Map TR.

[0134] In the model inference stage, that is, when the trained map data fusion model is applied to vector map data fusion, only the prediction results of the last layer are output. as model output.

[0135] From the above process, it can be seen that in this specific example, by building and training a map data fusion model, using neural networks and deep learning architecture, vector map data from multiple crowdsourcing platforms or collected multiple times are automatically fused to generate high-precision and consistent map data. Compared with the rule-based fusion method in the prior art, it has the following significant advantages: Higher accuracy: By extracting and fusing the deep features of the data through deep learning models, it is possible to more accurately reflect the map’s geometry, topological relationships, and semantic information, and improve the accuracy of the fusion results.

[0136] Stronger robustness: Deep learning models can process data from different sources and of different qualities, are more robust to data noise and anomalies, and can generate more reliable fusion results.

[0137] Better adaptability: The deep learning model can automatically adjust the fusion strategy according to the characteristics of the data, adapt to the data fusion requirements in different scenarios, and improve the versatility and flexibility of the method.

[0138] Higher efficiency: Through parallel computing and optimized deep learning framework, the computing efficiency of data fusion can be significantly improved to meet the real-time requirements of the autonomous driving system.

[0139] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0140] The present disclosure also provides a map data fusion method. Fig.10 is a flowchart of map data fusion according to some embodiments of the present disclosure, such as Fig.10 As shown, the map data fusion method can be applied to various electronic devices, including but not limited to terminal devices such as smart phones, computers, smart tablets, vehicles, and independent physical servers, server clusters or distributed systems composed of multiple physical servers, cloud servers and other types of servers. Fig.10 The map data fusion method shown includes the following steps.

[0141] In step S1002, a plurality of local vector maps to be fused are obtained.

[0142] It should be noted that each local vector map to be fused may be obtained from a crowdsourcing platform, by collecting map data provided by different users or by multiple collections, and multiple data collections may be performed on the same geographical area to obtain richer data.

[0143] In step S1004, multiple local vector maps to be fused are extracted to obtain category data and coordinate data of multiple map elements in the multiple local vector maps to be fused.

[0144] In some embodiments of the present disclosure, category data and coordinate data of multiple map elements in the multiple local vector maps to be fused can be obtained by preprocessing, parsing and extracting, dimensional alignment, etc. of the multiple local vector maps to be fused.

[0145] In step S1006, the category data and coordinate data of multiple map elements in the multiple local vector maps to be fused are input into the map data fusion model, and the global map data fused with the multiple local vector maps to be fused is output.

[0146] It should be noted that the map data fusion model is trained according to the model training method described in the above embodiment.

[0147] The disclosed embodiment uses the map data fusion model constructed and trained based on the deep learning architecture in the above embodiment to fuse the category data and coordinate data of multiple map elements in multiple local vector maps to be fused, and obtains global map data that fuses multiple local vector maps to be fused. It can achieve efficient and accurate fusion of crowdsourced vector map data, provide real-time and reliable map support for the autonomous driving system, and improve the safety and reliability of the autonomous driving system.

[0148] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0149] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0150] Fig.11 is a block diagram of a model training device according to some embodiments of the present disclosure. Fig.11 The device includes: a sample acquisition unit 1101, a sample data extraction unit 1102 and a sample training unit 1103.

[0151] The sample acquisition unit 1101 is used to acquire a training data set of a plurality of map data fusion training samples; the map data fusion training samples include at least two local vector maps and global map data; the global map data is obtained by fusing at least two local vector maps; The sample data extraction unit 1102 is used to extract at least two local vector maps to obtain category data and coordinate data of multiple map elements in the at least two local vector maps; The sample training unit 1103 is used to extract features from the category data and the coordinate data using the initial map data fusion model to obtain vector map features; perform rasterization processing based on the coordinate data to obtain raster map features; perform feature fusion and feature extraction on the vector map features and the raster map features to obtain refined map features; perform decoding and prediction processing on the refined map features to obtain global map prediction data; and train the initial map data fusion model based on the global map prediction data and the global map data to obtain a map data fusion model.

[0152] In some exemplary embodiments of the present disclosure, the sample data extraction unit 1102 is configured to: Extracting at least two local vector maps to obtain initial category data and initial coordinate data of a plurality of map elements in the at least two local vector maps; Dimensional alignment is performed on the initial category data and the initial coordinate data to obtain category data and coordinate data of a plurality of map elements in at least two local vector maps.

[0153] In some exemplary embodiments of the present disclosure, the sample training unit 1103 is configured to: Extract features from category data to obtain category features; Extract features from coordinate data to obtain position features; The category feature and the location feature are concatenated to obtain a vector map feature.

[0154] In some exemplary embodiments of the present disclosure, the sample training unit 1103 is further configured to: Based on the coordinate data, the center coordinates of each map element are calculated to obtain the 2D position information of multiple map elements.

[0155] Accordingly, the sample training unit 1103 is configured as follows: Perform feature fusion based on vector map features and raster map features to obtain initial map features; Based on the 2D position information and the raster map features, feature fusion is performed to obtain the 2D position code corresponding to the initial map features; Based on 2D position coding, the Transformer encoder is used to extract the initial map features to obtain the refined map features; Among them, the initial map data fusion model includes a Transformer encoder.

[0156] In some exemplary embodiments of the present disclosure, the sample training unit 1103 is configured to: Obtain pixel coordinate data of the raster map based on the coordinate data conversion to obtain a raster map image; Extracting image features based on the raster map image to obtain raster map image features and generating spatial position coordinate data corresponding to the raster map image features; The raster map features are obtained according to the raster map image features and the spatial position coordinate data.

[0157] In some exemplary embodiments of the present disclosure, the sample training unit 1103 is configured to: The refined map features are decoded using the Transformer decoder to obtain the prediction results of multiple map elements; The prediction head is used to classify and regress the prediction results of multiple map elements to obtain global map prediction data; Among them, the initial map data fusion model includes Transformer decoder and prediction head.

[0158] In some embodiments of the present disclosure, the global map prediction data includes: category prediction data, confidence score prediction data, and coordinate prediction data corresponding to map elements of the global map.

[0159] Further, in some exemplary embodiments of the present disclosure, the sample training unit 1103 is configured to: Obtain an initial content query vector; the initial content query vector is an all-zero vector; Based on the refined map features, generate the core parameters of the attention mechanism of the mutual attention layer of the Transformer decoder; Based on the core parameters of the attention mechanism, the initial content query vector is refined to obtain the refined content query vector as the prediction result of multiple map elements.

[0160] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0161] Fig.12 is a map data fusion block diagram according to some embodiments of the present disclosure. Fig.12The device includes: a vector map acquisition unit 1201, a map data extraction unit 1202 and a map fusion unit 1203.

[0162] A vector map acquisition unit 1201 is used to acquire multiple local vector maps to be fused; A map data extraction unit 1202 is used to extract the multiple local vector maps to be fused, and obtain the category data and coordinate data of multiple map elements in the multiple local vector maps to be fused; The map fusion unit 1203 is used to input the category data and coordinate data of multiple map elements in multiple local vector maps to be fused into the map data fusion model, and output the global map data fused with the multiple local vector maps to be fused; The map data fusion model is trained according to the model training method described in the above embodiment.

[0163] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0164] Fig.13 1 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. For example, the electronic device 1300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc., or a server.

[0165] Reference Fig.13 , the electronic device 1300 may include one or more of the following components: a processing component 1302 , a memory 1304 , a power component 1306 , a multimedia component 1308 , an audio component 1310 , an input / output (I / O) interface 1312 , a sensor component 1314 , and a communication component 1316 .

[0166] The processing component 1302 generally controls the overall operation of the electronic device 1300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1302 may include one or more processors 1320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1302 may include one or more modules to facilitate the interaction between the processing component 1302 and other components. For example, the processing component 1302 may include a multimedia module to facilitate the interaction between the multimedia component 1308 and the processing component 1302.

[0167] The memory 1304 is configured to store various types of data to support operations on the device 1300. Examples of such data include instructions for any application or method operating on the electronic device 1300, contact data, phone book data, messages, pictures, videos, etc. The memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0168] The power component 1306 provides power to the various components of the electronic device 1300. The power component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1300.

[0169] The multimedia component 1308 includes a screen that provides an output interface between the electronic device 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1308 includes a front camera and / or a rear camera. When the device 1300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0170] The audio component 1310 is configured to output and / or input audio signals. For example, the audio component 1310 includes a microphone (MIC), and when the electronic device 1300 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1304 or sent via the communication component 1316. In some embodiments, the audio component 1310 also includes a speaker for outputting audio signals.

[0171] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0172] The sensor assembly 1314 includes one or more sensors for providing various aspects of status assessment for the electronic device 1300. For example, the sensor assembly 1314 can detect the open / closed state of the device 1300, the relative positioning of components, such as the display and keypad of the electronic device 1300, and the sensor assembly 1314 can also detect the position change of the electronic device 1300 or a component of the electronic device 1300, the presence or absence of user contact with the electronic device 1300, the orientation or acceleration / deceleration of the electronic device 1300, and the temperature change of the electronic device 1300. The sensor assembly 1314 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0173] The communication component 1316 is configured to facilitate wired or wireless communication between the electronic device 1300 and other devices. The electronic device 1300 can access a wireless network based on a communication standard, such as Wi-Fi, 3G, 4G, 5G, other communication standards, or a combination thereof. In some embodiments of the present disclosure, the communication component 1316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of the present disclosure, the communication component 1316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0174] In some embodiments of the present disclosure, the electronic device 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0175] In some embodiments of the present disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, and the instructions can be executed by a processor 1320 of the electronic device 1300 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0176] In some embodiments of the present disclosure, a non-temporary computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute the above-mentioned model training method or map data fusion method.

[0177] In some embodiments of the present disclosure, a computer program product is also provided, including a computer program / instruction, which implements the above-mentioned model training method or map data fusion method when the computer program / instruction is executed by a processor.

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire a training data set of a plurality of map data fusion training samples; the map data fusion training samples include at least two local vector maps and global map data; The global map data is obtained by fusing the at least two local vector maps; Extracting the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps; Using an initial map data fusion model to extract features from the category data and the coordinate data to obtain vector map features; Performing rasterization processing based on the coordinate data to obtain raster map features; Performing feature fusion and feature extraction on the vector map features and the raster map features to obtain refined map features; Decoding and predicting the extracted map features to obtain global map prediction data; The initial map data fusion model is trained based on the global map prediction data and the global map data to obtain a map data fusion model.

2. The model training method according to claim 1, characterized in that: Extracting features from the category data and the coordinate data to obtain vector map features includes: Extracting features from the category data to obtain category features; Performing feature extraction on the coordinate data to obtain position features; The category feature and the location feature are concatenated to obtain the vector map feature.

3. The model training method according to claim 2, characterized in that: Also includes: Based on the coordinate data, the center coordinates of each of the map elements are calculated to obtain 2D position information of the multiple map elements.

4. The model training method according to claim 3, characterized in that: The vector map features and the raster map features are subjected to feature fusion and feature extraction to obtain the extracted map features, including: Perform feature fusion based on the vector map features and the raster map features to obtain initial map features; Perform feature fusion based on the 2D position information and the grid map features to obtain a 2D position code corresponding to the initial map feature; Based on the 2D position coding, using a Transformer encoder to extract the initial map features to obtain the extracted map features; Wherein, the initial map data fusion model includes the Transformer encoder.

5. The model training method according to claim 1, characterized in that: Performing rasterization processing based on the coordinate data to obtain raster map features includes: Obtain pixel coordinate data of a grid map based on the coordinate data conversion to obtain a grid map image; Extracting image features based on the grid map image to obtain grid map image features, and generating spatial position coordinate data corresponding to the grid map image features; The grid map features are obtained according to the grid map image features and the spatial position coordinate data.

6. The model training method according to claim 1, characterized in that: Decoding and predicting the extracted map features to obtain global map prediction data, including: Decoding the extracted map features using a Transformer decoder to obtain prediction results of the multiple map elements; Using the prediction head to classify and regress the prediction results of the plurality of map elements to obtain the global map prediction data; The initial map data fusion model includes the Transformer decoder and the prediction head.

7. The model training method according to claim 6, characterized in that: The refined map features are decoded using a Transformer decoder to obtain prediction results of the multiple map elements, including: Obtaining an initial content query vector; the initial content query vector is an all-zero vector; Based on the refined map features, generating core parameters of the attention mechanism of the mutual attention layer of the Transformer decoder; Based on the core parameters of the attention mechanism, the initial content query vector is refined to obtain a refined content query vector as a prediction result of the multiple map elements.

8. The model training method according to claim 1, characterized in that: The global map prediction data includes: Category prediction data, confidence score prediction data, and coordinate prediction data corresponding to the map elements of the global map.

9. The model training method according to claim 1, characterized in that: Extracting the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps includes: Extracting the at least two local vector maps to obtain initial category data and initial coordinate data of a plurality of map elements in the at least two local vector maps; Dimensionally aligning the initial category data and the initial coordinate data to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps.

10. A map data fusion method, characterized in that: include: Obtain multiple local vector maps to be fused; Extracting the multiple local vector maps to be fused, and obtaining category data and coordinate data of multiple map elements in the multiple local vector maps to be fused; Inputting category data and coordinate data of multiple map elements in the multiple local vector maps to be fused into a map data fusion model, and outputting global map data fused with the multiple local vector maps to be fused; Wherein, the map data fusion model is trained according to the model training method described in any one of claims 1 to 9.

11. A model training device, characterized in that: include: A sample acquisition unit, used to acquire a training data set of a plurality of map data fusion training samples; the map data fusion training samples include at least two local vector maps and global map data; The global map data is obtained by fusing the at least two local vector maps; A sample data extraction unit, used to extract the at least two local vector maps to obtain category data and coordinate data of a plurality of map elements in the at least two local vector maps; A sample training unit, used for extracting features from the category data and the coordinate data using an initial map data fusion model to obtain vector map features; Performing rasterization processing based on the coordinate data to obtain raster map features; Performing feature fusion and feature extraction on the vector map features and the raster map features to obtain refined map features; The extracted map features are decoded and predicted to obtain global map prediction data; the initial map data fusion model is trained based on the global map prediction data and the global map data to obtain a map data fusion model.

12. A map data fusion device, characterized in that: include: A vector map acquisition unit, used to acquire multiple local vector maps to be fused; A map data extraction unit, used for extracting the plurality of local vector maps to be fused, and obtaining category data and coordinate data of a plurality of map elements in the plurality of local vector maps to be fused; A map fusion unit, used for inputting the category data and coordinate data of a plurality of map elements in the plurality of local vector maps to be fused into a map data fusion model, and outputting global map data fused with the plurality of local vector maps to be fused; Wherein, the map data fusion model is trained according to the model training method described in any one of claims 1 to 9.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: implement the model training method described in any one of claims 1 to 9, or implement the map data fusion method described in claim 10.

14. A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute the model training method described in any one of claims 1 to 9, or implement the map data fusion method described in claim 10.

Images