Method for realizing BEV local map real-time perception by fusing SD Map

Abstract

The invention relates to a method for realizing BEV local map real-time perception by fusing SD Map, which comprises the following steps: encoding a navigation map to obtain a broken line sequence of the navigation map, and processing the broken line sequence of the navigation map by using a Transform encoder to obtain features of the navigation map; extracting features from the vehicle-mounted sensor data by using a map encoder, and converting the features into uniform BEV features; and based on a preset local map feature fusion network, fusing the features of the navigation map with the BEV features by using a multi-head cross attention mechanism to obtain fused BEV features, inputting the fused BEV features into a preset lane topology model decoder, and predicting a lane center line, traffic elements and an affinity matrix associated with the lane center line and the traffic elements. Therefore, the problems that in the prior art, a high-precision map is high in cost and difficult to maintain, and local map generation stability is poor are solved, and robustness and accuracy of a sensing system are improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method for realizing real-time perception of a local map of a BEV (Bird's Eye View) by integrating an SD Map (Standard Definition Map). Background Art

[0002] China's automotive industry is undergoing unprecedented changes, among which autonomous driving technology, as one of the core areas of intelligence, is developing rapidly. The environmental perception ability of autonomous vehicles, similar to the "eyes" and "touch" of a car, is the key to realizing intelligent driving. This perception ability enables the vehicle to perceive and integrate changes in the road environment in real time and accurately judge its own position and status.

[0003] One of the classic paradigms of current autonomous driving is to rely on high-definition maps (HDMaps) to provide detailed road geometry and semantic information, thereby enhancing the performance of autonomous driving modules such as vehicle positioning, perception, prediction, and planning. However, building and maintaining high-precision maps requires a lot of manpower and resources, which greatly limits its scalability and real-time performance. Therefore, the autonomous driving industry has begun to adopt a strategy of "heavy perception and light map", that is, reducing dependence on high-precision maps, thereby expanding the coverage of advanced autonomous driving functions.

[0004] In this context, the role of the computing platform and development system (Computing Brain Development System, CBDES) becomes crucial. CBDES consists of two parts: the computing base platform (Computing Base Brain, CBB) and the graphical software developer (Graphical ADAS-AD Software Developer, GAASD). CBB integrates the computing platform hardware, real-time kernel, middleware and functional software. Among them, the functional software is the core of the product and provides the basic algorithm components and framework required for the intelligent driving system. GAASD achieves efficient algorithm development by utilizing graphical methods.

[0005] Nowadays, there are more and more solutions for generating online local maps in real time. However, both pure vision and vision and radar fusion solutions face stability challenges in complex urban road environments, especially difficult to deal with occlusion problems, which urgently need to be solved. Summary of the invention

[0006] The present application provides a method for realizing real-time perception of BEV local maps by integrating SD Map, so as to solve the problems of high cost, difficult maintenance and poor stability of local map generation of high-precision maps in the prior art. Based on the advantages of relatively low cost, wide coverage and high reliability of navigation maps, the present invention combines navigation maps as auxiliary information sources and combines navigation maps with BEV local map perception algorithms to achieve more robust perception effects and cope with a wider range of driving environments and challenges.

[0007] The first aspect of the present application provides a method for integrating SD Map to realize real-time perception of BEV local map, including the following steps:

[0008] Acquire onboard sensor data of the current vehicle, and acquire a navigation map according to the current position of the current vehicle, wherein the navigation map includes road-level topology information and road type labels;

[0009] Encoding the navigation map to obtain a polyline sequence of the navigation map, and processing the polyline sequence of the navigation map using a Transformer encoder to obtain features of the navigation map;

[0010] extracting features from the onboard sensor data using a map encoder and converting the features into unified BEV features;

[0011] Based on a preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross attention mechanism to obtain fused BEV features, and the fused BEV features are input into a preset lane topology model decoder to predict the lane centerline, traffic elements, and an affinity matrix associated with the lane centerline and the traffic elements.

[0012] According to an embodiment of the present application, encoding the navigation map to obtain a polyline sequence includes:

[0013] Uniformly sampling each broken line in the navigation map to obtain a plurality of broken line points of each broken line, and performing sinusoidal embedding coding on the positions of the plurality of broken line points of each broken line;

[0014] Determine a one-hot vector of the road type label for each polyline;

[0015] The sinusoidal embedded coding of the fold point position of each fold line is connected with the one-hot vector of the corresponding road type label to obtain the sequence of each fold line, and the fold line sequence of the navigation map is obtained according to the sequence of each fold line.

[0016] According to an embodiment of the present application, the using of a Transformer encoder to process the broken line sequence of the navigation map to obtain features of the navigation map includes:

[0017] Embedding the polyline sequence of the navigation map into the linear layer of the Transformer encoder;

[0018] The multi-head self-attention mechanism is used to extract and encode global geometric and semantic information from the navigation map input to obtain the features of the navigation map.

[0019] According to one embodiment of the present application, the fused BEV features include at least one of 3D information and road level geometry information.

[0020] According to one embodiment of the present application, a cross-attention mechanism is applied between the BEV feature query and the navigation map feature in each intermediate layer of the Transformer encoder.

[0021] According to the fused SD Map of the embodiment of the present application, the real-time perception method of the BEV local map is implemented, the navigation map is encoded to obtain the broken line sequence of the navigation map, and the broken line sequence of the navigation map is processed by the Transformer encoder to obtain the features of the navigation map; the map encoder is used to extract features from the vehicle sensor data, and the features are converted into unified BEV features; based on the preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross attention mechanism to obtain fused BEV features, and input into the preset lane topology model decoder to predict the lane centerline, traffic elements, and the affinity matrix associated with the lane centerline and traffic elements. In this way, the problems of high cost, difficult maintenance, and poor stability of local map generation in the prior art are solved, and the robustness and accuracy of the perception system are improved while reducing costs.

[0022] The second aspect of the present application provides a device for realizing real-time perception of a BEV local map by integrating an SD Map, including:

[0023] An acquisition module, used to acquire onboard sensor data of a current vehicle, and acquire a navigation map according to a current position of the current vehicle, wherein the navigation map includes road-level topology information and road type labels;

[0024] A processing module, used for encoding the navigation map to obtain a polyline sequence of the navigation map, and processing the polyline sequence of the navigation map using a Transformer encoder to obtain features of the navigation map;

[0025] a feature extraction module, for extracting features from the vehicle-mounted sensor data using a map encoder and converting the features into a unified BEV feature;

[0026] A feature fusion module is used to fuse the features of the navigation map with the BEV features based on a preset local map feature fusion network using a multi-head cross attention mechanism to obtain fused BEV features, and input the fused BEV features into a preset lane topology model decoder to predict the lane centerline, traffic elements, and an affinity matrix associated with the lane centerline and the traffic elements.

[0027] According to one embodiment of the present application, the processing module is used to:

[0028] Uniformly sampling each broken line in the navigation map to obtain a plurality of broken line points of each broken line, and performing sinusoidal embedding coding on the positions of the plurality of broken line points of each broken line;

[0029] Determine a one-hot vector of the road type label for each polyline;

[0030] The sinusoidal embedded coding of the fold point position of each fold line is connected with the one-hot vector of the corresponding road type label to obtain the sequence of each fold line, and the fold line sequence of the navigation map is obtained according to the sequence of each fold line.

[0031] According to one embodiment of the present application, the processing module is used to:

[0032] Embedding the polyline sequence of the navigation map into the linear layer of the Transformer encoder;

[0033] The multi-head self-attention mechanism is used to extract and encode global geometric and semantic information from the navigation map input to obtain the features of the navigation map.

[0034] According to one embodiment of the present application, the fused BEV features include at least one of 3D information and road level geometry information.

[0035] According to one embodiment of the present application, a cross-attention mechanism is applied between the BEV feature query and the navigation map feature in each intermediate layer of the Transformer encoder.

[0036] According to the fused SD Map of the embodiment of the present application, a real-time perception device for a BEV local map is implemented, the navigation map is encoded to obtain a broken line sequence of the navigation map, and the broken line sequence of the navigation map is processed using a Transformer encoder to obtain the features of the navigation map; features are extracted from the on-board sensor data using a map encoder, and the features are converted into unified BEV features; based on a preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross-attention mechanism to obtain fused BEV features, and the fused BEV features are input into a preset lane topology model decoder to predict the lane centerline, traffic elements, and the affinity matrix associated with the lane centerline and traffic elements. Thus, the problems of high cost, difficult maintenance, and poor stability of local map generation in the prior art are solved, and the robustness and accuracy of the perception system are improved while reducing costs.

[0037] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for realizing real-time perception of BEV local maps by integrating SD Map as described in the above embodiment.

[0038] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the method for realizing real-time perception of BEV local maps by integrating SD Map as described in the above embodiments.

[0039] The fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, is used to implement the method for realizing real-time perception of BEV local maps by integrating SD Map as described in the above embodiments.

[0040] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A schematic diagram of components of a method for implementing a real-time perception of a BEV local map by integrating an SD Map according to an embodiment of the present application;

[0043] Figure 2 A flowchart of a method for implementing real-time perception of a BEV local map by integrating an SD Map according to an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a local map real-time perception network according to an embodiment of the present application;

[0045] Figure 4 It is a block diagram of a device for realizing real-time perception of a BEV local map by integrating an SD Map according to an embodiment of the present application;

[0046] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0048] The following describes a method for implementing real-time perception of BEV local maps by integrating SD Map according to an embodiment of the present application with reference to the accompanying drawings.

[0049] Before introducing the method for implementing BEV local map real-time perception by integrating SD Map in an embodiment of the present application, we first introduce the light map solution involved in the method for implementing BEV local map real-time perception by integrating SD Map in the present application and the components of the method for implementing BEV local map real-time perception by integrating SDMap.

[0050] Specifically, the light map solution refers to the use of lightweight navigation maps instead of heavyweight high-precision maps, combined with the powerful perception capabilities of a single vehicle to achieve autonomous driving. Navigation maps are low-cost to obtain and are available in most parts of the world. Different from the complete semantic and geometric lane-level details in high-precision maps, navigation maps use metadata to mark road-level topology, and do not need to be updated as frequently as high-precision maps unless the road topology changes significantly. The wider coverage and included road topology information of navigation maps can supplement on-board cameras for lane detection and topology reasoning. In particular, when roads merge or exit roads are obscured and invisible, navigation maps can provide prior information for more accurate downstream planning.

[0051] Therefore, this application uses navigation maps to improve online lane topology reasoning in the absence of high-precision maps, and proposes a method for integrating SD Map with a computing infrastructure platform to achieve real-time perception of BEV local maps.

[0052] like Figure 1As shown, the method for realizing real-time perception of BEV local map by integrating SD Map proposed in this application includes three parts: lane topology based on Transformer, SD Map input and encoding representation, and fusion of SD Map features and BEV features.

[0053] The following is a detailed introduction to the implementation process of the method of integrating SD Map to achieve real-time perception of BEV local maps, combining the Transformer-based lane topology, SD Map input and encoding representation, and the fusion of SD Map features and BEV features.

[0054] Specifically, Figure 2 A flow chart of a method for implementing real-time perception of BEV local maps by integrating SD Map provided in an embodiment of the present application.

[0055] like Figure 2 As shown, the method for realizing real-time perception of BEV local map by integrating SD Map includes the following steps:

[0056] In step S201, the onboard sensor data of the current vehicle is obtained, and a navigation map is obtained according to the current position of the current vehicle, wherein the navigation map includes road-level topology information and road type labels.

[0057] Specifically, the embodiment of the present application obtains standard-definition maps from OpenStreetMap (OSM), which provides a crowdsourcing platform for standard-definition maps and detailed geographic information of various places around the world. Specifically, OSM's navigation map contains road-level topology (i.e., road geometry and connectivity) and annotated road type information for each road segment (such as highways, residential roads, and crosswalks). For each frame, the embodiment of the present application extracts a local navigation map from OSM based on the ego-vehicle (current vehicle) position, and the resulting navigation map contains M polylines, each of which corresponds to a road segment. It should be noted that the embodiment of the present application uses G p The point locations of the polyline are converted to the coordinates of the ego vehicle. In addition, each polyline is further annotated with a specific road type label.

[0058] In step S202, the navigation map is encoded to obtain a polyline sequence of the navigation map, and the polyline sequence of the navigation map is processed using a Transformer encoder to obtain features of the navigation map.

[0059] Furthermore, in some embodiments, the navigation map is encoded to obtain a broken line sequence, including: uniformly sampling each broken line in the navigation map to obtain multiple broken line points of each broken line, and performing sinusoidal embedding encoding on the multiple broken line point positions of each broken line; determining the one-hot vector of the road type label of each broken line; connecting the sinusoidal embedding code of the broken line point position of each broken line with the one-hot vector of the corresponding road type label to obtain a sequence of each broken line, and obtaining the broken line sequence of the navigation map according to the sequence of each broken line.

[0060] Specifically, in order to encode the navigation map in a form that can be used by the following lane topology model, the embodiment of the present application introduces a broken line sequence representation and a Transformer encoder to obtain the final navigation map feature representation of the scene.

[0061] First, for a given scene's navigation map, a fixed number of N polyline points are uniformly sampled along M polylines, using Representation, and sinusoidal embeddings of different frequencies are used to encode the positions of broken line points. Sinusoidal embedding enhances the sensitivity to position changes, which is beneficial to the model, enabling it to effectively infer the structure of the broken line. The embodiment of the present application takes into account a vertical polyline with small curvature, which is characterized by very similar y-axis values ​​of all points. Directly inputting these point coordinates into the model may result in insufficient distinction of the curvature. Therefore, the embodiment of the present application uses sinusoidal embedding to make this distinction obvious, thereby improving the interpretability of the model for these features.

[0062] Among them, for a given coordinate position p∈{x i ,y i} and embedding dimension j∈{1···d / 2}, its sinusoidal embedding can be expressed as:

[0063]

[0064] Where d is the size of the sinusoidal embedding, T = 1000, is the temperature scale. This makes (x i ,y i ) coordinates can be transformed into their corresponding d-dimensional sinusoidal embeddings. In actual practice, the coordinates of each polyline relative to the BEV range are normalized to the range (0, 2π) before embedding.

[0065] Secondly, for the main lane types existing in OSM, the embodiment of the present application uses a one-hot vector of the road type label with a dimension of K to represent it, which not only ensures that the input value is standardized between 0 and 1, but also solves the situation where a road section may belong to multiple road types.

[0066] Finally, the polyline position embedding is concatenated with the road type as a one-hot vector representing the polyline sequence of the final navigation map, which has the shape of M × (N·d+K).

[0067] Furthermore, in some embodiments, a Transformer encoder is used to process a broken line sequence of a navigation map to obtain features of the navigation map, including: embedding the broken line sequence of the navigation map into a linear layer of the Transformer encoder; and extracting and encoding global geometric and semantic information from the navigation map input using a multi-head self-attention mechanism to obtain features of the navigation map.

[0068] For a given navigation map's polyline sequence representation, the present application embodiment uses a Transformer encoder to learn the feature representation of the downstream lane topology task, embedding the polyline sequence into a linear layer, which is a typical Transformer encoder architecture, ensuring that the discrete, single-hot representation of the road type can be meaningfully converted into a continuous space. Then, an L-layer multi-head self-attention is used to extract and encode global geometric and semantic information from the navigation map input, and the output shape obtained, that is, the features of the navigation map, is M×H, where H is the feature dimension generated by the self-attention layer.

[0069] In step S203 , a map encoder is used to extract features from the on-board sensor data and convert the features into unified BEV features.

[0070] Specifically, the embodiment of the present application refers to MapTRv2, an end-to-end framework for online vectorized high-precision map construction, to achieve real-time local map perception. First, the map elements are modeled as a set of equivalent permuted point sets to accurately describe the shape of the map elements and stabilize the learning process. Secondly, based on the new modeling, an end-to-end framework is used for efficient online vectorized high-precision map construction. The local map real-time perception network in the embodiment of the present application adopts the decoder-encoder paradigm, and the overall structure is as follows: Figure 3 As shown, it includes a map encoder and a map decoder.

[0071] Among them, the map encoder extracts features from the sensor data and converts the features into a unified BEV feature representation. Local map real-time perception is compatible with various on-board sensors and supports various PV2BEV conversion methods, such as CVT (Continuously Variable Transmission), LSS (Lateral Shift Suppression), deformable attention, GKT (Geometric Kernel Transform) and IPM (Inverse Perspective Mapping). In order to explicitly utilize depth information, the embodiment of the present application can select LSS-based BEVPoolv2 as the default conversion method.

[0072] Furthermore, the map decoder consists of a map query and several decoder layers. Each decoder layer updates the map query using self-attention and cross-attention.

[0073] MapTRv2 uses a hierarchical query embedding scheme to explicitly encode each map element. Specifically, a set of instance-level queries and a set of point-level queries shared by all instances are defined, and each map element corresponds to a set of hierarchical queries.

[0074] Furthermore, MapTR uses general self-attention to make hierarchical queries exchange information with each other (both between instances and within instances). As the number of queries increases, the computational cost and memory consumption increase dramatically. In MapTRv2, in order to reduce computational and memory consumption, decoupled self-attention is adopted, that is, along the two dimensions of between instances and within instances. Decoupled self-attention greatly reduces memory consumption and computational complexity, and has higher performance than ordinary self-attention.

[0075] Furthermore, the cross attention in the decoder is designed to make the map query interact with the input features. MapTRv2 studies three types of cross attention: BEV-based, PV-based, and hybrid cross attention. For BEV-based cross attention, deformable attention is used to make the hierarchical query interact with the BEV features. For 2D map construction, each query predicts the 2D normalized BEV coordinates of the reference point. For 3D map construction, each query predicts the 3D normalized 3D coordinates of the reference point. Then, the BEV features around the reference point are sampled and the query is updated. Map elements usually have irregular shapes and require long-range context. Each map element corresponds to a set of reference points with flexible and dynamic distribution. The reference points can adapt to the arbitrary shape of the map element and capture the information context for map element learning. For PV-based cross attention, the reference point is projected to the PV image, and then the features around the projected reference point are sampled, and the dense BEV features are deprecated. Hybrid cross attention is a combination of the above two cross attention methods.

[0076] In step S204, based on the preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross attention mechanism to obtain fused BEV features, and the fused BEV features are input into a preset lane topology model decoder to predict the lane centerline, traffic elements, and the affinity matrix associated with the lane centerline and traffic elements.

[0077] In some embodiments, the fused BEV features include at least one of 3D information and road level geometry information.

[0078] Specifically, the navigation map representation of the local map feature fusion network can now be used by any Transformer-based lane topology model. With the release of the lane topology task with the OpenLane-V2 dataset, major examples of lane detection and relational reasoning models have emerged, including the BEV Transformer encoder and the DETR (Detection Transformer) based map decoder. In this work, the present application uses the feature representation of the navigation map to enhance these advanced lane topology models.

[0079] In current methods, lane topology models mainly rely on multi-view camera inputs and have difficulty predicting occluded or distant areas. By introducing additional navigation map information, the local map feature fusion network of the embodiment of the present application can supplement these blind spots, allowing the model to reason about these areas more accurately.

[0080] In order to utilize the current advanced lane topology models, the local map feature fusion network of an embodiment of the present application fuses the navigation map features with the intermediate BEV feature representation by utilizing a multi-head cross-attention mechanism. This method applies cross-attention between the BEV feature query and the navigation map features in each intermediate layer of the model encoder, making the method compatible with almost all Transformer-based lane topology models.

[0081] In practice, the local map feature fusion network performs cross-attention on the navigation map features after each spatial cross-attention operation, so that the fused BEV features include not only the 3D information obtained from the image, but also the road-level geometric information extracted from the navigation map.

[0082] Finally, the decoder of the lane topology model will use these navigation map enhanced features as input to predict the lane centerline, traffic elements, and the associated affinity matrix between the lane centerline and traffic elements. The introduction of this method significantly improves the perception ability of the lane topology model in complex environments, especially in the face of occlusion and long-distance areas.

[0083] Therefore, the present invention explores the advantages of readily available and cost-effective navigation maps, and proposes a method for improving the real-time perception of BEV local maps by fusing navigation maps. The rich semantic information provided by the navigation map can help the vehicle more accurately identify and understand the surrounding environment. By fusing the navigation map, the reliance on high-precision sensors can be reduced to a certain extent, especially when the vehicle is blocked or in bad weather conditions, which can enable the vehicle to better predict potential dangers and complex traffic conditions and improve driving safety. In addition, this method can be integrated into any Transformer-based lane topology method without any additional conditions, and can significantly enhance the lane detection and topology prediction of the current most advanced local map real-time construction scheme.

[0084] According to the fused SD Map of the embodiment of the present application, the real-time perception method of the BEV local map is implemented, the navigation map is encoded to obtain the broken line sequence of the navigation map, and the broken line sequence of the navigation map is processed by the Transformer encoder to obtain the features of the navigation map; the map encoder is used to extract features from the vehicle sensor data, and the features are converted into unified BEV features; based on the preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross attention mechanism to obtain fused BEV features, and input into the preset lane topology model decoder to predict the lane centerline, traffic elements, and the affinity matrix associated with the lane centerline and traffic elements. In this way, the problems of high cost, difficult maintenance, and poor stability of local map generation in the prior art are solved, and the robustness and accuracy of the perception system are improved while reducing costs.

[0085] Next, a device for realizing real-time perception of BEV local map by integrating SD Map according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0086] Figure 4 It is a block diagram of a device for realizing real-time perception of BEV local maps by integrating SD Map in an embodiment of the present application.

[0087] like Figure 4 As shown, the device 10 for realizing real-time perception of BEV local map by integrating SD Map includes: an acquisition module 100, a processing module 200, a feature extraction module 300 and a feature fusion module 400.

[0088] Among them, the acquisition module 100 is used to obtain the on-board sensor data of the current vehicle, and obtain the navigation map according to the current position of the current vehicle, wherein the navigation map includes road-level topology information and road type labels; the processing module 200 is used to encode the navigation map to obtain a broken line sequence of the navigation map, and use the Transformer encoder to process the broken line sequence of the navigation map to obtain the features of the navigation map; the feature extraction module 300 is used to extract features from the on-board sensor data using a map encoder, and convert the features into unified BEV features; the feature fusion module 400 is used to fuse the features of the navigation map with the BEV features based on a preset local map feature fusion network using a multi-head cross-attention mechanism to obtain fused BEV features, and input the fused BEV features into a preset lane topology model decoder to predict the lane centerline, traffic elements, and the affinity matrix associated with the lane centerline and traffic elements.

[0089] Furthermore, in some embodiments, the processing module 200 is used to: uniformly sample each broken line in the navigation map to obtain multiple broken line points of each broken line, and perform sinusoidal embedding coding on the multiple broken line point positions of each broken line; determine the one-hot vector of the road type label of each broken line; connect the sinusoidal embedding coding of the broken line point position of each broken line with the one-hot vector of the corresponding road type label to obtain a sequence of each broken line, and obtain the broken line sequence of the navigation map according to the sequence of each broken line.

[0090] Furthermore, in some embodiments, the processing module 200 is used to: embed the broken line sequence of the navigation map into the linear layer of the Transformer encoder; and utilize a multi-head self-attention mechanism to extract and encode global geometric and semantic information from the navigation map input to obtain features of the navigation map.

[0091] Further, in some embodiments, the fused BEV features include at least one of 3D information and road level geometry information.

[0092] Further, in some embodiments, a cross-attention mechanism is applied between the BEV feature queries and the navigation map features in each intermediate layer of the Transformer encoder.

[0093] It should be noted that the aforementioned explanation of the embodiment of the method for implementing the real-time perception of BEV local maps by integrating SD Map is also applicable to the device for implementing the real-time perception of BEV local maps by integrating SD Map in this embodiment, and will not be repeated here.

[0094] According to the fused SD Map of the embodiment of the present application, a real-time perception device for a BEV local map is implemented, the navigation map is encoded to obtain a broken line sequence of the navigation map, and the broken line sequence of the navigation map is processed using a Transformer encoder to obtain the features of the navigation map; features are extracted from the on-board sensor data using a map encoder, and the features are converted into unified BEV features; based on a preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross-attention mechanism to obtain fused BEV features, and the fused BEV features are input into a preset lane topology model decoder to predict the lane centerline, traffic elements, and the affinity matrix associated with the lane centerline and traffic elements. Thus, the problems of high cost, difficult maintenance, and poor stability of local map generation in the prior art are solved, and the robustness and accuracy of the perception system are improved while reducing costs.

[0095] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0096] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0097] When the processor 502 executes the program, the method for realizing real-time perception of BEV local map by fusing SD Map provided in the above embodiment is implemented.

[0098] Furthermore, the electronic device further comprises:

[0099] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0100] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0101] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0104] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0105] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for realizing real-time perception of BEV local maps by integrating SD Map.

[0106] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned method for realizing real-time perception of BEV local maps by integrating SD Map.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for realizing real-time perception of BEV local maps by integrating SD Map, characterized in that: The following steps are involved: Acquire onboard sensor data of the current vehicle, and acquire a navigation map according to the current position of the current vehicle, wherein the navigation map includes road-level topology information and road type labels; Encoding the navigation map to obtain a polyline sequence of the navigation map, and processing the polyline sequence of the navigation map using a Transformer encoder to obtain features of the navigation map; extracting features from the onboard sensor data using a map encoder and converting the features into unified BEV features; Based on a preset local map feature fusion network, the features of the navigation map are fused with the BEV features using a multi-head cross attention mechanism to obtain fused BEV features, and the fused BEV features are input into a preset lane topology model decoder to predict the lane centerline, traffic elements, and an affinity matrix associated with the lane centerline and the traffic elements.

2. The method according to claim 1, characterized in that The step of encoding the navigation map to obtain a polyline sequence includes: Uniformly sampling each broken line in the navigation map to obtain a plurality of broken line points of each broken line, and performing sinusoidal embedding coding on the positions of the plurality of broken line points of each broken line; Determine a one-hot vector of the road type label for each polyline; The sinusoidal embedded coding of the fold point position of each fold line is connected with the one-hot vector of the corresponding road type label to obtain the sequence of each fold line, and the fold line sequence of the navigation map is obtained according to the sequence of each fold line.

3. The method according to claim 1, characterized in that The using of the Transformer encoder to process the broken line sequence of the navigation map to obtain features of the navigation map includes: Embedding the polyline sequence of the navigation map into the linear layer of the Transformer encoder; The multi-head self-attention mechanism is used to extract and encode global geometric and semantic information from the navigation map input to obtain the features of the navigation map.

4. The method according to claim 1, characterized in that: The fused BEV features include at least one of 3D information and road level geometry information.

5. The method according to claim 1, characterized in that A cross-attention mechanism is applied between the BEV feature queries and the navigation map features in each intermediate layer of the Transformer encoder.

6. A device for realizing real-time perception of BEV local maps by integrating SD Map, characterized by comprising: An acquisition module, used to acquire onboard sensor data of a current vehicle, and acquire a navigation map according to a current position of the current vehicle, wherein the navigation map includes road-level topology information and road type labels; A processing module, used for encoding the navigation map to obtain a polyline sequence of the navigation map, and processing the polyline sequence of the navigation map using a Transformer encoder to obtain features of the navigation map; a feature extraction module, for extracting features from the vehicle-mounted sensor data using a map encoder and converting the features into a unified BEV feature; A feature fusion module is used to fuse the features of the navigation map with the BEV features based on a preset local map feature fusion network using a multi-head cross attention mechanism to obtain fused BEV features, and input the fused BEV features into a preset lane topology model decoder to predict the lane centerline, traffic elements, and an affinity matrix associated with the lane centerline and the traffic elements.

7. The device according to claim 6, characterized in that The processing module is used to: Uniformly sampling each broken line in the navigation map to obtain a plurality of broken line points of each broken line, and performing sinusoidal embedding coding on the positions of the plurality of broken line points of each broken line; Determine a one-hot vector of the road type label for each polyline; The sinusoidal embedded coding of the fold point position of each fold line is connected with the one-hot vector of the corresponding road type label to obtain the sequence of each fold line, and the fold line sequence of the navigation map is obtained according to the sequence of each fold line.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for realizing real-time perception of BEV local maps by integrating SD Map as described in any one of claims 1 to 5.

9. A computer storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for realizing real-time perception of BEV local maps by integrating SD Map as described in any one of claims 1-5.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, is used to implement the SAR crop classification method based on a multi-scale time-varying information extraction network as described in any one of claims 1 to 5.

Images