Map construction method, device, vehicle, storage medium and computer program product

Abstract

This application relates to the field of assisted driving technology, and in particular to a map building method, device, vehicle, storage medium, and computer program product. The method includes: acquiring image features of the vehicle's surroundings and point cloud features perceived by LiDAR; converting image features using depth information from the point cloud features to obtain first BEV features, and optimizing the point cloud features based on the image features; fusing the first BEV features and the optimized point cloud features to obtain second BEV features; generating a semantic map based on the second BEV features and temporal BEV features; identifying one or more map elements from the semantic map; and constructing map information based on the map elements. This solves the problems of related technologies that use only current-time sensor data to build online maps, which are unsuitable for complex environments and suffer from poor accuracy.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and in particular to a map building method, device, vehicle, storage medium, and computer program product. Background Technology

[0002] Accurate map information is crucial for autonomous vehicles to accurately predict the future trajectories of surrounding vehicles or pedestrians, thereby safely planning their own routes. Traditional methods for building high-precision maps typically begin by using specialized data collection vehicles equipped with expensive equipment to conduct multiple scans to obtain road point cloud data. Then, SLAM (Simultaneous Localization and Mapping) technology is used for global point cloud alignment, and finally, manual semantic annotation is performed on the map. This method requires significant manpower and cannot provide real-time updates to changes in the road environment, making it difficult to meet the real-time and safety requirements of autonomous vehicles.

[0003] The general approach in related technologies involves extracting features from onboard sensor data, transforming these features into BEV (Battery, Vehicle, and Electricity) features, and then using a task-specific map network to generate the required high-precision map information. However, this approach has several shortcomings: 1) it often only considers using data from a single sensor; for camera sensors, some models even only use data from the forward-looking camera. 2) it considers using data from multiple sensors, but only fuses the features of each sensor at the BEV level to construct the map, often resulting in inaccuracies and missing information. Summary of the Invention

[0004] This application provides a map building method, apparatus, vehicle, storage medium, and computer program product to solve the problems in related technologies that use only sensor data at the current moment to build online maps, which are not suitable for complex environmental scenarios and have poor accuracy.

[0005] The first aspect of this application provides a map construction method, including the following steps: acquiring image features around a vehicle and point cloud features perceived by a LiDAR; converting image features using depth information of the point cloud features to obtain first BEV features, and optimizing the point cloud features based on the image features; fusing the first BEV features and the optimized point cloud features to obtain second BEV features, generating a semantic map based on the second BEV features and temporal BEV features; identifying the semantic map to obtain one or more map elements, and constructing map information based on the map elements.

[0006] Optionally, identifying one or more map elements from the semantic map includes: performing differentiable rasterization on the semantic map and generating one or more map elements based on the processed semantic map; or, identifying an ordered set of points in the semantic map, optimizing the ordered set of points using self-attention and cross-attention, and generating one or more map elements based on the optimized ordered set of points.

[0007] Optionally, before generating the semantic map based on the second BEV feature and the temporal BEV feature, the method further includes: obtaining historical frame BEV features prior to the second BEV feature; generating a query value based on the second BEV feature and generating key-value pairs based on the historical BEV features; establishing a target relationship between the query value and the query value using a cross-attention mechanism, and generating temporal BEV features based on the target relationship.

[0008] Optionally, generating time-series BEV features based on the target relationship includes: transforming the second BEV features and historical BEV features to the target coordinate system; generating a transformation matrix of the second BEV features and historical BEV features in the target coordinate system; updating the historical BEV features based on the transformation matrix and the target relationship; and optimizing the second BEV features using the updated historical BEV features to obtain time-series BEV features.

[0009] Optionally, the first BEV feature is obtained by converting image features using the depth information of point cloud data, including: projecting point cloud data onto an image plane to obtain a sparse depth map; processing the sparse depth map using a depth completion network to obtain a dense depth map; and generating the first BEV feature with depth information based on the dense depth map.

[0010] A second aspect of this application provides a map building apparatus, comprising: a first acquisition module for acquiring image features around a vehicle and point cloud features perceived by a lidar; a processing module for converting image features using depth information of the point cloud features to obtain first BEV features, and optimizing the point cloud features based on the image features; a fusion module for fusing the first BEV features and the optimized point cloud features to obtain second BEV features, and generating a semantic map based on the second BEV features and temporal BEV features; and a building module for identifying one or more map elements from the semantic map, and building map information based on the map elements.

[0011] Optionally, the building module is further configured to: perform differentiable rasterization on the semantic map, and generate one or more map elements based on the processed semantic map; or, identify ordered point sets in the semantic map, optimize the ordered point sets using self-attention and cross-attention, and generate one or more map elements based on the optimized ordered point sets.

[0012] Optionally, the map building apparatus further includes: a second acquisition module, used to acquire historical frame BEV features prior to the second BEV features before generating a semantic map based on the second BEV features and the temporal BEV features; a generation module, used to generate query values ​​based on the second BEV features and generate key-value pairs based on the historical BEV features; establish a target relationship between the query values ​​and the query values ​​using a cross-attention mechanism, and generate temporal BEV features based on the target relationship.

[0013] Optionally, the generation module is further configured to transform the second BEV feature and the historical BEV feature to the target coordinate system; generate a transformation matrix of the second BEV feature and the historical BEV feature in the target coordinate system; update the historical BEV feature based on the transformation matrix and the target relationship; and optimize the second BEV feature using the updated historical BEV feature to obtain the time-series BEV feature.

[0014] Optionally, the processing module is further configured to: project point cloud data onto an image plane to obtain a sparse depth map; process the sparse depth map using a depth completion network to obtain a dense depth map; and generate a first BEV feature with depth information based on the dense depth map.

[0015] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the map building method as described in the above embodiments.

[0016] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the map construction method as described in the above embodiments.

[0017] A fifth aspect of this application provides a computer program product having a computer program or instructions stored thereon, which, when executed, performs a map construction method as described in the above embodiments.

[0018] Therefore, this application has at least the following beneficial effects:

[0019] This application's embodiments can accurately predict the map information surrounding a vehicle by fusing BEV features from multiple sensors and temporal BEV features. It considers not only the spatial information at the current moment but also utilizes additional temporal information. Even when important elements in the current map are occluded, it can still obtain more comprehensive map information about the vehicle's surroundings. Therefore, it solves the problems of related technologies that only use sensor data at the current moment to construct online maps, which are unsuitable for complex environmental scenarios and suffer from poor accuracy.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 This is a map construction architecture diagram provided according to the embodiments of this application;

[0023] Figure 2 This is a flowchart of a map construction method provided according to an embodiment of this application;

[0024] Figure 3 This is a diagram illustrating the time-series BEV feature fusion process according to an embodiment of this application;

[0025] Figure 4 This is a block diagram of a map building apparatus provided according to an embodiment of this application;

[0026] Figure 5 This is a structural schematic diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The following description, with reference to the accompanying drawings, outlines a map building method, apparatus, vehicle, storage medium, and computer program product according to embodiments of this application. Addressing the problems mentioned in the background section, this application provides a map building method. This method, by fusing BEV features from multiple sensors and temporal BEV features, can accurately predict map information surrounding a vehicle. It considers not only spatial information at the current moment but also utilizes additional temporal information. Even when important elements in the current map are occluded, it can still obtain more comprehensive map information about the vehicle's surroundings. This solves the problems of related technologies that rely solely on current sensor data to build online maps, making them unsuitable for complex environments and resulting in poor accuracy.

[0029] Before introducing this application, the Encoder-Decoder framework involved in this application will first be explained. It can deeply integrate multiple sensor inputs such as cameras, LiDAR, and millimeter-wave radar. As a general, fully online map building framework, the embodiments of this application can be directly applied to multiple fields such as mass-produced autonomous vehicles and robots. For the sake of brevity, the core idea of ​​this invention will only be illustrated below using camera and LiDAR sensor inputs. The entire framework mainly includes five core modules: camera BEV encoder, LiDAR BEV encoder, BEV fusion, temporal BEV fusion, and map decoder. Figure 1 As shown.

[0030] 1. The camera BEV encoder first extracts the features of the panoramic multi-view camera images, and then uses the depth information obtained from the LiDAR point cloud to supervise the conversion process of image features to camera BEV features.

[0031] 2. The LiDAR BEV encoder first extracts the point cloud BEV features, and then uses the image features extracted by the camera encoder to further optimize the point cloud BEV features.

[0032] 3. Next, the BEV fusion module fuses the camera BEV features and the optimized point cloud BEV features to obtain the multi-sensor fused BEV features.

[0033] 4. In order to make better use of historical information to complete the construction of high-precision maps, the temporal BEV fusion module will use historical frame BEV features to update and optimize the current frame BEV features.

[0034] 5. The obtained temporal fusion BEV features can then be processed by a task-specific map decoder to obtain the required local high-precision map information.

[0035] Specifically, Figure 2 This is a flowchart illustrating a map construction method provided in an embodiment of this application.

[0036] like Figure 2 As shown, the map construction method includes the following steps:

[0037] In step S101, image features around the vehicle and point cloud features perceived by the lidar are acquired.

[0038] It is understandable that autonomous vehicles are typically equipped with multiple cameras covering the surrounding environment, as well as one or more LiDAR (Light Detection and Ranging) systems. Cameras usually possess rich semantic information, and surround-view camera data can acquire more comprehensive environmental information around the vehicle. However, camera data lacks accurate depth perception capabilities, while LiDAR has accurate depth information, but its point cloud data is very sparse. Therefore, embodiments of this application can acquire N surround-view camera images {I1,…,I...} around the vehicle using multiple cameras. N The system uses LiDAR to acquire point cloud data P around the vehicle, and the camera BEV encoder and LiDAR BEV encoder extract image features and point cloud features respectively.

[0039] In step S102, the image features are converted using the depth information of the point cloud features to obtain the first BEV features, and the point cloud features are optimized based on the image features.

[0040] In one embodiment of this application, the method of converting image features using depth information of point cloud data to obtain a first BEV feature includes: projecting point cloud data onto an image plane to obtain a sparse depth map; processing the sparse depth map using a depth completion network to obtain a dense depth map; and generating a first BEV feature with depth information based on the dense depth map.

[0041] For ease of description, the first BEV feature in the embodiments of this application can be the camera BEV feature.

[0042] Understandably, the camera BEV encoder first uses a traditional convolution-based backbone to extract 2D image features from each camera's viewpoint; then, it utilizes depth information obtained from LiDAR point clouds to supervise the transformation of 2D image features to camera BEV features. Specifically, for the image I acquired by the i-th camera at time t... t,i CNN Backbone (Generally, common convolutional networks such as ResNet and EfficientNet are used as the shared backbone) From image I t,i Extracting 2D image features Then, a 2D-3D viewpoint transformation is used to project the 2D image features from the camera space to the corresponding BEV space. Currently, typical pure vision viewpoint transformation solutions employ methods such as IPM, LSS, and BEVFormer. Geometry-based IPM assumes a level ground and uses a homography matrix to inversely map image information from the camera space to the BEV space; however, this assumption of a level ground is often unsuitable for complex real-world autonomous driving scenarios. Another geometry-based method (LSS) uses only semantic loss to implicitly supervise the learning of depth information. BEVFormer uses the Transformer paradigm to establish the connection between 2D image features and BEV features. However, all of these methods struggle to generate sufficiently accurate depth information. Furthermore, in adverse weather conditions such as darkness or rain, the camera's visual perception capabilities degrade to varying degrees.

[0043] Therefore, embodiments of this application can consider utilizing LiDAR sparse point cloud information to explicitly supervise the viewpoint transformation of 2D image features to BEV space. Specifically, the camera projection matrix corresponding to the viewpoint can be used to project the 3D LiDAR point cloud P t Projecting the image onto the corresponding image plane yields the corresponding sparse depth map. Then, a depth completion network (such as DepthNet) is used to obtain the corresponding dense depth map. Now, using this dense depth information, a BEV feature generation mechanism similar to that in the CaDDN model can be employed to directly generate camera BEV features with depth prior. At this point, fusing the BEV feature information from N cameras yields the final camera BEV feature. For example, a simple averaging operator can be used to fuse these feature information. No specific limitations are imposed.

[0044] In this embodiment, the LiDAR BEV encoder first extracts point cloud BEV features using a specific LiDAR backbone; then, it utilizes 2D image features to enhance and optimize the point cloud features. Specifically, for the point cloud data P acquired by the LiDAR at time t... t LiDAR Backbone (Generally using PointPillars or VoxelNet, etc.) from point cloud data P t Extracting LiDAR BEV features For example, first use a shared PointNet to aggregate LiDAR point clouds into pillars, and then use PointPillars to obtain... However, it is worth noting that LiDAR point cloud data itself is sparse, with very short effective measurements of the ground and a lack of rich semantic information. Therefore, embodiments of this application may consider utilizing rich image features to enhance the extraction of LiDAR point cloud features. The LiDAR BEV enhancement network here can adopt an Encoder-Decoder network structure similar to TransFusion.

[0045] In step S103, the first BEV feature and the optimized point cloud feature are fused to obtain the second BEV feature, and a semantic map is generated based on the second BEV feature and the temporal BEV feature.

[0046] Based on the above embodiments, the embodiments of this application can perform BEV fusion between the depth-supervised camera BEV features and the LiDAR point cloud features optimized from image features. There are many different methods for fusion, such as simple splicing or BEVFusion, without specific limitations.

[0047] Furthermore, the fused second BEV feature only considers the environmental information acquired by the sensors at the current moment. However, during the operation of an autonomous vehicle, the presence of other vehicles or obstacles around the vehicle may obstruct key information such as intersections or the ground, preventing the onboard sensors from capturing this crucial information. This inconsistency in the temporal sequence of local maps at different times poses a significant challenge to tasks such as motion planning for autonomous vehicles. To further ensure the accuracy of map construction, this embodiment can also incorporate historical information to supplement the perception of the current environment, i.e., by comprehensively generating a semantic map by acquiring the temporal BEV feature and the second BEV feature obtained in the above embodiment.

[0048] In one embodiment of this application, before generating a semantic map based on the second BEV feature and the temporal BEV feature, the method further includes: obtaining historical frame BEV features prior to the second BEV feature; generating a query value based on the second BEV feature and generating key-value pairs based on the historical BEV features; establishing a target relationship between the query value and the query value using a cross-attention mechanism, and generating temporal BEV features based on the target relationship.

[0049] The common practice in existing technologies to introduce historical information to supplement the perception of the current environment is to stack multiple consecutive historical frame information and pass them forward to the current frame together. However, this approach requires a lot of memory space and is difficult to apply to the construction of long-term online maps.

[0050] The embodiments of this application can introduce historical frame BEV features to enhance and update the current frame BEV features (i.e., the second BEV feature). For example... Figure 3 As shown, utilizing the current features Generate query value (query) and historical features. Generate key-value pairs (keys and values) and then use a standard cross-attention mechanism to establish the relationship between current BEV features and historical BEV features. here, It can enhance the spatial perception of feature locations of the time-series BEV fusion module, making it more trusting of current frame features in the environment near the vehicle and more reliant on historical frame features in the environment at a distance.

[0051] Furthermore, embodiments of this application can generate time-series BEV features based on a target relationship, including: transforming the second BEV feature and historical BEV feature to a target coordinate system; generating a transformation matrix of the second BEV feature and historical BEV feature in the target coordinate system; updating the historical BEV feature based on the transformation matrix and the target relationship; and optimizing the second BEV feature using the updated historical BEV feature to obtain the time-series BEV feature.

[0052] Specifically, embodiments of this application can use recurrent neural networks (such as LSTM, GRU, etc.) to weight and fuse the features optimized by the cross-attention model. And update the characteristics of the historical cache. Note that in actual implementation, this application uses the vehicle coordinate system. Therefore, each frame of BEV features needs to be appropriately transformed to align with the same BEV space. T∈SE(3) represents the transformation matrix between two consecutive frame coordinate systems. Finally, the RNN model outputs the updated and optimized BEV features. The embodiments of this application can optimize the second BEV feature using the updated historical BEV feature to obtain the time-series BEV feature, which can better predict the local map information around the vehicle.

[0053] In step S104, the semantic map is identified to obtain one or more map elements, and map information is constructed based on the map elements.

[0054] Map elements can include lane dividers, road boundaries, and pedestrian crossings.

[0055] In one embodiment of this application, identifying a semantic map to obtain one or more map elements includes: performing differentiable rasterization processing on the semantic map, and generating one or more map elements based on the processed semantic map; or, identifying an ordered set of points in the semantic map, optimizing the ordered set of points using self-attention and cross-attention, and generating one or more map elements based on the optimized ordered set of points.

[0056] In this application embodiment, the temporal BEV features obtained by spatiotemporal awareness fusion in the above embodiments It can be used by the map decoder module for semantic or vector map elements {L j The prediction of BEV features is described in this application. This embodiment can use existing map models to predict high-precision map information from temporal BEV features obtained through multi-sensor spatiotemporal perception fusion and second BEV features.

[0057] In practical implementation, this embodiment can directly use the traditional Deformable DETR as the core component of the map decoder, and the model outputs a local semantic map. Then, the obtained semantic map can be rasterized (e.g., appropriately expanded outward by a few pixels according to certain rules). After a series of post-processing operations, accurate map information required for path planning and navigation can be obtained. Alternatively, this embodiment can use ordered point sets to represent various map elements, such as lane dividers, lane center lines, and road boundaries, which can be represented by polylines, and pedestrian crossings, which can be represented by polygons. Then, a series of self-attention and cross-attention techniques are used to continuously optimize the map, and an appropriate binary matching strategy is used to learn map elements that match reality end-to-end. In addition, in order for the learned map elements to be closer to the real shape, especially in complex scenarios such as roundabouts, ramps, and intersections, this embodiment can also introduce shape prior characteristics such as multiple polynomials or Bezier curves into the map decoder, enabling it to predict complex map elements more accurately.

[0058] The map construction method proposed in this application can accurately predict the map information around a vehicle by fusing BEV features from multiple sensors and temporal BEV features. It not only considers the spatial information at the current moment but also utilizes additional temporal information. Even when important elements in the current map are occluded, it can still obtain more comprehensive map information about the vehicle's surroundings. Therefore, it solves the problems of related technologies that only use sensor data at the current moment to construct online maps, which are unsuitable for complex environmental scenarios and suffer from poor accuracy.

[0059] Next, the map building apparatus proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0060] Figure 4 A block diagram of a map building apparatus according to an embodiment of this application.

[0061] like Figure 4 As shown, the map building device 10 includes: a first acquisition module 100, a processing module 200, a fusion module 300, and a building module 400.

[0062] The first acquisition module 100 is used to acquire image features around the vehicle and point cloud features perceived by the LiDAR; the processing module 200 is used to convert image features using the depth information of the point cloud features to obtain first BEV features, and optimize the point cloud features based on the image features; the fusion module 300 is used to fuse the first BEV features and the optimized point cloud features to obtain second BEV features, and generate a semantic map based on the second BEV features and the temporal BEV features; the construction module 400 is used to identify the semantic map to obtain one or more map elements, and construct map information based on the map elements.

[0063] In one embodiment of this application, the construction module 400 is further configured to: perform differentiable rasterization processing on the semantic map, and generate one or more map elements based on the processed semantic map; or, identify an ordered set of points in the semantic map, optimize the ordered set of points using self-attention and cross-attention, and generate one or more map elements based on the optimized ordered set of points.

[0064] In one embodiment of this application, the map building apparatus 10 further includes a second acquisition module and a generation module.

[0065] The second acquisition module is used to acquire historical frame BEV features before the second BEV features before generating a semantic map based on the second BEV features and the temporal BEV features; the generation module is used to generate query values ​​based on the second BEV features and generate key-value pairs based on the historical BEV features; and to establish the query values ​​and the target relationship between the query values ​​using a cross-attention mechanism, and generate temporal BEV features based on the target relationship.

[0066] In one embodiment of this application, the generation module is further configured to transform the second BEV feature and the historical BEV feature to the target coordinate system; generate a transformation matrix of the second BEV feature and the historical BEV feature in the target coordinate system; update the historical BEV feature based on the transformation matrix and the target relationship; and optimize the second BEV feature using the updated historical BEV feature to obtain the time-series BEV feature.

[0067] In one embodiment of this application, the processing module 200 is further configured to: project point cloud data onto an image plane to obtain a sparse depth map; process the sparse depth map using a depth completion network to obtain a dense depth map; and generate a first BEV feature with depth information based on the dense depth map.

[0068] It should be noted that the foregoing explanation of the map building method embodiment also applies to the map building apparatus of this embodiment, and will not be repeated here.

[0069] The map building apparatus proposed in this application can accurately predict the map information around a vehicle by fusing BEV features from multiple sensors and temporal BEV features. It not only considers the spatial information at the current moment but also utilizes additional temporal information. Even when important elements in the current map are occluded, it can still obtain more comprehensive map information about the vehicle's surroundings. Therefore, it solves the problems of related technologies that only use sensor data at the current moment to build online maps, which are unsuitable for complex environmental scenarios and suffer from poor accuracy.

[0070] Figure 5 This application provides a schematic diagram of the structure of a vehicle. The vehicle may include:

[0071] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0072] When processor 502 executes the program, it implements the map construction method provided in the above embodiments.

[0073] Furthermore, the vehicle also includes:

[0074] Communication interface 503 is used for communication between memory 501 and processor 502.

[0075] The memory 501 is used to store computer programs that can run on the processor 502.

[0076] The memory 501 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0077] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0078] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0079] Processor 502 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0080] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the map construction described above.

[0081] This application also provides a computer program product that stores a computer program or instructions thereon. When the computer program or instructions are executed, the map construction method as described in the above embodiments is employed.

[0082] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0084] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0085] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0086] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0087] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A map construction method, characterized in that, Includes the following steps: Acquire image features of the area surrounding the vehicle and point cloud features perceived by LiDAR; The image features are transformed using the depth information of the point cloud features to obtain the first BEV feature, and the point cloud features are optimized based on the image features; The first BEV feature and the optimized point cloud feature are fused to obtain the second BEV feature, and a semantic map is generated based on the second BEV feature and the temporal BEV feature. The semantic map is identified to obtain one or more map elements, and map information is constructed based on the map elements. The step of identifying one or more map elements from the semantic map includes: performing differentiable rasterization processing on the semantic map, and generating the one or more map elements based on the processed semantic map; or, identifying an ordered set of points in the semantic map, optimizing the ordered set of points using self-attention and cross-attention, and generating the one or more map elements based on the optimized ordered set of points. Before generating a semantic map based on the second BEV feature and the time-series BEV feature, the method further includes: obtaining historical BEV features prior to the second BEV feature; generating a query value based on the second BEV feature and generating key-value pairs based on the historical BEV features; establishing a target relationship between the query value and the key-value pairs using a cross-attention mechanism; and generating the time-series BEV feature based on the target relationship.

2. The map construction method according to claim 1, characterized in that, The step of generating the temporal BEV features based on the target relationship includes: Transform the second BEV feature and the historical BEV feature to the target coordinate system; Generate the change matrix of the second BEV feature and the historical BEV feature in the target coordinate system; The historical BEV features are updated based on the change matrix and the target relationship, and the second BEV features are optimized using the updated historical BEV features to obtain the time-series BEV features.

3. The map construction method according to claim 1, characterized in that, The image features are transformed using the depth information of the point cloud features to obtain the first BEV feature, including: The depth information of the point cloud features is projected onto the image plane to obtain a sparse depth map; The sparse depth map is processed using a depth completion network to obtain a dense depth map, and a first BEV feature with depth information is generated based on the dense depth map.

4. A map building device, characterized in that, include: The first acquisition module is used to acquire image features around the vehicle and point cloud features perceived by the lidar. The processing module is used to convert the image features using the depth information of the point cloud features to obtain the first BEV feature, and to optimize the point cloud features based on the image features; The fusion module is used to fuse the first BEV feature and the optimized point cloud feature to obtain the second BEV feature, and generate a semantic map based on the second BEV feature and the temporal BEV feature. The construction module is used to identify the semantic map to obtain one or more map elements, and construct map information based on the map elements; The construction module is used to: perform differentiable rasterization processing on the semantic map, and generate one or more map elements based on the processed semantic map; Alternatively, identify the ordered set of points in the semantic map, optimize the ordered set of points using self-attention and cross-attention, and generate one or more map elements based on the optimized ordered set of points. The second acquisition module is used to acquire historical BEV features prior to the second BEV features before generating a semantic map based on the second BEV features and the temporal BEV features. The generation module is used to generate a query value based on the second BEV feature, generate key-value pairs based on the historical BEV feature, establish a target relationship between the query value and the key-value pairs using a cross-attention mechanism, and generate the time-series BEV feature based on the target relationship.

5. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the map construction method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the map construction method as described in any one of claims 1-3.

7. A computer program product having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the map construction method according to any one of claims 1-3.

Images