Real-time mapping method and device combined with navigation map, electronic device and storage medium

Abstract

The present disclosure provides a real-time mapping method and device combined with a navigation map, electronic equipment and a storage medium. In the present disclosure, the real-time mapping method combined with the navigation map comprises: acquiring navigation map data of a vehicle and point cloud data collected by a laser radar on the vehicle; obtaining BEV features of a region around the vehicle according to the navigation map data and the point cloud data through an encoding model; generating a driving region classification map of the BEV by using the BEV features through a first classification model, the driving region classification map containing driving category information of the region around the vehicle; and generating a road structure classification map of the BEV by using the BEV features through a second classification model, the road structure classification map containing road category information of the region around the vehicle. The present disclosure can obtain a real-time map that can be directly used for path planning in combination with a navigation map without using a high-definition map.

Description

Technical Field

[0001] This disclosure relates to a real-time mapping method, apparatus, electronic device, and storage medium that incorporates a navigation map. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) technology has important applications in fields requiring autonomous localization, such as autonomous driving and robotics. The core issue of SLAM technology is pose estimation for mobile devices (autonomous vehicles, robots, etc.). Mapping is the result of perceiving and describing the environment based on pose estimation using sensors such as radar and cameras.

[0003] High-definition maps (HDMaps) are a crucial component enabling the current implementation and application of autonomous driving. Level 2+ and higher autonomous driving solutions all rely heavily on HDMaps. Compared to standard maps, HDMaps provide higher precision and richer information required for autonomous driving. Building upon standard maps, HDMaps include more traffic elements such as streetlights, guardrails, and traffic lights, while also providing more precise lane and intersection information. HDMaps can serve various functional modules in autonomous driving, such as localization, perception, prediction, planning, and safety. From the current state of autonomous driving development, HDMaps play an extremely important role.

[0004] High-precision map production generally falls into two categories: centralized professional mapping and crowdsourced mapping. Centralized professional mapping involves specialized personnel using high-precision data acquisition equipment to collect road data and then process it into maps. Figure One Generally, mapping is divided into four stages: data collection, data processing, element identification, and manual verification. Crowdsourced mapping involves collecting data separately using many ordinary devices and personnel, then merging the data to create a map. Professional centralized mapping is very expensive due to its high requirements for personnel and equipment; crowdsourced mapping uses inexpensive equipment to synthesize maps, but the accuracy of these maps is much lower than that of professionally generated maps. Therefore, the vast majority of autonomous driving solutions use high-precision maps generated through professional centralized mapping.

[0005] Although high-definition maps play an important role in autonomous driving, they cannot be used on all road segments due to high construction costs and limitations on safety by government laws and regulations. Therefore, online driving map (ODM) is a viable solution. Unlike traditional SLAM mapping solutions, which provide positioning services and only include feature point clouds, real-time driving mapping provides more driving information to vehicles, such as lane lines, surrounding vehicle information, obstacle information, drivable areas, traffic lights, and the like. Moreover, without high-definition maps, subsequent path planning and other tasks need to rely on real-time driving maps. Existing real-time driving mapping solutions cannot provide reliable and directly usable real-time map information for planning, and rely on laser radar data, which is limited by the sensor characteristics and occlusion of laser radar, and cannot meet the actual requirements in terms of accuracy and real-time performance. SUMMARY

[0006] To solve at least one of the above technical problems, the present disclosure provides a real-time mapping method combined with a navigation map, a device, an electronic device, and a storage medium.

[0007] According to a first aspect of the present disclosure, a real-time mapping method combined with a navigation map is provided, comprising:

[0008] Obtaining navigation map data of a vehicle and point cloud data collected by a laser radar on the vehicle;

[0009] Obtaining BEV features of a region around the vehicle according to the navigation map data and the point cloud data through an encoding model;

[0010] Generating a driving area classification map of the BEV using the BEV features through a first classification model, the driving area classification map containing driving category information of the region around the vehicle;

[0011] Generating a road structure classification map of the BEV using the BEV features through a second classification model, the road structure classification map containing road category information of the region around the vehicle.

[0012] In some embodiments, the real-time mapping method combined with the navigation map further comprises: mapping the navigation map data of the vehicle into a navigation feature map, the navigation feature map containing road segment information and real-time distance information in a predetermined region corresponding to each real-time position of the vehicle within a predetermined period.

[0013] In some embodiments, elements in the navigation feature map are represented by key-value pairs Key-Value, the key Key in Key-Value indicating a road segment name, and the value Value indicating a real-time distance corresponding to the Key.

[0014] In some embodiments, the method further comprises: preprocessing the point cloud data and projecting to obtain point cloud pseudo image data, wherein the point cloud pseudo image data comprises a time sequence pseudo image, and the time sequence pseudo image implicitly contains time information of the point cloud data.

[0015] In some embodiments, the time sequence pseudo image is obtained by projecting each point in the point cloud data to the same channel according to a time index.

[0016] In some embodiments, the time sequence pseudo image is obtained by: calculating a time index of the point cloud data using time information of the point cloud data; processing the point cloud data to use the time index for the point cloud; and projecting and mapping the processed point cloud data to a channel as the time sequence pseudo image.

[0017] In some embodiments, the surrounding area of the vehicle comprises a laser radar trustable reachable area, a laser radar reachable edge area, and a laser radar unreachable area.

[0018] In some embodiments, the encoding model adopts a Transformer-Encoder structure with a double position encoding unit, and the double position encoding unit comprises a first position encoding unit for distinguishing a spatial sequence and a second position encoding unit for distinguishing a time sequence.

[0019] In some embodiments, the loss function of the first classification model comprises a first loss corresponding to a laser radar trustable reachable area, a second loss corresponding to a laser radar edge area, and a third loss corresponding to a laser radar unreachable area, and the weights of the first loss, the second loss, and the third loss are different.

[0020] In some embodiments, the second classification model is trained according to a preconfigured lane category dictionary, and the lane category dictionary comprises all pre-enumerated lane categories, and the lane categories are used to indicate types, positions, and / or driving directions of lanes.

[0021] According to a second aspect of the present disclosure, a real-time mapping device combined with a navigation map is provided, comprising:

[0022] A data acquisition module is configured to acquire navigation map data of a vehicle and point cloud data collected by a laser radar on the vehicle.

[0023] A BEV feature module is configured to obtain BEV features of a surrounding area of the vehicle by an encoding model according to the navigation map data and the point cloud data.

[0024] A driving area determination module is configured to generate a driving area classification map of the BEV by a first classification model using the BEV features, and the driving area classification map comprises driving category information of the surrounding area of the vehicle.

[0025] a road structure determination module configured to generate a road structure classification map of the BEV using BEV features through a second classification model, the road structure classification map containing road class information of an area surrounding the vehicle.

[0026] According to a second aspect of the present disclosure, an electronic device is provided, comprising:

[0027] a memory storing execution instructions; and

[0028] a processor executing the execution instructions stored in the memory, so that the processor executes the above-mentioned real-time mapping method in combination with a navigation map.

[0029] According to a fourth aspect of the present disclosure, a readable storage medium is provided, the readable storage medium storing execution instructions, the execution instructions being executed by a processor to implement the above-mentioned real-time mapping method in combination with a navigation map.

[0030] The present disclosure can obtain a real-time map that can be directly used for path planning in combination with a navigation map without using a high-precision map. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the general description of the disclosure given above and the detailed description of the disclosure given below, serve to explain the principles of the present disclosure. These drawings are included herewith and constitute a part of this specification.

[0032] Figure 1 is a flowchart of a real-time mapping method in combination with a navigation map according to some embodiments of the present disclosure.

[0033] Figure 2 is a key-value format diagram of pre-processed navigation map data according to some embodiments of the present disclosure.

[0034] Figure 3 is a visualization example diagram of a driving area class map according to some embodiments of the present disclosure.

[0035] Figure 4 is a visualization example diagram of a road structure class map according to some embodiments of the present disclosure.

[0036] Figure 5 is an exemplary architecture of a real-time mapping model and a processing process diagram thereof according to some embodiments of the present disclosure.

[0037] Figure 6 is an architecture example diagram of an encoding model according to some embodiments of the present disclosure.

[0038] Figure 7is a structural schematic block diagram of a real-time mapping device combined with a navigation map using a hardware implementation of a processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The present disclosure will be described in further detail below with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the present disclosure and are not to be taken in a limiting sense. It is further noted that, as used in this specification, "exemplary" means "serving as an instance or example," and should not necessarily be construed as preferred or advantageous over other examples.

[0040] It should be noted that the embodiments and features of the embodiments in the present disclosure can be combined if there is no conflict. The technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0041] Unless otherwise indicated, the exemplary embodiments / instances shown are to be understood as providing exemplary features of various details that can be implemented in practice to embody the technical concepts of the present disclosure. Thus, unless otherwise indicated, features of various embodiments / instances can be additionally combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0042] In the drawings, cross-hatching and / or shading are generally used for clarity and are not necessarily present to indicate one particular material, material property, dimension, ratio, etc. In addition, for clarity and the purpose of illustration, reference numbers can have been repeated in the drawings to indicate like portions of the devices. Moreover, dimensions of certain parts and relative dimensions can be exaggerated to highlight features of examples, and / or to illustrate one or more embodiments of the present disclosure. The primary purpose of the use of such visual aids is to illustrate, explain, and / or describe the examples of the present disclosure. As such, the description of examples of the present disclosure should not be construed to require or imply the inclusion of any particular feature, structure, or characteristic, unless explicitly recited by the claims.

[0043] When a component is referred to as being "on" or "over" another component, "connected to" or "coupled to" another component, it can be directly on, connected, or coupled to the other component, or intervening components can be present. When a component is referred to as being "directly on", "directly connected to", or "directly coupled to" another component, there are no intervening components present. By the term "connected" can mean physical, electrical, and / or the like, with or without intervening components.

[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "including," "includes," "having," "has," "a" or "an," or the like are used in the detailed description and / or claims, such terms are intended to be inclusive (i.e., to say that at least the stated feature is present), unless the context clearly indicates otherwise. It will also be noted that, as used herein, the terms "substantially," "approximately," and other like terms are used as terms of approximation and not as terms of degree, unless the context clearly indicates otherwise. Thus, at the very least, such terms are intended to account for the inherent deviations in measured and / or provided values that would be recognized by those of ordinary skill in the art.

[0045] Related Art

[0046] Related Art 1: MP3:A Unified Model to Map,Perceive,Predict and Plan published an autonomous driving scheme based on neural network, which does not rely on high-definition map. In this scheme, the real-time map is the intermediate output of the convolutional neural network, not the final output. The intermediate channel includes drivable area, intersection, reachable lane line, etc. The scheme calls it an interpretable map representation. The prediction and planning downstream of the pipeline are all based on this map representation. This scheme emphasizes that the map has strong interpretability, that is, the map representation is not trained by direct map information, and its reliability and accuracy are a big problem in the case of insufficient training data. And this scheme does not have an explicit spatiotemporal feature extraction structure in the model, so the effect is not ideal in time consistency and spatial consistency.

[0047] Related Art 2: HDMapNet:A Local Semantic Map Learning and Evaluation Framework proposed a HDMapNet network, which can generate a local high-definition map in real time, mainly including lane lines and lane line directions and lane line instance segmentation information. The goal of this scheme is mainly focused on the classification and vectorization of lane lines in high-definition maps, without considering other dynamic objects. Such a map cannot be directly used for planning. And this scheme focuses on networks containing visual information, and the effect of models with pure laser input is obviously weaker than other models containing visual information.

[0048] Related technology 3: MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps proposes a model combining perception and prediction tasks, which can predict the categories of obstacles such as vehicles and pedestrians on the real-time road without using the bounding box scheme, and output the grid map in the bird's eye view (BEV). At the same time, it can predict the speed and direction of the moving vehicle. The scheme proposes that the obstacle classification of urban roads should include other categories to buffer objects that the vehicle has not seen during training, so as to avoid the situation that the obstacle cannot be distinguished. However, this scheme focuses on the prediction of moving objects, and the output result does not include drivable areas and lane lines and other key driving information.

[0049] The above related technologies cannot provide sufficient reliable real-time map information for direct planning. Moreover, the map output results of the above related technologies are a direct reflection of the current laser radar data, which is limited by the sensor characteristics of the laser radar and factors such as occlusion. Direct real-time driving maps need to make more predictions on the road conditions to meet the data needs of planning.

[0050] Since the information range of the laser radar is limited by the range of the laser radar and the occlusion of other vehicles on the road, the ordinary real-time scene map, especially the real-time scene map of pure laser radar, contains insufficient information to meet the driving needs of the vehicle during driving, and the map construction is also difficult to meet the real-time requirements of the vehicle driving scene.

[0051] In view of this, the embodiments of the present disclosure provide a real-time mapping method, device, electronic device and storage medium as follows: when constructing a real-time driving map using a laser radar, the accuracy and real-time performance of the real-time driving map are improved by predicting the map that is not within the perception range of the laser radar, which can meet the planning needs and real-time needs of the vehicle.

[0052] Term explanation:

[0053] The data of the sensor needs to be converted by the extrinsic parameter of the sensor relative to the center of the device (for example, the electronic device below) to the center of the device. The present disclosure assumes that the sensor data has been converted by the extrinsic parameter.

[0054] Point cloud data frame: that is, laser frame, the data of the laser radar is stored and transmitted in the form of frames, and each frame of data includes the transmission distance of each point after the laser radar emits a laser beam to scan the environment within a fixed time.

[0055] Key frame: a frame with better quality in the laser frame, also used for the backend to construct a pose graph according to the key frame.

[0056] Coordinate system: The disclosure uses a rectangular coordinate system, which uses the coordinate system definition commonly used in the robot operating system, that is, a Cartesian rectangular coordinate system right-handed system can be adopted, the X axis is forward, the Y axis is left, and the Z axis is upward.

[0057] Pseudo map: The point cloud data is rasterized in the Z axis of the three-dimensional space, and the height of the point cloud data is rasterized at a resolution to form multiple layers of grids. Each layer of grid is a channel of the pseudo map, that is, the pseudo map of the point cloud data usually has multiple channels.

[0058] Application scenario example

[0059] The embodiments of the disclosure can be applied to automatic driving, assisted driving and the like. The embodiments of the disclosure can be applicable to mobile devices, which can be autonomous mobile devices such as automatic driving vehicles equipped with distance detection sensors and the like that need to perform environment perception mapping. The distance detection sensor can include but is not limited to a laser radar, a depth camera, a millimeter wave radar, an IMU and the like that can obtain environmental distance information.

[0060] Figure 1 The flowchart of the real-time mapping method combined with the navigation map of some embodiments of the disclosure is shown. As shown in Figure 1 The real-time mapping method of the embodiments of the disclosure can include the following steps S102-S108:

[0061] Step S102, acquiring navigation map data of a vehicle and point cloud data collected by a laser radar on the vehicle;

[0062] Step S104, obtaining BEV features of a region around the vehicle according to the navigation map data and the point cloud data through an encoding model;

[0063] Step S106, generating a driving region classification map of the BEV by using the BEV features through a first classification model, the driving region classification map including driving category information of the region around the vehicle;

[0064] Step S108, generating a road structure classification map of the BEV by using the BEV features through a second classification model, the road structure classification map including road category information of the region around the vehicle.

[0065] Among them, the driving region classification map and the road structure classification map can form a 2D multi-layer static map of the BEV.

[0066] The navigation map data obtained in step S102 can be a sequence formed by local navigation map data corresponding to each real-time position of the vehicle in a predetermined period. The local navigation map data can include navigation map prior information in a predetermined area corresponding to the current position of the vehicle, which can include but is not limited to the lane type of the current driving road of the vehicle, the driving direction of the lane, whether there is a roundabout, whether there is an intersection, the real-time distance of the roundabout from the current position of the vehicle, the real-time distance of the intersection from the current position of the vehicle, and the like. The predetermined area corresponding to the current position of the vehicle can be flexibly set as needed. For example, the predetermined area can be a circular area with the current position of the vehicle as the center and a preset distance (e.g., 3 kilometers, 5 kilometers, etc.) as the radius, and the like.

[0067] In a specific application, the local navigation map data can be obtained from the cloud or extracted from the navigation map built-in the vehicle.

[0068] Before step S104, it can further include: mapping the navigation map data obtained in step S102 into a navigation feature map, the navigation feature map containing feature data corresponding to each real-time position of the vehicle in a predetermined period, each element in the navigation feature map being represented by a key-value pair (Key-Value), the key (Key) in the Key-Value indicating a road segment name, and the value (Value) indicating a real-time distance corresponding to the Key, the real-time distance being the distance between the road segment represented by the Key and the real-time position of the vehicle. Thus, the navigation map data can be converted into 2D data, which can be directly processed by the encoding model.

[0069] In some embodiments, the local navigation map data can be converted into a data form that can be directly processed by the encoding model in the following manner: for the local navigation map data corresponding to a certain real-time position of the vehicle, the road segment name in the navigation map prior information thereof is mapped to Key, and the real-time distance in the navigation map prior information thereof is mapped to the Value corresponding to the Key, thereby converting the local navigation map data corresponding to the real-time position of the vehicle into Key-Value data.

[0070] For example, the Key can be used to represent the road segment name by a numerical value. For example, “crossroads” can be mapped to the numerical value “1”, “straight road” can be mapped to “2”, “roundabout” can be mapped to “3”, and “T-shaped intersection” can be mapped to “4”. Among them, “crossroads”, “straight road”, “roundabout”, and “T-shaped intersection” are road segment names included in the navigation map prior information, and “1”, “2”, “3”, and “4” are corresponding Keys.

[0071] In a specific application, the mapping of the road segment name to the Key can be achieved by enumerating all possible road segment names and configuring the numerical value corresponding to each road segment name. For example, the road segment name “crossroads” can be mapped to the numerical value “1”, the road segment name “straight road” can be mapped to the numerical value “2”, the road segment name “roundabout” can be mapped to the numerical value “3”, and the road segment name “T-shaped intersection” can be mapped to the numerical value “4”. Figure 3For example, all possible road segment names are found by enumeration, and there are 30 road segment names in total. The total number of Key values is set to 30, and each Key value represents a road segment name.

[0072] The Value corresponding to the Key indicates the real-time distance corresponding to the Key, that is, the distance between the road segment indicated by the Key and the real-time position of the corresponding vehicle. For example, assuming that the navigation map prior information contains the information "100 meters to the front crossroads", then the Value corresponding to the Key representing the crossroads is 100.

[0073] Figure 2 A visualization diagram of the key-value pair data converted from the local navigation map data corresponding to the real-time position of a certain vehicle is shown. Figure 2 In the example, the total number of Key values is 30, the Key is represented as the column number, and the value of the element (that is, the number "0", "30", etc. in the square) represents the Value corresponding to the Key. If a road segment name (for example, "straight road") does not exist, the Value can be set to 0 by default.

[0074] The local navigation map data corresponding to each real-time position of the vehicle within a predetermined period of time is respectively converted into Figure 2 key-value pair data as shown, and these key-value pair data are spliced in time sequence to form a navigation feature map. In the navigation feature map, the row of each element represents the time sequence, the column represents the Key, and the value of the element (that is, the value in the square in Figure 2 the example) represents the Value.

[0075] The point cloud data obtained in step S102 can include point cloud data frames corresponding to each real-time position of the vehicle within a predetermined period of time. In specific applications, the point cloud data frames can be collected in real time by a laser radar loaded on the vehicle.

[0076] In some embodiments of the present disclosure, before step S104, the point cloud data can also be preprocessed and projected to obtain point cloud pseudo map data, which can include pseudo map data frames corresponding to each real-time position of the vehicle within a predetermined period of time.

[0077] Most of the mobile devices are in motion during the collection of the point cloud data, and because the frequency of the laser radar is limited, the data in the same point cloud data frame is not collected at the same time, that is, the starting laser beam and the ending laser beam of the same point cloud data frame correspond to different positions of the moving object, and different laser points correspond to different coordinate systems, which will bring a large error to positioning and mapping. In view of this, in some embodiments of the present disclosure, the preprocessing of the point cloud data can include motion de-distortion processing of the point cloud data, that is, point cloud distortion compensation of the point cloud data.

[0078] In some embodiments, the point cloud distortion compensation can be: converting all data of the same point cloud data frame to the coordinate system of the starting laser point according to the motion model. For example, the motion speed of the mobile device provided by other sensors can be used to obtain the relative motion position by integration, and then a conversion matrix is calculated according to the relative relationship to convert all laser points to the coordinate system of the initial laser point. Through point cloud distortion compensation, the mapping accuracy can be effectively improved.

[0079] Exemplarily, the preprocessing of the point cloud data can further include voxelization, 3D to 2D pseudo map, etc. before the aforementioned motion distortion removal.

[0080] In some embodiments of the present disclosure, the pseudo map data frame can include a first pseudo map I1, a second pseudo map I2, and a time sequence pseudo map I t That is, the point cloud pseudo map data frame of the embodiment of the present disclosure can be expressed as “I1+I2+I t ”. Wherein, the first pseudo map can include the occupancy information (for example, the xyz mean value) of each point in the point cloud data frame, which can indicate whether there is a point at the corresponding position; the second pseudo map can include the intensity information (for example, the intensity information of the relative value of each point relative to the lane line intensity) of each point in the point cloud data frame, and the time sequence pseudo map can include the time information of the point cloud data frame.

[0081] In the embodiment of the present disclosure, the first pseudo map and the second pseudo map in the pseudo map data frame can be obtained by various methods capable of realizing point cloud to pseudo map.

[0082] In some embodiments, the process of point cloud to pseudo map can include: first, converting the current frame point cloud and the point cloud data frame (for example, 2 frames can be sampled per second) sampled in the past predetermined period (for example, 10 seconds) into a BEV format map converted to the current frame coordinate system, and then rasterizing the BEV format map according to the length, width and height to obtain the first pseudo map I1 and the second pseudo map I2.

[0083] In the embodiment of the present disclosure, the length, width and other parameters of the BEV format map are the same as the corresponding parameters of the predicted map such as the driving area classification map and the road structure classification map.

[0084] Suppose the range of the x and y directions of the BEV format map is [x1, y1], and the z direction range value is z1, which can be set as a fixed value (for example, 5), and the grid resolution of the three directions is [r x , r y , r z ], then the data format of the first pseudo map and the second pseudo map can be expressed as the following formula (1):

[0085]

[0086] wherein c is the number of channels of the pseudo image, w represents a width value of the pseudo image, and h represents a height value of the pseudo image.

[0087] In consideration of the fact that the map includes information of lane lines, the disclosure embodiments separately output the occupancy information in the point cloud data to the first pseudo image I1.

[0088] In consideration of the fact that the identification of the lane lines is related to both the height value and the intensity of the point cloud, the disclosure embodiments set the lane line intensity related value z as z2, and project the intensity information of the point cloud data relative to the lane line intensity related value into the second pseudo image I2.

[0089] In some embodiments, the time sequence pseudo image in the pseudo image data frame can be obtained by projecting each point in the point cloud data frame according to the time index into the same channel. That is, the time sequence pseudo image is generated by projecting the points in the point cloud data onto a separate channel according to the time index.

[0090] In some embodiments of the disclosure, it is assumed that there are n frames of current point cloud data frames and past sampled point cloud data frames in total, and the corresponding point cloud data frame set is {P i ,1<=i<=n}, P i is the identification of the point cloud data frame, the point cloud data frames in the point cloud data frame set P i are sorted according to their time information, the time information (for example, the time stamp or the collection time of the point cloud) of the i-th frame of point cloud data is denoted as t i , and the corresponding time sequence pseudo image I t can be obtained in the following manner:

[0091] Firstly, the time index T i of the point cloud can be calculated according to the following formula (2) using the time information of the point cloud data:

[0092] T i =(t i -t i )×scale (2)

[0093] wherein scale is a constant value that does not change with the frame number, t1 is the time information of the first frame of point cloud data, t i is the time information of the i-th frame of point cloud data, and it is assumed that there are n frames in total, so i = 1, 2, 3, …, n. As can be seen from formula (2), the closer the time index T i of the point cloud to the current frame, the greater the value.

[0094] Secondly, the point cloud data frame is processed according to the following formula (3) to use the time index for the point cloud:

[0095] P index =P i ×T i (3)

[0096] wherein the value of Pi in formula (3) can be set as 1 or other fixed value, and the value of P i is the same for all point cloud data frames.

[0097] Finally, the processed point cloud data frame P index is projected and mapped onto a separate layer (i.e., a separate channel) as time-sequenced data, denoted as time-sequenced pseudo image I t .

[0098] In the case of vehicle congestion, the laser radar cannot identify the type of lane line through the data of the point cloud hitting the ground, therefore, the embodiment of the present disclosure adds a time-sequenced pseudo image in the point cloud pseudo image data, which can judge the type of lane line through time feedback of vehicle motion. By adding time information factor in the processing of point cloud pseudo image data, i.e., adding time information factor in the conversion of point cloud data, the encoding model can help to more accurately judge the direction of traffic flow and the direction of lane line, thereby assisting in the identification of the direction of motion in the multi-lane congestion situation.

[0099] In step S104, the BEV feature of the surrounding area of the vehicle can be obtained by processing the navigation feature map converted from the point cloud pseudo image data and the navigation map data by the encoding model. The surrounding area of the vehicle not only includes the reachable and reliable area of the laser radar, the edge area reachable by the laser radar, but also includes the unreachable area of the laser radar. That is, in step S104, the BEV feature of the laser radar coverage area can be extracted by the encoding model, and the BEV feature of the laser radar unreachable area can also be predicted. In addition, since the time-sequenced pseudo image is included in the point cloud pseudo image data, the time information is implied in the time-sequenced pseudo image, and the time information can be fed back to the vehicle motion, therefore, the BEV feature obtained in step S104 also implies the features such as the direction of lane line and the direction of traffic flow.

[0100] In step S104 or before step S104, the point cloud pseudo image data and the navigation feature map can also be fused in time sequence. In this way, only the fusion result of the point cloud pseudo image data and the navigation feature map needs to be input into the encoding model to obtain the BEV feature, which can further improve the processing efficiency.

[0101] The encoding model of this disclosure can be implemented using various deep learning models. Preferably, the encoding model of this disclosure can adopt a Transformer-Encoder structure with dual positional encoding units. These dual positional encoding units include a first positional encoding unit for distinguishing spatial sequences and a second positional encoding unit for distinguishing time sequences. This allows for more accurate identification of information such as traffic flow direction and lane line direction implicit in BEV features. The specific architecture and training methods of the encoding model will be detailed below and will not be repeated here.

[0102] In step S106, the driving area classification map may include driving category information of the area surrounding the vehicle. Specifically, the area surrounding the vehicle can be divided into grids and the driving category of each grid can be labeled to obtain the driving area classification map. The driving category may include two types: drivable and non-drivable.

[0103] See Figure 3 For example, the area around the vehicle comprises three sub-regions, namely the LiDAR trusted reachable area (i.e., Figure 3 The "trusted reachable range" and the reachable edge area of ​​lidar (i.e., Figure 3 The "boundary reachable range" and the area inaccessible by lidar (i.e., Figure 3 The "inaccessible area" (as defined in the text) is divided into three sub-regions, each represented by a grid with different resolutions. The LiDAR-reliable reachable area is divided into N grids, the LiDAR-reachable edge area into M grids, and the LiDAR-inaccessible area into C grids. N can be greater than M, and M can be greater than C. For example, C can be 3 times N, and M can be 2 times N. In practical applications, the specific values ​​of N, M, and C are not limited and can be set as needed. The first classification model is used to classify the driving categories of the above three sub-regions and label the driving category of each grid in these three sub-regions, thereby generating a driving area classification map.

[0104] Among them, the unreachable area of ​​lidar refers to the geographical area beyond the range of lidar, the reliable reachable area of ​​lidar refers to the geographical area within the reliable range of lidar, and the reachable edge area of ​​lidar refers to the geographical area within the range of lidar but at the boundary of the range.

[0105] See Figure 3 As shown in the driving area classification map, the entire area around the vehicle is divided into a driving area (i.e., Figure 3 The "feasible zone" and the non-drivable zone (i.e., the "feasible zone") Figure 3 The "infeasible area" in the map shows that the driving area classification map can be directly applied to subsequent processing such as path planning.

[0106] In step S108, the road structure classification map can include road category information of the area around the vehicle. Specifically, the area around the vehicle can be divided into grids and the road category of each grid is marked to obtain the road structure classification map, which can include but is not limited to lane lines, lane categories, intersections, roundabouts, etc.

[0107] It is considered that the road structure classification map, although belonging to the classification result, involves various lane markings, and the shapes and sizes of different lane markings are quite different, and the lane position range and lane type need to be accurately calculated (for example, although the left lane and the middle lane both belong to straight lanes, their positions are different, and they still need to be accurately distinguished for subsequent processing such as path planning). Therefore, in some embodiments of the present disclosure, a block calculation method is preferably used to obtain the road structure classification map to at least accurately distinguish lanes at different positions.

[0108] Figure 4 A visualization example of the road structure classification map is shown. As shown in Figure 4 The part related to the road in the area around the vehicle can be divided into different road blocks, and each road block in the area around the vehicle is identified by the second classification model, so as to accurately distinguish lanes at different positions, lanes of different types, intersections, roundabouts, etc.

[0109] In addition, in order to ensure that the road structure has relatively obvious connectivity consistency, in the embodiment of the present disclosure, the second classification model is used to obtain the road structure classification map in step S108 without distinguishing the laser radar trustable reachable area, the laser radar reachable edge area and the laser radar unreachable area, etc.

[0110] In the embodiment of the present disclosure, the lane category can indicate the lane type, the lane position and / or the driving direction of the lane. For example, the lane category can represent “same direction left 1 straight lane”, which indicates that the lane type is a straight lane, located at the first left side of the ego vehicle, and the driving direction of the lane is the same as the driving direction of the ego vehicle.

[0111] In some embodiments of the present disclosure, the second classification model in step S108 can be trained according to a preconfigured lane category dictionary containing all enumerated lane category information, so as to accurately distinguish the lane type and the lane position by the second classification model. The identification of the lane structure and the lane line is counted as the lane category dictionary, and the second classification model can directly output the classification result according to the format defined by the dictionary, which simplifies the complexity of the road segment prediction and improves the efficiency.

[0112] In some embodiments, all possible lane categories can be enumerated in advance and mapped into one-hot codes, and the one-hot codes are used to form the lane category dictionary.

[0113] For example, in a certain scenario, there are at most four lanes in the same direction and at most four lanes in the opposite direction, and the lane types include left-turn lane, straight lane, right-turn lane, and U-turn lane. At this time, all possible lane categories can be integrated to give the following lane category dictionary: [left 1 straight lane, left 1 left-turn lane, left 1 U-turn lane, left 1 left-turn U-turn lane, left 1 left-turn U-turn straight lane, left 2 straight lane, left 2 right-turn lane, left 2 straight right-turn lane... pedestrian lane].

[0114] For ease of understanding, words are used in the lane category dictionary in this document to represent lane categories. Those skilled in the art should understand that these lane categories in the actually configured lane category dictionary can be represented by encoding such as one-hot code.

[0115] The driving area classification map and the road structure classification map generated by the method of the embodiments of the present disclosure not only cover the geographical range perceived by the lidar in real time, but also cover a larger geographical range that cannot be perceived by the lidar at present but is predicted by combining the data perceived by the lidar in real time and the navigation map data. Thus, the map obtained by the embodiments of the present disclosure not only has a large field of view, but also contains a large range of map information (such as intersections, roundabouts, etc.), and can be used as a lightweight real-time high-definition map (Lightweight real-time HD map, LRHD) and can be directly applied to subsequent processing such as path planning and prediction.

[0116] The training method, training process, etc. of the encoding model, the first classification model and the second classification model of the embodiments of the present disclosure are exemplarily described below.

[0117] The encoding model, the first classification model and the second classification model can form a real-time mapping model, and the training data of the real-time mapping model can include navigation map data, point cloud data, and local high-definition map data in the corresponding position and range obtained according to the pose. The local high-definition map data is used as the true value when the real-time mapping model is trained. At the same time, in order to obtain high-definition map information and navigation map information of the vehicle on urban roads, the RTK pose information of the vehicle can also be included in the collected data. In order to obtain the speed of the vehicle and the speed information of the surrounding vehicles, the data of the IMU can also be collected at the same time.

[0118] Preferably, the total amount of data meets the needs of model training, preventing underfitting of the model due to insufficient data. At the same time, the diversity of the data collection scenarios should also be ensured as much as possible, such as lane congestion sections, lidar occlusion, daytime and nighttime scenarios, rainy, foggy and sunny scenarios, etc., so that the generalization and adaptability of the model are stronger.

[0119] In the embodiments of the present disclosure, the real-time mapping model needs to process point cloud pseudo map data and navigation map data simultaneously, and can adopt a deep neural network structure with double-head input + backbone network + double-head output. Figure 5 An exemplary architecture of the real-time mapping model and its processing procedure are shown.

[0120] In some embodiments, the encoder module of the Transformer network can be selected as the backbone model of the real-time mapping model of the present disclosure, i.e., the aforementioned encoding model. Since it is a common practice to divide the picture structure into multiple patches and use the picture patches as the input of the network in the field of vision, etc., and since the task of determining the lane line direction by the traffic flow direction is involved in the embodiments of the present disclosure, the time sequence and spatial sequence relationship of the data need to be considered. Therefore, preferably, the backbone network of the real-time mapping model, i.e., the encoding model, can adopt the Transformer-Encoder structure with double position encoding as shown in Figure 6 . In which, two position encoding units (i.e., position encoding 1 and position encoding 2 in Figure 6 ) are arranged to distinguish the spatial sequence and the time sequence, and the input data thereof includes the fusion data of the point cloud pseudo map data and the navigation feature map at multiple time points (t0-tn).

[0121] As shown in Figure 5 , the double-input head of the real-time mapping model can include a navigation data mapping network and a multi-time point cloud input network. The navigation data mapping network can be used to map the navigation map data into a navigation feature map, to realize the preprocessing of the navigation map data, so that the preprocessed navigation data is mapped into a feature map with the same data specification as the preprocessed point cloud data. The multi-time point cloud input network can be used to perform the aforementioned preprocessing and mapping on the point cloud data to obtain the point cloud pseudo map data. In specific applications, the navigation data mapping network, the multi-time point cloud input network, etc. can adopt model architectures such as standard convolutional neural networks, etc.

[0122] As shown in Figure 5 , the real-time mapping model can further include a multi-source data combination network. The multi-source data combination network can be used to fuse the navigation feature map and the point cloud pseudo map data according to the time sequence to obtain the fusion result of the navigation feature map and the point cloud pseudo map data and provide the fusion result to the encoding model, so that the encoding model directly processes the fusion result to obtain the BEV feature.

[0123] In specific applications, the multi-source data combination network can adopt model architectures such as standard convolutional neural networks, etc. without limitation.

[0124] In a specific application, the training of the navigation data mapping network, the multi-time point cloud input network and the multi-source data combination network can be divided into two steps, that is, in the first step, the output results of these models are trained separately to meet certain requirements; in the second step, after these models are combined with the encoding model, the first classification model and the second classification model to form a real-time mapping model, the real-time mapping model is further trained until the real-time mapping model meets the predetermined convergence condition.

[0125] As shown in Figure 5 , the double output head of the real-time mapping model includes the first classification model and the second classification model, the first classification model can process the BEV features output by the encoding model to output a driving area classification map, and the second classification model can process the BEV features output by the encoding model to output a road structure classification map. The output resolution of the first classification model and the second classification model can be the same, and the network structure of the first classification model and the second classification model can both adopt 2 layers of 2D convolution layers.

[0126] The driving area classification map needs to distinguish between the laser hit area and the laser non-hit area. As described above, the driving area classification map needs to divide the entire vehicle surrounding area into three sub-areas with different confidence levels, i.e., the laser radar credible reachable area, the laser radar reachable edge area and the laser radar unreachable area. In view of this, in some embodiments of the present disclosure, the loss function used when training the first classification model can include three parts, i.e., the first loss corresponding to the laser radar credible reachable area, the second loss corresponding to the laser radar reachable edge area and the third loss corresponding to the laser radar unreachable area, and the weights of the first loss, the second loss and the third loss are different. Exemplarily, the above three parts of loss can adopt, but are not limited to, cross-entropy loss function.

[0127] Specifically, the loss function l zone of the first classification model can be represented as the following formula (4):

[0128]

[0129] Wherein, a is the weight of the laser radar credible reachable area, β is the weight of the laser radar reachable edge area, and γ is the weight of the laser radar unreachable area, a>β>γ; N represents the total number of grids of the laser radar credible reachable area, M represents the total number of grids of the laser radar reachable edge area, and C represents the total number of grids of the laser radar unreachable area; y i represents the driving class true value of the i-th grid of the laser radar credible reachable area, represents the driving class prediction value of the i-th grid of the laser radar credible reachable area, y j represents the driving class true value of the j-th grid of the laser radar reachable edge area, y represents the driving class prediction value of the jth grid of the laser radar reachable edge region k y represents the driving class true value of the kth grid of the laser radar unreachable region, y represents the driving class prediction value of the kth grid of the laser radar unreachable region.

[0130] The second classification model can be trained according to a pre-configured lane class dictionary. In some embodiments, the second classification model can also use a cross-entropy loss function.

[0131] For example, the loss function l line may be represented as the following formula (5)

[0132]

[0133] In formula (5), H represents the total number of grids of the second classification model, that is, the resolution; y h y represents the true value of the road class information of the hth grid, which can be obtained from the local high-definition map data; y represents the road class information output by the second classification model for the hth network.

[0134] The total loss function L of the real-time mapping model during training can be represented as the following formula (6).

[0135] L = l zone + l line (6)

[0136] It should be noted that the loss functions of formulas (4)-(6) are only examples. Those skilled in the art should understand that other forms of loss functions can also be used as needed.

[0137] Figure 7 is a structural schematic block diagram of a real-time mapping device of the present disclosure combined with a navigation map, which is a hardware implementation of an embodiment of the present disclosure.

[0138] The device can include corresponding modules for performing each or several steps in the above flowchart. Therefore, each or several steps in the above flowchart can be performed by corresponding modules, and the device can include one or more of these modules. The modules can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer readable medium for implementation by a processor, or implemented by some combination.

[0139] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and the overall design constraints. Bus 800 connects various circuits including one or more processors 900, memory 1000, and / or hardware modules together. Bus 800 can also connect various other circuits 1100 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0140] Bus 800 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0141] Any process or method descriptions or steps described in flow charts herein or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) of the above-described processes or methods. The various embodiments of the present disclosure can include additional or fewer processes, steps, operations, or functions, and the order of the processes, steps, operations, or functions can be changed. The scope of the disclosure is not intended to be limited to the particular order of operations set forth in the various processes, steps, operations, or functions described herein. The processes, steps, operations, or functions described herein can be implemented, at least in part, in hardware, software, or a combination of hardware and software. The various embodiments described herein can be implemented as software code to be executed on a processor or any other suitable computer. The software is maintained in one or more machine readable storage media as computer program instructions. The software can be written in any suitable programming language. The various embodiments described herein can also be implemented as a system on a chip, where each or many of the illustrated components are integrated onto a single integrated circuit or other semiconductor chip.

[0142] The logic and / or steps represented in flow charts or otherwise described herein can be embodied in any readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with the instruction execution system, apparatus, or device.

[0143] For the purposes of this specification, a "read only memory" can be any apparatus that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device. More specific examples (a non-exhaustive list) of the read only memory include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read only memory (CD ROM). In addition, a read only memory can even be paper or another suitable medium upon which the program can be printed, as the program can be electronically captured, for example by the optically scanning the paper or other medium, then electronically converted into a form which can be further processed by a computer into an electronically usable form which can be stored in the memory.

[0144] It should be understood that portions of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies known in the art or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits (ASICs) having appropriate combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0145] Those skilled in the art of the present technology can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0146] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing module, or each unit can be physically present separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a readable storage medium. The storage medium can be a read only memory, a magnetic disk or an optical disk, etc.

[0147] Figure 7 is a structural schematic diagram of a real-time mapping device 700 combined with a navigation map according to an embodiment of the present disclosure. As shown in Figure 7 According to some embodiments of the present disclosure, the real-time mapping device 700 combined with a navigation map can include:

[0148] The data acquisition module 702 is configured to acquire navigation map data of the vehicle and point cloud data collected by the laser radar on the vehicle.

[0149] The BEV feature module 704 is configured to obtain BEV features of the region around the vehicle according to the navigation map data and the point cloud data through an encoding model.

[0150] The driving category determination module 706 is configured to generate a driving region classification map of the BEV by using the BEV features through a first classification model, and the driving region classification map contains driving category information of the region around the vehicle.

[0151] The road category determination module 708 is configured to generate a road structure classification map of the BEV by using the BEV features through a second classification model, and the road structure classification map contains road category information of the region around the vehicle.

[0152] In some embodiments, the real-time mapping device in combination with a navigation map 700 can further include a navigation data preprocessing module 710 configured to map the navigation map data of the vehicle into a navigation feature map, and the navigation feature map contains road segment information and real-time distance information in a predetermined region corresponding to each real-time position of the vehicle within a predetermined period.

[0153] In some embodiments, the real-time mapping device in combination with a navigation map 700 can further include a point cloud preprocessing module 712 configured to preprocess and project the point cloud data to obtain point cloud pseudo map data, and the point cloud pseudo map data contains a time sequence pseudo map, and the time sequence pseudo map implicitly contains time information of the point cloud data.

[0154] The specific technical details of the real-time mapping device in combination with a navigation map 700 and each module thereof can be found in the method section above, and will not be repeated here.

[0155] The present disclosure also provides an electronic device, which includes a memory storing execution instructions, and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module performs the above-mentioned real-time mapping method in combination with a navigation map.

[0156] The present disclosure also provides a readable storage medium, which stores execution instructions, and the execution instructions are executed by a processor to implement the above-mentioned real-time mapping method in combination with a navigation map.

[0157] In the description of the present specification, the description of the term "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. It is not necessarily the same embodiment / way or example in the illustrative description of the above terms in the present specification. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0158] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0159] The person skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made on the basis of the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method of combining a navigation map with real-time mapping, characterized by, The method comprises: acquiring navigation map data of a vehicle and point cloud data collected by a laser radar on the vehicle; obtaining BEV features of a region around the vehicle according to the navigation map data and the point cloud data through an encoding model; generating a driving region classification map of the BEV through a first classification model using the BEV features, the driving region classification map containing driving category information of the region around the vehicle; generating a road structure classification map of the BEV through a second classification model using the BEV features, the road structure classification map containing road category information of the region around the vehicle; The real-time mapping method further comprises: mapping the navigation map data of the vehicle into a navigation feature map, the navigation feature map containing road segment information and real-time distance information in a predetermined region corresponding to each real-time position of the vehicle within a predetermined period; Elements in the navigation feature map are represented by a key-value pair Key-Value, the key Key in the Key-Value pair indicating a road segment name and the value Value indicating a real-time distance corresponding to the key; The real-time mapping method further comprises: preprocessing the point cloud data and projecting to obtain point cloud pseudo-map data, the point cloud pseudo-map data containing a time sequence pseudo-map, the time sequence pseudo-map implicitly containing time information of the point cloud data; The time sequence pseudo-map is obtained by projecting each point in the point cloud data according to a time index to the same channel. 2.The method of claim 1, wherein, The time sequence pseudo-map is obtained by: calculating a time index of the point cloud data using time information of the point cloud data; processing the point cloud data to use the time index for the point cloud; projecting and mapping the processed point cloud data to a channel as a time sequence pseudo-map. 3.The method of claim 1, wherein, The region around the vehicle includes a laser radar reachable and reliable region, a laser radar reachable edge region and a laser radar unreachable region.

4. The method of claim 1 to 3, wherein, The encoding model adopts a Transformer-Encoder structure with a double position encoding unit, the double position encoding unit including a first position encoding unit for distinguishing spatial sequences and a second position encoding unit for distinguishing time sequences.

5. The method of claim 1 or 2, wherein the real-time mapping method is combined with a navigation map. The loss function of the first classification model includes a first loss corresponding to the laser radar reachable and reliable region, a second loss corresponding to the laser radar edge region and a third loss corresponding to the laser radar unreachable region, the weights of the first loss, the second loss and the third loss being different. 6.The method of claim 1, wherein, The second classification model is trained according to a preconfigured lane category dictionary, the lane category dictionary containing all pre-enumerated lane categories, the lane categories being used to indicate the type, position and / or driving direction of a lane.

7. An electronic device, comprising: The method comprises: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the real-time mapping method combined with a navigation map according to any one of claims 1 to 6.

8. A readable storage medium, characterized by, The readable storage medium stores execution instructions, the execution instructions being executed by the processor to implement the real-time mapping method combined with a navigation map according to any one of claims 1 to 6.

Images