Real-time mapping method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a real-time mapping method, device, electronic equipment and storage medium. The real-time mapping method of the present disclosure comprises: acquiring point cloud data currently collected by a sensor loaded on a vehicle; obtaining real-time mapping information based on the point cloud data by using a pre-trained real-time mapping model; wherein the real-time mapping information comprises: category information of an obstacle, motion prediction information of the obstacle, local static map information and map information outside the sensing range of the sensor. The present disclosure can obtain more complete real-time mapping information without using an external high-precision map.

Description

Technical Field

[0001] The present disclosure relates to a real-time mapping method, device, electronic device and storage medium. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology has important applications in fields such as autonomous driving and robotics that require autonomous positioning. The core of SLAM technology is the pose estimation of mobile devices such as autonomous vehicles and robots. Mapping is based on pose estimation, using sensors such as radar and cameras to perceive the environment and describe the recorded results.

[0003] High-definition map technology is a crucial component of the current implementation and application of autonomous driving. Autonomous driving solutions at levels L2+ and above clearly rely on HD maps. Compared to standard maps, HD maps provide more accurate and comprehensive information required for autonomous driving. Beyond standard maps, HD maps also include additional traffic elements such as streetlights, guardrails, and traffic lights, while providing more accurate lane and intersection information. HD maps can serve various functional modules of autonomous driving, including positioning, perception, prediction, planning, and safety. Judging from the current state of autonomous driving development, HD maps play an extremely important role in autonomous driving.

[0004] There are generally two types of high-precision map production: professional centralized mapping and crowdsourcing mapping. Professional centralized mapping is the process of professionals using professional high-precision data acquisition equipment to collect road data and then process the map. Figure 1 Mapping generally involves four stages: data collection, data processing, element identification, and manual verification. Crowdsourcing mapping involves collecting data separately using a variety of standard equipment and personnel, then merging the data to construct a map. Because professional centralized mapping requires a high level of personnel and equipment, the cost of this approach is high. Crowdsourcing mapping, on the other hand, uses inexpensive equipment to create maps, but the accuracy of these maps is significantly lower than that of professional centralized mapping. Therefore, the vast majority of autonomous driving solutions rely on high-precision maps produced by professional centralized mapping.

[0005] Although high-definition maps play an important role in autonomous driving, they cannot be used on all road segments due to high construction costs and limitations on safety by government laws and regulations. Therefore, real-time driving mapping (ODM) is a viable solution. Real-time driving mapping is different from the mapping scheme of traditional SLAM, which is for positioning services and generally only contains feature point clouds. Real-time driving mapping tries to provide more driving information to vehicles, such as lane lines, surrounding vehicle information, obstacle information, drivable areas, traffic lights, etc. Moreover, without high-definition maps, subsequent path planning and other tasks need to rely on real-time generated driving maps. However, current real-time driving mapping technology cannot provide sufficient reliable real-time map information that can be directly applied to tasks such as path planning. Not only is the amount of information contained in the real-time map obtained by the real-time driving mapping technology unable to meet the driving needs of vehicles during driving, but the real-time nature also cannot meet the needs of real-time driving. Especially in the pure laser radar scene, the information range of the laser radar is limited by the range of the laser radar and the shielding situation of other vehicles on the road. The amount of information of the real-time map is smaller, and the real-time nature is worse. The reliability and accuracy of the real-time map cannot meet the driving needs. SUMMARY

[0006] To solve the above technical problems, the present disclosure provides a real-time mapping method, device, electronic equipment and storage medium.

[0007] According to one aspect of the present disclosure, a real-time mapping method is provided, comprising:

[0008] Obtaining point cloud data currently collected by a sensor mounted on a vehicle;

[0009] Obtaining real-time mapping information based on the point cloud data using a pre-trained real-time mapping model;

[0010] The real-time mapping information includes category information of obstacles, motion prediction information of obstacles, local static map information, and map information outside the sensor perception range.

[0011] According to the real-time mapping method of at least one embodiment of the present disclosure, the category information of the obstacles includes category information of in-lane obstacles; and / or, the local static map information includes map information within a sensor ROI region; and / or, the map information outside the sensor perception range includes map information within a predetermined range around the sensor ROI region; and / or, the motion prediction information of the obstacles includes position information of the obstacles within a future predetermined period.

[0012] According to the real-time mapping method of at least one embodiment of the present disclosure, the category information of the obstacle is used to indicate that the obstacle belongs to one of the following categories: vehicle, person, bicycle, motorcycle, and other categories; wherein the other category includes all types of obstacles except vehicles, people, bicycles / motorcycles.

[0013] According to the real-time mapping method of at least one embodiment of the present disclosure, the first preprocessing includes one or more of the following: motion de-distortion, voxelization, and conversion to pseudo-graph.

[0014] According to the real-time mapping method of at least one embodiment of the present disclosure, the real-time mapping model includes: a backbone network and first, second, third, and fourth output heads connected at the output end of the backbone network, the first output head is used to generate the category information of the obstacle, the second output head is used to generate the motion prediction information of the obstacle, the third output head is used to generate the local static map information, and the fourth output head is used to generate the map information outside the sensor perception range.

[0015] According to the real-time mapping method of at least one embodiment of the present disclosure, the backbone network of the real-time mapping model includes: a Transformer encoder module based on a self-attention mechanism, a first position encoding module, and a second position encoding module, the first position encoding module is used to encode the input data of the Transformer encoder module to distinguish the spatial position relationship between the internal patches of the input data, and the second position encoding module is used to encode the aggregation result of the output data of the Transformer encoder module to distinguish the time sequence relationship between different output data.

[0016] According to the real-time mapping method of at least one embodiment of the present disclosure, the first, second, and / or third output heads adopt a network structure of two two-dimensional convolution layers; and / or, the fourth output head adopts a network structure combining convolution and deconvolution.

[0017] According to the real-time mapping method of at least one embodiment of the present disclosure, further comprising: obtaining a local high-definition map of the position where the vehicle is located according to the RTK pose information of the vehicle; projecting the road information in the local high-definition map onto the BEV map to obtain reference map information outside the sensor perception range for training the real-time mapping model.

[0018] According to the real-time mapping method of at least one embodiment of the present disclosure, the real-time mapping model is trained according to the following loss function:

[0019] L all =lcls +l motion +αl sp +βl mp

[0020]

[0021]

[0022] Among them, L all represents the total loss of the real-time mapping model, l mp Place of expression Figure 1 Causative loss, l cls Including the classification loss of the first output head, the third output head and the fourth output head, l sp represents the spatial consistency loss of motion prediction, l motion represents the regression loss of the second output head, α is the weight hyperparameter of the spatial consistency loss, and β is the ground Figure 1 The weight hyperparameter of the consistency loss; X i,j and X i′,j′ Respectively represent the predicted values ​​of position (i, j) and position (i′, j′) on the same obstacle, represents the predicted value of the boundary point (i, j) in the local static map information, Represents the predicted value of the position (i+1, j) in the map information outside the sensor's perception range that is connected to the boundary point (i, j) in the local static map information.

[0023] According to a real-time mapping method of at least one embodiment of the present disclosure, cls Using cross entropy loss function; and / or, l motion The smooth L1 loss function is used.

[0024] According to one aspect of the present disclosure, a real-time mapping device is provided, comprising:

[0025] A data acquisition unit, used to acquire point cloud data currently collected by sensors mounted on the vehicle;

[0026] A mapping information determination unit is configured to obtain real-time mapping information based on the point cloud data using a pre-trained real-time mapping model; wherein the real-time mapping information includes: obstacle category information, obstacle motion prediction information, local static map information, and map information outside the sensor perception range.

[0027] According to one aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor performs the above-mentioned real-time mapping method.

[0028] According to one aspect of the present disclosure, a readable storage medium is provided, which stores execution instructions for implementing the real-time mapping method described above when executed by a processor.

[0029] The embodiments of the present disclosure can obtain real-time mapping information containing obstacle categories, obstacle motion prediction information, local static map information, and map information outside the sensor perception range without using an external high-precision map, can make up for the lack of real-time mapping information caused by the limitations of sensors such as lidar, and can make the real-time mapping information applicable to real-time driving more abundant and complete, thereby effectively improving the reliability and rationality of subsequent tasks such as path planning, and thus improving driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the general description of the present disclosure given above, explain the principles of the present disclosure. These drawings are provided so that those skilled in the art can understand the present disclosure and are included as part of the present disclosure.

[0031] Figure 1 FIG. 1 is a flowchart of a real-time mapping method according to one embodiment of the present disclosure.

[0032] Figure 2 FIG. 2 is an example diagram of map information perceived by a sensor according to one embodiment of the present disclosure.

[0033] Figure 3 FIG. 3 is an example diagram of map information containing map information inside and outside the sensor perception range obtained by a real-time mapping model according to one embodiment of the present disclosure.

[0034] Figure 4 FIG. 4 is an example diagram of a local static map according to one embodiment of the present disclosure.

[0035] Figure 5 FIG. 5 is an example diagram of a map outside the sensor perception range and a local static map according to one embodiment of the present disclosure.

[0036] Figure 6 FIG. 6 is an example diagram of a network structure of a real-time mapping model according to one embodiment of the present disclosure.

[0037] Figure 7 FIG. 7 is an example diagram of the architecture of a backbone network in a real-time mapping model according to one embodiment of the present disclosure.

[0038] Figure 8 FIG. 8 is an example diagram of a real-time mapping device in a hardware implementation of a processing system according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] Unless otherwise stated, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0042] The use of cross hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise indicated, the presence or absence of cross hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the order described. In addition, the same figure numbers represent the same components.

[0043] When a component is referred to as being “on,” “over,” “connected to,” or “coupled to” another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. For this purpose, the term “connected” may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "including," "includes," "having," "has," "a" or "an," or the like are used in the detailed description and / or claims, such terms are intended to be inclusive (i.e., to say that at least the stated feature is present), unless the context clearly indicates otherwise. It is also to be noted that, as used herein, the terms "substantially," "approximately," and other similar terms are used as terms of approximation and not as terms of degree, unless the context clearly indicates otherwise, and as such, they are used to account for inherent variations in measurement, calculation, and / or other sources of variation.

[0045] Brief Description of Related Art:

[0046] Related Art 1: MP3:A Unified Model to Map,Perceive,Predict and Plan discloses an automatic driving scheme based on neural network, which is independent of high-precision map and integrates perception, prediction and planning. The real-time map is the intermediate output of the convolutional neural network, not the final output. The intermediate channel includes drivable area, intersection, reachable lane line and other information, which is called an interpretable map representation in the scheme. The prediction and planning downstream of the pipeline are all based on this map representation. The scheme emphasizes that the map has strong interpretability, that is, the map representation is not trained by direct map information, and in the case where the training data such as direct map information is insufficient, the reliability and accuracy of the related information of the map representation may be low. Moreover, the scheme does not explicitly extract the structure of the spatio-temporal features in the model, so the effect on time consistency and spatial consistency is not ideal.

[0047] Related Art 2: HDMapNet:A Local Semantic Map Learning and Evaluation Framework proposes a network of HDMapNet, which can generate a local high-precision map in real time, mainly including lane lines and lane line directions and lane line instance segmentation information. The input data includes laser, vision or laser + vision sensor data. The goal of this scheme is mainly focused on the classification and vectorization of lane lines in the high-precision map, without considering other dynamic objects. Such a map cannot be directly used for path planning and other tasks. Moreover, this scheme focuses on the network containing visual information, and the effect is obviously poor for pure laser input scenes.

[0048] Related Technology 3: MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps proposes a model that combines perception and prediction tasks, which can predict the categories of obstacles such as vehicles and people on the real-time road without using the bounding box scheme, and output the grid map in the bird’s eye view (BEV). At the same time, it can predict the speed and direction of moving vehicles. This scheme proposes that the obstacle classification of urban roads should include other categories to buffer objects that vehicles have not seen during training, so as to avoid situations where obstacles cannot be distinguished. However, this scheme focuses on the prediction of moving objects, and the output result does not include important information such as drivable areas and lane lines that are closely related to driving safety.

[0049] As can be seen from the above, the related technologies cannot provide sufficient reliable real-time map information that can be directly applied to planning. Moreover, the map output results of the related technologies are a direct reflection of the current lidar data. However, due to the limitations of the sensor characteristics of the lidar and factors such as occlusion, the direct real-time driving map needs to make more predictions on the road conditions to meet the data needs of tasks such as path planning. Therefore, it is necessary to improve the real-time mapping technology so that the directly obtained real-time driving map can make more comprehensive predictions on the road conditions to meet the data needs of tasks such as path planning.

[0050] In view of this, the embodiments of the present disclosure propose a real-time mapping method, device, electronic equipment and storage medium as follows: the embodiments of the present disclosure use a lidar to construct a real-time driving map, and unlike the real-time driving map in the related technology, the embodiments of the present disclosure can make predictions on map information that is not within the perception range of the lidar.

[0051] Related explanations and explanations of the embodiments of the present disclosure:

[0052] Sensor data: including point cloud data; in the embodiments of the present disclosure, the sensor data has been converted by external parameters by default, which means converting the sensor data from the external parameters of the sensor relative to the center of the mobile device to the center of the device.

[0053] Frame: sensor data is stored and transmitted in the form of frames. Taking a lidar as an example, each frame of point cloud data collected by the lidar (i.e., each laser frame) includes the transmission distance of each point after the lidar emits a laser beam to scan the environment within a fixed time.

[0054] Coordinate system: The embodiments of the present disclosure use a rectangular coordinate system defined using the coordinate system commonly used in the robot operating system. For example, a right-handed Cartesian rectangular coordinate system can be used, i.e., the X axis is forward, the Z axis is upward, and the Y axis is left.

[0055] Region of Interest (ROI): In the embodiments of the present disclosure, the ROI refers to a region in which point cloud information is reliably dense within the range of the laser point cloud.

[0056] Pseudo image: The three-dimensional space processing of the point cloud is rasterized in the Z axis. If each layer is considered as a channel of an image, it is a pseudo image with multiple channels.

[0057] Training set: The training set is a data set used for model training, and has ground truth.

[0058] Validation set: The data distribution of the validation set is the same as that of the training set, and has ground truth. However, the validation set is not used for training, but is used to evaluate the training results of the model.

[0059] Batchsize: The number of training samples input at a time during model training.

[0060] Epoch: One epoch refers to training the entire training set once.

[0061] Figure 1 is a flowchart of a real-time mapping method according to an embodiment of the present disclosure. As shown in Figure 1 , the real-time mapping method of the embodiments of the present disclosure can include the following steps:

[0062] Step S102, acquiring point cloud data currently collected by a sensor mounted on a vehicle;

[0063] Step S104, obtaining real-time mapping information based on the point cloud data using a pre-trained real-time mapping model;

[0064] The real-time mapping information of the vehicle can include: obstacle category information, obstacle motion prediction information, local static map information, and map information outside the sensor perception range.

[0065] In some embodiments, the category information of the obstacle includes category information of the obstacle in the lane. Specifically, the category information of the obstacle can be used to indicate that the obstacle belongs to one of the following categories: vehicle, person, bicycle, motorcycle, and other categories; wherein the other categories can include all types of obstacles other than vehicles, people, bicycles, and motorcycles. In this way, not only can the important information of the obstacle in the scene such as an urban road be covered, but other categories of obstacles that the model fails to learn can be buffered, and the accuracy and completeness of the obstacle category information can be improved.

[0066] For example, the category information of the obstacle in the lane can be, but is not limited to, semantic category information of the obstacle in the lane. Taking an urban road as an example, the semantic category of the obstacle in the lane can be represented as {car, person, bicycle / motorcycle, other obstacles}. On an urban road, for a general supervised learning classification problem, the category of the obstacle is fixed, and if the category of the obstacle is a general whitelist of traffic categories (for example, vehicle, person, bicycle / motorcycle), the prediction of the category of the obstacle is prone to error once an obstacle type that has not been learned is encountered, which can have serious consequences for autonomous driving or assisted driving. Therefore, the obstacle category of the embodiments of the present disclosure not only includes important types in the whitelist of traffic categories, but also includes “other categories”, which can buffer all obstacle categories that have not been learned and do not belong to important information.

[0067] In some embodiments, the motion prediction information of the obstacle can include position information of the obstacle in a future predetermined period. For example, the motion prediction information of the obstacle can include the position of the moving obstacle (for example, a vehicle in motion) on the map in a period t from a certain time (for example, from the time when the point cloud data is collected). The value of the period t can be flexibly set as needed.

[0068] In some embodiments, the local static map information can include map information in the ROI region of the sensor. Taking an urban road as an example, the local static map information can include, but is not limited to, position information of lane lines, drivable areas, and non-drivable areas in the map within the ROI region of the sensor. Since the local static map information is completely from the data of the laser radar and other sensors in its detection range, it has very high credibility.

[0069] In some embodiments, the map information outside the sensor perception range can include map information of an invisible area of the sensor such as a laser radar, which can include an occluded area or an area beyond the range of the laser radar. In this way, without using an external high-definition map, the map information outside the sensor perception range can be obtained from the data collected by the sensor, effectively making up for the lack of important map information caused by the limited range and perception range of the sensor such as a laser radar, effectively widening the map range covered by the real-time mapping information, helping the vehicle to expand the field of view and obtain more valuable real-time map information, thereby providing more map information to the downstream automatic driving module, and effectively improving driving safety.

[0070] The map information outside the sensor perception range does not need to cover the entire range. Since the map information within the sensor perception range is already included in the local static map, in some embodiments of the present disclosure, the map information outside the sensor perception range can only include map information that is not within the ROI area of the sensor. Preferably, the map information outside the sensor perception range can only include map information within a predetermined range around the ROI area of the sensor. For example, the map information outside the sensor perception range can include, but is not limited to, map information such as intersections and roundabouts within a predetermined range around the outside of the ROI area of the sensor.

[0071] Figure 2 The map information sensed by the sensor such as a laser radar when the vehicle is about to approach a position near a crossroads is shown in FIG. 1A, Figure 2 The gray square represents the vehicle, and the arrow on the gray square indicates the driving direction of the vehicle. By Figure 2 As can be seen, the sensor such as a laser radar can only perceive part of the road along the crossroads that the vehicle is about to approach, and cannot perceive the overall situation of the crossroads. If the vehicle needs to turn left at a position outside the sensor perception range (for example, 100 meters away from the current position of the vehicle), the overall situation of the crossroads needs to be predicted, that is, the map information outside the sensor perception range needs to be predicted, so that accurate path planning can be performed in advance. As can be seen, the real-time mapping information used for path planning needs to include map information outside the sensor perception range to ensure driving safety as much as possible.

[0072] For Figure 2 For the scenario shown in FIG. 1A in which the vehicle is about to approach a position near a crossroads, the embodiment of the present disclosure can recover the road structure of the crossroads according to the road along the crossroads and other information detected from the point cloud data collected by the sensor by using a pre-trained real-time mapping model, infer the width and position of the feasible lane according to the road width or the driving trajectory of the vehicle, and obtain the map information shown in FIG. 1B, such as Figure 3 Figure 3 ​As shown, the map information obtained by the embodiments of the present disclosure not only contains map information within the sensor perception range (for example, part of the road in front of the intersection, the road along the intersection, etc.), but also contains map information outside the sensor perception range (for example, the overall view of the intersection). When the vehicle needs to turn left, the map information shown can accurately plan the left-turn path of the vehicle in advance. Figure 3 As shown, the map information shown can accurately plan the left-turn path of the vehicle in advance.

[0073] In comparison Figure 2 and Figure 3 It can be seen that, by obtaining map information outside the sensor perception range, the embodiments of the present disclosure can effectively expand the map range covered by the real-time mapping information, expand the field of view of the vehicle, and improve the accuracy, reliability and safety of subsequent tasks such as path planning.

[0074] Figure 4 and Figure 5 respectively show an example of a local static map based on the same sensor data and an example of a map within and outside the sensor perception range. Figure 4 The local static map information contains map information within the ROI region of the sensor (i.e., the sensor perception range), Figure 5 The map outside the sensor perception range and the map within the sensor perception range (i.e., Figure 4 the local static map) in the above are shown, Figure 5 The information of the map outside the sensor perception range in the above can only contain map information outside the ROI region of the sensor, containing map information within a certain distance to the left, right and front of the ROI region of the sensor.

[0075] In the real-time mapping information, the local static map information and the map information outside the sensor perception range can be distinguished according to the BEV range. The BEV range of the local static map information is the ROI region of the sensor such as a laser radar, and the BEV range of the map information outside the sensor perception range is greater than the ROI region of the sensor such as a laser radar and does not contain the ROI region of the sensor such as a laser radar.

[0076] Suppose the ROI range of the local static map in the x direction and the y direction is [x1, y1], and the range in the z direction is z1, which can be set to 5 or other values, and the grid resolution in the three directions is [r x ,r y ,r z ], then the data format of the local static map information can be represented as the following formula (1):

[0077]

[0078] In formula (1), w l represents the width of the local static map, h lrepresents the height of the local static map, c l represents the number of lanes of the local static map.

[0079] The coverage of the map information outside the sensor perception range is farther and wider than the local static map. Therefore, the BEV range of the map information outside the sensor perception range needs to refer to the road width and the area of the traffic intersection of the applied area. For example, the BEV range of the map information outside the sensor perception range in the x direction and the y direction can be set as [x2, y2], and the resolution of the two directions is also [r x ,r y ] respectively. The data format of the map information outside the sensor perception range can be represented as the following formula (2):

[0080]

[0081] In formula (2), w hd represents the width of the map outside the sensor perception range, and h hd represents the height of the map outside the sensor perception range.

[0082] In some embodiments, before step S104, a first preprocessing can also be performed on the point cloud data, which can include one or more of the following: motion de-distortion, voxelization, and conversion to a pseudo map.

[0083] The mobile device is mostly in motion during the collection of the point cloud data, and because the frequency of the sensor such as the laser radar is limited, the same frame of point cloud data is not collected at the same time, the starting laser beam and the ending laser beam of the same frame correspond to different positions of the moving object (for example, a vehicle), and different laser points correspond to different coordinate systems. Therefore, there will be a large error in positioning and mapping without considering the distortion of the laser point cloud. Therefore, compensation and elimination of the point cloud distortion are needed.

[0084] The motion de-distortion of the point cloud data can also be referred to as point cloud distortion compensation. Exemplarily, the point cloud distortion compensation can include: pushing all points in the same frame of point cloud data to the coordinate system of the starting laser beam according to a motion model. Specifically, the motion speed of the mobile device provided by other sensors can be used to obtain the relative motion position by integration, and a conversion matrix is calculated according to the relative relationship to convert all points in the laser frame to the coordinate system of the initial point (i.e., the starting laser beam). Through the point cloud distortion compensation, the accuracy of real-time mapping can be effectively improved.

[0085] The pseudo map refers to converting the point cloud data into a pseudo map. That is, the current frame point cloud and the point cloud data converted to the past 1 second in the current frame coordinate system are converted into a BEV map. In the embodiment of the present disclosure, the parameters (for example, length, width, and other parameters) of the BEV map converted from the point cloud data and the parameters of the BEV map of the remaining channels other than the map information outside the sensor perception range (for example, the BEV format of the local static map information) can be the same.

[0086] Exemplarily, the process of converting the point cloud data into a pseudo map can include: obtaining a pseudo map I1 based on length-width-height rasterization. Since the local static map includes information of lane lines, the intensity information in the point cloud needs to be output separately into the pseudo map. The identification of the lane line is related to both the point height value and the point intensity. The z value related to the lane line intensity information is set as z2. According to the channel segmentation result of the pseudo map I1, the height value in each channel is changed to an intensity value to obtain a pseudo map I2. In this way, the maximum height value in the current pixel grid is stored in the pseudo map I1, and the maximum intensity value is stored in the pseudo map I2. Finally, the pseudo map I1 and the pseudo map I2 are combined to form a pseudo map corresponding to the point cloud data.

[0087] In some embodiments, in step S104, the map information outside the sensor perception range can be obtained by: obtaining a high-precision map of the current location of the vehicle through a real-time mapping model; and projecting the road information in the high-precision map onto the BEV map to obtain the map information outside the sensor perception range. Here, the road information of the high-precision map can include but is not limited to lane lines, intersections, roundabouts, drivable areas, road edges, and the like in the high-precision map.

[0088] The deep neural network has multiple fields, and each field has multiple models. For example, the main representative in the visual field is CNN, and the RNN series model is more suitable for the language text field. Therefore, the model category needs to be selected according to the characteristics of the task. At the same time, the complexity of the model needs to be determined according to the difficulty of the task to prevent overfitting caused by selecting a too complex model.

[0089] The real-time mapping model of the embodiment of the present disclosure involves tasks including semantic segmentation, semantic instance, motion prediction, and long-distance data association. Therefore, the real-time mapping model of the embodiment of the present disclosure is preferably a deep neural network having a feature extraction capability related to semantic recognition, a capability of associating long-range features, and the ability to meet certain temporal and spatial consistency.

[0090] Since the real-time mapping model of the embodiment of the present disclosure needs to process a composite task, the real-time mapping model of the embodiment of the present disclosure can adopt a deep neural network architecture of “Backbone + multi-head output”.

[0091] In some embodiments, the real-time mapping model can include a backbone network and a first output head, a second output head, a third output head and a fourth output head connected at the output end of the backbone network, the first output head can be used to generate category information of obstacles, the second output head can be used to generate motion prediction information of obstacles, the third output head can be used to generate local static map information, and the fourth output head can be used to generate map information outside the sensor perception range.

[0092] Figure 6 A network architecture diagram of the real-time mapping model of the embodiments of the present disclosure is shown. The input data of the real-time mapping model is multi-frame point cloud data in BEV format, which is processed by the skeleton network and then enters each output head for further processing, and finally the category information of obstacles, the motion prediction information of obstacles, the local static map and the map information outside the sensor perception range are obtained.

[0093] Exemplarily, as shown in Figure 6 The first output head can adopt a network structure of two two-dimensional convolution layers. The second output head can adopt a network structure of two two-dimensional convolution layers. The third output head can adopt a network structure of two two-dimensional convolution layers. Since the resolution of the map information outside the sensor perception range is higher than the input resolution, the fourth output head can preferably adopt a network structure combining convolution and deconvolution.

[0094] In some embodiments, the encoder module of the Transformer can be selected as the Backbone model of the real-time mapping model.

[0095] The common practice of using the structure of the Transformer in the field of vision is to divide the picture structure into multiple patches, and then use the picture patches as the input of the network. Such a method is desirable for tasks that do not need to consider the time continuity. Since motion prediction of moving obstacles needs to be considered during real-time mapping, the time continuity needs to be considered. In view of this, the input data of the backbone network of the real-time mapping model is defined as a pseudo picture of continuous multiple frames of point clouds in the embodiments of the present disclosure, and each frame of point cloud is further processed by a patch block. At the same time, in order to ensure the particularity of the time sequence and the spatial sequence, two position encodings can be used inside the backbone network of the real-time mapping model, i.e. inside the Transformer Encoder network, to effectively extract spatial features and time features.

[0096] In some embodiments, the backbone network of the real-time mapping model can include a Transformer encoder module based on a self-attention mechanism, a first position encoding module, and a second position encoding module. The first position encoding module can be used to encode the input data of the Transformer encoder module to distinguish the spatial position relationship between the internal patches of the input data. The second position encoding module is used to encode the aggregation result of the output data of the Transformer encoder module to distinguish the time sequence relationship between different output data. Thus, according to the above, the embodiments of the present disclosure comprehensively consider the spatial distribution relationship, the time sequence consistency, and the map consistency of the point cloud data, and use the Transformer network with two introduced position encodings to realize the distinction of the internal spatial information of the same frame point cloud, the position consistency of the same patch of different frame data, and the distinction of the time sequence information of different frames.

[0097] Figure 7 An architecture example diagram of the backbone network of the real-time mapping model of the embodiments of the present disclosure is shown. As shown in Figure 7 The backbone network of the real-time mapping model of the embodiments of the present disclosure includes n (n is an integer greater than 2) Transformer Encoder modules of the first level and one Transformer Encoder module of the second level. Two position encoding modules are set to distinguish the spatial sequence and the time sequence. Position encoding 1 (i.e., the first position encoding module) is connected between the n Transformer Encoder modules of the first level, and position encoding 2 (i.e., the first position encoding module) is connected between the Transformer Encoder module of the first level and the Transformer Encoder module of the second level. By Figure 7 It can be seen that the position encoding 1 can be added to the point cloud data of each input data of the backbone network of the real-time mapping model, which can be used to distinguish the position relationship of the internal patches of each point cloud, and the position relationship is consistent between different frames. The position encoding 2 (i.e., the second position encoding module) can be added to the aggregation result of the multi-frame point cloud, which can be used to distinguish the time sequence relationship between the point clouds.

[0098] The two position encoding modules introduced in the backbone network of the real-time mapping model of the embodiments of the present disclosure have the following effects: 1> distinguishing the spatial position relationship between the internal patches of the same frame point cloud; 2> the spatial consistency of the patches corresponding to different frame point clouds; 3> distinguishing the time sequence relationship of different frame point clouds. It can be seen that the embodiments of the present disclosure use two kinds of position encodings to comprehensively consider the spatial distribution relationship, the time sequence relationship, and the spatial consistency, which can ensure the time continuity and the spatial consistency in the real-time mapping process.

[0099] The training data set of the real-time mapping model can use an applicable public data set, or can be constructed by calibrating sensor data of the vehicle, a real high-definition map, etc. When constructing the training data set, the total amount of data is ensured to meet the needs of model training, to prevent the real-time mapping model from being under-fitted due to insufficient data. At the same time, the diversity of the data collection scene is also ensured as much as possible, such as a lane congestion section, a situation in which a laser radar and other sensors are blocked, a daytime scene, a nighttime scene, a rainy scene, a foggy scene, a sunny scene, etc., so that the generalization and adaptability of the real-time mapping model are stronger.

[0100] Before training the real-time mapping model using the training data set, preprocessing can be performed on the training data set, and the preprocessing manner is the same as the preprocessing before step S104 in the foregoing. That is, preprocessing such as motion distortion correction, voxelization, and conversion of pseudo maps can be performed on the point cloud data in the training data set; data related to semantic segmentation in the training data set can be labeled according to IMU data, such as data related to moving objects such as road surface segmentation, vehicles, and pedestrians; and local static reference map information at the corresponding position and reference map information outside the sensor perception range can be obtained according to the corresponding RTK pose information and external high-definition map. The training data set after the foregoing preprocessing can be directly used for training of the real-time mapping model.

[0101] When a laser radar and other sensors are used as the only environment acquisition sensor, the training data set of the real-time mapping model can be constructed based on the point cloud data collected by the laser radar and other sensors, real-time dynamic (RTK) pose information, IMU data, and an external high-definition map. The local high-definition map of the urban road where the vehicle is located can be obtained through the RTK pose information, and the self-vehicle speed and other-vehicle speed during vehicle driving can be obtained through the IMU data. Therefore, the point cloud data is combined with the RTK pose information, IMU data, etc. to construct the training data set, which can improve the accuracy of the training data of the real-time mapping model, thereby improving the training efficiency and training effect of the real-time mapping model.

[0102] In some embodiments, the local static reference map information and the reference map information outside the sensor perception range used for training the real-time mapping model can be obtained using the BEV projection of the ground truth of the high-definition map. Specifically, the real-time mapping method of the embodiments of the present disclosure can further include: obtaining a local high-definition map of a position where the vehicle is located according to RTK pose information of the vehicle; and projecting road information in the local high-definition map onto a BEV map to obtain reference map information outside the sensor perception range used for training the real-time mapping model.

[0103] For example, as the ground truth of training data, a local high-definition map of the vehicle's current location can be obtained based on the vehicle's RTK posture information, and road information such as lane lines, intersections, roundabouts, drivable areas, and curbs in the local high-definition map can be projected onto the BEV map to obtain reference map information outside the sensor's perception range.

[0104] The setting of the loss function is related to the model output. In some embodiments, the real-time mapping model can be trained according to the loss function of the following formula (3):

[0105] L all =l cls +l motion + αl sp + βl mp (3)

[0106] Among them, L all represents the total loss of the real-time mapping model, l mp Place of expression Figure 1 Causative loss, l cls Including the classification loss of the first output head, the third output head and the fourth output head, l sp represents the spatial consistency loss of motion prediction, l motion represents the regression loss of the second output head, α is the weight hyperparameter of the spatial consistency loss, and β is the ground Figure 1 The weight hyperparameter of the consistency loss.

[0107] Since the motion prediction problem needs to consider spatial consistency, that is, the motion prediction of points on the same object should be the same, the above loss function of the embodiment of the present disclosure adds a spatial consistency loss l to the output of the motion prediction. sp .

[0108] In some implementations, the spatial consistency loss l sp The loss function can be the function shown in the following formula (4):

[0109] l sp =∑ (i,j) ∑ (i′,j′) |X i,j -X i′,j′ | (4)

[0110] Among them, X i,j and X i′,j′ They represent the predicted values ​​of position (i, j) and position (i′, j′) on the same obstacle respectively.

[0111] The predicted output of the map information outside the sensor perception range and the predicted output of the local static map information both belong to classification problems. If the predicted output of the map information outside the sensor perception range is in a non-ROI area, which does not overlap with the local static map at all, then there is no need to calculate the overlapping position with the local static map when calculating the classification loss of the predicted output of the map information outside the sensor perception range. Although the predicted output of the map information outside the sensor perception range and the predicted output of the local static map information do not overlap in area, there is continuity within the map. For example, the lane lines, curbs, etc. at the connection position of the local static map and the map outside the sensor perception range should remain consistent. In view of this, the embodiment of the present disclosure adds a ground error in the loss function of the real-time mapping model. Figure 1 Consistency loss function l mp .

[0112] In some embodiments, the Figure 1 Consistency loss function l mp The loss function can be the function shown in the following formula (5):

[0113]

[0114] in, represents the predicted value of the boundary point (i, j) in the local static map information, Represents the predicted value of the position (i+1, j) in the map information outside the sensor's perception range that is connected to the boundary point (i, j) in the local static map information.

[0115] The embodiment of the present disclosure constructs a map by using the map connection edge data to keep consistent. Figure 1 The consistency loss function can ensure the consistency of map information outside the sensor's perception range and local static map information.

[0116] The first, third, and fourth output heads in the real-time mapping model all belong to the task of segmentation. Therefore, the classification loss l cls Can be selected as a loss function suitable for classification tasks. For example, the classification loss l cls A cross entropy loss function can be used.

[0117] In some implementations, the classification loss l cls The loss function can be the function shown in the following formula (6):

[0118]

[0119] In formula (6), N refers to the number of pixels in the pseudo image participating in the classification, i is the ergodic variable, and formula (6) represents the sum of the result errors of each pixel.

[0120] The motion prediction of the obstacle belongs to regression prediction, that is, the second output head in the real-time mapping model belongs to a regression prediction task, and thus the regression loss l motion The loss function can be selected as a loss function suitable for a regression task. Exemplarily, the regression loss l motion The smooth L1 loss function can be used.

[0121] In some embodiments, the regression loss l motion The loss function can be a function shown in the following formula (7):

[0122]

[0123] In formula (7), N represents the number of pixels of the pseudo map participating in classification, and i is a traversal variable.

[0124] The real-time mapping model can be set with multiple sets of hyperparameters. For example, the hyperparameters of the real-time mapping model can include, but are not limited to, Batchsize, training epoch, learning rate, and the like. For the same set of parameters, different initialization methods can also be used to improve the training effect of the real-time mapping model. During the training process, the parameters can also be adjusted in real time according to the real-time training error and the validation error.

[0125] After training, the real-time mapping model can be tested on a validation set. Specifically, multiple sets of trained real-time mapping models can be tested on the validation set, and the evaluation indicators and the loss function are consistent. The real-time mapping model with the smallest loss function has better generalization ability and better performance in the actual use stage, and thus the real-time mapping model with the smallest loss function is preferably applied to the actual real-time mapping.

[0126] The embodiments of the present disclosure can obtain real-time mapping information containing obstacle categories, obstacle motion prediction information, local static map information, and map information outside the sensor perception range without using an external high-precision map. The real-time mapping information can not only make up for the insufficient information caused by the limitations of sensors such as laser radars, but also enrich and complete the information for real-time driving, and can effectively improve the reliability and rationality of subsequent tasks such as path planning, thereby improving driving safety.

[0127] Figure 8 An example diagram of a real-time mapping device using a hardware implementation of a processing system is shown.

[0128] The apparatus can include respective means for performing each of or several of the steps in the above-described flow charts. Accordingly, those steps of the flow charts above-including each and every step in the flow charts—can be performed by a respective means, and the apparatus can include one or more such means. The means can be one or more hardware modules specially configured to carry out the associated steps, or can be implemented by a processor configured to perform the associated steps by way of program instructions stored in a memory medium, or some combination thereof.

[0129] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 900 connects various circuits including one or more processors 1000, memory 1100, and / or hardware modules together. Bus 900 can also connect various other circuits 1200 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0130] Bus 900 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0131] Any process or method described in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps) of the process, and the various embodiments of the present disclosure can include additional implementation in which the functions are performed in a different order, or are performed substantially concurrently, or are performed in reverse order, as will be appreciated by those skilled in the art of the field of the present disclosure. Processors perform various methods described above. For example, the method embodiments in the present disclosure can be implemented as a software program tangibly embodied in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps of the above-described methods can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above-described methods by other any appropriate means, such as by means of firmware.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0133] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0134] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0135] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0136] In addition, each of the functional units in the various embodiments of the present disclosure can be integrated into one processing module, or each unit can be physically present separately, or two or more units can be integrated into one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0137] Figure 8 is a structural schematic diagram of a real-time mapping device according to an embodiment of the present disclosure. As shown in Figure 8 , the real-time mapping device 800 according to the present disclosure can include:

[0138] a data acquisition unit 802 configured to acquire point cloud data currently collected by a sensor loaded on a vehicle;

[0139] a mapping information determination unit 804 configured to obtain real-time mapping information based on the point cloud data by using a pre-trained real-time mapping model;

[0140] The real-time mapping information includes: category information of an obstacle, motion prediction information of the obstacle, local static map information, and map information outside a sensor perception range.

[0141] The present disclosure also provides an electronic device including a memory storing execution instructions, and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the real-time mapping method described above.

[0142] The present disclosure also provides a readable storage medium having execution instructions stored therein, and the execution instructions are executed by a processor to implement the real-time mapping method described above.

[0143] In the description of the present specification, the description of the terms "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily the same embodiment / way or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0144] Furthermore, the terms "first", "second", etc. are used herein for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined with "first", "second" etc. can explicitly or implicitly include at least one of such features. In the description of the application, the meaning of "plurality" is at least two, for example two, three, etc., unless explicitly and specifically defined otherwise.

[0145] Those skilled in the art should understand that the above-mentioned embodiments are only for clearly illustrating the present disclosure, but not for limiting the scope of the present disclosure. Based on the above disclosure, other changes or modifications can also be made by those skilled in the art, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method of real-time mapping, characterized in that, The method comprises: acquiring point cloud data currently collected by a sensor mounted on a vehicle; obtaining real-time mapping information based on the point cloud data by using a pre-trained real-time mapping model; the real-time mapping information comprises: category information of an obstacle, motion prediction information of the obstacle, local static map information, and map information outside a sensor perception range; the category information of the obstacle comprises category information of an in-lane obstacle; the local static map information comprises map information within a sensor ROI region; and the map information outside the sensor perception range comprises map information within a predetermined range around the sensor ROI region; the motion prediction information of the obstacle comprises position information of the obstacle within a future predetermined period of time; the method further comprises: acquiring a local high-definition map of a location where the vehicle is located according to RTK pose information of the vehicle; projecting road information in the local high-definition map onto a BEV map to obtain reference map information outside the sensor perception range for training the real-time mapping model; the real-time mapping model is trained according to the following loss function: wherein, denotes the total loss of real-time mapping model, denotes the map consistency loss, denotes the classification loss including the first output head, the third output head and the fourth output head, denotes the spatial consistency loss of motion prediction, denotes the regression loss of the second output head, α is a weight hyperparameter of the spatial consistency loss, and β is a weight hyperparameter of the map consistency loss; and respectively denote the predicted values of two different positions, i.e. position (i, j) and position (i+1, j), on the same obstacle, denotes the predicted value of the boundary point (i, j) in the local static map information, denotes the predicted value of the position (i+1, j) connected with the boundary point (i, j) in the local static map information, and denotes the predicted value of the position (i+1, j) connected with the boundary point (i, j) in the local static map information.

2. The real-time mapping method of claim 1, wherein, the category information of the obstacle is used to indicate that the obstacle belongs to one of the following categories: vehicle, person, bicycle, motorcycle, and other categories; the other categories include all types of obstacles except vehicles, people, bicycles / motorcycles.

3. The real-time mapping method according to claim 1, characterized in that: Further comprising: performing first preprocessing on the point cloud data, the first preprocessing comprising one or more of the following: motion de-distortion, voxelization, and conversion to pseudo-map.

4. The real-time mapping method of claim 1, wherein, The real-time mapping model comprises: a backbone network, and first, second, third, and fourth output heads connected to an output end of the backbone network, the first output head being used to generate the category information of the obstacle, the second output head being used to generate the motion prediction information of the obstacle, the third output head being used to generate the local static map information, and the fourth output head being used to generate the map information outside the sensor perception range.

5. The real-time mapping method of claim 4, wherein, The backbone network of the real-time mapping model comprises: a Transformer encoder module based on a self-attention mechanism, a first position encoding module, and a second position encoding module, the first position encoding module being used to encode input data of the Transformer encoder module to distinguish spatial position relationships between internal patches of the input data, and the second position encoding module being used to encode an aggregation result of output data of the Transformer encoder module to distinguish time sequence relationships between different output data.

6. The real-time mapping method of claim 4, wherein, The first, second, and / or third output heads adopt a network structure of two two-dimensional convolution layers; and / or, the fourth output head adopts a network structure combining convolution and deconvolution.

7. The method of real-time mapping of claim 1, wherein, Using a cross entropy loss function; and / or, The smooth L1 loss function is used.

8. An electronic device, comprising: The method comprises: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the real-time mapping method according to any one of claims 1 to 7.

9. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are used for realizing the real-time mapping method in any one of claims 1 to 7 when executed by the processor.

Images