Method for constructing a perception map

Abstract

A method for constructing a perception map, comprising: obtaining a sequence of laser point cloud frames and a sequence of image frames in real time; matching the laser point cloud frames and the image frames to determine the current laser point cloud frame and its corresponding current image frame; determining static points of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame; constructing a current grid map based on the static points and determining parameter information of each grid; and constructing a perception map based on a grid AGE update mechanism.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and more particularly, to a method for constructing a perception map. Background Art

[0002] In the autonomous driving scenario, due to limitations in computing power and sensor layout, on-vehicle sensors inevitably have perception blind spots; static obstacles (such as traffic cones) outside the blind spots are promptly identified and "cached", and a perception map is constructed. When the static obstacle enters the perception blind spot, the obstacle can be quickly and accurately identified by querying the perception map; when the vehicle operates on a fixed route, by using the historically constructed perception map, the problem of detecting and identifying static obstacles can be transformed into a query problem, significantly reducing the computational cost and improving the recognition accuracy. Summary of the Invention

[0003] Some embodiments of the present disclosure provide a method for constructing a perception map, including:

[0004] Obtaining a sequence of laser point cloud frames and a sequence of image frames in real time;

[0005] Matching the laser point cloud frame and the image frame to determine the current laser point cloud frame and its corresponding current image frame;

[0006] Determining the static points of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame;

[0007] Constructing a current grid map based on the static points and determining parameter information of each grid; and

[0008] Constructing a perception map based on the grid AGE update mechanism.

[0009] In some embodiments, the matching the laser point cloud frame and the image frame to determine the current laser point cloud frame and its corresponding current image frame includes:

[0010] Determining a first timestamp of the current laser point cloud frame; and

[0011] Traversing the sequence of image frames, and when the difference between the timestamp of the image frame and the first timestamp is less than a first threshold, determining the image frame as the current image frame corresponding to the current laser point cloud frame.

[0012] In some embodiments, the determining the static points of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame includes:

[0013] Performing semantic segmentation on the current image frame to obtain a semantic segmentation mask map;

[0014] Projecting the current laser point cloud frame onto the semantic segmentation mask map through coordinate transformation; and

[0015] Filter the static points of the current laser point cloud frame.

[0016] In some embodiments, the static points of the current laser point cloud frame are given class information and / or height information.

[0017] In some embodiments, constructing the current grid map based on the static points and determining the parameter information of each grid includes:

[0018] Construct the current grid map according to a predetermined grid size; and

[0019] Determine the parameter information of each grid based on the parameter information of the static points projected onto each grid and the historical parameter information of each grid.

[0020] In some embodiments, the parameter information of each grid includes class information and / or height information.

[0021] In some embodiments, constructing the perception map based on the grid AGE update mechanism includes performing the following operations on each grid in the current grid map:

[0022] Obtain the AGE value of the grid;

[0023] Update the AGE value of the grid based on whether the grid corresponds to the static points of the current laser point cloud frame;

[0024] Cache the AGE value of the grid.

[0025] In some embodiments, obtaining the AGE value of the grid includes:

[0026] When the grid exists in the previous grid map, use the updated AGE value of the grid in the previous grid map as the AGE value before update in the current grid map; and

[0027] When the grid does not exist in the previous grid map, set the AGE value of the grid to 0.

[0028] In some embodiments, the updating the AGE value of the grid based on whether the grid corresponds to the static points of the current laser point cloud frame includes:

[0029] When the grid corresponds to the static points of the current laser point cloud frame, increment the AGE value of the grid by 1; and

[0030] When the grid does not correspond to the static points of the current laser point cloud frame, decrement the AGE value of the grid by 1.

[0031] In some embodiments, the updating the AGE value of the grid based on whether the grid corresponds to the static points of the current laser point cloud frame further includes:

[0032] When the updated AGE value is less than 0, set the updated AGE value to 0; and

[0033] When the updated AGE value is greater than a predetermined value, set the updated AGE value to the predetermined value.

[0034] The above solution of the embodiments of the present disclosure has at least the following beneficial effects compared with the related art:

[0035] The present disclosure uses the fusion of laser point cloud information and image information to perform real-time online mapping of static obstacles, identify and refresh the information of the above static obstacles, and judge the motion state of the obstacles by fusing the timing information of multiple frames of data through the Age update mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. In the drawings:

[0037] Figure 1 is a flowchart of a method for constructing a perception map provided by some embodiments of the present disclosure;

[0038] Figure 2 is Figure 1 a flowchart of step S102 in

[0039] Figure 3 is Figure 1 a flowchart of step S103 in

[0040] Figure 4 is Figure 1 a flowchart of step S104 in

[0041] Figure 5 is Figure 1 a flowchart of step S105 in DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] To more clearly elaborate the purpose, technical solutions and advantages of the present disclosure, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts belong to the scope of protection of the present disclosure. It should be understood that the following description of the embodiments is intended to explain and illustrate the general concept of the present disclosure, rather than being construed as a limitation to the present disclosure. In the specification and drawings, the same or similar reference numerals refer to the same or similar components or elements. For clarity, the drawings are not necessarily drawn to scale, and some well-known components and structures may be omitted in the drawings.

[0043] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should have the ordinary meaning understood by those of ordinary skill in the art to which the present disclosure pertains. The terms "first", "second" and similar terms used in the present disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. The word "a" or "an" does not exclude a plurality. The terms such as "including" or "comprising" mean that the elements or items appearing before this word cover the elements or items listed after this word and their equivalents, without excluding other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", "right", "top" or "bottom" etc. are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly. When an element is referred to as being "on" or "under" another element, the element may be "directly" on or under the other element, or there may be intermediate elements.

[0044] Currently, most of the maps constructed and used for autonomous driving are high-precision maps (HDMaps). Such maps accurately identify information such as curbs and lanes. Through the identified curb information, the areas where some static obstacles such as green plants and curbs are located can be approximately inferred, and it has a strong dependence on positioning accuracy. Although high-precision maps (HDMaps) can accurately identify road traffic elements, they have high maintenance costs and long production cycles. In view of the phenomenon that low-growing green plants beside the road grow vigorously and invade the road area, due to the long production cycle of HDMaps, the boundaries of the actual drivable area of the vehicle cannot be corrected in time. At the same time, for temporary static obstacles (such as traffic cones, etc.), since it is difficult to estimate their existence period during the production of HDMaps, the information of temporary roadblocks is often not recorded.

[0045] The present disclosure provides a method for constructing a perception map, including: obtaining a sequence of laser point cloud frames and a sequence of image frames in real time; matching the laser point cloud frames and the image frames to determine a current laser point cloud frame and its corresponding current image frame; determining static points of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame; constructing a current grid map based on the static points and determining parameter information of each grid; and constructing a perception map based on a grid AGE update mechanism.

[0046] The present disclosure uses the fusion of laser point cloud information and image information to perform real-time online mapping of static obstacles, identify and refresh the information of the above-mentioned static obstacles, and judge the motion state of the obstacles by fusing the timing information of multiple frames of data through an Age update mechanism.

[0047] The optional embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0048] Figure 1 It is a flowchart of a method for constructing a perception map provided for some embodiments of the present disclosure. As Figure 1 shown, the method for constructing a perception map provided for some embodiments of the present disclosure includes the following steps S101 to S105:

[0049] S101: Obtain a sequence of laser point cloud frames and a sequence of image frames in real time.

[0050] One or more lidars and one or more image detectors are provided on the vehicle. During the process of autonomous driving of the vehicle, the lidar is used to obtain the laser point cloud data around the vehicle in real time, and the image detector is used to obtain the image data around the vehicle in real time. The laser point cloud data can be cached in a data frame sequence to obtain a sequence of laser point cloud frames, and the sequence of laser point cloud frames has its corresponding timestamp. The image data can also be cached in a data frame sequence to obtain a sequence of image frames, and each sequence of image frames has its corresponding timestamp.

[0051] S102: Match the laser point cloud frames and the image frames to determine a current laser point cloud frame and its corresponding current image frame.

[0052] Specifically, matching the laser point cloud frames and the image frames facilitates the fusion of the laser point cloud data and the image data obtained at the same or similar times. Figure 2 For Figure 1 is a flowchart of step S102 in Figure 2 shown, step S102 specifically includes the following steps S1021 to S1022:

[0053] S1021: Determine the first timestamp of the current laser point cloud frame;

[0054] Obtain the timestamp t l of the current laser point cloud frame, that is, at tl Obtain the laser point cloud data of the current laser point cloud frame at all times.

[0055] S1022: traverse the image frame sequence, and when the difference between the timestamp of an image frame and the first timestamp is less than a first threshold, determine the image frame as the current image frame corresponding to the current laser point cloud frame.

[0056] Specifically, the cached image frame sequence is traversed to obtain the timestamp t of each image frame in the image frame sequence. c , when the difference between the timestamp of the image frame and the first timestamp is less than the first threshold value t thred Time, that is, t l -t c │<t thred , determine the image frame as the current image frame corresponding to the current laser point cloud frame.

[0057] Since the frequency and phase of data sent by the camera and lidar are different, and the lidar may drop frames, the timestamps of the laser point cloud frame and the image frame will not be exactly the same, and there will most likely be deviations. In order to fuse the laser point cloud data with the image data, the applicant associates the image frames and laser point cloud frames whose timestamp differences are less than a predetermined threshold.

[0058] In some embodiments, if a qualified image frame corresponding to the current laser point cloud frame cannot be found, the current laser point cloud frame is discarded, and a new laser point cloud frame is obtained as the current laser point cloud frame to perform steps S1021 and S1022 until the current laser point cloud frame is corresponded to the current image frame.

[0059] S103: Determine a static point of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame.

[0060] The laser point cloud frame data only reflects the distance information between the obstacle and the laser radar, the shape information of the obstacle, etc., but does not reflect the category information of the obstacle. It is necessary to combine the image frame information to determine the category of the obstacle, and then obtain the static obstacle information, so as to facilitate the subsequent accurate generation of the perception map.

[0061] Figure 3 for Figure 1 The flowchart of step S103 in FIG. Figure 3 As shown, step S103 specifically includes the following steps S1031 to S1033:

[0062] S1031: Perform semantic segmentation on the current image frame to obtain a semantic segmentation mask image;

[0063] Specifically, semantic segmentation is performed on the current image frame to calculate the category of each pixel in the current image frame, and then a semantic segmentation mask map is obtained to mark the category information of each object on the current image frame, such as green plants, buildings, curbs, roads, utility poles, traffic signs, vehicles, pedestrians, etc.

[0064] S1032: Project the current lidar point cloud frame onto the semantic segmentation mask map through coordinate transformation.

[0065] Specifically, project the current lidar point cloud frame into the image coordinate system. Since the vehicle itself is in a moving state, there is a difference between the position where the lidar collects the lidar point cloud and the position where the camera captures the image. Therefore, first combine the lidar point cloud data at time t l , that is, the lidar point cloud data of the current lidar point cloud frame, with the pose change of the vehicle, and project it into the world coordinate system. The specific calculation is as follows:

[0066]

[0067] In the above formula, for any point (x l , y l , z l ) in the lidar point cloud data, by looking up the pose of the vehicle at time t l , according to the rotation matrix (a 3*3 matrix) and the translation matrix (a 3*1 matrix) between the lidar coordinate system and the world coordinate system, obtain the value (x w , y w , z w ) of this point in the world coordinate system;

[0068] Then look up the pose of the vehicle provided by the positioning module at time t c , and project the point in the above world coordinate system into the camera coordinate system according to the rotation matrix and the translation matrix to obtain the corresponding point (x c , y c , z c ) in the camera coordinate system. The calculation is as follows:

[0069]

[0070] The camera coordinate system is a three-dimensional coordinate system with the camera optical center as the origin. It is necessary to project the point in the camera coordinate system into the undistorted image (two-dimensional) coordinate system with the help of the camera internal parameters to obtain the corresponding pixel position (x im , y im ). The specific calculation process is as follows:

[0071]

[0072] where f x 、f y 、u0, and v0 are all internal camera parameters obtained through camera calibration;

[0073] Through the above process, the points in the current lidar point cloud frame within the camera's field of view can be projected onto the semantic segmentation mask map of the current image frame.

[0074] S1033: Filter the static points in the current lidar point cloud frame.

[0075] Traverse each point in the current lidar point cloud frame within the camera's field of view, find the corresponding semantic segmentation mask map of the image frame, obtain the category corresponding to each point, and determine whether it belongs to a static category, such as green plants, buildings, curbs, roads, poles, traffic signs, etc. Filter the points belonging to the static category as the static points of the current lidar point cloud frame.

[0076] In some embodiments, the static points of the current lidar point cloud frame are given category information and / or height information. The category information represents the category to which the static point belongs, such as green plants, buildings, curbs, roads, poles, traffic signs, etc. The height information represents the height at which the static point is located. For example, the height relative to the ground.

[0077] S104: Construct the current grid map based on the static points and determine the parameter information of each grid.

[0078] To reduce the computational complexity, a grid map can be constructed to discretize the point cloud of the current lidar point cloud frame, for example, discretize it in the world coordinate system.

[0079] Figure 4 For Figure 1 the flowchart of step S104 in Figure 3 as shown, step S103 specifically includes the following steps S1041 to S1022:

[0080] S1041: Construct the current grid map according to a predetermined grid size;

[0081] Specifically, divide the space around the vehicle body into grids according to a predetermined grid size (for example, the size of each grid is 10 cm × 10 cm), and only select the grids in the adjacent space (such as 40 meters in front, behind, left, and right of the vehicle) to construct the current grid map, and project all the static points in the current lidar point cloud frame into the current grid map.

[0082] S1042: Determine the parameter information of each grid based on the parameter information of the static points projected onto each grid and the historical parameter information of each grid.

[0083] When constructing its grid map and determining the grid parameter information, there are cases where multiple static points are projected onto the same grid; at the same time, there is an overlapping area between the currently constructed grid map and the previously constructed grid map. To enhance the robustness of the system to projection and positioning errors, this case designs a fusion strategy that fuses the information of all static points in the current grid, such as including category information and / or height information, and the historical information of the grid, such as including category information and / or height information. The specific method is as follows:

[0084] First, for the case where multiple static points (such as m static points, m is a natural number greater than or equal to 2) are projected onto the same grid when constructing the current grid map, some embodiments of the present disclosure first perform the following measures to fuse the information of the points:

[0085]

[0086] In the above formula, cls p1 , cls p2 , cls p3 ,..., cls pm are the category IDs corresponding to the static points p1, p2, p3... pm in the current grid, and mode is the mode function, that is, the category ID with the highest frequency of occurrence is selected as the category ID corresponding to the grid; if there is more than one category ID with the highest frequency of occurrence, then one of them is randomly selected as the category ID after fusion, that is

[0087] Even when the vehicle is in a moving state, there will be an overlapping area when constructing the grid map in each round, that is, there are some grids that will be updated multiple times. To enhance the robustness of the system to visual segmentation misdetection, some embodiments of the present disclosure not only fuse the information of all points in the grid when constructing the current grid map in the current round, but also fuse the historical category information of the grid, that is: cache the category information obtained from the composition in the last n rounds (n is a natural number greater than or equal to 2, for example, n = 10) for each grid Vote on it, and the calculation process is as follows:

[0088]

[0089] That is the category ID obtained by the above two fusions for the grid in the current grid map.

[0090] In some embodiments of the present disclosure, in the current upper grid map, in addition to storing the corresponding category information for each grid, the height information of the grid can also be stored to facilitate height verification when using the grid map. For example, when a pedestrian stands on the curb and several points are scanned by a lidar, forming a cluster of point clouds, it can be quickly determined whether the cluster of point clouds is a static obstacle by querying the grid map. The specific method is as follows: By querying the grid map constructed historically, it can be known that there is a static obstacle "curb" in this area. By verifying the height difference between the cluster of point clouds and the static area in the grid, it can be quickly determined that the obstacle is not the "curb".

[0091] The fusion strategy for the height of each grid is as follows:

[0092] First, for the case where multiple static points (such as m static points, where m is a natural number greater than or equal to 2) are projected onto the same grid when constructing the current grid map, some embodiments of the present disclosure first perform the following measures to fuse the information of the points:

[0093]

[0094] In the above formula, high p1 ,high p2 ,high p3 ,...,high pm are the heights corresponding to the static points p1, p2, p3... pm in the current grid, and max is the maximum value function, that is, the height of the static point with the highest height is selected as the height corresponding to the grid

[0095] Even when the vehicle is in a moving state, there will be overlapping areas when constructing the grid map in each round, that is, there are some grids that will be updated multiple times; to enhance the robustness of the system to misdetection in visual segmentation, some embodiments of the present disclosure, in addition to fusing the information of all points in the grid when constructing the current grid map in the current round, also fuse and use the historical category information of the grid, that is: cache the height information obtained from mapping in the past n rounds (n is a natural number greater than or equal to 2, for example, n = 10) for each grid Vote on it, and the calculation process is as follows:

[0096]

[0097] max is the maximum function, That is, the height information obtained by the above two fusions for the grid in the current grid map.

[0098] S105: Construct a perception map based on the grid AGE update mechanism.

[0099] Since it is difficult to judge the motion attributes of obstacles through single-frame data, some embodiments of the present disclosure design an AGE update mechanism. When constructing a perception map in each round of update, it is counted whether static point clouds are input into each grid. If a grid at a fixed position in space is hit by static points for multiple consecutive rounds, it indicates that the position has been occupied by an obstacle for a long time, and there is a static obstacle at this position; since there is a situation where a temporary static obstacle may resume motion or be displaced again, there should be a corresponding deletion mechanism for the duration of occupancy of this grid position to avoid the existence of false static obstacles; in order to simulate various situations of the occupancy duration, detect and approximately depict the life cycle of static obstacles, the present disclosure designs an AGE update mechanism.

[0100] Figure 5 For Figure 1 the flowchart of step S105 in Figure 5 as shown, in step S105, for each grid in the current grid map, the following steps S1051 to S1053 are specifically included:

[0101] S1051: Obtain the AGE value of the grid.

[0102] Specifically, when the grid exists in the previous grid map, the updated AGE value of the grid in the previous grid map is used as the AGE value before update in the previous grid map, that is, the AGE value of the grid that is not newly appeared in the current grid map is taken as the AGE value of this grid in the previous grid map; and when the grid does not exist in the previous grid map, the AGE value of the grid is set to 0, that is, for the grid that first appears in the current grid map, the initial value of the grid AGE is set to 0.

[0103] S1052: Update the AGE value of the grid based on whether the grid corresponds to the static points of the current laser point cloud frame;

[0104] Specifically, when the grid corresponds to the static points of the current laser point cloud frame, that is, when the static points of the current laser point cloud frame are projected onto the grid, the AGE value of the grid is incremented by 1; and when the grid does not correspond to the static points of the current laser point cloud frame, that is, when there are no static points of the current laser point cloud frame projected onto the grid, the AGE value of the grid is decremented by 1.

[0105] In some embodiments, when the updated AGE value is less than 0, the updated AGE value is set to 0; and when the updated AGE value is greater than a predetermined value, for example, 200, the updated AGE value is set to the predetermined value.

[0106] S1053: Cache the AGE value of the grid.

[0107] Cache the updated AGE value of the grid in the current grid map.

[0108] A plurality of grid map sequences assigned with AGE to can construct a perception map. The method for constructing the perception map provided by the embodiments of the present disclosure can be used to identify static obstacles (such as green plants, curbs, fences, etc.), and store and update them in the form of grids.

[0109] Finally, it should be noted that the embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same and similar parts among the embodiments, reference can be made to each other. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and reference can be made to the description of the method part for the relevant parts.

[0110] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for constructing a perception map, characterized in that, Including: Obtaining a sequence of laser point cloud frames and a sequence of image frames in real time; Matching the laser point cloud frames and the image frames to determine the current laser point cloud frame and its corresponding current image frame; Determining static points of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame; Constructing a current grid map based on the static points and determining parameter information of each grid; And Constructing a perception map based on a grid AGE update mechanism; The matching of the laser point cloud frames and the image frames to determine the current laser point cloud frame and its corresponding current image frame includes: Determining a first timestamp of the current laser point cloud frame; And Traversing the sequence of image frames, and when the difference between the timestamp of an image frame and the first timestamp is less than a first threshold, determining the image frame as the current image frame corresponding to the current laser point cloud frame.

2. The construction method according to claim 1, wherein, The determining of the static points of the current laser point cloud frame based on the current laser point cloud frame and its corresponding current image frame includes: Performing semantic segmentation on the current image frame to obtain a semantic segmentation mask map; Projecting the current laser point cloud frame onto the semantic segmentation mask map through coordinate transformation; and Filtering the static points of the current laser point cloud frame.

3. The construction method according to claim 2, wherein, The static points of the current laser point cloud frame are assigned class information and / or height information.

4. The construction method according to claim 1, wherein, Constructing a current grid map based on the static points and determining parameter information of each grid includes: Constructing a current grid map according to a predetermined grid size; and Determining parameter information of each grid based on parameter information of static points projected onto each grid and historical parameter information of each grid.

5. The construction method according to claim 4, wherein, The parameter information of each grid includes class information and / or height information.

6. The construction method according to claim 1, wherein, Constructing a perception map based on a grid AGE update mechanism includes performing the following operations on each grid in the current grid map: Obtaining the AGE value of the grid; Updating the AGE value of the grid based on whether the grid corresponds to a static point of the current laser point cloud frame; Caching the AGE value of the grid.

7. The construction method according to claim 6, wherein, Obtaining the AGE value of the grid includes: When the grid exists in the previous grid map, using the updated AGE value of the grid in the previous grid map as the AGE value before update in the current grid map; and When the grid does not exist in the previous grid map, setting the AGE value of the grid to 0.

8. The construction method according to claim 6, wherein, The updating of the AGE value of the grid based on whether the grid corresponds to a static point of the current laser point cloud frame includes: When the grid corresponds to a static point of the current laser point cloud frame, adding 1 to the AGE value of the grid; and When the grid does not correspond to a static point of the current laser point cloud frame, subtracting 1 from the AGE value of the grid.

9. The construction method according to claim 8, wherein, The updating of the AGE value of the grid based on whether the grid corresponds to a static point of the current laser point cloud frame further includes: When the updated AGE value is less than 0, setting the updated AGE value to 0; and When the updated AGE value is greater than a predetermined value, setting the updated AGE value to the predetermined value.

Images