Grid map correction method and device, equipment and storage medium

Abstract

The invention provides a raster map correction method and device, equipment and a storage medium, a large-range accumulative raster map is maintained, the size of the accumulative raster map is larger than that of a raster map obtained according to point cloud data, and raster information in the accumulative raster map is continuously accumulated as time goes on, so that the raster map is corrected. Therefore, the perception range of the vehicle can be expanded, holes existing in the accumulative grid map are reduced, and the grid map is corrected by combining the grid information in the accumulative grid map at the historical moment, so that the accuracy and robustness of the output perception result can be improved, and the accuracy and robustness of the vehicle perception result can be improved. Therefore, stable input can be provided for a downstream planning control module, frequent re-planning is avoided, and the working efficiency and automatic driving safety of the mine vehicle are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent mine driving technology, and in particular to a method, device, equipment and storage medium for correcting a grid map. Background Art

[0002] The haul roads in open-pit mines are usually unpaved dirt roads, and the loading and unloading areas at both ends change rapidly as the excavation progresses. In such an environment, the work site is often characterized by severe dust, irregular sites, and blurred road boundaries.

[0003] In the related technology, the perception algorithm of mine autonomous driving usually takes single-frame point cloud data as input and obstacle boxes or point cloud segmentation categories as output. However, using single-frame point cloud as the input of the perception algorithm has the problem of insufficient feature information. At the same time, due to the limitation of lidar resolution and the influence of dust occlusion, the real obstacle point cloud is often very sparse. Therefore, the perception results output by the perception algorithm will have problems such as poor accuracy, short perception distance, many holes in the output grid map, and poor robustness. The above problems will cause the downstream planning and control module to need frequent re-planning, and the vehicle's operating efficiency and autonomous driving safety are low. Summary of the invention

[0004] In view of this, the present application provides a method, device, equipment and storage medium for correcting a raster map to solve the deficiencies in the related art.

[0005] In a first aspect of the present application, a method for correcting a grid map is provided, which is applied to an electronic device installed in a vehicle, wherein the vehicle further includes a laser radar for outputting point cloud data, and the method includes:

[0006] Obtain a first grid map of a first preset size at time t, wherein the first grid map is obtained according to the point cloud data;

[0007] Constructing a second grid map of a second preset size with the position of the laser radar in the world coordinate system at time t as the center, and initializing the grid information in the second grid map using the overlapping area with the accumulated grid map of the second preset size at time t-1, the center of the accumulated grid map at time t-1 being the position of the laser radar in the world coordinate system at time t-1, and the second preset size being larger than the first preset size;

[0008] According to the position and posture of the laser radar at time t, mapping the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system;

[0009] For each grid in the first grid map at time t after mapping, the second grid map is corrected according to grid information of the grid in the first grid map at time t after mapping and the cumulative grid map before time t to obtain the cumulative grid map at time t.

[0010] According to one embodiment of the present application, obtaining a first grid map at time t includes:

[0011] Merge the point cloud data at time t with the point cloud data at N historical times, and encode the time information into each point to obtain the data to be processed at time t;

[0012] The data to be processed at time t is input into a preset perception model to obtain a first grid map at time t, wherein grid information in the first grid map at time t includes categories and height values.

[0013] According to one embodiment of the present application, merging the point cloud data at time t with the point cloud data at N historical times includes:

[0014] According to the positions and postures of the laser radar at N historical moments, the point cloud data at N historical moments are respectively converted into the world coordinate system to obtain global point cloud data;

[0015] According to the position and posture of the laser radar at time t, the global point cloud data is converted to the laser radar coordinate system at time t to obtain converted point cloud data;

[0016] The converted point cloud data is superimposed on the point cloud data at time t to obtain merged point cloud data.

[0017] According to one embodiment of the present application, the preset perception model includes a feature extraction module, a conversion module and a prediction module, wherein the feature extraction module is used to map each point in the data to be processed at time t to a pre-divided grid, and extract three-dimensional feature information from the pre-divided grid, the conversion module is used to flatten the three-dimensional feature information in the height dimension to obtain corresponding two-dimensional feature information, and the prediction module is used to output the grid information in each grid in the first grid map at time t based on the two-dimensional feature information.

[0018] According to one embodiment of the present application, the feature extraction module includes a voxelization module and a backbone network, wherein the voxelization module is used to map each point in the data to be processed at time t to the pre-divided grid, and for each grid in the pre-divided grid, perform feature aggregation on all points in the grid, and the backbone network is used to extract three-dimensional feature information from the pre-divided grid after feature aggregation.

[0019] According to one embodiment of the present application, the prediction module includes a segmentation head and a height head, the segmentation head is used to output the category of each grid in the first grid map at time t according to the two-dimensional feature information, and the height head is used to output the height value of each grid in the first grid map at time t according to the two-dimensional feature information.

[0020] According to one embodiment of the present application, the method further includes:

[0021] Determine an intersection-and-union ratio of grids classified as obstacles in the first grid map at time t after the mapping and the cumulative grid map at time t-1;

[0022] Determining whether the vehicle has lane deviation according to a comparison result of the intersection-to-merge ratio and a first preset threshold;

[0023] If the intersection-to-merge ratio is less than the first preset threshold, it is determined that the vehicle has lane deviation.

[0024] According to one embodiment of the present application, the initializing grid information in the second grid map using the overlapping area with the cumulative grid map at time t-1 includes:

[0025] Determine an overlapping area between the second grid map and the cumulative grid map at time t-1;

[0026] For each grid in the overlapping area, the grid information of the grid in the second grid map is set as the grid information of the grid in the cumulative grid map at the time t-1.

[0027] According to an embodiment of the present application, the grid information includes a category and a height value, and the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the cumulative grid map before time t, including:

[0028] Determine the probability and height value of the grid in the second grid map being an obstacle according to the preset weight values ​​at each moment and the grid information of the grid in the first grid map at moment t after the mapping and the accumulated grid map before moment t;

[0029] When the probability that the grid in the second grid map is an obstacle is less than a second preset threshold and the height value is less than a third preset threshold, the grid in the second grid map is corrected to a road.

[0030] According to an embodiment of the present application, the grid information includes a category, and after the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the accumulated grid map before time t, the method further includes:

[0031] Determine the connected area of ​​the grid classified as obstacles from the corrected second grid map;

[0032] Determining whether the size of the connected area is less than a fourth preset threshold;

[0033] If the size of the connected area is smaller than the fourth preset threshold, the category of each grid in the connected area is corrected to road.

[0034] According to an embodiment of the present application, the grid information includes a category, and after the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the accumulated grid map before time t, the method further includes:

[0035] Determine whether a preset target point is located in the corrected second grid map, the target point being the end point of the vehicle;

[0036] If the target point is located in the corrected second grid map, the category of the grid within a preset range centered on the target point in the corrected second grid map is corrected to a road.

[0037] In a second aspect of the present application, a grid map correction device is provided, which is applied to an electronic device installed in a vehicle, wherein the vehicle further includes a laser radar for outputting point cloud data, and the device includes:

[0038] An acquisition unit, configured to acquire a first grid map of a first preset size at time t, wherein the first grid map is obtained according to the point cloud data;

[0039] an initialization unit, configured to construct a second grid map of a second preset size with the position of the laser radar in the world coordinate system at time t as the center, and initialize the grid information in the second grid map using the overlapping area with the accumulated grid map of the second preset size at time t-1, wherein the center of the accumulated grid map at time t-1 is the position of the laser radar in the world coordinate system at time t-1, and the second preset size is larger than the first preset size;

[0040] A conversion unit, used for mapping the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system according to the position and posture of the laser radar at time t;

[0041] The correction unit is used to correct the second grid map for each grid in the first grid map at time t after mapping according to the grid information of the grid in the first grid map at time t after mapping and the cumulative grid map before time t, so as to obtain the cumulative grid map at time t.

[0042] In a third aspect of the present application, an electronic device is provided, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the steps of the method proposed in the above embodiment.

[0043] In a fourth aspect of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the steps of the method proposed in the above embodiment are implemented.

[0044] In a fifth aspect of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method proposed in the above embodiment when executed by a processor.

[0045] It can be seen from the above technical solution that by obtaining a first grid map of a first preset size at time t, a second grid map of a second preset size is constructed with the position of the lidar in the world coordinate system at time t as the center, and the grid information in the second grid map is initialized using the overlapping area with the cumulative grid map of the second preset size at time t-1. For each grid in the mapped first grid map at time t, the second grid map is corrected according to the grid information in the first grid map at time t after mapping and the cumulative grid map before time t, so as to obtain the cumulative grid map at time t. By maintaining a large-scale cumulative grid map, the size of which is larger than the grid map obtained based on point cloud data, and the grid information in the cumulative grid map will continue to accumulate over time, the vehicle's perception range can be expanded and the holes in the cumulative grid map can be reduced. In addition, by correcting the grid map in combination with the grid information in the cumulative grid map at historical moments, the accuracy and robustness of the output perception results can be improved, which can provide stable input for the downstream planning and control module to avoid frequent re-planning, thereby improving the operating efficiency of mining vehicles and the safety of autonomous driving.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of a method for correcting a grid map provided in an embodiment of the present application;

[0048] Figure 2 It is a framework diagram of a perception model provided in an embodiment of the present application;

[0049] Figure 3 It is a flowchart of a method for correcting a grid map provided in an embodiment of the present application;

[0050] Figure 4 It is a structural schematic diagram of a grid map correction device provided in an embodiment of the present application;

[0051] Figure 5 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0054] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0055] The haul roads in open-pit mines are usually unpaved dirt roads, and the loading and unloading areas at both ends change rapidly as the excavation progresses. In such an environment, the work site is often characterized by severe dust, irregular sites, and blurred road boundaries.

[0056] In related technologies, mine autonomous driving perception algorithms usually take single-frame point cloud data as input and obstacle boxes or point cloud segmentation categories as output.

[0057] However, dust can significantly scatter the laser beam emitted by the LiDAR, causing the LiDAR to receive a large amount of noise points. In addition, the feature information carried by a single frame of point cloud data is limited. If only a single frame of point cloud data is used as the input of the perception algorithm in a dusty environment, the perception algorithm is prone to false detection and has low accuracy. Specifically, due to the limited feature information, the perception algorithm may mistake the noise points generated by dust for obstacles, which will not only cause the downstream planning and control module to frequently trigger braking, accelerating the loss of the vehicle's braking system, but also cause driving setbacks, affecting the vehicle's operating efficiency and safety.

[0058] At the same time, due to the limitation of LiDAR resolution and the influence of dust occlusion, the point cloud data of real obstacles is often very sparse, which means that the number of information points representing obstacles in the point cloud data is small. Due to the sparse point cloud data, the perception algorithm can only reliably identify obstacles at a closer distance, which will lead to problems such as short perception distance, many holes in the output grid map, and poor robustness. The above problems will cause the downstream planning and control module to need to re-plan frequently, and the vehicle's operating efficiency and safety will be low.

[0059] In view of this, an embodiment of the present application discloses a method for correcting a grid map to address the deficiencies in the related art.

[0060] like Figure 1 As shown, Figure 1 It is a flow chart of a method for correcting a grid map provided in an embodiment of the present application. The method for correcting a grid map can be applied to electronic equipment installed in a vehicle, wherein the vehicle further includes a laser radar for outputting point cloud data.

[0061] The embodiments of the present application do not specifically limit the above-mentioned electronic device. The above-mentioned electronic device may be a device with functions such as data processing, control execution and communication, and illustratively, the above-mentioned electronic device may be an electronic control unit ECU (Electronic Control Unit), a vehicle-mounted computer, etc.

[0062] The method for correcting the grid map may include the following steps:

[0063] S101: Acquire a first grid map of a first preset size at time t, where the first grid map is obtained based on point cloud data.

[0064] The point cloud data at time t output by the vehicle laser radar can be obtained, and a first grid map of a first preset size at time t can be obtained based on the point cloud data at time t.

[0065] For example, the first grid map may be a grid map under a bird's-eye view (BEV). The BEV view can allow the system to understand the surrounding environment more clearly and intuitively, and help make more accurate decisions.

[0066] It should be noted that the embodiments of the present application do not specifically limit the viewing angle selection of the first grid map. The viewing angle of the first grid map can be flexibly selected according to actual needs. For example, the first grid map can also be a grid map under the front view (Frontal View), a customized special view, etc.

[0067] Exemplarily, the grid information in the first grid map may include categories and height values. It should be noted that the embodiments of the present application do not specifically limit the grid information in the first grid map. For example, the grid information in the first grid map may also include other information such as position offset, angle, speed, etc.

[0068] Exemplarily, the first grid map of the first preset size may be a first grid map including M×M grids.

[0069] It should be noted that the point cloud data output by the laser radar is based on the laser radar coordinate system, that is, the point cloud data output by the laser radar is point cloud data centered on the laser radar in the laser radar coordinate system. Therefore, the first grid map obtained based on the point cloud data output by the laser radar can be a grid map of the first preset size centered on the laser radar in the laser radar coordinate system.

[0070] In some embodiments, the point cloud data at time t may be merged with the point cloud data at N historical times, and the time information may be encoded into each point to obtain the data to be processed at time t;

[0071] The data to be processed at time t is input into the preset perception model to obtain the first grid map at time t.

[0072] In this embodiment, the point cloud data at time t is merged with the point cloud data at N historical moments to obtain merged point cloud data. Since the merged point cloud data contains the point cloud data at time t and N historical moments, the problem of insufficient feature information when using single-frame point cloud data as input data for the perception model can be avoided.

[0073] If the merged point cloud data is used as the input data of the perception model, the perception model also needs to consider the potential time information, otherwise the input data will have information loss, affecting the output effect of the perception model. Therefore, in this embodiment, after obtaining the merged point cloud data, the time dimension can be expanded for it, that is, the time information can be encoded into each point in the merged point cloud data, so that the merged point cloud data with time information can be used as the input data of the perception model.

[0074] For example, assuming that time t refers to the tth frame, the N historical moments are t-1 frame, t-2 frame, ..., tN frame, and the point cloud data output by the lidar from the tth frame to the tNth frame are P t , P t-1 , ..., P t-N . t For example, P t Each point in can be denoted as p t [x t ,y t ,z t ,i t ] T , where x, y, z are the coordinates of the point in the x, y, z directions in the laser radar coordinate system at time t, i (intensity) represents the intensity of the point, and the superscript T represents the transpose of the matrix.

[0075] The dimension of each point in the original point cloud data output by the laser radar is 4-dimensional, and after obtaining the merged point cloud data, the time dimension can be expanded. Encode the time information "1" into each point of the point cloud data of the tth frame in the merged point cloud data, and encode the time information "2" into each point of the point cloud data of the t-1th frame in the merged point cloud data. And so on, encode the time information "N+1" into each point of the point cloud data of the tNth frame in the merged point cloud data. Finally, each point in the data to be processed at time t has five-dimensional information.

[0076] Since time information is naturally contained between frames, in this embodiment, it is only necessary to introduce additional time information for each point in the merged point cloud data for encoding, and use this as input data for the perception model, so that the perception model can naturally learn the time information.

[0077] In this embodiment, the point cloud data at time t can be merged with the point cloud data at N historical times, and the time information can be encoded into each point to obtain the data to be processed at time t. The data to be processed at time t is input into the preset perception model to obtain the first grid map at time t, so that the input data of the perception model is converted from a single frame to a multi-frame accumulation, and the time information is encoded into each laser point, so that the perception model implicitly considers the time information, which not only avoids complex perception model design, but also improves the perception ability of the perception model in an environment with severe dust, that is, improves the accuracy of the perception results.

[0078] In some embodiments, merging the point cloud data at time t with the point cloud data at N historical times may include the following steps:

[0079] According to the positions and postures of the laser radar at N historical moments, the point cloud data at N historical moments are respectively converted to the world coordinate system to obtain the global point cloud data;

[0080] According to the position and posture of the laser radar at time t, the global point cloud data is converted to the laser radar coordinate system at time t to obtain the converted point cloud data;

[0081] The converted point cloud data is superimposed with the point cloud data at time t to obtain the merged point cloud data.

[0082] Specifically, for each of the N historical moments, the transformation matrix from the laser radar coordinate system to the world coordinate system at the historical moment can be determined according to the position and posture of the laser radar at the historical moment. The position and posture of the laser radar can include the position and posture of the laser radar in the world coordinate system. A translation matrix can be constructed according to the position information of the laser radar, and the translation matrix is ​​used to describe the position of the origin of the laser radar coordinate system relative to the origin of the world coordinate system. A rotation matrix can be constructed according to the posture information of the laser radar, and the rotation matrix is ​​used to describe the rotation of the laser radar coordinate system relative to the world coordinate system. Finally, the translation matrix and the rotation matrix are combined to obtain the transformation matrix from the laser radar coordinate system to the world coordinate system.

[0083] The point cloud data at this historical moment is converted to the world coordinate system using the conversion matrix from the laser radar coordinate system at this historical moment to the world coordinate system.

[0084] After converting the point cloud data of N historical moments into the world coordinate system, the transformation matrix from the lidar coordinate system at time t to the world coordinate system can be determined according to the position and posture of the lidar at time t, and the point cloud data of N historical moments in the world coordinate system can be converted into the lidar coordinate system at time t using the transformation matrix.

[0085] Finally, the point cloud data of N historical moments in the laser radar coordinate system at time t can be superimposed with the point cloud data at time t to obtain the merged point cloud data.

[0086] For example, assuming that time t refers to the tth frame, the N historical moments are t-1 frame, t-1 frame, ..., tN frame, and the point cloud data output by the lidar from the tth frame to the tNth frame are P t , P t-1 , ..., P t-N , the transformation matrices from the LiDAR coordinate system to the world coordinate system from the tth frame to the tNth frame are w T t , w T t-1 , ..., w T t-N .

[0087] Use formula (1) to convert the point cloud data of N historical moments into the world coordinate system to obtain the global point cloud data W P j . Formula (1) is as follows:

[0088] W P j = W T j ×P j (j=tN,...,t-1) (1)

[0089] in, W P j Represents the position of the point cloud data of the jth frame in the world coordinate system, j = t-1, t-2, ..., tN.

[0090] Use formula (2) to transform the global point cloud data into the laser radar coordinate system at time t, and obtain the transformed point cloud data t P j . Formula (2) is as follows:

[0091] t P j =( W T t ) -1 × W P j (j=tN,...,t-1) (2)

[0092] in, t P j Represents the position of the point cloud data of the jth frame in the lidar coordinate system at time t, j = t-1, t-2, ..., tN.

[0093] Convert point cloud datat P j and the point cloud data P at time t t The combined point cloud data P is obtained by stacking them up in columns, where P = {P t , t P t-1 , t P t-2 ,..., t P t-N}.

[0094] S102: Construct a second grid map of a second preset size with the position of the laser radar in the world coordinate system at time t as the center, and initialize the grid information in the second grid map using the overlapping area with the cumulative grid map of the second preset size at time t-1, the center of the cumulative grid map at time t-1 is the position of the laser radar in the world coordinate system at time t-1, and the second preset size is larger than the first preset size.

[0095] At time t, a second grid map of a second preset size is constructed with the position of the laser radar in the world coordinate system at time t as the center. At this time, the grid information of each grid in the second grid map has not been determined, that is, the grid information is unknown or empty. The grid information of each grid in the overlapping area of ​​the second grid map can be initialized by using the overlapping area between the second grid map and the accumulated grid map of the second preset size with the position of the laser radar in the world coordinate system at time t-1 as the center.

[0096] In some embodiments, the overlapping area between the second grid map and the cumulative grid map at time t-1 can be determined, and the grid information of each grid in the overlapping area in the cumulative grid map at time t-1 can be copied to the corresponding grid of the second grid map, thereby initializing the grid information of each grid in the overlapping area in the second grid map.

[0097] It should be noted that the present application does not specifically limit the method for determining the overlapping area between the second grid map and the cumulative grid map at time t-1. For example, a spatial index, such as an octree, a KD tree, etc., can be used to store the grid maps, and the overlapping area between the grid maps can be determined by querying the spatial index, or the boundaries of the grid maps can be determined first, and then the intersection of the boundaries of the grid maps can be calculated to determine the overlapping area between the grid maps, etc.

[0098] Exemplarily, the second grid map of the second preset size may be a second grid map including K×K grids, where K>M.

[0099] Specifically, the cumulative grid map of the 0th frame can be preset. The cumulative grid map of the 0th frame can be a grid map containing K×K grids centered on the initial position of the lidar in the world coordinate system. The grid information of each grid in the cumulative grid map of the 0th frame can be preset or empty.

[0100] In the first frame, a second grid map containing K×K grids is constructed with the position of the lidar in the world coordinate system in the first frame as the center. The cumulative grid map of the 0th frame is obtained, the overlapping area between the second grid map of the 1st frame and the cumulative grid map of the 0th frame is determined, and the grid information of each grid in the overlapping area in the cumulative grid map of the 0th frame is copied to the corresponding grid of the second grid map of the 1st frame.

[0101] According to S103 and S104, the second grid map of the first frame is corrected to obtain the cumulative grid map of the first frame. The specific steps of S103 and S104 will be described in detail below and will not be repeated here.

[0102] In the second frame, a second grid map containing K×K grids is constructed with the position of the second frame lidar in the world coordinate system as the center. The previously obtained cumulative grid map of the first frame is obtained, the overlapping area between the second grid map of the second frame and the cumulative grid map of the first frame is determined, and the grid information of each grid in the overlapping area in the cumulative grid map of the first frame is copied to the corresponding grid of the second grid map of the second frame.

[0103] According to S103 and S104, the second grid map of the second frame is corrected to obtain the accumulated grid map of the second frame.

[0104] Similarly, at time t, a second grid map containing K×K grids is constructed with the position of the lidar in the world coordinate system at time t as the center. The previously obtained cumulative grid map at time t-1 is obtained, the overlapping area between the second grid map at time t and the cumulative grid map at time t-1 is determined, and the grid information of each grid in the overlapping area in the cumulative grid map at time t-1 is copied to the corresponding grid of the second grid map at time t, thereby initializing the grid information of each grid in the overlapping area in the second grid map at time t.

[0105] In an embodiment of the present application, by copying the grid information of each grid in the overlapping area of ​​the cumulative grid map at time t-1 to the corresponding grid of the second grid map at time t, the grid information in the cumulative grid map at the historical moment can be used in the second grid map constructed at the current moment, so that the grid information in the cumulative grid map will continue to accumulate over time, thereby expanding the perception range of the vehicle and reducing the holes in the cumulative grid map.

[0106] S103: Mapping the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system according to the position and posture of the laser radar at time t.

[0107] The position and posture of the laser radar at time t may include the position and posture of the laser radar in the world coordinate system at time t. A translation matrix can be constructed according to the position information of the laser radar at time t, and a rotation matrix can be constructed according to the posture information of the laser radar at time t. The translation matrix and the rotation matrix are combined to obtain a transformation matrix from the laser radar coordinate system at time t to the world coordinate system. The transformation matrix is ​​used to map the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system.

[0108] It should be noted that if the first grid map at time t is a grid map of the first preset size centered on the laser radar in the laser radar coordinate system, then the mapped first grid map at time t is a grid map of the first preset size centered on the position of the laser radar in the world coordinate system at time t.

[0109] S104: For each grid in the first grid map at time t after mapping, the second grid map is corrected according to grid information of the grid in the first grid map at time t after mapping and the cumulative grid map before time t to obtain the cumulative grid map at time t.

[0110] In some embodiments, the grid information includes a category and a height value, and for each grid in the first grid map at time t after mapping, the probability and height value of the category of the grid in the second grid map being an obstacle can be determined based on the preset weight value at each time, and the grid information in the first grid map at time t after mapping and the accumulated grid map before time t;

[0111] When the probability that the grid in the second grid map is an obstacle is less than the second preset threshold and the height value is less than the third preset threshold, the grid in the second grid map is set to be a road.

[0112] Otherwise, the category of the grid in the second grid map is set to an obstacle.

[0113] It should be noted that the first grid map at time t after mapping is a grid map of a first preset size centered on the position of the laser radar in the world coordinate system at time t, and the center of the second grid map at time t is also the position of the laser radar in the world coordinate system at time t. Therefore, for each grid in the first grid map at time t after mapping, the grid is also in the second grid map at time t. Therefore, the probability and height value of the category of the grid in the second grid map being an obstacle can be determined based on the preset weight values ​​at each time, as well as the grid information in the first grid map at time t after mapping and the accumulated grid map before time t.

[0114] Specifically, for each grid in the first grid map at time t after mapping, the probability and height value of the grid in the second grid map at time t can be calculated respectively using the following formulas (3) and (4):

[0115] P new =α×P current +(1-α)×P previ ous (3)

[0116] H new =α×H current +(1-α)×H previ ous (4)

[0117] Among them, α is the weight value, and the weight value α corresponding to each moment can be pre-set. For example, the weight value α corresponding to the first 5 frames can be set to a larger value (such as 0.3), and the weight value α corresponding to the 5th to 50th frames can be linearly reduced to a smaller value (such as 0.01). The corresponding weight value α after the 50th frame remains unchanged at a smaller value (such as 0.01).

[0118] P new is the probability that the grid in the second grid map at time t is an obstacle, P current is the probability that the category of the grid in the first grid map at time t after mapping is an obstacle, P previous is the probability that the category of the grid in the second grid map at time t-1 is an obstacle.

[0119] H new is the height value of the grid in the second grid map at time t, H current is the height value of the grid in the first grid map at time t after mapping, H previous is the height value of the grid in the second grid map at time t-1.

[0120] Assume that the second preset threshold is 0.5, the third preset threshold is 0.5, and the P value corresponding to the 0th frame is preset.new is 0, the H corresponding to the 0th frame new is 0.2.

[0121] The category of grid A in the first grid map of the first frame after mapping is road, and the height value is 0.2. The category of grid A in the first grid map of the second frame after mapping is obstacle, and the height value is 0.6. The category of grid A in the first grid map of the third frame after mapping is obstacle, and the height value is 0.6.

[0122] In the first frame, the weight value α is 0.3, and P in formula (3) previous is the P corresponding to the 0th frame new , at this time P previous is 0, P current is the probability that the category of grid A in the first grid map of the first frame after mapping is an obstacle. At this time, P current =0, therefore, according to formula (3), we can determine the P corresponding to the first frame. new is equal to 0.

[0123] Similarly, according to formula (4), the H corresponding to the first frame can be determined new Equal to 0.2.

[0124] Since the P corresponding to the first frame new is less than the second preset threshold, and the H corresponding to the first frame new is smaller than the third preset threshold, so the category of grid A in the second grid map of the first frame is still road.

[0125] In the second frame, the weight value α is 0.3, and P in formula (3) previous is the P corresponding to the first frame new , that is, P previous =0, P current is the probability that the category of grid A in the first grid map of the second frame after mapping is an obstacle. At this time, P current =1, therefore, according to formula (3), we can determine the P corresponding to the second frame. new Equal to 0.3.

[0126] Similarly, according to formula (4), the H corresponding to the second frame can be determined new Equal to 0.48.

[0127] Although the category of grid A in the first grid map of the second frame after mapping is an obstacle, due to the P corresponding to the second frame new is less than the second preset threshold, and the H corresponding to the second frame new is smaller than the third preset threshold, so the category of grid A in the second grid map of the second frame needs to be set to road.

[0128] In the third frame, the weight value α is 0.3, and P in formula (3) previous is the P corresponding to the second frame new , that is, P previous =0.3, P current is the probability that the category of grid A in the first grid map of the third frame after mapping is an obstacle. At this time, P current =1, therefore, according to formula (3), we can determine the P corresponding to the third frame. new Equal to 0.51.

[0129] Similarly, according to formula (4), the H corresponding to the third frame can be determined new Equal to 0.516.

[0130] Since the P corresponding to the third frame new is greater than the second preset threshold, the H corresponding to the second frame new is greater than the third preset threshold, so the category of grid A in the second grid map of the third frame is still an obstacle.

[0131] At time t, and so on.

[0132] It should be noted that the present application does not specifically limit the method for correcting the second grid map based on the grid information in the first grid map at time t after the mapping and the cumulative grid map before time t. For example, the grid information in the grid map can be corrected by using filtering techniques such as Kalman filtering, particle filtering or extended Kalman filtering, combined with current and historical grid information, or the grid information in the grid map can be predicted and corrected by using a deep learning model, etc.

[0133] In an embodiment of the present application, by acquiring a first grid map of a first preset size at time t, a second grid map of a second preset size is constructed with the position of the laser radar in the world coordinate system at time t as the center, and the grid information in the second grid map is initialized using the overlapping area with the cumulative grid map of the second preset size at time t-1. For each grid in the mapped first grid map at time t, the second grid map is corrected according to the grid information in the first grid map at time t after mapping and the cumulative grid map before time t, so as to obtain the cumulative grid map at time t. By maintaining a large-scale cumulative grid map, the size of which is larger than the grid map obtained based on point cloud data, and the grid information in the cumulative grid map will continue to accumulate over time, the vehicle's perception range can be expanded and the holes in the cumulative grid map can be reduced. In addition, by correcting the grid map in combination with the grid information in the cumulative grid map at historical moments, the accuracy and robustness of the output perception results can be improved, which can provide stable input for the downstream planning and control module to avoid frequent re-planning, thereby improving the operating efficiency of mining vehicles and the safety of autonomous driving.

[0134] In some embodiments, the point cloud data at time t can be merged with the point cloud data at N historical times, and the time information can be encoded into each point to obtain the data to be processed at time t, and the data to be processed at time t can be input into a preset perception model to obtain the first grid map at time t.

[0135] like Figure 2 As shown, Figure 2 It is a framework diagram of a perception model provided in an embodiment of the present application. The perception model may include a feature extraction module 21, a conversion module 22 and a prediction module 23.

[0136] The feature extraction module 21 is used to map each point in the data to be processed at time t to a pre-divided grid, and extract three-dimensional feature information from the pre-divided grid.

[0137] Specifically, the feature extraction module 21 may include a voxelization module 211 and a backbone network 212 .

[0138] The voxelization module 211 may be used for:

[0139] (1) Spatial division

[0140] A voxel is a volume element in a three-dimensional space and can be considered as a small cube in the 3D space, similar to a pixel in a two-dimensional image. The voxelization module 211 can divide the three-dimensional space into a series of voxels of the same size, each voxel representing a fixed three-dimensional area. After the three-dimensional space is voxelized, a voxel can be considered as a three-dimensional grid.

[0141] (2) Point cloud mapping

[0142] Map each point in the data to be processed at time t to the pre-divided grid obtained in step (1). A grid may contain one or more points, or may have no points.

[0143] (3) Feature aggregation

[0144] For each grid in the pre-divided grid, feature aggregation is performed on all points in the grid.

[0145] Specifically, for multiple points falling in the same grid, the features of the grid can be calculated by averaging. These features can contain geometric information (such as point density, location, etc.) and other attributes (such as color, reflection intensity, etc.).

[0146] For example, suppose there are two points in a grid:

[0147] Point 1, x:100, y:100, z:-1, intensity:200

[0148] Point 2, x:20, y:30, z:1, intensity:100

[0149] By taking the average value, we can get the characteristics of the grid as: x:60, y:65, z:0, intensity:150.

[0150] (4) Sparse Representation

[0151] Since most of the grid may be empty, a sparse matrix or hash table can be used to efficiently store the voxelization results to avoid storing and calculating empty grids.

[0152] The backbone network 212 may be used to extract three-dimensional feature information from the pre-divided grid after feature aggregation.

[0153] The backbone network 212 is mainly composed of sparse convolution layers, which are responsible for extracting three-dimensional feature information from the grid processed by the voxelization module 211. Since there may be a large number of empty grids in the grid processed by the voxelization module 211, and the sparse convolution only operates on non-empty grids, the efficiency of processing sparse data can be significantly improved. In sparse convolution, the convolution kernel slides on the valid position of the input data. During the sliding process, each convolution kernel only performs operations on the non-zero or valid area of ​​the data. The sparse convolution kernel calculates the convolution result based on the neighborhood of these valid points.

[0154] The conversion module 22 is used to flatten the three-dimensional feature information output by the feature extraction module 21 in the height dimension to obtain corresponding two-dimensional feature information.

[0155] The dimension of the three-dimensional feature information output by the feature extraction module 21 is (N, C, D, H, W). The conversion module 22 can flatten the above three-dimensional feature information in the height dimension. Flattening refers to the process of converting data from a multi-dimensional structure to a one-dimensional structure. The dimension after flattening in the height dimension is (N, C*D, H, W).

[0156] Among them, N is batch_size, which means the number of data samples processed at a time, C is channel, which means the number of channels of the feature, and H / W / D are the length, width, and height of the three-dimensional space respectively. C*D means multiplying the original number of channels C and the depth D to get the new number of channels, which means that the features in each height dimension are flattened into a separate channel.

[0157] The prediction module 23 is used to output the grid information of each grid in the first grid map at time t according to the two-dimensional feature information output by the conversion module 22. Exemplarily, the grid information may include the category and height value of the grid.

[0158] Specifically, the prediction module 23 may include a segmentation head 231 and a height head 232 .

[0159] Among them, the segmentation head 231 can be used to output the category of each grid in the first grid map at time t according to the two-dimensional feature information, and the height head 232 can be used to output the height value of each grid in the first grid map at time t according to the two-dimensional feature information.

[0160] For the segmentation head 231 and the height head 232, the same network structure can be used, and different activation functions can be added at the output end.

[0161] For the segmentation head 231, the following activation function may be used:

[0162]

[0163] Among them, z iis the i-th component of the input vector. j Represents the input vector except z i All other elements except

[0164] For the height head 232, the following activation function can be used:

[0165] Softplus(x)=log(1+e x ) (6)

[0166] For the training of the preset perception model, the cross entropy loss function can be used to calculate the overall cost for the output of the segmentation head 231, and the formula is as follows:

[0167]

[0168] Among them, y i is the true value label, is the predicted value.

[0169] For the output of the height head 232, the L1 loss function can be used to calculate the overall cost, and the formula is as follows:

[0170]

[0171] Among them, y i is the true value label, is the predicted value.

[0172] In some embodiments, the intersection-and-merge ratio of the grids classified as obstacles in the first grid map at time t after mapping and the cumulative grid map at time t-1 can be determined, and based on the comparison result of the above-mentioned intersection-and-merge ratio and the first preset threshold, it is judged whether the vehicle has lane deviation. If the above-mentioned intersection-and-merge ratio is less than the first preset threshold, it is determined that the vehicle has lane deviation; otherwise, it is determined that the vehicle has not lane deviation.

[0173] Specifically, the set S of grids classified as obstacles in the first grid map at time t after mapping can be extracted: current , extract the set S of grids with obstacle category in the cumulative grid map at time t-1 accumulated , calculate S current With S accumulated The intersection area S intersection , where S intersection =S current ∩S accumulated , calculate S current With S accumulated The union area S of union , where S union =S current ∪S accumulatedFinally, the overlap IOU (Intersection over Union) index is calculated using formula (9), which is as follows:

[0174]

[0175] In this embodiment, the IOU index can be calculated to determine whether the vehicle has lane deviation. If it is determined that the vehicle has lane deviation, a deviation warning can be sent to the downstream planning control module to enable the downstream planning control module to replan the route, thereby improving the safety factor of mine autonomous driving and reducing the risk of accidents.

[0176] In some embodiments, after the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the cumulative grid map before time t, the connected area of ​​the grids classified as obstacles can be determined from the corrected second grid map, and it is judged whether the size of the connected area is less than a fourth preset threshold. If the size of the connected area is less than the fourth preset threshold, the category of each grid in the connected area is corrected to a road, thereby obtaining the cumulative grid map at time t.

[0177] Specifically, a preset algorithm, such as a depth-first search DFS (Depth-First-Search) algorithm, a breadth-first search BFS (Breadth-First Search) algorithm, etc., can be used to determine the connected area of ​​the grid classified as an obstacle from the corrected second grid map.

[0178] For each connected area found, the bounding rectangle of the connected area is calculated. The bounding rectangle is the smallest rectangle that can completely surround the connected area.

[0179] It is determined whether the diagonal length of the circumscribed rectangle corresponding to each connected area is less than a fourth preset threshold. If less than, the connected area is considered to be a small area obstacle, which can be filtered, and therefore, the category of each grid in the connected area can be corrected to a road.

[0180] In this embodiment, the grid connected area whose size is less than the fourth preset threshold and is classified as an obstacle can be filtered, and the category of each grid in the connected area can be corrected to a road, thereby reducing interference and false alarms caused by noise, thereby improving the accuracy and robustness of the output perception results, and further providing a stable input for the downstream planning and control module to avoid frequent re-planning, thereby improving the operating efficiency of mining vehicles and the safety of autonomous driving.

[0181] In some embodiments, after the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the cumulative grid map before time t, it can also be determined whether a preset target point is located in the corrected second grid map, and the target point is the end point of the vehicle. If the target point is located in the corrected second grid map, the category of the grid within a preset range centered on the target point in the corrected second grid map is corrected to a road, thereby obtaining the cumulative grid map at time t.

[0182] Specifically, it is possible to detect in real time whether a target point (such as a vehicle's pickup / dumping point in a mining environment) is located in a corrected second grid map. If it is detected that the target point is located in the corrected second grid map, the category of the C×C grids centered on the target point in the corrected second grid map is corrected to a road.

[0183] In this embodiment, if the target point is detected to be located in the corrected second grid map, the category of the grid within a preset range centered on the target point in the corrected second grid map will be corrected to a road, thereby ensuring that the downstream planning control module can include the target point in the planned route when performing route planning when inaccurate perception results occur at long-distance points.

[0184] like Figure 3 As shown, Figure 3 It is a flow chart of a method for correcting a grid map provided in an embodiment of the present application. The method for correcting a grid map can be applied to electronic equipment installed in a vehicle, wherein the vehicle further includes a laser radar for outputting point cloud data.

[0185] S301: Obtaining point cloud data at time t

[0186] Get the point cloud data output by the LiDAR at time t, and get the position and posture of the LiDAR in the world coordinate system at time t.

[0187] S302: Merge the point cloud data at time t with the point cloud data at N historical times

[0188] Obtain the point cloud data of N historical moments and the positions of the laser radars in the world coordinate system at these N historical moments. According to the positions of the laser radars in the world coordinate system at N historical moments, determine the transformation matrices from the laser radar coordinate system to the world coordinate system at N historical moments. Use these transformation matrices to transform the point cloud data of N historical moments to the world coordinate system.

[0189] According to the position of the laser radar in the world coordinate system at time t, the transformation matrix from the laser radar coordinate system at time t to the world coordinate system is determined. According to the transformation matrix, the point cloud data of N historical moments in the world coordinate system are transformed to the laser radar coordinate system at time t.

[0190] The point cloud data of N historical moments in the laser radar coordinate system at time t are superimposed with the point cloud data at time t to obtain the merged point cloud data.

[0191] In this embodiment, the point cloud data at time t is merged with the point cloud data at N historical moments. Since the merged point cloud data contains the point cloud data at time t and N historical moments, the problem of insufficient feature information when using single-frame point cloud data as input data for the perception model can be avoided.

[0192] S303: Expand the time dimension of the merged point cloud data to obtain the data to be processed at time t

[0193] The dimension of each point in the original point cloud data is 4-dimensional, including x t ,y t , z t ,i t Since the merged point cloud data contains the accumulated point cloud data of N historical moments, the time information needs to be encoded into each point of the merged point cloud data. The time encoding of the point cloud data at time t in the merged point cloud data is "1", the time encoding of the point cloud data at time t-1 is "2", and so on. The time encoding of the point cloud data at time tN is "N+1". After dimensional expansion, each point has five-dimensional information. The expanded point cloud data, that is, the data to be processed at time t, will be used as the input data of the preset perception model.

[0194] In this embodiment, the time dimension of the merged point cloud data is expanded to obtain the data to be processed at time t, and the data to be processed at time t is used as the input data of the perception model, so that the perception model implicitly considers the time information, which not only avoids the complex perception model design, but also improves the perception ability of the perception model in an environment with severe dust, that is, improves the accuracy of the perception results.

[0195] S304: Input the data to be processed at time t into a preset perception model to obtain the first grid map at time t

[0196] The data to be processed at time t is used as the input data of the preset perception model, and its output is the first grid map at time t containing M×M grids, and the grid information in the first grid map at time t includes the category and height value of the grid. Among them, the category of the grid in the first grid map is output by the segmentation head in the preset perception model, and the height value of the grid is output by the height head in the preset perception model.

[0197] S310: Calculate the IOU index to determine whether the lane is offset

[0198] Get the cumulative grid map containing K×K grids at time t-1, and use the first grid map at time t and the cumulative grid map at time t-1 to calculate the IOU index. Only the number of obstacles (such as earth walls) in the first grid map at time t and the cumulative grid map at time t-1 are considered during the calculation.

[0199] Extract the earth wall grid set in the first grid map at time t, extract the earth wall grid set in the cumulative grid map at time t-1, calculate the intersection area and union area between the two earth wall grid sets, and calculate the IOU index based on the intersection area and union area. Compare the calculated IOU with the first preset threshold to determine whether the vehicle has lane deviation.

[0200] If the IOU is lower than the first preset threshold, it is considered that the vehicle has lane deviation. In this case, a deviation warning can be sent to the downstream planning control module to enable the downstream planning control module to replan the route.

[0201] If the IOU is not lower than the first preset threshold, it is considered that the vehicle has not deviated from the lane.

[0202] In this embodiment, the IOU index can be calculated to determine whether the vehicle has lane deviation. If it is determined that the vehicle has lane deviation, a deviation warning can be sent to the downstream planning control module to enable the downstream planning control module to replan the route, thereby improving the safety factor of mine autonomous driving and reducing the risk of accidents.

[0203] S305: Construct the second grid map and initialize it

[0204] A second grid map containing K×K grids is constructed with the position of the laser radar in the world coordinate system at time t as the center, where K>M. At this time, the grid information of each grid in the second grid map just constructed has not been determined, that is, the grid information is unknown or empty.

[0205] The cumulative grid map containing K×K grids at time t-1 is obtained, the overlapping area between the second grid map at time t and the cumulative grid map at time t-1 is determined, and the grid information of each grid in the overlapping area in the cumulative grid map at time t-1 is copied to the corresponding grid of the second grid map at time t, thereby initializing the grid information of each grid in the overlapping area in the second grid map.

[0206] In this embodiment, by constructing and initializing the second grid map, that is, copying the grid information of each grid in the overlapping area of ​​the cumulative grid map at time t-1 to the corresponding grid of the second grid map at time t, the grid information in the cumulative grid map at the historical moment can be used in the second grid map constructed at the current moment, so that the grid information in the cumulative grid map will continue to accumulate over time, thereby expanding the perception range of the vehicle and reducing the holes in the cumulative grid map.

[0207] S306: Coordinate system conversion

[0208] According to the position and posture of the laser radar in the world coordinate system at time t, a transformation matrix from the laser radar coordinate system at time t to the world coordinate system is determined. The first grid map is mapped from the laser radar coordinate system at time t to the world coordinate system using the transformation matrix.

[0209] S307: Correct the category of the second grid map

[0210] For each grid in the first grid map at time t after mapping, the probability and height value of the grid category in the second grid map being an obstacle can be determined based on the preset weight values ​​at each time, as well as the grid information in the first grid map at time t after mapping and the cumulative grid map before time t.

[0211] If the probability that the grid in the second grid map is an obstacle is less than the second preset threshold and the height value is less than the third preset threshold, the grid in the second grid map is set to be a road. Otherwise, the grid in the second grid map is set to be an obstacle.

[0212] In this embodiment, the grid map is corrected by combining the grid information in the cumulative grid map at historical moments, so as to improve the accuracy and robustness of the output perception results, thereby providing a stable input for the downstream planning and control module to avoid frequent re-planning, thereby improving the operating efficiency of mining vehicles and the safety of autonomous driving.

[0213] S308: Filter small obstacles

[0214] The DFS algorithm is used to determine the connected area of ​​the grids classified as obstacles from the second grid map processed by S307. For each connected area found, the bounding rectangle of the connected area is calculated. It is determined whether the diagonal length of the bounding rectangle corresponding to each connected area is less than the fourth preset threshold. If less than, the connected area is considered to be a small area obstacle, and the small area obstacles can be filtered. Therefore, the category of each grid in the connected area can be corrected to a road.

[0215] In this embodiment, the grid connected area whose size is less than the fourth preset threshold and is classified as an obstacle can be filtered, and the category of each grid in the connected area can be corrected to a road, thereby reducing interference and false alarms caused by noise, thereby improving the accuracy and robustness of the output perception results, and further providing a stable input for the downstream planning and control module to avoid frequent re-planning, thereby improving the operating efficiency of mining vehicles and the safety of autonomous driving.

[0216] S309: Obstacle removal around the receiving / discharging point

[0217] Real-time detection is performed to determine whether the vehicle's pickup / dumping point is located in the second grid map processed by S308. If it is detected that the vehicle's pickup / dumping point is located in the second grid map, the categories of the C×C grids centered on the vehicle's pickup / dumping point in the second grid map are corrected to roads.

[0218] In this embodiment, if the target point is detected to be located in the corrected second grid map, the category of the grid within a preset range centered on the target point in the corrected second grid map will be corrected to a road, thereby ensuring that the downstream planning control module can include the target point in the planned route when performing route planning when inaccurate perception results occur at long-distance points.

[0219] Finally, the cumulative grid map at time t is obtained.

[0220] The above content describes the method provided by the present application. The following describes the device provided by the present application:

[0221] See also Figure 4 , is a structural schematic diagram of a grid map correction device provided in an embodiment of the present application.

[0222] like Figure 4 As shown, the device may include:

[0223] An acquisition unit 410 is used to acquire a first grid map of a first preset size at time t, where the first grid map is obtained based on point cloud data;

[0224] an initialization unit 420, configured to construct a second grid map of a second preset size with the position of the laser radar in the world coordinate system at time t as the center, and initialize grid information in the second grid map using an overlapping area with the accumulated grid map of the second preset size at time t-1, wherein the center of the accumulated grid map at time t-1 is the position of the laser radar in the world coordinate system at time t-1, and the second preset size is greater than the first preset size;

[0225] A conversion unit 430 is used to map the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system according to the position and posture of the laser radar at time t;

[0226] The correction unit 440 is used to correct the second grid map for each grid in the first grid map at time t after mapping according to the grid information of the grid in the first grid map at time t after mapping and the cumulative grid map before time t, so as to obtain the cumulative grid map at time t.

[0227] In some embodiments, the acquisition unit 410 is specifically configured to:

[0228] Merge the point cloud data at time t with the point cloud data at N historical times, and encode the time information into each point to obtain the data to be processed at time t;

[0229] The data to be processed at time t is input into a preset perception model to obtain a first grid map at time t, wherein grid information in the first grid map at time t includes categories and height values.

[0230] In some embodiments, the acquisition unit 410 is specifically configured to:

[0231] According to the positions and postures of the laser radar at N historical moments, the point cloud data at N historical moments are respectively converted into the world coordinate system to obtain global point cloud data;

[0232] According to the position and posture of the laser radar at time t, the global point cloud data is converted to the laser radar coordinate system at time t to obtain converted point cloud data;

[0233] The converted point cloud data is superimposed on the point cloud data at time t to obtain merged point cloud data.

[0234] In some embodiments, the preset perception model includes a feature extraction module, a conversion module and a prediction module, wherein the feature extraction module is used to map each point in the data to be processed at time t to a pre-divided grid, and extract three-dimensional feature information from the pre-divided grid, the conversion module is used to flatten the three-dimensional feature information in the height dimension to obtain corresponding two-dimensional feature information, and the prediction module is used to output the grid information in each grid in the first grid map at time t based on the two-dimensional feature information.

[0235] In some embodiments, the feature extraction module includes a voxelization module and a backbone network, wherein the voxelization module is used to map each point in the data to be processed at time t to the pre-divided grid, and for each grid in the pre-divided grid, perform feature aggregation on all points in the grid, and the backbone network is used to extract three-dimensional feature information from the pre-divided grid after feature aggregation.

[0236] In some embodiments, the prediction module includes a segmentation head and a height head, the segmentation head is used to output the category of each grid in the first grid map at time t according to the two-dimensional feature information, and the height head is used to output the height value of each grid in the first grid map at time t according to the two-dimensional feature information.

[0237] In some embodiments, the device further includes a judgment unit, and the judgment unit is specifically configured to:

[0238] Determine an intersection-and-union ratio of grids classified as obstacles in the first grid map at time t after the mapping and the cumulative grid map at time t-1;

[0239] Determining whether the vehicle has lane deviation according to a comparison result of the intersection-to-merge ratio and a first preset threshold;

[0240] If the intersection-to-merge ratio is less than the first preset threshold, it is determined that the vehicle has lane deviation.

[0241] In some embodiments, the initialization unit 420 is specifically configured to:

[0242] Determine an overlapping area between the second grid map and the cumulative grid map at time t-1;

[0243] The grid information of each grid in the overlapping area in the cumulative grid map at time t-1 is copied to the corresponding grid in the second grid map.

[0244] In some embodiments, the correction unit 440 is specifically configured to:

[0245] Determine the probability and height value of the grid in the second grid map being an obstacle according to the preset weight values ​​at each moment and the grid information of the grid in the first grid map at moment t after the mapping and the accumulated grid map before moment t;

[0246] When the probability that the grid in the second grid map is an obstacle is less than a second preset threshold and the height value is less than a third preset threshold, the grid in the second grid map is set to be a road.

[0247] In some embodiments, the correction unit 440 is specifically configured to:

[0248] Determine the connected area of ​​the grid classified as obstacles from the corrected second grid map;

[0249] Determining whether the size of the connected area is less than a fourth preset threshold;

[0250] If the size of the connected area is smaller than the fourth preset threshold, the category of each grid in the connected area is corrected to road.

[0251] In some embodiments, the correction unit 440 is specifically configured to:

[0252] Determine whether a preset target point is located in the corrected second grid map, the target point being the end point of the vehicle;

[0253] If the target point is located in the corrected second grid map, the category of the grid within a preset range centered on the target point in the corrected second grid map is corrected to a road.

[0254] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0255] The present application embodiment also provides a hardware structure. Figure 5 , Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0256] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0257] Exemplarily, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0258] It should be noted that, in this article, relational terms such as target and target are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0259] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for correcting a grid map, applied to an electronic device installed in a vehicle, wherein the vehicle further comprises a laser radar for outputting point cloud data, characterized in that: The method includes: Obtain a first grid map of a first preset size at time t, wherein the first grid map is obtained according to the point cloud data; Constructing a second grid map of a second preset size with the position of the laser radar in the world coordinate system at time t as the center, and initializing the grid information in the second grid map using the overlapping area with the accumulated grid map of the second preset size at time t-1, the center of the accumulated grid map at time t-1 being the position of the laser radar in the world coordinate system at time t-1, and the second preset size being larger than the first preset size; According to the position and posture of the laser radar at time t, mapping the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system; For each grid in the first grid map at time t after mapping, the second grid map is corrected according to grid information of the grid in the first grid map at time t after mapping and the cumulative grid map before time t to obtain the cumulative grid map at time t.

2. The method according to claim 1, characterized in that The obtaining of the first grid map at time t includes: Merge the point cloud data at time t with the point cloud data at N historical times, and encode the time information into each point to obtain the data to be processed at time t; The data to be processed at time t is input into a preset perception model to obtain a first grid map at time t, wherein grid information in the first grid map at time t includes categories and height values.

3. The method according to claim 2, characterized in that The step of merging the point cloud data at time t with the point cloud data at N historical times includes: According to the positions and postures of the laser radar at N historical moments, the point cloud data at N historical moments are respectively converted into the world coordinate system to obtain global point cloud data; According to the position and posture of the laser radar at time t, the global point cloud data is converted to the laser radar coordinate system at time t to obtain converted point cloud data; The converted point cloud data is superimposed on the point cloud data at time t to obtain merged point cloud data.

4. The method according to claim 2, characterized in that: The preset perception model includes a feature extraction module, a conversion module and a prediction module, wherein the feature extraction module is used to map each point in the data to be processed at time t to a pre-divided grid, and extract three-dimensional feature information from the pre-divided grid, the conversion module is used to flatten the three-dimensional feature information in the height dimension to obtain corresponding two-dimensional feature information, and the prediction module is used to output the grid information in each grid in the first grid map at time t according to the two-dimensional feature information.

5. The method according to claim 4, characterized in that The feature extraction module includes a voxelization module and a backbone network, wherein the voxelization module is used to map each point in the data to be processed at time t to the pre-divided grid, and for each grid in the pre-divided grid, perform feature aggregation on all points in the grid, and the backbone network is used to extract three-dimensional feature information from the pre-divided grid after feature aggregation.

6. The method according to claim 4, characterized in that The prediction module includes a segmentation head and a height head. The segmentation head is used to output the category of each grid in the first grid map at time t according to the two-dimensional feature information, and the height head is used to output the height value of each grid in the first grid map at time t according to the two-dimensional feature information.

7. The method according to claim 1, characterized in that The method further comprises: Determine an intersection-and-union ratio of grids classified as obstacles in the first grid map at time t after the mapping and the cumulative grid map at time t-1; Determining whether the vehicle has lane deviation according to a comparison result of the intersection-to-merge ratio and a first preset threshold; If the intersection-to-merge ratio is less than the first preset threshold, it is determined that the vehicle has lane deviation.

8. The method according to claim 1, characterized in that: The step of initializing the grid information in the second grid map by using the overlapping area with the cumulative grid map at time t-1 includes: Determine an overlapping area between the second grid map and the cumulative grid map at time t-1; The grid information of each grid in the overlapping area in the cumulative grid map at time t-1 is copied to the corresponding grid in the second grid map.

9. The method according to claim 1, characterized in that: The grid information includes a category and a height value, and the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the accumulated grid map before time t, including: Determine the probability and height value of the grid in the second grid map being an obstacle according to the preset weight values ​​at each moment and the grid information of the grid in the first grid map at moment t after the mapping and the accumulated grid map before moment t; When the probability that the grid in the second grid map is an obstacle is less than a second preset threshold and the height value is less than a third preset threshold, the grid in the second grid map is set to be a road.

10. The method according to claim 1, characterized in that The grid information includes categories, and after the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the accumulated grid map before time t, the method further includes: Determine the connected area of ​​the grid classified as obstacles from the corrected second grid map; Determining whether the size of the connected area is less than a fourth preset threshold; If the size of the connected area is smaller than the fourth preset threshold, the category of each grid in the connected area is corrected to road.

11. The method according to claim 1, characterized in that: The grid information includes categories, and after the second grid map is corrected according to the grid information in the first grid map at time t after the mapping and the accumulated grid map before time t, the method further includes: Determine whether a preset target point is located in the corrected second grid map, the target point being the end point of the vehicle; If the target point is located in the corrected second grid map, the category of the grid within a preset range centered on the target point in the corrected second grid map is corrected to a road.

12. A grid map correction device, applied to an electronic device installed in a vehicle, wherein the vehicle further comprises a laser radar for outputting point cloud data, characterized in that: The device includes: An acquisition unit, configured to acquire a first grid map of a first preset size at time t, wherein the first grid map is obtained according to the point cloud data; an initialization unit, configured to construct a second grid map of a second preset size with the position of the laser radar in the world coordinate system at time t as the center, and initialize the grid information in the second grid map using the overlapping area with the accumulated grid map of the second preset size at time t-1, wherein the center of the accumulated grid map at time t-1 is the position of the laser radar in the world coordinate system at time t-1, and the second preset size is larger than the first preset size; A conversion unit, used for mapping the first grid map at time t from the laser radar coordinate system at time t to the world coordinate system according to the position and posture of the laser radar at time t; The correction unit is used to correct the second grid map for each grid in the first grid map at time t after mapping according to the grid information of the grid in the first grid map at time t after mapping and the cumulative grid map before time t, so as to obtain the cumulative grid map at time t.

13. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 11.

14. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images