A semantic grid map optimization method and system based on multi-task road surface information

Abstract

The application provides a semantic grid map optimization method and system based on multi-task road surface information, and belongs to the technical field of automatic driving navigation. The method comprises the following steps: processing a road environment image to obtain lane line segmentation results and drivable area segmentation results; establishing a coordinate projection relationship from an OCC space to a pixel coordinate system based on a rotation and translation matrix between sensors; projecting surface layer grid center points of a semantic grid map to the pixel coordinate system based on the coordinate projection relationship, and supplementing lane line information according to a projection result; performing semantic correction on a drivable area based on the coordinate projection relationship and the drivable area segmentation results; performing instance extraction and semantic association processing on unknown obstacles; and outputting an optimized three-dimensional semantic grid map. Based on the method, the application further provides a semantic grid map optimization system based on multi-task road surface information. The application improves the quality and practicability of a semantic map without significantly increasing the burden of a system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving navigation, and particularly relates to a semantic grid map optimization method and system based on multi-task road surface information. BACKGROUND

[0002] Precise environmental perception is the solid foundation of automatic driving. The most mature multi-modal fusion algorithm at present is the bird's eye view series algorithm, i.e. BEV. The OCC (Occupancy Prediction) algorithm derived from the BEV algorithm uses semantic grids to construct a dense three-dimensional occupancy map. In the task of constructing a three-dimensional semantic grid map, the OCC algorithm sufficiently expresses the obstacles with geometric information in the automatic driving environment. With the aid of an image multi-task road surface information extraction model, the perception of lane lines and drivable areas can be well supplemented. In addition, when the category semantics of obstacles in the OCC task need to be supplemented, the image multi-task model is selected to supplement this semantics. Then, the detection result of the image multi-task model on this semantics is associated with the corresponding position of the semantic grid result, which can avoid the high cost of repeated training of the OCC model and achieve more efficient representation.

[0003] However, due to the limitation of the computing power of the vehicle-mounted platform, the grid resolution of the semantic grid map is usually controlled at 0.5m x 0.5m x 0.5m. This accuracy cannot represent the information of narrow lane lines. In the existing open-source OCC dataset, the definition of the drivable area during manual labeling is the flat road surface (between the road edges). The no-entry area, such as the emergency lane on the highway, is not distinguished. In addition, due to the complexity of the OCC task, the training cost of the OCC model is much higher than that of the conventional two-dimensional image detection and three-dimensional point cloud detection models. In the actual application process, it is difficult to avoid the situation that new obstacle semantics need to be supplemented. At this time, the cost of repeatedly retraining the OCC model is too high. SUMMARY

[0004] To solve the above technical problems, the present application proposes a semantic grid map optimization method and system based on multi-task road surface information. The information supplement and result optimization of the three-dimensional semantic grid map are realized.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] A semantic grid map optimization method based on multi-task road surface information, comprising the following steps:

[0007] Using a computer vision model to process the obtained road environment image to obtain lane line segmentation results and drivable area segmentation results;

[0008] Based on the rotation and translation matrix between sensors, a coordinate projection relationship from the OCC space to the pixel coordinate system is established;

[0009] Based on the coordinate projection relationship, the surface layer grid center points of the semantic grid map are projected to the pixel coordinate system, and lane line information is supplemented according to the projection result;

[0010] Based on the coordinate projection relationship and the drivable area segmentation result, the drivable area is semantically corrected;

[0011] Unknown obstacles in the initial map are extracted, and the obstacle instances in the non-driving area are filtered out by using the corrected drivable area semantic information; the remaining unknown obstacle instances are projected to the image space by using the coordinate projection relationship again, and are semantically associated with the detection result of the computer vision model;

[0012] The supplemented lane line information, the corrected drivable area semantics and the associated obstacle semantics are integrated to generate a final optimized three-dimensional semantic grid map and output.

[0013] The embodiment of the application also provides a semantic grid map optimization system based on multi-task road surface information, which comprises:

[0014] A preprocessing module is configured to process the obtained road environment image by using a computer vision model to obtain lane line segmentation results and drivable area segmentation results;

[0015] A mapping module is configured to establish a coordinate projection relationship from the OCC space to the pixel coordinate system based on a rotation and translation matrix between sensors;

[0016] A supplement module is configured to project the surface layer grid center points of the semantic grid map to the pixel coordinate system based on the coordinate projection relationship, and supplement lane line information according to the projection result;

[0017] A correction module is configured to correct the semantics of the drivable area based on the coordinate projection relationship and the drivable area segmentation result;

[0018] A semantic association module is configured to extract unknown obstacle instances in the initial map, filter out obstacle instances in the non-driving area by using the corrected drivable area semantic information, project the remaining unknown obstacle instances to the image space by using the coordinate projection relationship again, and semantically associate the obstacle instances with the detection result of the computer vision model;

[0019] An output map module is configured to integrate the supplemented lane line information, the corrected drivable area semantics and the associated obstacle semantics, generate a final optimized three-dimensional semantic grid map and output.

[0020] The effects provided in the summary are only the effects of the embodiments, not all the effects of the application. One of the above technical solutions has the following advantages or beneficial effects:

[0021] The application provides a semantic grid map optimization method and system based on multi-task road surface information. The method comprises the following steps: processing a road environment image obtained by using a computer vision model to obtain lane line segmentation results and drivable area segmentation results; establishing a coordinate projection relationship from an OCC space to a pixel coordinate system based on a rotation and translation matrix between sensors; projecting a surface layer grid center point of a semantic grid map to the pixel coordinate system based on the coordinate projection relationship, and supplementing lane line information according to the projection results; performing semantic correction on a drivable area based on the coordinate projection relationship and the drivable area segmentation results; performing instance extraction on unknown obstacles in an initial map, and filtering out obstacle instances in a non-driving area by using the corrected drivable area semantic information; projecting the remaining unknown obstacle instances to an image space again by using the coordinate projection relationship, and performing semantic association with detection results of the computer vision model; integrating the supplemented lane line information, the corrected drivable area semantics and the associated obstacle semantics, generating a final optimized three-dimensional semantic grid map and outputting the same. Based on the method, the application further provides a semantic grid map optimization system based on multi-task road surface information. By closely integrating visual perception results and grid map structures, the application constructs an efficient, automated and lightweight semantic map optimization pipeline. The core effect is that the quality and practicability of the semantic map are significantly improved without significantly increasing the system burden, thereby providing a safer and more reliable environment recognition basis for an automatic driving system. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A semantic grid map optimization method based on multi-task road surface information is provided for the embodiment 1 of the application, and a flowchart thereof is shown in the figure;

[0023] Figure 2 A schematic diagram of lane line fitting is provided for the embodiment 1 of the application;

[0024] Figure 3 An image association semantic supplement schematic diagram is provided for the embodiment 1 of the application;

[0025] Figure 4 A semantic grid map optimization system based on multi-task road surface information is provided for the embodiment 2 of the application, and a schematic diagram thereof is shown in the figure. DETAILED DESCRIPTION

[0026] For the purpose of clearly illustrating the technical features of the present application, the present application will be described in detail below with specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing the different structures of the present application. In order to simplify the disclosure of the present application, the components and arrangements of specific examples are described below. In addition, reference numerals and / or letters can be repeated in different examples. Such repetition is for the purpose of simplification and clarity, and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. The present application omits the description of well-known components and processing techniques and processes to avoid unnecessarily limiting the present application.

[0027] Embodiment 1

[0028] Embodiment 1 of the present application proposes a semantic grid map optimization method based on multi-task road surface information. The method is an occupancy grid map (OCC) post-processing method for optimizing the initial three-dimensional semantic grid map generated by the OCC model. The grid resolution of the initial map is 0.5m×0.5m×0.5m. The method solves the problems of lane line information loss, drivable area definition ambiguity, and high cost of unknown obstacle semantic supplement in the initial map by integrating lane line information supplement, drivable area correction, and unknown obstacle semantic association functions.

[0029] Figure 1 A flowchart of a semantic grid map optimization method based on multi-task road surface information is proposed for embodiment 1 of the present application;

[0030] In step S1, the lane line segmentation result and the drivable area segmentation result are obtained by processing the acquired road environment image using a computer vision model. The specific process is as follows:

[0031] The pre-trained computer vision model is deployed on a vehicle-mounted computing unit. The original image of the road environment is collected by a vehicle-mounted surround-view camera system, and the original image is input into the computer vision model.

[0032] The computer vision model performs parallel processing on the input image and performs the following inference tasks:

[0033] The probability map of each pixel belonging to different categories of lane lines is output, and different ID lane lines are distinguished based on instance segmentation algorithm;

[0034] The binary segmentation map of each pixel belonging to drivable area or non-drivable area is output;

[0035] The lane line instance segmentation result and the drivable area binary segmentation result are extracted from the model output, and morphological post-processing is performed on the segmentation result to eliminate noise and small cavities.

[0036] The post-processed lane line segmentation results and drivable area segmentation results are output in matrix form and cached in a specified memory area for subsequent semantic raster map optimization processes.

[0037] In step S2, a coordinate projection relationship from OCC space to pixel coordinate system is established based on the rotation and translation matrix between sensors;

[0038] Among them, the OCC spatial coordinate system is ;

[0039] The three-dimensional spatial coordinate system of semantic raster maps typically has the center of the vehicle's rear axle as the origin. The axis points in the direction the vehicle is moving; The axis points to the left side of the vehicle; The axis is perpendicular to the ground and pointing upwards;

[0040] The radar coordinate system is ;

[0041] in, Represents radar coordinate system Axis coordinates, pointing in the direction of vehicle movement; Represents radar coordinate system The axis coordinate points to the left side of the vehicle. Represents radar coordinate system Axis coordinates, perpendicular to the ground upwards;

[0042] Camera coordinate system is ;

[0043] The origin is the optical center of the camera. Axis to the right; Axial downwards; The axis points in the shooting direction along the optical axis.

[0044] Pixel coordinate system is :in, Horizontal pixel coordinates These are the vertical pixel coordinates.

[0045] The extrinsic transformation matrix is :

[0046] The extrinsic transformation matrix from radar coordinate system to camera coordinate system is ,in , It is a 3×3 rotation matrix. It is a 3×1 translation vector, in meters, representing the transformation relationship from the radar coordinate system to the camera coordinate system; Represents the camera intrinsic parameter matrix Principal coordinates ;

[0047] The camera intrinsic matrix is : wherein, is a 3x3 matrix containing focal length and principal point coordinates , that is,

[0048] ;

[0049] The grid resolution is , the map origin is The coordinates in the radar coordinate system are .

[0050] Determine the three-dimensional coordinates of the grid center point in the semantic grid map:

[0051] Let the index of a certain grid in the semantic grid map be , then the coordinates of the center point in the semantic grid map space are: ;

[0052] wherein, are the indices of the grid in the axis direction respectively; is the point in the OCC space coordinate system.

[0053] Convert to the radar coordinate system: if the OCC space and the radar coordinate system coincide, that is, and are the same point, and the axis systems are consistent, then ;

[0054] If there is an offset, the point in the radar coordinate system is calculated by the transformation matrix from the OCC space coordinate system to the radar coordinate system:

[0055] ; obtain the point in the radar coordinate system.

[0056] Conversion from the radar coordinate system to the camera coordinate system: use the extrinsic transformation matrix to convert the point in the radar coordinate system to the camera coordinate system:

[0057] ;

[0058] Perspective projection from the camera coordinate system to the pixel coordinate system:

[0059] Project the point in the camera coordinate system to the pixel coordinate system by the camera intrinsic matrix .

[0060] Normalized image coordinate calculation: first project the point in the camera coordinate system to the imaging plane of the camera, i.e. normalized coordinate

[0061]

[0062] Pixel coordinate conversion; use intrinsic matrix Convert normalized coordinate to pixel coordinate:

[0063]

[0064] After unfolding:

[0065]

[0066] Through the above process, a one-to-one correspondence between any grid center point in the OCC space and the image pixel can be established, providing a coordinate conversion basis for subsequent lane line fitting, obstacle association and other algorithms.

[0067] In step S3, based on the coordinate projection relationship, the surface grid center points of the semantic grid map are projected to the pixel coordinate system, and lane line information is supplemented according to the projection result. Figure 2 The schematic diagram of fitting lane lines for the embodiment 1 of the present application;

[0068] The point cloud is projected to the pixel coordinate system, the center points falling on the lane line pixels are retained, and the retained center points are classified according to the number of lane line instance segmentation. In the classification process, a filtering rule based on the farthest point sampling is established, and the surface semantic grid center points are screened using the instance lane line segmentation result, to ensure that the screened center points can completely represent the geometric shape of the lane line.

[0069] For each batch of classified center points, a cubic function curve fitting is performed according to and coordinates. Before fitting, an outlier rejection operation is performed, the outliers are the center points deviating from the regular trend of the lane line, which are determined and rejected by calculating the Euclidean distance and angle deviation of the center points and adjacent center points. After obtaining the overhead lane line in the OCC space, the intersection line of the lane line surface and the surface plane of the drivable area grid is taken as the OCC three-dimensional lane line, and the lane line information supplement is completed.

[0070] In step S4, based on the coordinate projection relationship and the drivable area segmentation result, the drivable area is semantically corrected; Figure 3 The image association semantic supplement schematic diagram for the embodiment 1 of the present application;

[0071] ​​​​The semantics of the drivable area grid is revised once based on the coordinate projection relationship and the drivable area segmentation result; and the semantics of the drivable area on both sides of the OCC three-dimensional lane line is checked for consistency and revised again.

[0072] The semantics of the drivable area grid is revised once based on the coordinate projection relationship and the drivable area segmentation result, specifically: the center point of the drivable area grid is projected to the pixel coordinate system; if the projection point does not fall on the drivable area segmentation pixel, the corresponding grid semantics is modified to non-drivable area, realizing the first revision of the drivable area.

[0073] The semantics of the drivable area on both sides of the OCC three-dimensional lane line is checked for consistency and revised again, specifically: the OCC three-dimensional lane line and the lane line on both sides of the region semantics output by the computer vision model, if the left side of a certain instance lane line in the image is a drivable area and the right side is a non-drivable area, a preset color mask is added to the drivable area grid on the right side of the lane line under the overhead view of the semantic grid map to mark it as an actual non-drivable area; the mask marking uses a preset color different from the basic color difference of the drivable area and the non-drivable area; after the revision is completed, a continuity check is performed, and if there is an isolated drivable area grid, the revision operation is re-executed.

[0074] After the revision of the drivable area is completed, the continuity of the revised drivable area grid is checked, and if there is an isolated drivable area grid, it is determined that the revision is abnormal and the drivable area revision operation needs to be re-executed. Based on the semantics of the drivable area revised twice, the unknown obstacle class in the initial three-dimensional semantic grid map is extracted, single grids and sheet grids located in non-drivable areas are filtered, and unknown obstacle grid instances in the drivable area are retained.

[0075] In step S5, unknown obstacles in the initial map are extracted, and the revised drivable area semantic information is used to filter out obstacle instances in the non-drivable area; the remaining unknown obstacle instances are projected to the image space again using the coordinate projection relationship, and are associated with the detection results of the computer vision model in terms of semantics;

[0076] The unknown obstacles in the initial map are extracted, specifically: the unknown obstacle class is extracted; single grids and sheet grids in the non-drivable area are filtered; and unknown obstacle grid instances in the drivable area are retained.

[0077] If new obstacle semantics need to be supplemented, only the annotation information corresponding to the new obstacle semantics to be supplemented in the image detection dataset true value (i.e. the target detection true value in the multi-task dataset) needs to be modified, and the true value data of other obstacle classes in the dataset does not need to be modified. The computer vision model is retrained with the new obstacle semantics supplemented as the training target to obtain an optimized model with new semantic detection function.

[0078] The minimum enclosure of the extracted unknown obstacle grid instance is dimensionally projected to the corresponding image by the projection logic of step S1, and if the grid instance is in multiple camera perspectives, multiple projections are performed.

[0079] The distance intersection over union of the projection frame obtained by projection and the prediction frame output by the optimized model is calculated, and when the distance intersection over union is greater than 0.6, it is determined that the semantics of the two are the same and the association is established, if there are multiple distance intersection over unions meeting the requirements, the highest value corresponding association relationship is taken. If all the distance intersection over unions calculated are not greater than 0.6, the corresponding unknown obstacle grid instance is not associated with semantics, and its "unknown obstacle" semantic state is maintained, and the spatial position information of the instance is recorded for subsequent data statistics and analysis, and the unknown obstacle semantic supplement is completed.

[0080] The distance intersection over union DIoU is an index for measuring the spatial position similarity of two bounding boxes in an image, which is used to determine whether the projection frame of the unknown obstacle in the image and the prediction frame output by the computer vision model match, so that the regression becomes more stable.

[0081] ;

[0082] wherein, represents the center point of the prediction frame, represents the center point of the real frame, represents the Euclidean distance between the two center points, represents the diagonal distance of the smallest closed region that can contain the prediction frame and the real frame.

[0083] The scope of protection of the present application is not limited to the numerical values listed in Example 1, and persons skilled in the art can make reasonable choices according to actual conditions.

[0084] In step S6, the supplemented lane line information, the corrected drivable area semantics and the associated obstacle semantics are integrated to generate a final optimized three-dimensional semantic grid map and output.

[0085] The method of Example 1 of the present application adapts to the computing power limit of the vehicle-mounted platform, and all the calculation processes of the steps use lightweight data processing logic to avoid the running delay of the vehicle-mounted system caused by excessive calculation.

[0086] The embodiment 1 of the present application proposes a semantic grid map optimization method based on multi-task road surface information, constructs an efficient, automated and lightweight semantic map optimization pipeline by closely integrating visual perception results and grid map structure, and the core effect is that the quality and practicability of the semantic map are significantly improved without significantly increasing the system burden, thereby providing a safer and more reliable environment recognition basis for the automatic driving system.

[0087] Embodiment 2

[0088] Based on the semantic grid map optimization method based on multi-task road surface information proposed in the embodiment 1 of the present application, the embodiment 2 of the present application further proposes a semantic grid map optimization system based on multi-task road surface information, Figure 4 The embodiment 2 of the present application proposes a semantic grid map optimization system based on multi-task road surface information, and the system includes:

[0089] A preprocessing module is configured to process the obtained road environment image by using a computer vision model to obtain lane line segmentation results and drivable area segmentation results.

[0090] A mapping module is configured to establish a coordinate projection relationship from the OCC space to the pixel coordinate system based on a rotation and translation matrix between sensors.

[0091] A supplement module is configured to project the surface layer grid center point of the semantic grid map to the pixel coordinate system based on the coordinate projection relationship, and supplement the lane line information according to the projection result.

[0092] A correction module is configured to perform semantic correction on the drivable area based on the coordinate projection relationship and the drivable area segmentation results.

[0093] A semantic association module is configured to perform instance extraction on unknown obstacles in the initial map, filter out obstacle instances in the non-driving area by using the corrected drivable area semantic information, project the remaining unknown obstacle instances to the image space by using the coordinate projection relationship again, and perform semantic association with the detection results of the computer vision model.

[0094] An output map module is configured to integrate the supplemented lane line information, the corrected drivable area semantics and the associated obstacle semantics, generate a final optimized three-dimensional semantic grid map and output the same.

[0095] In the preprocessing module of the present application: a pre-trained computer vision model is deployed on a vehicle-mounted computing unit; raw images of the road environment are collected by a vehicle-mounted surround-view camera system, and the raw images are input into the computer vision model; the computer vision model performs parallel processing on the input images and performs the following inference tasks; a probability map of each pixel belonging to different categories of lane lines is output, and different ID lane lines are distinguished based on instance segmentation algorithm; a binary segmentation map of each pixel belonging to drivable area or non-drivable area is output; the lane line instance segmentation result and the drivable area binary segmentation result are extracted from the model output; and morphological post-processing is performed on the segmentation result to eliminate noise and small cavities; the post-processed lane line segmentation result and the drivable area segmentation result are output in matrix form and cached in a specified memory area for subsequent semantic grid map optimization process calling.

[0096] In the mapping module, the coordinate projection relationship from the OCC space to the pixel coordinate system is established, specifically: the point cloud composed of the center points of the surface semantic grid is regarded as the radar point; based on the external parameter calibration data of the laser radar and the camera, the coordinate projection relationship from the OCC space to the pixel coordinate system is established.

[0097] In the supplement module, the lane line information is supplemented, specifically: the grid center points are projected to the pixel coordinate system, the points falling on the lane line pixels are retained and classified according to the instance segmentation number; the classified points are subjected to outlier rejection and cubic function curve fitting; the intersection line of the lane line surface and the drivable area grid surface is calculated to generate the OCC three-dimensional lane line.

[0098] The classified points are subjected to outlier rejection, specifically: the Euclidean distance and angular deviation of the center point and the adjacent points are calculated; the abnormal points deviating from the regular trend of the lane line are rejected; the farthest point sampling rule is adopted to ensure that the screened points can completely represent the geometric shape of the lane line.

[0099] In the correction module, the drivable area is semantically corrected based on the coordinate projection relationship and the drivable area segmentation result, specifically: the semantics of the drivable area grid are corrected once based on the coordinate projection relationship and the drivable area segmentation result; and according to the OCC three-dimensional lane line, the drivable area semantics on both sides of the lane line are verified for consistency and corrected twice.

[0100] The semantics of the drivable area grid are corrected once based on the coordinate projection relationship and the drivable area segmentation result, specifically: the center points of the drivable area grid are projected to the pixel coordinate system;

[0101] If the projected point does not fall on the drivable area segmentation pixel, the corresponding grid semantics is modified to non-drivable area, realizing the first correction of the drivable area.

[0102] According to the OCC three-dimensional lane line, the consistency of the drivable area semantics on both sides of the lane line is verified and secondary correction is performed, specifically: the OCC three-dimensional lane line and the area semantics on both sides of the lane line output by the computer vision model, if the left side of a certain instance lane line in the image is a drivable area and the right side is a non-drivable area, then under the overhead perspective of the semantic grid map, a preset color mask is added to the drivable area grid on the right side of the lane line to mark it as an actual non-drivable area; the mask marking uses a preset color that is different from the basic color difference between the drivable area and the non-drivable area; after the correction is completed, continuity verification is performed, and if there are isolated drivable area grids, the correction operation is re-executed.

[0103] In the correction module, unknown obstacle instances in the initial map are extracted, specifically: unknown obstacle categories are extracted; single grids and piecewise grids in the non-drivable area are filtered; unknown obstacle grid instances in the drivable area are retained.

[0104] Semantic association, specifically: the smallest covering body of the unknown obstacle instance is projected to the corresponding image; the distance intersection ratio of the projection frame and the predicted frame output by the image multi-task model is calculated; when the distance intersection ratio is greater than a preset value, semantic association is established.

[0105] Embodiment 2 of the present application provides a semantic grid map optimization system based on multi-task road surface information, which tightly integrates visual perception results and grid map structure to construct an efficient, automated and lightweight semantic map optimization pipeline. The core effect is to significantly improve the quality and practicality of the semantic map without significantly increasing the system burden, thereby providing a safer and more reliable environment for the automatic driving system.

[0106] The related part of the semantic grid map optimization system based on multi-task road surface information provided in Embodiment 2 of the present application can refer to the detailed description of the corresponding part in the semantic grid map optimization method based on multi-task road surface information provided in Embodiment 1 of the present application, which will not be repeated here.

[0107] It is to be noted that, in the present text, the terms such as first and second, and the like are used merely to differentiate one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Moreover, the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements in the list. Without more limitations, an element defined by an expression "comprising a..." does not exclude the existence of additional identical elements in the process, method, article, or apparatus including the element. In addition, the above-described technical solutions provided by the embodiments of the present application have not been described in detail, which are consistent with the implementation principles of the corresponding technical solutions in the prior art, so as not to be too verbose.

[0108] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the protection scope of the present application. Based on the above description, those skilled in the art can make other different forms of modifications or changes. Here, it is not necessary or possible to exhaust all the embodiments. Various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A semantic grid map optimization method based on multi-task road surface information, characterized in that, The method comprises the following steps: processing the acquired road environment image by using a computer vision model to obtain lane line segmentation results and drivable area segmentation results; establishing a coordinate projection relationship from the OCC space to the pixel coordinate system based on the rotation and translation matrix between sensors; projecting the surface layer grid center points of the semantic grid map to the pixel coordinate system based on the coordinate projection relationship, and supplementing lane line information according to the projection results; performing semantic correction on the drivable area based on the coordinate projection relationship and the drivable area segmentation results; extracting unknown obstacle instances in the initial map, filtering out obstacle instances in the non-driving area by using the corrected drivable area semantic information, projecting the remaining unknown obstacle instances to the image space again by using the coordinate projection relationship, and performing semantic association with the detection results of the computer vision model; wherein the process of semantic association is: projecting the minimum covering body of the unknown obstacle instance to the corresponding image; calculating the distance intersection ratio of the projection frame and the predicted frame output by the image multi-task model; the distance intersection ratio is used to measure the spatial position similarity of two boundary boxes in the image; when the distance intersection ratio is greater than a preset value, the semantic association is established; integrating the supplemented lane line information, the corrected drivable area semantics and the associated obstacle semantics to generate a final optimized three-dimensional semantic grid map and output the final optimized three-dimensional semantic grid map.

2. The method of claim 1, wherein, The coordinate projection relationship from the OCC space to the pixel coordinate system is established based on the rotation and translation matrix between sensors, and specifically: the point cloud composed of the surface layer semantic grid center points is regarded as a radar point; the coordinate projection relationship from the OCC space to the pixel coordinate system is established based on the external parameter calibration data of the lidar and the camera.

3. The method of claim 1, wherein, The lane line information is supplemented, and specifically: the grid center points are projected to the pixel coordinate system, the points falling on the lane line pixels are retained and classified according to the instance segmentation number; abnormal value elimination and cubic function curve fitting are performed on the classified points. The intersection line of the lane line surface and the drivable area grid surface layer plane is calculated to generate an OCC three-dimensional lane line.

4. The method of claim 3, wherein, The abnormal value elimination is performed on the classified points, and specifically: the Euclidean distance and the angle deviation of the center point and the adjacent point are calculated; the abnormal points deviating from the regular trend of the lane line are eliminated; the farthest point sampling rule is adopted to ensure that the screened points can completely represent the geometric shape of the lane line.

5. The method of claim 1, wherein, The drivable area is semantically corrected based on the coordinate projection relationship and the drivable area segmentation results, and specifically: the semantic of the drivable area grid is corrected once based on the coordinate projection relationship and the drivable area segmentation results; the consistency verification and the secondary correction of the drivable area semantic on both sides of the lane line are performed again according to the OCC three-dimensional lane line.

6. The method of claim 5, wherein, The semantic of the drivable area grid is corrected once based on the coordinate projection relationship and the drivable area segmentation results, and specifically: the drivable area grid center points are projected to the pixel coordinate system; if the projection point does not fall on the drivable area segmentation pixel, the semantic of the corresponding grid is modified to be a non-drivable area, and the drivable area is corrected once.

7. The method of claim 5, wherein, The consistency verification and the secondary correction of the drivable area semantic on both sides of the lane line are performed again according to the OCC three-dimensional lane line, and specifically: The OCC three-dimensional lane line and the lane line two-side area semantics output by the computer vision model, if the left side of a certain instance lane line in the image is a drivable area and the right side is a non-drivable area, then in the overhead view of the semantic grid map, a preset color mask is added to the drivable area grid on the right side of the lane line to mark it as an actual non-drivable area; the mask marking uses a preset color that is different from the basic color difference between the drivable area and the non-drivable area; After the correction is completed, a continuity check is performed, and if there is an isolated drivable area grid, the correction operation is re-executed.

8. The method of claim 1, wherein, Instance extraction is performed on unknown obstacles in the initial map, specifically: Instance extraction is performed on unknown obstacle categories; single grids and non-drivable area piece grids are filtered; unknown obstacle grid instances in the drivable area are retained.

9. A multi-task road surface information based semantic grid map optimization system for performing the multi-task road surface information based semantic grid map optimization method of any one of claims 1 to 8. It includes: A preprocessing module is configured to process a road environment image obtained by using a computer vision model to obtain lane line segmentation results and drivable area segmentation results; A mapping module is configured to establish a coordinate projection relationship from an OCC space to a pixel coordinate system based on a rotation and translation matrix between sensors; A supplement module is configured to project the surface grid center points of the semantic grid map to the pixel coordinate system based on the coordinate projection relationship, and supplement lane line information according to the projection result; A correction module is configured to perform semantic correction on the drivable area based on the coordinate projection relationship and the drivable area segmentation result; A semantic correlation module is configured to perform instance extraction on unknown obstacles in the initial map, filter out obstacle instances in the non-drivable area by using the corrected drivable area semantic information, and project the remaining unknown obstacle instances to the image space by using the coordinate projection relationship again, and perform semantic correlation with the detection result of the computer vision model; An output map module is configured to integrate the supplemented lane line information, the corrected drivable area semantics, and the correlated obstacle semantics, generate a final optimized three-dimensional semantic grid map, and output the map.

Images