Lane line segmentation optimization method and device, electronic equipment and computer program product

Abstract

The invention discloses a lane line segmentation optimization method and device, electronic equipment and a computer program product, and the method comprises the steps: obtaining a current frame road image collected by a vehicle, and carrying out the lane line segmentation, and obtaining a lane line segmentation result of the current frame; filtering the lane line segmentation result of the current frame by using a preset filtering strategy according to the lane line segmentation result of the previous frame; positioning displacement generated by the vehicle in the current frame and the previous frame is determined, and the filtered lane line segmentation result of the current frame and the filtered lane line segmentation result of the previous frame are matched according to the positioning displacement to obtain a lane line matching result; and compensating the filtered lane line segmentation result of the current frame according to the lane line matching result. According to the method, the lane line misrecognition information of the current frame is filtered in combination with the historical frame data, and the missing part is matched and compensated in combination with the positioning displacement, so that the missing result of lane line segmentation can be effectively recalled, and the accuracy of lane line segmentation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of lane line segmentation, and in particular to a lane line segmentation optimization method, device, electronic equipment, and computer program product. Background Art

[0002] High-precision positioning is very important for autonomous vehicles. The higher the positioning accuracy and the more stable it is, the safer the autonomous vehicle will be. At present, the mainstream positioning solution is mainly based on combined navigation, that is, using Kalman filtering to fuse high-frequency IMU (Inertial Measurement Unit) information with low-frequency GNSS signals to output high-frequency positioning results without cumulative errors.

[0003] Due to the multipath effect caused by high-rise buildings, viaducts, etc. in cities, the accuracy of GNSS signals will be greatly reduced. With the development of hardware and multi-sensor fusion positioning technology, SLAM (Simultaneous Localization and Mapping) methods based on sensor devices such as cameras and lidar are gradually adopted to provide auxiliary positioning information to make up for the shortcomings of GNSS (Global Navigation Satellite System) signals when they are poor.

[0004] However, the premise of using laser SLAM or visual SLAM is to have map data, such as laser-built maps, high-precision maps, or semantic maps. In addition, in many scenarios, reliable results cannot be obtained, such as when there is snow or water on the road, or when there are many dynamic objects around the autonomous vehicle, which will affect the recognition of road elements such as lane lines, and thus affect the positioning accuracy of the autonomous vehicle. Summary of the invention

[0005] The embodiments of the present application provide a lane line segmentation optimization method, device, electronic device, and computer program product to improve the accuracy of lane line segmentation, thereby providing effective support for auxiliary positioning of autonomous driving vehicles.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a lane line segmentation optimization method, wherein the lane line segmentation optimization method includes:

[0008] Acquire a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame;

[0009] Obtaining the lane line segmentation result of the previous frame, and filtering the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame, to obtain a filtered lane line segmentation result of the current frame;

[0010] Determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result;

[0011] The lane line segmentation result of the filtered current frame is compensated according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

[0012] Optionally, the lane line segmentation result includes a lane line semantic segmentation result and a lane line instance segmentation result, and the lane line segmentation result of the current frame is filtered using a preset filtering strategy according to the lane line segmentation result of the previous frame to obtain the filtered lane line segmentation result of the current frame, including:

[0013] According to the lane line semantic segmentation result of the previous frame, the lane line semantic segmentation result of the current frame is filtered using a first preset filtering strategy to obtain a filtering result of the lane line semantic segmentation result of the current frame;

[0014] Based on the filtering result of the lane line semantic segmentation result of the current frame, the lane line instance segmentation result of the current frame is filtered using a second preset filtering strategy to obtain the filtering result of the lane line instance segmentation result of the current frame.

[0015] Optionally, filtering the lane line semantic segmentation result of the current frame using a first preset filtering strategy according to the lane line semantic segmentation result of the previous frame to obtain a filtering result of the lane line semantic segmentation result of the current frame includes:

[0016] Projecting the lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame to the vehicle body coordinate system to obtain the projected lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame;

[0017] Matching the projected lane line semantic segmentation result of the previous frame with the lane line semantic segmentation result of the current frame to obtain a matching result of the lane line semantic segmentation result;

[0018] Determine the lane line semantic segmentation length of the current frame according to the lane line semantic segmentation result of the current frame;

[0019] According to the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame, the lane line semantic segmentation result of the current frame is filtered to obtain a filtering result of the lane line semantic segmentation result of the current frame.

[0020] Optionally, filtering the lane line semantic segmentation result of the current frame according to the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame to obtain the filtering result of the lane line semantic segmentation result of the current frame includes:

[0021] If there are lane line pixels that are not successfully matched in the current frame in the matching result of the lane line semantic segmentation result, the lane line pixels that are not successfully matched in the current frame are filtered out from the lane line semantic segmentation result of the current frame;

[0022] If the lane line semantic segmentation length of the current frame is less than the preset segmentation length threshold, the lane line pixels whose lane line semantic segmentation length of the current frame is less than the preset segmentation length threshold are filtered out from the lane line semantic segmentation result of the current frame.

[0023] Optionally, the filtering result based on the lane line semantic segmentation result of the current frame uses a second preset filtering strategy to filter the lane line instance segmentation result of the current frame, and the filtering result of the lane line instance segmentation result of the current frame includes:

[0024] Filtering the lane line instance segmentation result of the current frame based on the filtering result of the lane line semantic segmentation result of the current frame to obtain a first filtering result of the lane line instance segmentation result of the current frame;

[0025] Clustering the first filtered result of the lane line instance segmentation result of the current frame using a preset clustering algorithm to obtain a clustering result of the lane line instance segmentation result of the current frame;

[0026] Determine the lane line instance segmentation length of the current frame according to the lane line instance segmentation result of the current frame;

[0027] According to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame, the first filtering result of the lane line instance segmentation result of the current frame is filtered to obtain the second filtering result of the lane line instance segmentation result of the current frame.

[0028] Optionally, the clustering result includes the number of clustering core points, and filtering the first filtering result of the lane line instance segmentation result of the current frame according to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame to obtain the second filtering result of the lane line instance segmentation result of the current frame includes:

[0029] If the number of cluster core points corresponding to the lane line instance segmentation result of the current frame is less than a preset number threshold, the lane line instance segmentation result with the number of cluster core points less than the preset number threshold is filtered out from the lane line instance segmentation result of the current frame;

[0030] If the lane line instance segmentation length of the current frame is less than a preset instance length threshold, lane line pixels whose lane line instance segmentation length is less than the preset instance length threshold are filtered out from the lane line instance segmentation result of the current frame.

[0031] Optionally, matching the lane line segmentation result of the current frame after filtering with the lane line segmentation result of the previous frame according to the positioning displacement to obtain the lane line matching result includes:

[0032] Based on the filtered lane line segmentation result of the current frame and the positioning displacement, the lane line segmentation result of the previous frame is restored to obtain a restored result of the lane line segmentation result of the previous frame;

[0033] The lane line segmentation result of the filtered current frame is matched with the restored result of the lane line segmentation result of the previous frame to obtain the lane line matching result.

[0034] Optionally, compensating the filtered lane line segmentation result of the current frame according to the lane line matching result to obtain the compensated lane line segmentation result of the current frame includes:

[0035] Generate candidate lane line pixel points of the current frame according to the lane line matching result;

[0036] The candidate lane line pixel points of the current frame are combined with the filtered lane line segmentation result of the current frame to obtain the compensated lane line segmentation result of the current frame.

[0037] In a second aspect, an embodiment of the present application further provides a lane line segmentation optimization device, wherein the lane line segmentation optimization device comprises:

[0038] A segmentation unit is used to obtain a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame;

[0039] A filtering unit, used to obtain the lane line segmentation result of the previous frame, and filter the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame to obtain a filtered lane line segmentation result of the current frame;

[0040] A matching unit, used to determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result;

[0041] A compensation unit is used to compensate the lane line segmentation result of the filtered current frame according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

[0042] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0043] Processor; and

[0044] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform any of the lane segmentation optimization methods described above.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements any of the aforementioned lane line segmentation optimization methods.

[0046] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the lane line segmentation optimization method in the embodiments of the present application first obtains the current frame road image collected by the autonomous driving vehicle, and uses the lane line segmentation model to perform lane line segmentation on the current frame road image to obtain the lane line segmentation result of the current frame; then obtains the lane line segmentation result of the previous frame, and according to the lane line segmentation result of the previous frame, uses a preset filtering strategy to filter the lane line segmentation result of the current frame to obtain the filtered lane line segmentation result of the current frame; then determines the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and matches the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain the lane line matching result; finally, compensates the filtered lane line segmentation result of the current frame according to the lane line matching result to obtain the compensated lane line segmentation result of the current frame. The lane line segmentation optimization method of the embodiment of the present application combines the lane line segmentation data of the historical frame to filter the lane line misidentification information of the current frame, and further combines the positioning displacement of the adjacent frames to match and compensate for the missing parts. It can effectively recall the missing results of the lane line segmentation, improve the accuracy of the lane line segmentation, and provide more reliable information for subsequent lane line fitting and auxiliary positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A schematic diagram of a process flow of a lane line segmentation optimization method in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a lane matching result in an embodiment of the present application;

[0050] Figure 3 This is a schematic diagram of the structure of a lane segmentation optimization device in an embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0053] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0054] The present application embodiment provides a lane line segmentation optimization method, such as Figure 1 As shown, a flow chart of a lane line segmentation optimization method in an embodiment of the present application is provided, and the lane line segmentation optimization method at least includes the following steps S110 to S140:

[0055] Step S110, obtaining a current frame road image collected by the autonomous driving vehicle, and performing lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame.

[0056] When performing lane line segmentation optimization in the embodiment of the present application, it is necessary to first obtain the current frame road image collected by the autonomous driving vehicle in real time. The road image collected here mainly refers to the road image directly in front of the autonomous driving vehicle. Then, the lane lines in the road image are segmented using a pre-trained lane line segmentation model to obtain the lane line segmentation result of the current frame. The lane line segmentation result can specifically include the lane line semantic segmentation result and the lane line instance segmentation result, which can be obtained through different segmentation models. The specific segmentation model to be adopted can be flexibly selected by those skilled in the art in combination with the existing technology, and is not specifically limited here.

[0057] Step S120, obtaining the lane line segmentation result of the previous frame, and filtering the lane line segmentation result of the current frame using a preset filtering strategy based on the lane line segmentation result of the previous frame to obtain a filtered lane line segmentation result of the current frame.

[0058] Considering that if the optimization is only based on the lane line segmentation result of a single image, it will not be possible to accurately distinguish between lane line misidentification or missing information due to the lack of prior information, the embodiment of the present application further obtains the lane line segmentation results of the historical frame. Since the relative changes of lane lines in adjacent frames are generally small, in order to facilitate calculation and improve calculation efficiency, the lane line segmentation results of the previous frame are mainly obtained here. The lane line segmentation results of the previous frame are used as prior information, and a certain filtering strategy is adopted to filter out the misidentification information in the lane line segmentation results of the current frame to obtain the filtered lane line segmentation results of the current frame.

[0059] Step S130, determining the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and matching the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result.

[0060] Taking into account the actual scene, due to changes in occlusion, lighting conditions, etc., there may be missing information in the lane line segmented by the current frame. Therefore, the embodiment of the present application needs to compensate for the missing lane line information based on the lane line segmentation result obtained after the above filtering, so as to further optimize the lane line segmentation result.

[0061] Compensation for missing lane line information also requires the lane line segmentation results of the historical frames, and mainly the lane line segmentation results of the previous frame. Because the missing lane line information that is not segmented in the current frame may be recognized in the previous frame, the lane line missing information of the current frame can be compensated by combining the lane line segmentation results of the previous frame. Since the lane line segmentation results usually contain lane line pixels corresponding to multiple segmented lane lines, it is necessary to match the lane line segmentation results of the current frame with the lane line segmentation results of the previous frame to determine the corresponding lane line segmentation results. In addition, since the autonomous driving vehicle is driving in real time, the lane line area recognized in the current frame is different from the lane line area recognized in the previous frame. Therefore, the positioning displacement of the autonomous driving vehicle in these two frames also needs to be considered when matching.

[0062] Step S140, compensating the filtered lane line segmentation result of the current frame according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

[0063] After obtaining the matching result of the lane line segmentation result of the current frame and the previous frame, the missing information of the lane line segmentation result of the current frame can be compensated based on the matching result, so as to obtain the compensated lane line segmentation result of the current frame.

[0064] The lane line segmentation optimization method of the embodiment of the present application combines the lane line segmentation data of the historical frame to filter the lane line misidentification information of the current frame, and further combines the positioning displacement of the adjacent frames to match and compensate for the missing parts. It can effectively recall the missing results of the lane line segmentation, improve the accuracy of the lane line segmentation, and provide more reliable information for subsequent lane line fitting and auxiliary positioning.

[0065] In some embodiments of the present application, the lane line segmentation result includes a lane line semantic segmentation result and a lane line instance segmentation result. The lane line segmentation result of the current frame is filtered according to the lane line segmentation result of the previous frame using a preset filtering strategy to obtain the filtered lane line segmentation result of the current frame, including: according to the lane line semantic segmentation result of the previous frame, the lane line semantic segmentation result of the current frame is filtered using a first preset filtering strategy to obtain the filtered result of the lane line semantic segmentation result of the current frame; based on the filtered result of the lane line semantic segmentation result of the current frame, the lane line instance segmentation result of the current frame is filtered using a second preset filtering strategy to obtain the filtered result of the lane line instance segmentation result of the current frame.

[0066] The lane line segmentation results of the embodiments of the present application mainly include lane line semantic segmentation results and lane line instance segmentation results. The lane line semantic segmentation model is used to perform lane line semantic segmentation on the current frame road image, and the lane line semantic segmentation results of the current frame can be obtained. The main goal of the lane line semantic segmentation model is to separate the lane lines in the image from the background. Usually, the output is a binary segmentation map, in which white represents the lane lines and black represents the background. This type of model is mainly based on deep learning technology, especially the convolutional neural network (CNN) architecture.

[0067] The lane line instance segmentation model is used to perform lane line instance segmentation on the current frame road image, and the lane line instance segmentation result of the current frame can be obtained. The lane line instance segmentation model not only needs to separate the lane line from the background, but also needs to further distinguish different lane line instances (i.e. different lanes). This type of model usually outputs a lane line mask at the pixel level, each mask corresponds to a lane line, which can be implemented based on the LaneNet model, etc.

[0068] Of course, the specific form of the lane semantic segmentation model and instance segmentation model can be flexibly selected by those skilled in the art in combination with the existing technology, and no specific limitation is made here.

[0069] Since the lane line semantic segmentation result only contains the segmented lane line foreground pixels and background pixels, and does not distinguish the specific lane line categories, the lane line instance segmentation result will not only segment the lane line foreground pixels and background pixels, but also further segment the specific lane line types. Therefore, the information contained in the lane line semantic segmentation result is simpler. When filtering the outliers in the lane line segmentation result of the current frame, the lane line semantic segmentation result of the previous frame can be used to perform a rough filtering on the lane line semantic segmentation result of the current frame, which can filter out obvious misidentification information, and then further filter the instance segmentation results on this basis. Compared with the method of directly filtering the instance segmentation results, the amount of data for filtering the instance segmentation results can be reduced, the processing efficiency can be improved, and the real-time requirements of the scene can be met.

[0070] In some embodiments of the present application, the lane line semantic segmentation results of the previous frame are filtered using a first preset filtering strategy to obtain the filtered results of the lane line semantic segmentation results of the current frame, including: projecting the lane line semantic segmentation results of the previous frame and the lane line semantic segmentation results of the current frame to the vehicle body coordinate system to obtain the projected lane line semantic segmentation results of the previous frame and the lane line semantic segmentation results of the current frame; matching the projected lane line semantic segmentation results of the previous frame and the lane line semantic segmentation results of the current frame to obtain the matching results of the lane line semantic segmentation results; determining the lane line semantic segmentation length of the current frame according to the lane line semantic segmentation results of the current frame; filtering the lane line semantic segmentation results of the current frame according to the matching results of the lane line semantic segmentation results and the lane line semantic segmentation length of the current frame to obtain the filtered results of the lane line semantic segmentation results of the current frame.

[0071] When filtering the lane line semantic segmentation result of the current frame, the embodiment of the present application can first use the inverse perspective transformation (IPM) to project the lane line semantic segmentation results of the previous frame and the current frame to the vehicle body coordinate system to avoid the impact of image distortion on the lane line segmentation optimization. Then, the projected lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame are matched to obtain a matching result of the lane line semantic segmentation result. The matching result can reflect to a certain extent whether there is misidentification information in the lane line semantic segmentation result of the current frame that was not segmented in the previous frame. The specific matching algorithm can be flexibly selected in combination with the existing technology and is not specifically limited here.

[0072] In addition, the lane line semantic segmentation length of the current frame can also be determined based on the lane line semantic segmentation result of the current frame. The lane line semantic segmentation length can also reflect the misidentification of the lane line to a certain extent. Finally, the lane line semantic segmentation result of the current frame is filtered by combining the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame, thereby obtaining the filtered result of the lane line semantic segmentation result of the current frame.

[0073] In some embodiments of the present application, the lane line semantic segmentation result of the current frame is filtered according to the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame to obtain the filtered result of the lane line semantic segmentation result of the current frame, including: if there are lane line pixels in the matching result of the lane line semantic segmentation result that are not successfully matched in the current frame, then the lane line pixels in the current frame that are not successfully matched are filtered out from the lane line semantic segmentation result of the current frame; if the lane line semantic segmentation length of the current frame is less than a preset segmentation length threshold, then the lane line pixels in the current frame whose lane line semantic segmentation length is less than the preset segmentation length threshold are filtered out from the lane line semantic segmentation result of the current frame.

[0074] On the one hand, the feature changes of normal lane lines between two adjacent frames are relatively small, so they can generally be matched successfully. If there are unmatched lane line pixels in the matching results of the lane line semantic segmentation results of the current frame, it means that the unmatched lane line pixels are likely to be misidentified pixels. In this case, the unmatched lane line pixels of the current frame are filtered out from the lane line semantic segmentation results of the current frame.

[0075] On the other hand, a normal lane line has a certain length. If the lane line semantic segmentation length of the current frame is less than a certain segmentation length threshold, it means that the lane line pixel is also likely to be a misidentified pixel. In this way, the lane line pixel whose lane line semantic segmentation length of the current frame is less than the preset segmentation length threshold is filtered out from the lane line semantic segmentation result of the current frame. In this way, the misidentified information in the lane line semantic segmentation result is filtered out to a certain extent.

[0076] In some embodiments of the present application, the filtering result based on the lane line semantic segmentation result of the current frame uses a second preset filtering strategy to filter the lane line instance segmentation result of the current frame to obtain the filtering result of the lane line instance segmentation result of the current frame, including: filtering the lane line instance segmentation result of the current frame based on the filtering result of the lane line semantic segmentation result of the current frame to obtain a first filtering result of the lane line instance segmentation result of the current frame; clustering the first filtering result of the lane line instance segmentation result of the current frame using a preset clustering algorithm to obtain a clustering result of the lane line instance segmentation result of the current frame; determining the lane line instance segmentation length of the current frame according to the lane line instance segmentation result of the current frame; filtering the first filtering result of the lane line instance segmentation result of the current frame according to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame to obtain a second filtering result of the lane line instance segmentation result of the current frame.

[0077] Since the lane line semantic segmentation results and the instance segmentation results have the same presentation form, except that the instance segmentation results further distinguish different lane line categories, if the pixels filtered out in the semantic segmentation results should also be filtered out in the instance segmentation results, when filtering the lane line instance segmentation results of the current frame, the embodiment of the present application can first filter the lane line instance segmentation results of the current frame based on the filtering results of the lane line semantic segmentation results of the current frame to obtain the first filtering results of the lane line instance segmentation results of the current frame. This process is equivalent to applying the filtering results of the semantic segmentation results to the instance segmentation results, thereby improving the subsequent processing efficiency of the instance segmentation results.

[0078] In order to further improve the processing efficiency of instance segmentation results, the first filtered result of the lane line instance segmentation result of the current frame can be subjected to image thinning processing, aiming to reduce the amount of information in the image while retaining the key features and details of the original data as much as possible. Then, clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise) are used to cluster the lane line instance segmentation results to obtain the clustering results of the lane line instance segmentation results of the current frame.

[0079] In addition, considering that different lane line instance types have different lengths, the embodiment of the present application can also further determine the segmentation length of each lane line instance of the current frame according to the lane line instance segmentation result of the current frame. Finally, the first filtering result of the lane line instance segmentation result of the current frame is filtered by combining the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length to obtain a second filtering result.

[0080] In some embodiments of the present application, the clustering result includes the number of clustering core points, and the first filtering result of the lane line instance segmentation result of the current frame is filtered according to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame to obtain the second filtering result of the lane line instance segmentation result of the current frame including: if the number of clustering core points corresponding to the lane line instance segmentation result of the current frame is less than a preset number threshold, then the lane line instance segmentation result with the number of clustering core points less than the preset number threshold is filtered out from the lane line instance segmentation result of the current frame; if the lane line instance segmentation length of the current frame is less than the preset instance length threshold, then the lane line pixel points with the lane line instance segmentation length less than the preset instance length threshold are filtered out from the lane line instance segmentation result of the current frame.

[0081] On the one hand, the DBSCAN algorithm is a density-based clustering algorithm. Therefore, for the pixels of normal lane line instances, after clustering based on the DBSCAN algorithm, the number of clustering core points should be able to reach a certain threshold. Therefore, if the number of clustering core points corresponding to the lane line instance segmentation result of the current frame is less than the preset threshold, it means that the lane line instance is likely to be a misidentified lane line instance, and it will be filtered out from the lane line instance segmentation result of the current frame.

[0082] On the other hand, normal lane lines have a certain length, and different types of lane lines have different lengths. Therefore, if the lane line instance segmentation length of the current frame is less than the corresponding length threshold, it means that the lane line instance is also likely to be a misidentified lane line instance, and it will be filtered out from the lane line semantic segmentation result of the current frame. In this way, on the basis of filtering the semantic segmentation results, the misidentified information in the lane line instance segmentation results is further filtered out.

[0083] In some embodiments of the present application, the lane line segmentation result of the filtered current frame is matched with the lane line segmentation result of the previous frame according to the positioning displacement to obtain the lane line matching result, including: based on the filtered lane line segmentation result of the current frame and the positioning displacement, the lane line segmentation result of the previous frame is restored to obtain the restored result of the lane line segmentation result of the previous frame; the lane line segmentation result of the filtered current frame is matched with the restored result of the lane line segmentation result of the previous frame to obtain the lane line matching result.

[0084] When matching lane lines in the lane line compensation stage, the lane line segmentation result of the previous frame can be restored based on the filtered lane line segmentation result and positioning displacement of the current frame, and then the filtered lane line segmentation result of the current frame is matched with the restored result of the lane line segmentation result of the previous frame, and the edge extraction result of the lane line is used to calculate the overlapping area of ​​the previous and next frames, and the largest area is taken as the best match.

[0085] In some embodiments of the present application, the lane line segmentation result of the filtered current frame is compensated according to the lane line matching result to obtain the compensated lane line segmentation result of the current frame, including: generating candidate lane line pixel points of the current frame according to the lane line matching result; merging the candidate lane line pixel points of the current frame with the lane line segmentation result of the filtered current frame to obtain the compensated lane line segmentation result of the current frame.

[0086] After lane line matching, the union of lane line pixel points in the lane line boundary box in the x direction in the vehicle body coordinate system is taken, and candidate points are added along the x direction to obtain the lane line segmentation result of the current frame after compensation.

[0087] To facilitate understanding of the above embodiments, Figure 2 As shown, a schematic diagram of a lane line matching result in an embodiment of the present application is provided. Figure 2 The x-axis and y-axis in the figure are the coordinate axes in the vehicle coordinate system. The two rectangular boxes are the lane line edge extraction results of the previous and next frames respectively. The gray area is the maximum overlapping area of ​​the match. The dotted line outside the lane line boundary box of the current frame is the supplementary lane line information.

[0088] In summary, the key points of the lane segmentation optimization method of this application are:

[0089] 1) Combine historical frame data to filter out lane line misidentification information in the current frame and compensate for missing lane line information;

[0090] 2) The inverse perspective transformation is used to convert the missing lane segmentation results into the vehicle coordinate system instead of adding them directly to the original image. At the same time, the positioning displacement is used as the matching compensation of the previous and next frames, which can effectively recall the missing lane segmentation results.

[0091] This application can achieve effective filtering of outliers in lane segmentation results and effective supplementation of missing lane lines in straight road section scenarios, which has a positive effect on subsequent curve fitting and positioning assistance for lane lines.

[0092] The present application also provides a lane segmentation optimization device 300, such as Figure 3 As shown, a schematic diagram of the structure of a lane line segmentation optimization device in an embodiment of the present application is provided, wherein the lane line segmentation optimization device 300 includes: a segmentation unit 310, a filtering unit 320, a matching unit 330 and a compensation unit 340, wherein:

[0093] The segmentation unit 310 is used to obtain a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame;

[0094] A filtering unit 320 is used to obtain the lane line segmentation result of the previous frame, and filter the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame to obtain a filtered lane line segmentation result of the current frame;

[0095] A matching unit 330 is used to determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result;

[0096] The compensation unit 340 is used to compensate the lane line segmentation result of the filtered current frame according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

[0097] In some embodiments of the present application, the lane line segmentation result includes a lane line semantic segmentation result and a lane line instance segmentation result, and the filtering unit 320 is specifically used to: filter the lane line semantic segmentation result of the current frame using a first preset filtering strategy based on the lane line semantic segmentation result of the previous frame to obtain a filtered result of the lane line semantic segmentation result of the current frame; based on the filtered result of the lane line semantic segmentation result of the current frame, filter the lane line instance segmentation result of the current frame using a second preset filtering strategy to obtain a filtered result of the lane line instance segmentation result of the current frame.

[0098] In some embodiments of the present application, the filtering unit 320 is specifically used to: project the lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame to the vehicle body coordinate system to obtain the projected lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame; match the projected lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame to obtain the matching result of the lane line semantic segmentation result; determine the lane line semantic segmentation length of the current frame according to the lane line semantic segmentation result of the current frame; filter the lane line semantic segmentation result of the current frame according to the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame to obtain the filtering result of the lane line semantic segmentation result of the current frame.

[0099] In some embodiments of the present application, the filtering unit 320 is specifically used to: if there are lane line pixels that have not been successfully matched in the current frame in the matching results of the lane line semantic segmentation results, then the lane line pixels that have not been successfully matched in the current frame are filtered out from the lane line semantic segmentation results of the current frame; if the lane line semantic segmentation length of the current frame is less than a preset segmentation length threshold, then the lane line pixels whose lane line semantic segmentation length of the current frame is less than the preset segmentation length threshold are filtered out from the lane line semantic segmentation results of the current frame.

[0100] In some embodiments of the present application, the filtering unit 320 is specifically used to: filter the lane line instance segmentation result of the current frame based on the filtering result of the lane line semantic segmentation result of the current frame to obtain a first filtering result of the lane line instance segmentation result of the current frame; cluster the first filtering result of the lane line instance segmentation result of the current frame using a preset clustering algorithm to obtain a clustering result of the lane line instance segmentation result of the current frame; determine the lane line instance segmentation length of the current frame according to the lane line instance segmentation result of the current frame; filter the first filtering result of the lane line instance segmentation result of the current frame according to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame to obtain a second filtering result of the lane line instance segmentation result of the current frame.

[0101] In some embodiments of the present application, the clustering result includes the number of clustering core points, and the filtering unit 320 is specifically used to: if the number of clustering core points corresponding to the lane line instance segmentation result of the current frame is less than a preset number threshold, then the lane line instance segmentation result with the number of clustering core points less than the preset number threshold is filtered out from the lane line instance segmentation result of the current frame; if the lane line instance segmentation length of the current frame is less than the preset instance length threshold, then the lane line pixel points with the lane line instance segmentation length less than the preset instance length threshold are filtered out from the lane line instance segmentation result of the current frame.

[0102] In some embodiments of the present application, the matching unit 330 is specifically used to: restore the lane line segmentation result of the previous frame based on the filtered lane line segmentation result of the current frame and the positioning displacement to obtain the restored result of the lane line segmentation result of the previous frame; match the filtered lane line segmentation result of the current frame with the restored result of the lane line segmentation result of the previous frame to obtain the lane line matching result.

[0103] In some embodiments of the present application, the compensation unit 340 is specifically used to: generate candidate lane line pixel points of the current frame based on the lane line matching results; merge the candidate lane line pixel points of the current frame with the filtered lane line segmentation results of the current frame to obtain the compensated lane line segmentation results of the current frame.

[0104] It can be understood that the above-mentioned lane line segmentation optimization device can implement the various steps of the lane line segmentation optimization method provided in the aforementioned embodiment, and the relevant explanations about the lane line segmentation optimization method are applicable to the lane line segmentation optimization device and will not be repeated here.

[0105] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 4At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0106] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0107] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0108] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a lane segmentation optimization device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0109] Acquire a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame;

[0110] Obtaining the lane line segmentation result of the previous frame, and filtering the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame, to obtain a filtered lane line segmentation result of the current frame;

[0111] Determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result;

[0112] The lane line segmentation result of the filtered current frame is compensated according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

[0113] The above application Figure 1 The method performed by the lane line segmentation optimization device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0114] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the lane segmentation optimization device in the illustrated embodiment is specifically used to perform:

[0115] Acquire a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame;

[0116] Obtaining the lane line segmentation result of the previous frame, and filtering the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame, to obtain a filtered lane line segmentation result of the current frame;

[0117] Determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result;

[0118] The lane line segmentation result of the filtered current frame is compensated according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

[0119] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0125] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0128] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A lane segmentation optimization method, wherein: The lane line segmentation optimization method comprises: Acquire a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame; Obtaining the lane line segmentation result of the previous frame, and filtering the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame, to obtain a filtered lane line segmentation result of the current frame; Determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result; The lane line segmentation result of the filtered current frame is compensated according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

2. The lane segmentation optimization method according to claim 1, wherein: The lane line segmentation result includes a lane line semantic segmentation result and a lane line instance segmentation result. The lane line segmentation result of the current frame is filtered using a preset filtering strategy according to the lane line segmentation result of the previous frame, and the filtered lane line segmentation result of the current frame includes: According to the lane line semantic segmentation result of the previous frame, the lane line semantic segmentation result of the current frame is filtered using a first preset filtering strategy to obtain a filtering result of the lane line semantic segmentation result of the current frame; Based on the filtering result of the lane line semantic segmentation result of the current frame, the lane line instance segmentation result of the current frame is filtered using a second preset filtering strategy to obtain the filtering result of the lane line instance segmentation result of the current frame.

3. The lane segmentation optimization method according to claim 2, wherein: The filtering result of the lane line semantic segmentation result of the current frame by using a first preset filtering strategy according to the lane line semantic segmentation result of the previous frame to obtain the lane line semantic segmentation result of the current frame includes: Projecting the lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame to the vehicle body coordinate system to obtain the projected lane line semantic segmentation result of the previous frame and the lane line semantic segmentation result of the current frame; Matching the projected lane line semantic segmentation result of the previous frame with the lane line semantic segmentation result of the current frame to obtain a matching result of the lane line semantic segmentation result; Determine the lane line semantic segmentation length of the current frame according to the lane line semantic segmentation result of the current frame; According to the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame, the lane line semantic segmentation result of the current frame is filtered to obtain a filtering result of the lane line semantic segmentation result of the current frame.

4. The lane segmentation optimization method according to claim 3, wherein: The filtering of the lane line semantic segmentation result of the current frame according to the matching result of the lane line semantic segmentation result and the lane line semantic segmentation length of the current frame to obtain the filtering result of the lane line semantic segmentation result of the current frame includes: If there are lane line pixels that are not successfully matched in the current frame in the matching result of the lane line semantic segmentation result, the lane line pixels that are not successfully matched in the current frame are filtered out from the lane line semantic segmentation result of the current frame; If the lane line semantic segmentation length of the current frame is less than the preset segmentation length threshold, the lane line pixels whose lane line semantic segmentation length of the current frame is less than the preset segmentation length threshold are filtered out from the lane line semantic segmentation result of the current frame.

5. The lane segmentation optimization method according to claim 2, wherein: The filtering result based on the lane line semantic segmentation result of the current frame, using the second preset filtering strategy to filter the lane line instance segmentation result of the current frame, to obtain the filtering result of the lane line instance segmentation result of the current frame includes: Filtering the lane line instance segmentation result of the current frame based on the filtering result of the lane line semantic segmentation result of the current frame to obtain a first filtering result of the lane line instance segmentation result of the current frame; Clustering the first filtered result of the lane line instance segmentation result of the current frame using a preset clustering algorithm to obtain a clustering result of the lane line instance segmentation result of the current frame; Determine the lane line instance segmentation length of the current frame according to the lane line instance segmentation result of the current frame; According to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame, the first filtering result of the lane line instance segmentation result of the current frame is filtered to obtain the second filtering result of the lane line instance segmentation result of the current frame.

6. The lane segmentation optimization method according to claim 5, wherein: The clustering result includes the number of cluster core points. The first filtering result of the lane line instance segmentation result of the current frame is filtered according to the clustering result of the lane line instance segmentation result of the current frame and the lane line instance segmentation length of the current frame to obtain the second filtering result of the lane line instance segmentation result of the current frame, including: If the number of cluster core points corresponding to the lane line instance segmentation result of the current frame is less than a preset number threshold, the lane line instance segmentation result with the number of cluster core points less than the preset number threshold is filtered out from the lane line instance segmentation result of the current frame; If the lane line instance segmentation length of the current frame is less than a preset instance length threshold, lane line pixels whose lane line instance segmentation length is less than the preset instance length threshold are filtered out from the lane line instance segmentation result of the current frame.

7. The lane segmentation optimization method according to claim 1, wherein: The lane line segmentation result of the filtered current frame is matched with the lane line segmentation result of the previous frame according to the positioning displacement to obtain the lane line matching result, which includes: Based on the filtered lane line segmentation result of the current frame and the positioning displacement, the lane line segmentation result of the previous frame is restored to obtain a restored result of the lane line segmentation result of the previous frame; The lane line segmentation result of the filtered current frame is matched with the restored result of the lane line segmentation result of the previous frame to obtain the lane line matching result.

8. The lane segmentation optimization method according to claim 1, wherein: The step of compensating the filtered lane line segmentation result of the current frame according to the lane line matching result to obtain the compensated lane line segmentation result of the current frame includes: Generate candidate lane line pixel points of the current frame according to the lane line matching result; The candidate lane line pixel points of the current frame are combined with the filtered lane line segmentation result of the current frame to obtain the compensated lane line segmentation result of the current frame.

9. A lane segmentation optimization device, wherein: The lane line segmentation optimization device comprises: A segmentation unit is used to obtain a current frame road image collected by the autonomous driving vehicle, and perform lane line segmentation on the current frame road image using a lane line segmentation model to obtain a lane line segmentation result of the current frame; A filtering unit, used to obtain the lane line segmentation result of the previous frame, and filter the lane line segmentation result of the current frame using a preset filtering strategy according to the lane line segmentation result of the previous frame to obtain a filtered lane line segmentation result of the current frame; A matching unit, used to determine the positioning displacement generated by the autonomous driving vehicle in the current frame and the previous frame, and match the filtered lane line segmentation result of the current frame with the lane line segmentation result of the previous frame according to the positioning displacement to obtain a lane line matching result; A compensation unit is used to compensate the lane line segmentation result of the filtered current frame according to the lane line matching result to obtain a compensated lane line segmentation result of the current frame.

10. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor performs the lane line segmentation optimization method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the lane line segmentation optimization method according to any one of claims 1 to 8 is implemented.

Images