Drivable area detection method based on semantic segmentation

Abstract

The present invention discloses a method for detecting a drivable area based on semantic segmentation, and the steps are as follows: 1) regional segmentation: segmenting the input original image to obtain a segmentation result image containing a road surface label, a moving target label, an obstacle label and a background label; 2) regional pixel scanning: scanning the segmented image, scanning column by column, and finding the key point pairs of each column, that is, the paired road edge points in the pixel column; 3) key point extraction: after regional scanning, one or more pairs of key points are obtained to determine the specific drivable area range; 4) back-projection transformation: the drivable area information in the pixel coordinate system is restored to the vehicle coordinate system by back-projection transformation to obtain a drivable area image based on semantic segmentation. The present invention forms a key point pair after scanning a specific area, and then screens out the final drivable area boundary, effectively eliminating the road blank noise introduced by the detection.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile intelligent driving exterior visual perception application, and in particular to a drivable area detection method based on semantic segmentation. Background Art

[0002] In recent years, technologies such as autonomous driving and intelligent assisted driving have gradually become research hotspots in the automotive industry. To develop such technologies, it is inseparable from the auxiliary support of driving scene and vehicle exterior environment perception technology. The perception of the exterior environment mainly relies on two types of sensors: radar and camera. Compared with radar, cameras have the advantages of low cost and rich detectable information. In the field of ADAS visual perception, the detection of static scene information includes lane lines, traffic signs, road signs and drivable areas. Among them, the drivable area provides a reference driving area for intelligent driving control decisions to avoid various obstacles, which is one of the important road environment information.

[0003] In the prior art, there are corresponding technical solutions for the detection of drivable areas, such as CN202010726575.7, Method for determining drivable areas, intelligent driving systems and intelligent vehicles, which need to rely on multi-sensor collaboration. Its sensing devices include cameras, infrared sensors, chemical detectors, microphones, lidars and related speed and angle sensors, etc., as well as positioning devices such as global positioning systems (GPS) and inertial navigation systems (INS). After the intelligent driving system detects the current drivable area, it queries the historical drivable area and superimposes the two according to certain rules to obtain the current final drivable area. The hardware system of this solution is complex, the cost is relatively high, and it is subject to certain restrictions in application.

[0004] Another solution of the prior art, such as CN202011478180.6, a method and device for detecting a drivable area, uses a neural network model to infer the pictures collected by a monocular camera, detects the image information to obtain the segmented area blocks, and then constructs the pixel contour of the drivable area. This solution focuses on the structural design of the neural network and does not involve how to obtain the pixel edges of the specific drivable area. If there is a certain error in the result of the network detection, it will affect the final drivable area detection accuracy. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a drivable area detection method based on semantic segmentation, which is suitable for the scene of external environment perception of a monocular camera and can extract the drivable area information of the road on the image after semantic segmentation; further, it can also solve the problem of inaccurate detection of image areas after segmentation (for example, detecting holes in the road surface) and the like.

[0006] The technical solution of the present invention is achieved in this way:

[0007] The method for detecting a drivable area based on semantic segmentation is characterized by comprising the following steps:

[0008] 1) Region segmentation: Perform semantic segmentation based on deep learning or image threshold segmentation based on traditional algorithms on the input original image to obtain a segmentation result image containing road surface labels, moving target labels, obstacle labels, and background labels;

[0009] 2) Regional pixel scanning: Scan the segmented image column by column to find the key point pairs in each column, that is, the paired road edge points in the pixel column;

[0010] 3) Key point extraction: After scanning the area, one or more pairs of key points are obtained to determine the specific drivable area;

[0011] 4) Back-projection transformation: The drivable area information in the pixel coordinate system is restored to the vehicle coordinate system through back-projection transformation to obtain a drivable area image based on semantic segmentation.

[0012] In this way, the present invention forms key point pairs after scanning a specific area, and then screens out the final drivable area boundary, effectively eliminating the blank road noise introduced by detection; on the premise of classifying and labeling obstacles, the method can give the dynamic and static attributes of the drivable area with only one traversal, providing more references for subsequent control decisions.

[0013] Further: it also includes step 5) testing data, and performing actual road driving detection on the drivable area image obtained in step 4). The present invention screens out the final drivable area boundary, and then performs actual road driving detection, which can be continuously updated and iterated.

[0014] Further: the obstacles of the obstacle label mainly include road posts, cones, water barriers, and fences.

[0015] Further: in the scanning operation of step 2), first, a scanning stop line is set on the segmented image; then two loops are used to traverse the lower half of the image, first the columns, then the rows, and so on. Setting a scanning stop line first can scan only the area below the scanning stop line, saving scanning time and improving efficiency.

[0016] Further: Before starting to scan each column, it is necessary to initialize all the top and bottom values ​​of the key point pair coordinate cache array of the column to 0, and set a flag label with an initial value of 1; then start scanning from bottom to top, set the flag to 0 when encountering a road marker, and record the pixel coordinates of the bottom point in the cache array; segment the image to identify the moving target label, and also record this attribute; when encountering a background label, obstacle label or stopping the scan line, set the flag to 1, and record the position coordinates of the pixel point in the next row of the same column of the point in the top cache array. At this point, a key point pair has been formed, and the number of key point pair caches needs to be increased by 1. In this way, the key point pairs of each column can be obtained faster and more accurately when scanning again.

[0017] Further: the position of the scanning stop line is half of the image pixel height, or the position of the image principal point, or a few more lines below the principal point to ensure the maximum distance requirement. Further narrowing the area of ​​the scanning stop line can scan only the area below the scanning stop line, saving scanning time and improving efficiency.

[0018] Further: After scanning each column, first determine whether the number of key point pairs in the current column is greater than 0; if the number of key point pairs is equal to 0, it means that there is no boundary point of the drivable area in this column; if the number of key point pairs is equal to 1, take the top point as the final output, and directly extract the top point of the key point pair as the pixel boundary of the drivable area; if the number of key point pairs is greater than 1, there are multiple pairs of key points. Explain and operate on one or more key point pairs that may exist after scanning.

[0019] Further: If there are multiple pairs of key points, at this time, if the road isolation belt or the railings and green belts in the middle of the road are segmented and labeled, it can be judged that there are void areas on the road surface due to detection errors, and the top point coordinates of the last pair of key points at the location of the isolation belt or the railings and green belts in the middle of the road should be output; if the isolation belt or the railings and green belts in the middle of the road are not separately segmented, it is necessary to judge whether the road void at this time is noise or isolation belt based on pixel distance. All holes in the column can be traversed first to find the maximum hole length, and the maximum value is judged. If the value does not exceed the set self-set threshold, all holes can be considered as detection errors, and the top point of the topmost key point pair is output as the pixel boundary of the drivable area; if the maximum hole length is greater than the set self-set threshold, it means that this is the central isolation belt of the road, and the top point of the key point pair below the isolation belt needs to be output.

[0020] Further: when traversing all holes, the top and bottom points on both sides are transformed into the vehicle coordinate system through back projection transformation, and then the absolute value of the difference between the longitudinal coordinates is taken and compared with the self-set threshold.

[0021] Further: in step 4), the road point corresponding to the midpoint of the rear axle of the vehicle is selected as the origin of the vehicle coordinate system, and the coordinate transformation adopts the ground point method to establish the correspondence between the road pixel coordinates and the vehicle physical coordinates. In this way, the correspondence between the road pixel coordinates and the vehicle physical coordinates can be established with high precision through back projection transformation.

[0022] In summary, the present invention has the following beneficial effects:

[0023] 1. The detection scheme proposed in the present invention can be realized by only relying on a monocular camera in terms of equipment, and has the advantages of simple operation and low cost. In terms of algorithm, the scheme processes the image after regional segmentation, but has no restrictions on the segmentation method, and has a certain degree of universality.

[0024] 2. The present invention forms key point pairs after scanning a specific area, and then screens out the final drivable area boundary, effectively eliminating the blank road noise introduced by detection; on the premise of classifying and labeling obstacles, the method can give the dynamic and static attributes of the drivable area with only one traversal, providing more references for subsequent control decisions.

[0025] 3. The drivable area scanning method proposed in the present invention is very flexible. If the resolution of the input scene image is high, in order to save running time and improve efficiency, the result image can be downsampled and then scanned. The pixel boundary of the drivable area is then restored to the corresponding position of the original image, and then a back-projection transformation is performed to obtain the drivable area in the world coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall framework flow chart of the present invention;

[0027] Figure 2 It is a comparison diagram of the regional segmentation example of the present invention;

[0028] Figure 3 It is a diagram illustrating the scanning process of the present invention;

[0029] Figure 4 2 is a comparative diagram of examples of results obtained by the treatment method of the present invention. DETAILED DESCRIPTION

[0030] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0031] See also Figure 1 — Figure 4 , Figure 1The overall process framework diagram of the drivable area detection method based on semantic segmentation of the present invention is briefly shown; the drivable area detection method based on semantic segmentation of the present invention mainly includes the following steps: image region segmentation, region pixel scanning, key point pair processing and back projection transformation, wherein region segmentation and back projection transformation can select other algorithms according to needs, and the implementation process of each step is described in detail below:

[0032] Step 1) Region segmentation: Perform semantic segmentation based on deep learning or image threshold segmentation based on traditional algorithms on the input original image to obtain a segmentation result image containing road surface labels, moving target labels (moving objects such as vehicles and pedestrians), obstacle labels (mainly such as road posts, cones, water barriers, fences, etc.) and background labels;

[0033] The input original image is subjected to semantic segmentation based on deep learning or image threshold segmentation based on traditional algorithms to obtain a segmentation result image containing road surface labels, vehicle and pedestrian moving target labels, road stakes, cones, water barriers, fences and other obstacle labels, and background labels. Figure 2 An example of semantic segmentation is shown, with a comparison of the original image and the image after region segmentation. The number of classification labels can be appropriately increased or decreased according to different usage requirements. If there is no requirement for dynamic and static separation in the drivable area, moving targets such as pedestrians and vehicles can be classified as background labels.

[0034] Step 2) Regional pixel scanning: Scan the segmented image column by column to find the key point pairs in each column, that is, the paired road edge points in the pixel column, so as to obtain the specific drivable area range;

[0035] First, set a scan stop line on the segmented image. The scan stop line can be half the pixel height of the image, or the position of the main point of the image, or a few more rows below the main point to ensure the maximum distance requirement. Then use two loops to traverse the lower half of the image, first the columns, then the rows, and so on. Before starting the scan of each column, you need to initialize all the values ​​of the top and bottom coordinate cache arrays of the key point pair of the column to 0, and set a flag label with an initial value of 1. Then start scanning from bottom to top, such as Figure 3As shown, when encountering a road marker (or road markers such as lane lines, curbs, and road signs), set the flag to 0 and record the pixel coordinates of the bottom point in the cache array. If the segmented image recognizes a moving target label, this attribute is also recorded (the same below); when encountering a background label, an obstacle label, or stopping the scan line, set the flag to 1, and record the position coordinates of the pixel point in the next row of the same column of the point in the top cache array. At this point, a key point pair has been formed, and the key point pair cache number needs to be increased by 1. After the inner loop exits, it means that the column has been completely scanned. At this time, you can proceed to the next step to filter the key points and extract the pixel boundaries of the drivable area.

[0036] Step 3) Extraction of key points: After scanning the area, one or more pairs of key points are obtained to determine the specific range of the drivable area;

[0037] After scanning each column, first determine whether the number of key point pairs in the current column is greater than 0. If the number of key point pairs is equal to 0, it means that there is no boundary point of the drivable area in this column; if the number of key point pairs is equal to 1, take the top point as the final output, and directly extract the top point of the key point pair as the pixel boundary of the drivable area; if the number of key point pairs is greater than 1, that is, there are multiple pairs of key points, at this time, if the isolation belt of the road or the railings and green belts in the middle of the road are segmented and labeled, it can be judged that there are errors in the detection, resulting in a hole area (hole noise) on the road surface, and the top point coordinates of the last pair of key points at the isolation belt or the railings and green belts in the middle of the road should be output. If no separate segmentation labels are made for the isolation belt or the railings and green belts in the middle of the road, this situation requires special processing. It is necessary to determine whether the road hole at this time is noise or isolation belt based on the pixel distance. You can first traverse all the holes in the column, find the maximum hole length, and judge the maximum value. If the value does not exceed the set value (self-set threshold), all holes can be considered as detection errors, and the top point of the topmost key point pair is output as the pixel boundary of the drivable area. If the maximum hole length is greater than the set value (self-set threshold), it means that it is very likely to be the isolation belt in the middle of the road, and the top point of the key point pair below the isolation belt needs to be output. The specific data selection of the self-set threshold can refer to the actual size of the isolation belt in the vehicle coordinate system, such as a specific threshold set to be 1.2 meters. The specific method is to traverse all holes, and transform the top and bottom points on both sides of them into the vehicle coordinate system through back projection transformation, and then take the absolute value of the difference of the longitudinal coordinates, and compare it with the self-set threshold. If the difference is greater than the self-set threshold, it means that there is an isolation belt in the current scene, and the pixel boundary of the drivable area is the top point of the current point pair. If the difference is less than the self-set threshold, it means that the hole is detection noise, and the pixel boundary of the drivable area is the top point of the last pair of key points in the column. If the number of holes in a column is greater than 1, you can first traverse the largest hole, repeat the above judgment for the top and bottom points of the hole, and you can get the final drivable area.

[0038] Step 4) Back-projection transformation: restore the pixel position of the drivable area information in the pixel coordinate system to the vehicle coordinate system (world coordinate system) through back-projection transformation to obtain the drivable area based on semantic segmentation;

[0039] The pixel coordinate points are mapped to the vehicle coordinate system through back-projection transformation. This application selects the road surface point corresponding to the midpoint of the rear axle of the vehicle as the origin of the vehicle coordinate system (world coordinate system). Because it is monocular vision, the coordinate transformation adopts the grounding point method to establish the correspondence between the road surface pixel coordinates and the vehicle physical coordinates. The actual coordinate conversion accuracy depends on the calibration of the camera and the calculation of real-time dynamic external parameters. During the transformation process, an appropriate distance threshold can be set for screening. After the coordinate transformation, if the longitudinal distance is less than the lower limit of the distance threshold or greater than the self-set upper limit, these points are eliminated according to the distance, such as eliminating farther points greater than 120 meters, so that the final fan-shaped area is smoother.

[0040] Step 5) Test data: Perform actual road driving test on the drivable area image obtained in step 4);

[0041] In the actual road driving process, the vehicle obtains the external environment perception scene through the monocular camera, and the method of the present invention is used for processing to obtain a better effect; the vehicle can extract the drivable area information of the road on the image after semantic segmentation through the monocular camera external environment perception scene, thereby improving the safety of intelligent driving. The present invention can solve the problem of inaccurate detection of the segmented image area (for example, detecting holes in the road surface), etc.

[0042] Finally, it should be noted that the above examples of the present invention are merely examples for illustrating the present invention, and are not intended to limit the embodiments of the present invention. Although the applicant has described the present invention in detail with reference to the preferred embodiments, for those of ordinary skill in the art, other different forms of changes and modifications can be made based on the above description. It is impossible to list all the embodiments here. Any obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. The drivable area detection method based on semantic segmentation is characterized by The steps include: 1) Region segmentation: Perform semantic segmentation based on deep learning or image threshold segmentation based on traditional algorithms on the input original image to obtain a segmentation result image containing road surface labels, moving target labels, obstacle labels, and background labels; 2) Regional pixel scanning: Scan the segmented image column by column to find the key point pairs in each column, that is, the paired road edge points in the pixel column; Before starting to scan each column, initialize all the top and bottom values ​​of the key point pair coordinate cache array of the column to 0, and set a flag label with an initial value of 1; then start scanning from bottom to top, set the flag to 0 when encountering a road marker, and record the pixel coordinates of the bottom point in the cache array; segment the image to identify the moving target label, and also record this attribute; when encountering a background label, obstacle label or stopping the scan line, set the flag to 1, and record the position coordinates of the pixel point in the next row of the same column of the point into the top cache array. At this point, a key point pair has been formed, and the number of key point pair caches needs to be increased by 1 Key point extraction: After scanning the area, one or more pairs of key points are obtained to determine the specific range of the drivable area; If there are multiple pairs of key points, at this time, if the road isolation belt or the railings and green belts in the middle of the road are segmented and labeled, it can be judged that there are void areas on the road surface due to errors in detection, and the top point coordinates of the last pair of key points at the location of the isolation belt or the railings and green belts in the middle of the road should be output; if the isolation belt or the railings and green belts in the middle of the road are not separately segmented, it is necessary to judge whether the road void at this time is noise or isolation belt based on pixel distance. All holes in the column can be traversed first to find the maximum value of the hole length, and the maximum value is judged. If the value does not exceed the set self-set threshold, all holes can be considered as detection errors, and the top point of the topmost key point pair is output as the pixel boundary of the drivable area; if the maximum value of the hole length is greater than the set self-set threshold, it means that the place is the isolation belt in the middle of the road, and the top point of the key point pair below the isolation belt needs to be output; 4) Back-projection transformation: The drivable area information in the pixel coordinate system is restored to the vehicle coordinate system through back-projection transformation to obtain a drivable area image based on semantic segmentation.

2. The method for detecting a drivable area based on semantic segmentation according to claim 1, characterized in that: The method also includes the test data of step 5), and performs actual road driving detection on the drivable area image obtained in step 4).

3. The method for detecting a drivable area based on semantic segmentation according to claim 1, characterized in that: The obstacles of the obstacle label include road posts, cones, water barriers, and fences.

4. The method for detecting a drivable area based on semantic segmentation according to claim 1, characterized in that: In the scanning operation of step 2), first, a scanning stop line is set on the segmented image; then two loops are used to traverse the lower half of the image, first the columns and then the rows, and so on, scanning in sequence by columns.

5. The method for detecting a drivable area based on semantic segmentation according to claim 4, characterized in that: The position of the scanning stop line is half of the image pixel height, or the position of the image principal point, or a few more lines below the principal point to ensure the maximum distance requirement.

6. The method for detecting a drivable area based on semantic segmentation according to any one of claims 1 to 5, characterized in that: After scanning each column, first determine whether the number of key point pairs in the current column is greater than 0; if the number of key point pairs is equal to 0, it means that there is no boundary point of the drivable area in this column; If the number of key point pairs is equal to 1, take the top point as the final output, and directly extract the top point of the key point pair as the pixel boundary of the drivable area; if the number of key point pairs is greater than 1, there are multiple pairs of key points.

7. The method for detecting a drivable area based on semantic segmentation according to claim 6, characterized in that: When traversing all holes, the top and bottom points on both sides are transformed into the vehicle coordinate system through back projection transformation, and then the absolute value of the difference between the longitudinal coordinates is taken and compared with the self-set threshold.

8. The method for detecting a drivable area based on semantic segmentation according to any one of claims 1 to 5, characterized in that: In step 4), the road surface point corresponding to the midpoint of the rear axle of the vehicle is selected as the origin of the vehicle coordinate system, and the coordinate transformation adopts the ground point method to establish the corresponding relationship between the road surface pixel coordinates and the vehicle physical coordinates.

Images