Lane dividing device and method

Abstract

The embodiment of the invention provides a lane division device and method, and the device comprises an extraction unit which extracts a road surface region and a lane line region from an input image based on a semantic segmentation model; a first determination unit that determines the contour of the road surface region based on the binarized image of the road surface region; a second determination unit that determines the contour of the lane line based on the binarized image of the lane line region; and a dividing unit that divides the lane according to the contour of the road surface region and the contour of the lane line.

Description

Technical Field

[0001] The present application relates to the field of image processing. Background Art

[0002] Video surveillance technology has provided great help to road traffic management. By analyzing video surveillance images, traffic safety hazards can be discovered in advance, traffic accidents can be discovered in time, and the pressure of road traffic management can be reduced. For example, by analyzing surveillance images, the upward and downward directions of the lane and the position and travel direction of the vehicle in the lane can be determined to determine whether the vehicle is traveling in the wrong direction.

[0003] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of this document. Summary of the invention

[0004] The inventors found that the prior art usually determines the upward and downward directions of the lane by extracting optical flow vectors from an image. However, the optical flow vector method can only be applied to situations where there are vehicles moving on the road. If there are no vehicles moving on the road, the optical flow mask cannot be extracted, and the direction of the lane cannot be determined. In addition, the current semantic segmentation model has inaccurate segmentation under conditions such as poor lighting, weather or road conditions, resulting in incomplete extraction of the road area, resulting in inaccurate lane division.

[0005] In response to at least one of the above problems, an embodiment of the present application provides a lane division device and method.

[0006] According to a first aspect of an embodiment of the present application, a lane dividing device is provided, the device comprising:

[0007] An extraction unit, which extracts a road surface area and a lane line area from an input image based on a semantic segmentation model;

[0008] a first determining unit, which determines a contour of the road surface area based on a binarized image of the road surface area;

[0009] A second determining unit, which determines the contour of the lane line based on the binary image of the lane line area;

[0010] A dividing unit divides lanes according to the contour of the road surface area and the contour of the lane line.

[0011] According to a second aspect of an embodiment of the present application, a lane division method is provided, the method comprising:

[0012] Extract road surface areas and lane line areas from input images based on semantic segmentation models;

[0013] determining a contour of the road surface area based on a binary image of the road surface area;

[0014] Determine the outline of the lane line based on the binary image of the lane line area;

[0015] Lanes are divided according to the contours of the road surface area and the contours of the lane lines.

[0016] One of the beneficial effects of the embodiments of the present application is that lane division is achieved by identifying the outline of the road surface area and the outline of the lane line in the image, and using the outline of the lane line to divide the area enclosed by the outline of the road surface area. Even if there is no vehicle movement on the road surface and the optical flow mask cannot be extracted, the lanes can still be accurately divided.

[0017] With reference to the following description and accompanying drawings, the specific embodiments of the present application are disclosed in detail, indicating the way in which the principles of the present application can be adopted. It should be understood that the embodiments of the present application are not limited in scope. Within the scope of the spirit and clauses of the appended claims, the embodiments of the present application include many changes, modifications and equivalents.

[0018] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0019] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 is a schematic diagram of a lane division method according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of labels used for semantic segmentation in an embodiment of the present application;

[0023] Figure 3 is another schematic diagram of the lane division method according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of extracting the contour of a road surface area in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of checking the vertices of a polygonal outline according to an embodiment of the present application;

[0026] Figure 6 is another schematic diagram of checking the vertices of a polygonal outline according to an embodiment of the present application;

[0027] Figure 7 is another schematic diagram of checking the vertices of a polygonal outline according to an embodiment of the present application;

[0028] Figure 8 is a flow chart of checking polygon outline vertices according to an embodiment of the present application;

[0029] Fig. 9 is another schematic diagram of checking the vertices of a polygonal outline according to an embodiment of the present application;

[0030] Fig.10 is another schematic diagram of the lane division method according to an embodiment of the present application;

[0031] Fig.11 is a schematic diagram of extracting the outline of a lane line in an embodiment of the present application;

[0032] Fig.12 is another schematic diagram of the lane division method according to an embodiment of the present application;

[0033] Fig.13 is a schematic diagram of dividing lane areas according to an embodiment of the present application;

[0034] Fig.14 is a schematic diagram of a lane dividing device according to an embodiment of the present application;

[0035] Fig.15 is a schematic diagram of an extraction unit according to an embodiment of the present application;

[0036] Fig.16 is a schematic diagram of a first determination unit in an embodiment of the present application;

[0037] Fig.17 is a schematic diagram of a second determination unit in an embodiment of the present application;

[0038] Fig.18 is a schematic diagram of a division unit in an embodiment of the present application;

[0039] Fig.19 is a schematic diagram of a lane dividing line unit according to an embodiment of the present application;

[0040] Fig. 20 A schematic diagram of an electronic device according to an embodiment of the present application;

[0041] Fig.21 It is a schematic block diagram of the system structure of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0042] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or temporal order of these elements, etc., and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0043] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0044] With reference to the accompanying drawings, the above and other features of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the present application includes all modifications, variations and equivalents falling within the scope of the attached claims.

[0045] Embodiments of the first aspect

[0046] An embodiment of the present application provides a lane division method. Figure 1 Schematic diagram of a lane division method according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0047] 101: Extract road surface area and lane line area from input image based on semantic segmentation model;

[0048] 102: Determine a contour of the road surface area based on the binary image of the road surface area;

[0049] 103: Determine the contour of the lane line based on the binary image of the lane line area;

[0050] 104: Divide lanes according to the contours of the road surface area and the contours of the lane lines.

[0051] It can be seen from the above embodiments that lane division is achieved by identifying the outline of the road surface area and the outline of the lane line in the image, and using the outline of the lane line to divide the area enclosed by the outline of the road surface area. Even if the optical flow mask extraction is incomplete due to light, weather and other conditions, the lanes can be accurately divided.

[0052] In an embodiment of the present application, the input image may be obtained from a camera device that photographs the area where the road is located, for example, by a surveillance camera installed above the road.

[0053] In addition, the method of the embodiment of the present application can process at least one frame of the surveillance video captured by the surveillance camera, for example, process the interval frames of the surveillance video.

[0054] In the embodiment of the present application, the interval frames refer to multiple frames with at least one frame between each other, that is, non-continuous frames, for example, semantic segmentation is performed every 10 frames. In this way, moving objects and non-moving objects can be clearly distinguished, which is conducive to maintaining the integrity of the extracted road area.

[0055] In the embodiment of the present application, the semantic segmentation model can be a model based on a deep learning network, for example, semantic segmentation is performed based on a DeepLab V3 network. The specific implementation method can refer to the relevant technology, and the embodiment of the present application is not limited to this.

[0056] In some embodiments, operation 101 may include:

[0057] Extracting a road surface area corresponding to a first label from the input image based on a semantic segmentation model, where the first label includes at least one of a road, a fence, a moving object, a solid line, a dotted line, and a guide line;

[0058] A lane line area corresponding to a second label is extracted from the input image based on a semantic segmentation model, where the second label includes at least one of a dotted line, a solid line, a guide line, and a fence.

[0059] Figure 2 is a schematic diagram of labels used for semantic segmentation in an embodiment of the present application.

[0060] In the embodiments of the present application, Figure 2As shown in FIG. 1 , the labels used for semantic segmentation may include road, fence, vegetation, person, bicycle, solid line, sidewalk, pole, terrain, car, water, guide line, building, traffic sign, sky, truck, dot-line, background, etc., wherein the labels corresponding to the road surface area may be roads, fences, vehicles and other moving bodies, solid lines, dot-lines and guide lines, etc., and the road surface area may be obtained by summarizing the areas corresponding to these labels, and the labels corresponding to the lane line area may be dot-lines, solid lines, guide lines and fences, etc., and the lane line area may be obtained by summarizing the areas corresponding to these labels.

[0061] In some embodiments, in order to avoid vehicles blocking the road and leaving holes in the middle of the road mask, the type of the pixel can be determined for the label type of the pixel at the same position in multiple segmentation results, so as to complete the area belonging to the road. For example, considering the mobility of the vehicle, the semantic segmentation results of every 10 frames of input images can be obtained, and multiple semantic segmentation results can be obtained, such as obtaining 30 segmentation results corresponding to 30 interval frames, and then the semantic segmentation results are scanned pixel by pixel, and the type of the pixel is determined for all pixels in these 30 frames one by one, and the type with the highest frequency in the semantic segmentation results of the pixel is used as the type of the pixel. For example, for the 30 frames, the type of each pixel in each frame is determined, for example, the pixel belongs to the road, or the pixel belongs to the vehicle. For a certain pixel, the number of times it is divided into belonging to the road in the 30 semantic segmentation results is 25 times, and the number of times it is divided into belonging to the vehicle is 5 times, then the type of the pixel is determined as the road. In this way, the area belonging to the road can be completed, so as to obtain a complete road surface area.

[0062] Figure 3 It is another schematic diagram of the lane dividing method according to an embodiment of the present application.

[0063] In some embodiments, Figure 3 As shown, operation 102 may include:

[0064] 1021: Binarize the input image based on the road surface area to obtain a binary image of the road surface area;

[0065] 1022: Searching for the outermost circle of the binary image of the road surface area to obtain a preliminary contour of the road surface area;

[0066] 1023: converting the preliminary contour of the road surface area to obtain a polygonal contour;

[0067] 1024: for each vertex of the polygonal outline, determine whether to retain the vertex according to a positional relationship between the vertex and adjacent vertices;

[0068] 1025: Determine the polygonal outline enclosed by the retained vertices as the outline of the road surface area.

[0069] Figure 4 It is a schematic diagram of extracting the contour of a road surface area in an embodiment of the present application.

[0070] In the embodiment of the present application, the binarization process refers to setting the pixel values ​​of some pixels to 255 and setting the pixel values ​​of other pixels except these pixels to 0. Figure 4 As shown in (a), the pixel values ​​of the road surface area are set to 255, and the pixel values ​​of other areas are set to 0, thereby obtaining a binary image of the road surface area. Sometimes the "binarized image" is also called a "binarized mask."

[0071] In operation 1022, the outermost circle in the binary image is searched while ignoring the inner contour contained in the outer contour, thereby roughly determining the contour of the road surface area. In addition, small contours can also be removed based on the number and size of pixels in the contour to determine the preliminary contour of the lane area.

[0072] In operation 1023, the "conversion process" may include at least one of morphological transformation, smoothing, and polygon fitting. For example, the morphological transformation includes opening the preliminary contour of the road surface area to remove noise, and closing the preliminary contour of the road surface area to fill small holes in the road surface area; the smoothing process includes average blurring or Gaussian blurring of the preliminary contour of the road surface area; polygon fitting, for example, decomposes the continuous smooth contour curve into straight lines, and the contour points of the image can be regarded as point sets and polygon fitting, and then, polygon approximation is used to fit the specified point set, and the accuracy of the approximation is set by adjusting parameters. The implementation of the conversion process can refer to the relevant technology, and the embodiments of the present application are defined in this regard.

[0073] In some cases, there is a problem of incoherent road extraction due to poor lighting, so the road contour is irregular. Usually, in order to optimize the smoothness of the contour, some points need to be deleted. However, if the deleted points are some key points, the optimized area will be inconsistent with the original road area. In order to prevent this over-correction of the contour, the accuracy of road extraction can be improved by judging whether each triangular area formed by each vertex and its two adjacent vertices is a convex triangle or a concave triangle, and then deciding whether to retain the vertex based on the area of ​​the triangular area and a predetermined threshold.

[0074] For example, in operation 1024, when the vertex satisfies a first condition, the vertex is retained, wherein the first condition includes:

[0075] The triangle formed by the vertex and two adjacent vertices is a concave triangle, and the ratio of the area occupied by the road surface in the concave triangle to the area of ​​the concave triangle is less than a first threshold; or

[0076] The triangle formed by the vertex and two adjacent vertices is a convex triangle, and the ratio of the area occupied by the road surface in the convex triangle to the area of ​​the convex triangle is greater than a second threshold.

[0077] In the embodiment of the present application, a concave triangle refers to a triangular area formed by a vertex of a polygon and two vertices adjacent to the vertex. If the vertex is removed and the area of ​​the new polygon is increased compared to the area of ​​the original polygon after connecting the two adjacent vertices of the vertex, the triangular area corresponding to the vertex is a concave triangle. Otherwise, it is a convex triangle, that is, if the vertex is removed and the area of ​​the new polygon is reduced compared to the area of ​​the original polygon after connecting the two adjacent vertices of the vertex, the triangular area corresponding to the vertex is a convex triangle. In addition, the first threshold and the second threshold are not equal, and the specific values ​​can be set according to actual needs, which is not limited in the embodiment of the present application.

[0078] Figures 5 to 7 is a schematic diagram of checking the vertices of a polygonal outline according to an embodiment of the present application, Figure 5 It's the original picture. Figure 6 This is a diagram without vertex checking. Figure 7 This is a schematic diagram showing vertex checking.

[0079] Combine the following Figures 5 to 7 The method for optimizing the vertices of a polygonal outline according to an embodiment of the present application is described.

[0080] In the embodiments of the present application, Figure 5 The polygon outline shown in Figure 1 lists some vertices S1-S8. In some cases, to optimize the smoothness of the outline, points S1 and S2 may be deleted, resulting in Figure 6The optimized polygon outline shown, however Figure 6 The area enclosed by the polygonal outline shown does not correspond to the original road surface area.

[0081] In some embodiments, each vertex on the contour is checked at operation 1024 to determine whether to retain the vertex. Figure 5 As shown, for vertex S1, the two adjacent vertices of vertex S1 are S4 and S6, and the triangle surrounded by points S1, S4, and S6 is a convex triangle. The ratio of the road surface area in the convex triangle to the convex triangle is calculated. If the ratio is greater than the second threshold, the vertex is retained, otherwise the vertex is deleted. The second threshold is, for example, 0.7. In this example, if the ratio of the road surface area in the triangle surrounded by points S1, S4, and S6 to the area of ​​the triangle is greater than 0.7, then point S1 is retained. In addition, the road surface area in the triangle can be obtained based on the area of ​​the area corresponding to the first label in the triangle area.

[0082] For vertex S2, the two adjacent vertices of vertex S2 are S7 and S8, and the triangle surrounded by points S2, S7, and S8 is a concave triangle. The ratio of the road surface area in the concave triangle to the concave triangle is calculated. If the ratio is less than the first threshold, the vertex is retained, otherwise the vertex is deleted. The first threshold is, for example, 0.3. In this example, the ratio of the road surface area in the triangle surrounded by points S2, S7, and S8 to the area of ​​the triangle is less than 0.3, so point S2 is retained. In addition, the values ​​of the first threshold and the second threshold can be set according to actual needs, and the embodiments of the present application are not limited to this.

[0083] In addition, for vertex S4, the triangle enclosed by vertices S4, S1, and S3 is a concave triangle. In this example, the ratio of the road surface area in the concave triangle to the area of ​​the concave triangle is greater than 0.3, so point S4 is deleted.

[0084] In addition, for vertex S3, the triangle surrounded by vertices S3, S4, and S5 is a concave triangle. In this example, the ratio of the road surface area in the concave triangle to the area of ​​the concave triangle is greater than 0.3, so point S3 is deleted.

[0085] This process is repeated for other vertices until all vertices have been checked.

[0086] In the above example, after the above vertex check, we get Figure 7 The optimized polygon outline is shown.

[0087] Figure 8 4 is a flow chart of checking contour vertices according to an embodiment of the present application.

[0088] like Figure 8 As shown, the process may include:

[0089] 801: Start to optimize the contour of the road surface polygon;

[0090] 802: for a vertex, start to determine whether the triangle corresponding to the vertex is a concave triangle or a convex triangle, for example, assuming that the vertex is removed;

[0091] 803: Calculate whether the area of ​​the polygon increases after the vertex is removed. If yes, it is determined to be a concave triangle and the process goes to 804. If no, it is determined to be a convex triangle and the process goes to 808.

[0092] 804: Calculate the area of ​​a triangle formed by the vertex and adjacent vertices;

[0093] 805: Determine whether the ratio of the road surface area to the triangle area is less than 0.3, if "yes", proceed to 806, if "no", proceed to 807;

[0094] 806: Keep the vertex in the contour;

[0095] 807: Remove the vertex from the contour;

[0096] 808: Calculate the area of ​​the triangle formed by the vertex and adjacent vertices;

[0097] 809: Determine whether the ratio of the road surface area to the triangle area is greater than 0.7, if yes, proceed to 810, if no, proceed to 811;

[0098] 810: Keep the vertex in the contour;

[0099] 811: Remove the vertex from the contour;

[0100] 812: Output the corrected polygon outline.

[0101] The implementation manner of steps 801 to 812 can refer to the aforementioned embodiments and will not be described in detail here.

[0102] In addition, there may still be some irregular shapes in the optimized polygonal contour, such as small-angle spikes, short edges, sawtooth, etc., and further judgment can be made on the boundary vertices to eliminate these irregular shapes and make the contour smooth.

[0103] For example, in operation 1024, it is determined whether the vertex is an outermost boundary vertex; when the vertex is a vertex other than a boundary vertex and satisfies a second condition, the vertex is discarded, wherein the second condition includes at least one of the following:

[0104] The triangle formed by the vertex and two adjacent vertices is a convex triangle and the angle between the vertex and the line connecting the two adjacent vertices is less than a third threshold;

[0105] The distances between the vertex and two adjacent vertices are both less than a fourth threshold;

[0106] The triangle formed by the vertex and two adjacent vertices is a concave triangle, the angle between the vertex and the line connecting the two adjacent vertices is an acute angle, and at least one of the distances between the vertex and the two adjacent vertices is less than a fifth threshold.

[0107] Fig. 9 This is another schematic diagram of checking the vertices of a polygonal contour according to an embodiment of the present application.

[0108] like Fig. 9 As shown in (a), point c is not a boundary vertex, the triangle formed by vertex c and two adjacent vertices is a convex triangle, and the angle θ between vertex c and the line connecting the two adjacent vertices is less than a third threshold, the third threshold is, for example, 10°, and vertex c is removed.

[0109] like Fig. 9 As shown in (b), point c is not a boundary vertex. The distances between vertex c and its two adjacent vertices are dist1 and dist2 respectively. Both dist1 and dist2 are less than the fourth threshold, so vertex c is removed.

[0110] like Fig. 9 As shown in (c), point c is not a boundary vertex, the triangle formed by vertex c and its two adjacent vertices is a concave triangle, and the angle θ between vertex c and the line connecting its two adjacent vertices is an acute angle, and at least one of the distances between vertex c and its two adjacent vertices is less than the fifth threshold, for example, dist2 is less than the fifth threshold, and vertex c is removed.

[0111] In addition, in operation 1024, when the vertex is a boundary vertex and satisfies a third condition, the vertex is retained, wherein the third condition includes: a triangle formed by the vertex and two adjacent vertices is a convex triangle, and a triangle formed by a vertex adjacent to the vertex and two adjacent vertices is a concave triangle, and a distance between the vertex and the adjacent vertex is less than a sixth threshold. When the vertex is a boundary vertex and satisfies the third condition, the adjacent vertex is discarded.

[0112] For example, Fig. 9As shown in (d), point c is a boundary vertex, the triangle formed by vertex c and its two adjacent vertices is a convex triangle, and the triangle formed by a vertex p2 adjacent to vertex c and its two adjacent vertices is a concave triangle, and the distance dist2 between vertex c and the adjacent vertex p2 is less than the sixth threshold, vertex c is retained, and vertex p2 is removed. Alternatively, the following judgment can be made: point c is a boundary vertex, the angle between vertex c and the line connecting its two adjacent vertices is θ, the angle between vertex p2 adjacent to vertex c and the line connecting its two adjacent vertices is θ1, θ*θ1 is less than 0, and the distance dist2 between vertex c and the adjacent vertex p2 is less than the sixth threshold, vertex c is retained, and vertex p2 is removed.

[0113] The value of the above threshold can be set according to actual needs, and the embodiment of the present application does not limit this.

[0114] Fig.10 It is another schematic diagram of the lane dividing method according to an embodiment of the present application.

[0115] In some embodiments, Fig.10 As shown, operation 103 includes:

[0116] 1031: Binarize the input image based on the lane line region to obtain a binary image of the lane line region;

[0117] 1032: Based on the embedded vector corresponding to the lane line region obtained by the semantic segmentation model, clustering is performed on the lane line region of the binary image of the lane line region to obtain points on the contour of the lane line.

[0118] 1033: Connect the points to obtain the outline of the lane line.

[0119] Fig.11 It is a schematic diagram of extracting the outline of lane lines in an embodiment of the present application.

[0120] In the embodiment of the present application, the method of obtaining the binary image of the lane line area is similar to the method of obtaining the binary image of the road surface area, for example, Fig.11 As shown in (b), the pixel value of the lane line area is set to 255, and the pixel values ​​of other areas are set to 0, thereby obtaining a binary image of the lane line area.

[0121] In the embodiments of the present application, Fig.11 As shown, the input image (such as Fig.11 , for example, based on the LaneNet network model. The LaneNet network structure is divided into two branches, one is the semantic segmentation branch, through which the binary image is obtained, as shown in (a). Fig.11The other is the embedding branch, through which the pixels are represented by vectors to obtain the embedded vectors, as shown in (b). Fig.11 As shown in (c), the embedding branch and the segmentation branch, for example, share the first and second stages of ENet by applying the same stage operation to different output channels respectively. The segmentation branch has 2 channel outputs and the embedding branch has 4 channel output tensors. The results of the two branches are then clustered to obtain the result of instance segmentation. For example, the binary image is used as a mask to filter out the embedded vectors that are lane line points in the embedded vector, and then the embedded vectors of these filtered points are clustered until all lane line pixels are assigned to corresponding lanes.

[0122] In the embodiment of the present application, the clustering algorithm can adopt unsupervised clustering, such as the mean shift clustering algorithm and the Dbscan (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm, or supervised clustering, such as spectral clustering. The implementation method of the above clustering algorithm can refer to the relevant technology, and the embodiment of the present application does not limit this.

[0123] In addition, before clustering, the binary image of the lane line area can also be post-processed, for example, morphological transformation, smoothing, etc. For example, different lines are separated by opening and closing operations, and holes in the lane line are filled by blurring operations to smooth the lane line mask. Afterwards, the connected contours in the mask are searched and sorted to remove small contours. The implementation of the above post-processing can refer to the relevant technology, and the embodiments of the present application are not limited to this.

[0124] In the embodiments of the present application, accurate extraction of lane line contours is achieved through the above embodiments.

[0125] In order to verify the accuracy of the extracted lane line contours, confusion matrices TP, FN, FP, and TN can be defined, which reflect the prediction results between the binary mask and the real data:

[0126]

[0127] Where C represents the number of correctly predicted lane line points,

[0128] M represents the number of distortions of lane line points,

[0129] W represents the number of incorrectly predicted lane lines,

[0130] N represents the number of correctly predicted non-lane line points;

[0131] T pred Indicates the number of points where the prediction is true;

[0132] F pred Indicates the number of points where the prediction was false.

[0133] Table 1 is the prediction result of extracting lane line contours using the method of the embodiment of the present application.

[0134]

[0135] Table 1

[0136] It can be seen from Table 1 that the method of the embodiment of the present application can achieve high-precision lane line contour extraction. The specific implementation of the above confusion matrix can refer to the relevant technology, and the embodiment of the present application does not limit this.

[0137] In addition, the accuracy acc can be used to measure the prediction results of instance segmentation results.

[0138]

[0139] C im Indicates the correct number of points.

[0140] S im Indicates real points.

[0141] Table 2 shows the corresponding accuracy of some clustering methods.

[0142]

[0143] Table 2

[0144] It can be seen from Table 2 that the comprehensive performance of the Meanshift clustering algorithm is better.

[0145] Fig.12 It is another schematic diagram of the lane dividing method according to an embodiment of the present application.

[0146] In some embodiments, Fig.12 As shown, operation 104 includes:

[0147] 1041: Determine a preliminary lane dividing line according to a distance from the outline of the lane line to a middle area of ​​an area enclosed by the outline of the road surface area;

[0148] 1042: Update the point on the preliminary lane dividing line according to the distance relationship between the point on the preliminary lane dividing line and surrounding pixels to obtain a lane dividing line;

[0149] 1043: Merge the lane dividing line and the contour of the road surface area, where the area enclosed by the contour of the road surface area is divided into a plurality of lane areas by the lane dividing line.

[0150] In the embodiment of the present application, the "middle area" is an area surrounded by the outline of the road surface area and located in a predetermined range in the middle in the horizontal direction, for example, an area within 30% of the middle of the road surface area.

[0151] Fig.13 It is a schematic diagram of dividing lane areas according to an embodiment of the present application.

[0152] like Fig.13 As shown in (a), the contours of the lane lines are recorded as L1, L2 and L3, the contour of the road surface area is recorded as L4, and the middle area M is, for example, an area surrounded by the contour L4 and located in the middle of a predetermined range in the horizontal direction. For example, the size of the contour L4 in the horizontal direction is calculated at intervals in the vertical direction, and two points in the predetermined range in the middle of the horizontal direction in each calculation are selected, and the selected points are sequentially connected in the vertical direction to obtain the middle area M. The middle area M can also be selected in other ways, and the embodiment of the present application is not limited to this.

[0153] In the embodiment of the present application, lane lines divide lanes into different directions from the middle of the road. Therefore, the average distance from each lane line to the middle area of ​​the road is arranged from near to far. If a line label is a straight line or a fence, and the horizontal coordinates are mainly distributed near the middle third of the road, then the line closest to the middle distance and with the shortest average distance is selected as the dividing line.

[0154] For example, in operation 1041, the distance from the contour of each lane line to the middle area is calculated, and the contour of the lane line whose distance to the middle area is within a predetermined distance and the shortest distance to the middle area is determined as the preliminary lane dividing line.

[0155] For example, Fig.13 As shown in (a), the distances from the lane line profiles L1 to L3 to the middle area M are calculated respectively. For example, for each point on the lane line profile, the distance from the point to the two boundaries of the middle area is calculated, and the average value of the distances is taken as the distance from the point to the middle area. The average value of the distances from all points on the lane line profile to the middle area is calculated as the distance from the lane line profile to the middle area M. In the above example, the distance from the lane line profile L2 to the middle area M is the shortest, so the lane line profile L2 is determined as the preliminary lane dividing line.

[0156] During the clustering process, in order to avoid noise, some discrete points are sometimes ignored, especially the points at the top and bottom of the line, which leads to the loss of some information and thus the missing of some lane lines.

[0157] In the embodiment of the present application, the lane dividing line can be corrected by searching for a vanishing point on the preliminary lane dividing line and adding some points near the vanishing point.

[0158] For example, in operation 1042, a vanishing point on the preliminary lane dividing line is determined, the vanishing point is the endpoint of the preliminary lane dividing line and is located within a predetermined area of ​​the area enclosed by the outline of the road surface area; starting from the vanishing point, the preliminary lane dividing line is searched upward and downward for a predetermined number of times every first step length pixel; when the horizontal distance between the current point and the previous point is within a predetermined number of pixels, the current point is added to the preliminary lane dividing line, and all points of the preliminary lane dividing line are connected to obtain the lane dividing line.

[0159] like Fig.13 As shown in (b), the road surface extends far away, but the middle lane line contour does not extend far away. Fig.13 As shown in (c), there are some lane line pixels vaguely present in regions R1 and R2, but due to the presence of fences, light, etc., the lane lines are discontinuous, resulting in the middle lane line contour not extending to the distance. That is, there are some vanishing points at the end of the lane line contour. In addition, sometimes at the nearest end of the lane line contour (i.e., the bottom of the picture) due to the presence of green belts or fences, there may also be vanishing points. In this embodiment of the application, the endpoints of the lane line contour are referred to as vanishing points. For example, Fig.13 In (b), point N1 is the vanishing point. The vanishing point is usually located in the upper 1 / 3 area of ​​the picture or in the lower 1 / 3 area of ​​the picture. The "predetermined area" is, for example, an area within 30% of the upper boundary in the vertical direction and / or an area within 30% of the lower boundary in the vertical direction in the area surrounded by the outline of the road area.

[0160] like Fig.13As shown in (d), starting from the vanishing point N1, every first step length pixel searches upward and downward for a predetermined number of times along the preliminary lane dividing line L2. For example, at point N1, new points are searched upward and downward for 10 times every 3 pixels along the extension direction of the preliminary lane dividing line L2. When the distance between point D1 and point N1 in the horizontal direction is within 10 pixels, point D1 is added to the preliminary lane dividing line. Starting from point D1, every first step length pixel searches upward and downward for a predetermined number of times along the preliminary lane dividing line L2. When the distance between point D2 and point D1 in the horizontal direction is within a predetermined number of pixels, point D2 is added to the preliminary lane dividing line. This is analogous to point D2 until the farthest boundary of the road surface area is searched. All points are connected to obtain the lane dividing line V. The size of the first step length can be set according to actual needs, and the embodiment of the present application does not limit this.

[0161] In addition, since many points are redundant when forming polygons and may even cause jagged edges on lane lines, these points should be reduced. For example, redundant points on preliminary lane dividing lines can be detected and deleted.

[0162] In some embodiments, in operation 1042, for each point on the preliminary lane dividing line, if the distance between the point and the two adjacent points is less than the seventh threshold, or if the angle between the point and the line connecting the two adjacent points is less than the eighth threshold or greater than the ninth threshold, the point is deleted; the remaining points on the preliminary lane dividing line are connected to obtain the lane dividing line.

[0163] For example, in Fig.13 In (c), the distances between points D4, D5, D7, and D8 on the preliminary lane dividing line L2 and their adjacent points are all less than the seventh threshold, for example, less than 25 pixels. Then, points D4, D5, D7, and D8 are deleted, thereby obtaining Fig.13 (d) shows the lane dividing line V. In addition, on the preliminary lane dividing line, when the angle between the point and the connecting line of the adjacent point is less than 90 degrees and or greater than 170 degrees, the point can also be removed.

[0164] In some embodiments, the method further includes: determining the type of each divided lane according to predetermined conditions.

[0165] For example, Fig.13 As shown in (d), in the figure, according to the traffic rules of the location where the input image is located, it is determined whether the left and right lanes are up lanes or down lanes. For example, in Japan, the left lane LL shown in (d) is the up lane, and the right lane RL is the down lane; in China, the left lane LL shown in (d) is the down lane, and the right lane RL is the up lane.

[0166] The above only describes the steps or processes related to the present application, but the present application is not limited thereto. The lane division method may also include other steps or processes, and the specific contents of these steps or processes may refer to the prior art. In addition, the above only uses some structures of the model used in the lane division method as an example to illustrate the embodiments of the present application, but the present application is not limited to these structures, and these structures may be appropriately modified, and the implementation methods of these modifications should all be included in the scope of the embodiments of the present application.

[0167] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0168] It can be seen from the above embodiments that lane division is achieved by identifying the outline of the road surface area and the outline of the lane line in the image and using the outline of the lane line to divide the area enclosed by the outline of the road surface area. Even if there is no vehicle moving on the road surface and the optical flow mask cannot be extracted, the lanes can be accurately divided.

[0169] Embodiments of the second aspect

[0170] An embodiment of the present application also provides a lane dividing device, which corresponds to the lane dividing method of the embodiment of the first aspect. Fig.14 Schematic diagram of a lane dividing device according to an embodiment of the present application. Fig.14 As shown, the lane dividing device 1400 includes:

[0171] An extraction unit 1410 extracts a road surface area and a lane line area from an input image based on a semantic segmentation model;

[0172] A first determining unit 1420, which determines the contour of the road surface area based on the binary image of the road surface area;

[0173] A second determining unit 1430, which determines the contour of the lane line based on the binary image of the lane line area;

[0174] The dividing unit 1440 divides the lane according to the contour of the road surface area and the contour of the lane line.

[0175] Fig.15 It is a schematic diagram of an extraction unit according to an embodiment of the present application.

[0176] like Fig.15 As shown, the extraction unit 1410 includes:

[0177] A road surface extraction unit 1411, which extracts a road surface area corresponding to a first label from the input image based on a semantic segmentation model, wherein the first label includes at least one of a road, a fence, a moving object, a solid line, a dotted line, and a guide line;

[0178] The lane line extraction unit 1412 extracts a lane line area corresponding to a second label from the input image based on a semantic segmentation model, where the second label includes at least one of a dotted line, a solid line, a guide line, and a fence.

[0179] Fig.16 It is a schematic diagram of a first determination unit in an embodiment of the present application.

[0180] like Fig.16 As shown, the first determining unit 1420 includes:

[0181] A first processing unit 1421, which performs binarization processing on the input image based on the road surface area to obtain a binarized image of the road surface area;

[0182] A first searching unit 1422, which searches the outermost circle of the binary image of the road surface area to obtain a preliminary contour of the road surface area;

[0183] A conversion unit 1423, which converts the preliminary contour of the road surface area to obtain a polygonal contour;

[0184] A first judging unit 1424, which determines, for each vertex of the polygonal outline, whether to retain the vertex according to a positional relationship between the vertex and adjacent vertices;

[0185] The first contour determining unit 1425 determines the polygonal contour surrounded by the retained vertices as the contour of the road surface area.

[0186] In some embodiments, the first determination unit 1424 retains the vertex when the vertex satisfies the first condition.

[0187] The first condition includes:

[0188] The triangle formed by the vertex and two adjacent vertices is a concave triangle, and the ratio of the area occupied by the road surface in the concave triangle to the area of ​​the concave triangle is less than a first threshold; or

[0189] The triangle formed by the vertex and two adjacent vertices is a convex triangle, and the ratio of the area occupied by the road surface in the convex triangle to the area of ​​the convex triangle is greater than a second threshold.

[0190] In some embodiments, the first determining unit 1424 determines whether the vertex is an outermost boundary vertex.

[0191] When the vertex is a vertex other than a boundary vertex and satisfies the second condition, the first judgment unit 1424 discards the vertex.

[0192] The second condition includes at least one of the following:

[0193] The triangle formed by the vertex and two adjacent vertices is a convex triangle and the angle between the vertex and the line connecting the two adjacent vertices is less than a third threshold;

[0194] The distances between the vertex and two adjacent vertices are both less than a fourth threshold;

[0195] The triangle formed by the vertex and two adjacent vertices is a concave triangle, the angle between the vertex and the line connecting the two adjacent vertices is an acute angle, and at least one of the distances between the vertex and the two adjacent vertices is less than a fifth threshold.

[0196] In some embodiments, when the vertex is a boundary vertex and satisfies the third condition, the first determination unit 1424 retains the vertex.

[0197] Among them, the third condition includes: the triangle formed by the vertex and two adjacent vertices is a convex triangle, and the triangle formed by an adjacent vertex of the vertex and its two adjacent vertices is a concave triangle, and the distance between the vertex and the adjacent vertex is less than a sixth threshold.

[0198] In some embodiments, when the vertex is a boundary vertex and satisfies the third condition, the first judgment unit 1424 discards the adjacent vertex.

[0199] Fig.17 is a schematic diagram of a second determination unit in an embodiment of the present application.

[0200] like Fig.17 As shown, the second determining unit 1430 includes:

[0201] A second processing unit 1431 performs binarization processing on the input image based on the lane line region to obtain a binarized image of the lane line region;

[0202] A clustering unit 1432, based on the embedded vector corresponding to the lane line region obtained by the semantic segmentation model, clusters the lane line region in the binary image of the lane line region to obtain points on the contour of the lane line,

[0203] The second contour determining unit 1433 connects the points to obtain the contour of the lane line.

[0204] Fig.18 It is a schematic diagram of a division unit according to an embodiment of the present application.

[0205] like Fig.18 As shown, the division unit 1440 includes

[0206] A preliminary lane dividing line unit 1441, which determines a preliminary lane dividing line according to a distance from the outline of the lane line to a middle area of ​​an area surrounded by the outline of the road surface area;

[0207] A lane dividing line unit 1442 is configured to update the points on the preliminary lane dividing line according to the distance relationship between the points on the preliminary lane dividing line and surrounding pixels to obtain a lane dividing line;

[0208] The lane area division unit 1443 combines the lane dividing line with the contour of the road surface area, and the area surrounded by the contour of the road surface area is divided into multiple lane areas by the lane dividing line.

[0209] In some embodiments, the preliminary lane dividing line unit 1441 calculates the distance from the contour of each lane line to the middle area, and determines the contour of the lane line whose distance to the middle area is within a predetermined distance and the shortest distance to the middle area as the preliminary lane dividing line.

[0210] Fig.19 is a schematic diagram of a lane dividing line unit according to an embodiment of the present application.

[0211] like Fig.19 As shown, the lane dividing line unit 1442 includes:

[0212] a vanishing point determining unit 14421, which determines a vanishing point on the preliminary lane dividing line, wherein the vanishing point is an end point of the preliminary lane dividing line and is located in a predetermined area of ​​an area enclosed by the outline of the road surface area;

[0213] A second search unit 14422, which searches upward and downward along the preliminary lane dividing line for a predetermined number of times every first step length pixel from the vanishing point;

[0214] The adding point unit 14423 adds the current point to the preliminary lane dividing line when the horizontal distance between the current point and the previous point is within a predetermined number of pixels.

[0215] The connection unit 14424 connects all points of the preliminary lane dividing line to obtain the lane dividing line.

[0216] In some embodiments, Fig.19 As shown, the lane dividing line unit 1442 may also include:

[0217] The point deletion unit 14425 deletes each point on the preliminary lane dividing line when the distance between the point and the two adjacent points is less than the seventh threshold, or when the angle between the point and the line connecting the two adjacent points is less than the eighth threshold or greater than the ninth threshold.

[0218] The connection unit 14424 connects the remaining points on the preliminary lane dividing line to obtain the lane dividing line.

[0219] In this embodiment, the specific implementation method of each of the above-mentioned units is the same as that described in the embodiment of the first aspect and will not be repeated here.

[0220] It is worth noting that the above only describes the components or modules related to the present application, but the present application is not limited thereto. The lane dividing device 1400 may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.

[0221] To keep it simple, Fig.14 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned various components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0222] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0223] It can be seen from the above embodiments that lane division is achieved by identifying the outline of the road surface area and the outline of the lane line in the image and using the outline of the lane line to divide the area enclosed by the outline of the road surface area. Even if there is no vehicle moving on the road surface and the optical flow mask cannot be extracted, the lanes can be accurately divided.

[0224] Embodiments of the third aspect

[0225] The embodiment of the present application also provides an electronic device, Fig. 20 A schematic diagram of an electronic device according to an embodiment of the present application. Fig. 20 As shown, the electronic device 2000 includes a lane dividing device 2001. The structure and function of the lane dividing device 2001 are the same as those of the lane dividing device 400 in the embodiment of the second aspect. For specific contents, please refer to the description in the embodiment of the second aspect, and will not be repeated here.

[0226] Fig.211 is a schematic block diagram of the system structure of the electronic device according to the embodiment of the present application. Fig.21 As shown, the electronic device 2100 may include a processor 2101 and a memory 2102; the memory 2102 is coupled to the processor 2101. This figure is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0227] In some embodiments, processor 2101 includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU).

[0228] like Fig.21 As shown, the electronic device 2100 may further include: an input device 2103 , a display 2104 , and a power supply 2105 .

[0229] In one embodiment, the function of the lane division device can be integrated into the processor 2101. The processor 2101 can be configured to: extract the road surface area and the lane line area from the input image based on the semantic segmentation model; determine the contour of the road surface area based on the binary image of the road surface area; determine the contour of the lane line based on the binary image of the lane line area; divide the lane according to the contour of the road surface area and the contour of the lane line. For other embodiments, please refer to the embodiment of the second aspect, which will not be repeated here.

[0230] In another embodiment, the lane dividing device may be configured separately from the processor 2101 . For example, the lane dividing device may be configured as a chip connected to the processor 2101 , and the function of the lane dividing device may be implemented through the control of the processor 2101 .

[0231] In this embodiment, the electronic device 2100 does not necessarily include Figure 6 All parts shown in .

[0232] like Fig.21 As shown, the processor 2101 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 2101 receives inputs and controls the operations of various components of the electronic device 2100.

[0233] The memory 2102 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The processor 2101 may execute the program stored in the memory 2102 to implement information storage or processing. The functions of other components are similar to those of the prior art and are not described in detail herein. The components of the electronic device 2100 may be implemented by dedicated hardware, firmware, software or a combination thereof without departing from the scope of the present application.

[0234] It can be seen from the above embodiments that lane division is achieved by identifying the outline of the road surface area and the outline of the lane line in the image and using the outline of the lane line to divide the area enclosed by the outline of the road surface area. Even if there is no vehicle moving on the road surface and the optical flow mask cannot be extracted, the lanes can be accurately divided.

[0235] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a lane dividing device or an electronic device, the program enables a computer to execute the lane dividing method described in the first aspect embodiment in the lane dividing device or the electronic device.

[0236] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the lane dividing method described in the first aspect of the embodiment in a lane dividing device or an electronic device.

[0237] The lane division device or the lane division method executed in the electronic device described in the embodiments of the present application can be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, Fig.14 One or more of the functional block diagrams shown in and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to Figure 1 These hardware modules can be implemented by solidifying these software modules using, for example, a field programmable gate array (FPGA).

[0238] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the electronic device uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0239] against Fig.14One or more of the functional block diagrams and / or one or more combinations of the functional block diagrams described herein may be implemented as a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any appropriate combination thereof for performing the functions described in the present application. Fig.14 One or more of the described functional block diagrams and / or one or more combinations of the functional block diagrams may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0240] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the spirit and principles of the present application, and these modifications and variations are also within the scope of the present application.

[0241] This application also provides the following notes:

[0242] 1. A lane division method, wherein the method comprises:

[0243] Extract road surface areas and lane line areas from input images based on semantic segmentation models;

[0244] determining a contour of the road surface area based on a binary image of the road surface area;

[0245] Determine the outline of the lane line based on the binary image of the lane line area;

[0246] Lanes are divided according to the contours of the road surface area and the contours of the lane lines.

[0247] 2. The method according to Note 1, wherein the step of extracting the road surface area and the lane line area from the input image based on the semantic segmentation model comprises:

[0248] Extracting a road surface area corresponding to a first label from the input image based on a semantic segmentation model, where the first label includes at least one of a road, a fence, a moving object, a solid line, a dotted line, and a guide line;

[0249] A lane line area corresponding to a second label is extracted from the input image based on a semantic segmentation model, where the second label includes at least one of a dotted line, a solid line, a guide line, and a fence.

[0250] 3. The method according to Supplement 1, wherein the step of determining the contour of the road surface area based on the binary image of the road surface area comprises:

[0251] Binarizing the input image based on the road surface area to obtain a binary image of the road surface area;

[0252] Searching the outermost circle of the binary image of the road surface area to obtain a preliminary outline of the road surface area;

[0253] Converting the preliminary contour of the road surface area to obtain a polygonal contour;

[0254] For each vertex of the polygonal outline, determining whether to retain the vertex according to a positional relationship between the vertex and adjacent vertices;

[0255] A polygonal outline enclosed by the retained vertices is determined as the outline of the road surface area.

[0256] 4. The method according to Supplementary Note 3, wherein determining whether to retain the vertex according to the positional relationship between the vertex and adjacent vertices comprises:

[0257] When the vertex satisfies the first condition, retain the vertex.

[0258] The first condition includes:

[0259] The triangle formed by the vertex and two adjacent vertices is a concave triangle, and the ratio of the area occupied by the road surface in the concave triangle to the area of ​​the concave triangle is less than a first threshold; or

[0260] The triangle formed by the vertex and two adjacent vertices is a convex triangle, and the ratio of the area occupied by the road surface in the convex triangle to the area of ​​the convex triangle is greater than a second threshold.

[0261] 5. The method according to Supplementary Note 3, wherein the step of determining whether to retain the vertex according to the positional relationship between the vertex and adjacent vertices comprises:

[0262] Determine whether the vertex is an outermost boundary vertex;

[0263] When the vertex is a vertex other than a boundary vertex and satisfies the second condition, the vertex is discarded.

[0264] The second condition includes at least one of the following:

[0265] The triangle formed by the vertex and two adjacent vertices is a convex triangle and the angle between the vertex and the line connecting the two adjacent vertices is less than a third threshold;

[0266] The distances between the vertex and two adjacent vertices are both less than a fourth threshold;

[0267] The triangle formed by the vertex and two adjacent vertices is a concave triangle, the angle between the vertex and the line connecting the two adjacent vertices is an acute angle, and at least one of the distances between the vertex and the two adjacent vertices is less than a fifth threshold.

[0268] 6. The method according to Supplementary Note 5, wherein the step of determining whether to retain the vertex based on the positional relationship between the vertex and adjacent vertices comprises:

[0269] When the vertex is a boundary vertex and satisfies the third condition, retain the vertex.

[0270] Among them, the third condition includes: the triangle formed by the vertex and two adjacent vertices is a convex triangle, and the triangle formed by an adjacent vertex of the vertex and its two adjacent vertices is a concave triangle, and the distance between the vertex and the adjacent vertex is less than a sixth threshold.

[0271] 7. The method according to Supplementary Note 6, wherein the step of determining whether to retain the vertex based on the positional relationship between the vertex and adjacent vertices further comprises:

[0272] When the vertex is a boundary vertex and satisfies the third condition, the adjacent vertex is discarded.

[0273] 8. The method according to Supplementary Note 1, wherein the step of determining the contour of the lane line based on the binary image of the lane line area comprises:

[0274] Binarizing the input image based on the lane line area to obtain a binary image of the lane line area;

[0275] Based on the embedding vector corresponding to the lane line area obtained by the semantic segmentation model, the lane line area of ​​the binary image of the lane line area is clustered to obtain the points on the outline of the lane line.

[0276] The points are connected to obtain the outline of the lane line.

[0277] 9. The method according to Supplementary Note 1, wherein the lane division according to the contour of the road surface area and the contour of the lane line includes:

[0278] Determine a preliminary lane dividing line according to the distance from the outline of the lane line to the middle area of ​​the area enclosed by the outline of the road surface area;

[0279] Update the point on the preliminary lane dividing line according to the distance relationship between the point on the preliminary lane dividing line and surrounding pixels to obtain a lane dividing line;

[0280] The lane dividing line is combined with the contour of the road surface area, and the area enclosed by the contour of the road surface area is divided into a plurality of lane areas by the lane dividing line.

[0281] 10. The method according to Supplementary Note 9, wherein the determining of the preliminary lane dividing line according to the distance from the outline of the lane line to the middle area of ​​the area enclosed by the outline of the road surface area comprises:

[0282] Calculate the distance from the contour of each lane line to the middle area,

[0283] The contour of a lane line whose distance to the middle area is within a predetermined distance and whose distance to the middle area is the shortest is determined as the preliminary lane dividing line.

[0284] 11. The method according to Supplementary Note 9, wherein the updating of the points on the preliminary lane dividing line according to the distance relationship between the points on the preliminary lane dividing line and surrounding pixels to obtain the lane dividing line comprises:

[0285] Determine a vanishing point on the preliminary lane dividing line, the vanishing point being an end point of the preliminary lane dividing line and being located within a predetermined area of ​​an area enclosed by a contour of a road surface area;

[0286] Starting from the vanishing point, searching upward and downward along the preliminary lane dividing line for a predetermined number of times every first step length pixel;

[0287] When the horizontal distance between the current point and the previous point is within a predetermined number of pixels, the current point is added to the preliminary lane dividing line.

[0288] Connect all the points of the preliminary lane dividing line to get the lane dividing line.

[0289] 12. The method according to Supplementary Note 9, wherein the updating of the points on the preliminary lane dividing line according to the distance relationship between the points on the preliminary lane dividing line and surrounding pixels to obtain the lane dividing line comprises:

[0290] For each point on the preliminary lane dividing line, if the distance between the point and two adjacent points is less than the seventh threshold, or if the angle between the point and the line connecting the two adjacent points is less than the eighth threshold or greater than the ninth threshold, delete the point;

[0291] The remaining points on the preliminary lane dividing line are connected to obtain the lane dividing line.

[0292] 13. The method according to Note 1, wherein the method further comprises:

[0293] The type of each lane divided is determined according to predetermined conditions.

[0294] 14. A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor coupled to the storage medium to execute the following method:

[0295] Extract road surface areas and lane line areas from input images based on semantic segmentation models;

[0296] determining a contour of the road surface area based on a binary image of the road surface area;

[0297] Determine the outline of the lane line based on the binary image of the lane line area;

[0298] Lanes are divided according to the contours of the road surface area and the contours of the lane lines.

Claims

1. A lane dividing device, wherein: The device comprises: An extraction unit, which extracts a road surface area and a lane line area from an input image based on a semantic segmentation model; a first determining unit that determines a contour of the road surface area based on a binarized image of the road surface area; A second determining unit, which determines the contour of the lane line based on the binary image of the lane line area; A dividing unit divides lanes according to the contour of the road surface area and the contour of the lane line.

2. The device according to claim 1, wherein: The extraction unit comprises: a road surface extraction unit, which extracts a road surface area corresponding to a first label from the input image based on a semantic segmentation model, wherein the first label includes at least one of a road, a fence, a moving object, a solid line, a dotted line, and a guide line; A lane line extraction unit extracts a lane line area corresponding to a second label from the input image based on a semantic segmentation model, wherein the second label includes at least one of a dotted line, a solid line, a guide line, and a fence.

3. The device according to claim 1, wherein: The first determining unit includes: A first processing unit, which performs binarization processing on the input image based on the road surface area to obtain a binarized image of the road surface area; A first search unit, which searches the outermost circle of the binary image of the road surface area to obtain a preliminary contour of the road surface area; A conversion unit, which converts the preliminary contour of the road surface area to obtain a polygonal contour; A first judgment unit, for each vertex of the polygonal outline, determines whether to retain the vertex according to a positional relationship between the vertex and adjacent vertices; A first contour determining unit determines a polygonal contour surrounded by the retained vertices as a contour of the road surface area.

4. The device according to claim 3, wherein: The first judgment unit retains the vertex when the vertex meets the first condition, The first condition includes: The triangle formed by the vertex and two adjacent vertices is a concave triangle, and the ratio of the area occupied by the road surface in the concave triangle to the area of ​​the concave triangle is less than a first threshold; or The triangle formed by the vertex and two adjacent vertices is a convex triangle, and the ratio of the area occupied by the road surface in the convex triangle to the area of ​​the convex triangle is greater than a second threshold.

5. The device according to claim 3, wherein: The first judging unit judges whether the vertex is a boundary vertex located at the outermost side. When the vertex is a vertex other than a boundary vertex and satisfies a second condition, the first judgment unit discards the vertex. The second condition includes at least one of the following: The triangle formed by the vertex and two adjacent vertices is a convex triangle and the angle between the vertex and the line connecting the two adjacent vertices is less than a third threshold; The distances between the vertex and two adjacent vertices are both less than a fourth threshold; The triangle formed by the vertex and two adjacent vertices is a concave triangle, the angle between the vertex and the line connecting the two adjacent vertices is an acute angle, and at least one of the distances between the vertex and the two adjacent vertices is less than a fifth threshold.

6. The device according to claim 5, wherein: When the vertex is a boundary vertex and satisfies a third condition, the first judgment unit retains the vertex. The third condition includes: the triangle formed by the vertex and two adjacent vertices is a convex triangle, and the triangle formed by a vertex adjacent to the vertex and two adjacent vertices is a concave triangle, and the distance between the vertex and the adjacent vertex is less than a sixth threshold, When the vertex is a boundary vertex and satisfies the third condition, the first judgment unit discards the adjacent vertex.

7. The device according to claim 1, wherein: The second determining unit includes: A second processing unit, which performs binarization processing on the input image based on the lane line area to obtain a binarized image of the lane line area; A clustering unit, which clusters the lane line region in the binary image of the lane line region to obtain points on the contour of the lane line based on the embedded vector corresponding to the lane line region obtained by the semantic segmentation model, A second contour determining unit is configured to connect the points to obtain a contour of the lane line.

8. The device according to claim 1, wherein: The division unit includes a preliminary lane dividing line unit, which determines a preliminary lane dividing line according to a distance from the outline of the lane line to a middle area of ​​an area enclosed by the outline of the road surface area; A lane dividing line unit updates the points on the preliminary lane dividing line according to the distance relationship between the points on the preliminary lane dividing line and surrounding pixels to obtain a lane dividing line; A lane area division unit is configured to merge the lane dividing line with the contour of the road surface area, wherein the area enclosed by the contour of the road surface area is divided into a plurality of lane areas by the lane dividing line.

9. The device according to claim 8, wherein: The lane dividing line unit comprises: a vanishing point determination unit, which determines a vanishing point on the preliminary lane dividing line, wherein the vanishing point is an end point of the preliminary lane dividing line and is located in a predetermined area of ​​an area enclosed by a contour of a road surface area; a second search unit, which searches upward and downward along the preliminary lane dividing line for a predetermined number of times every first step length pixel from the vanishing point; an adding point unit, which adds the current point to the preliminary lane dividing line when the distance between the current point and the previous point in the horizontal direction is within a predetermined number of pixels; a point deletion unit, for each point on the preliminary lane dividing line, deleting the point if the distance between the point and two adjacent points is less than a seventh threshold, or if the angle between the point and the line connecting the two adjacent points is less than an eighth threshold or greater than a ninth threshold; A connection unit connects all points of the preliminary lane dividing line to obtain a lane dividing line.

10. A lane division method, wherein: The method comprises: Extract road surface areas and lane line areas from input images based on semantic segmentation models; determining a contour of the road surface area based on a binary image of the road surface area; Determine the outline of the lane line based on the binary image of the lane line area; Lanes are divided according to the contours of the road surface area and the contours of the lane lines.

Images