Lane line automatic identification system and method based on semantic segmentation

Abstract

The invention provides an automatic lane line recognition system based on semantic segmentation, and the system comprises an image collection module which is used for collecting a lane image through camera equipment; the lane image recognition module is used for inputting a lane image into the semantic segmentation model and outputting the lane image of lane line contour information and lane line category information; the lane line skeleton extraction and fitting module is used for extracting a lane line skeleton from the lane image of the lane line contour information and the lane line category information by using the image processing library, and fitting the lane line skeleton to obtain a lane vector line; and the lane line classification module is used for calculating the distance between the vector lines of each lane and classifying the lane lines according to the distance. According to the invention, by using the semantic segmentation model, the system can carry out pixel-level identification on the lane marking and provide a more accurate lane marking extraction result. Therefore, it is ensured that the extracted lane marking is consistent with the actual road condition, and more reliable information is provided for an automatic driving system to use.

Description

Technical Field

[0001] The present invention relates to the technical field of road recognition, and in particular to a lane line automatic recognition system and method based on semantic segmentation. Background Art

[0002] Currently, autonomous driving is developing rapidly, and the most important thing for real-line autonomous driving is automatic road recognition.

[0003] When using traditional machine learning algorithms to extract lane markings, they may not perform well for complex scenes and images under different lighting conditions. For example, in strong light, shadows, or at night, the contrast and edge information of lane markings may be weak. Traditional algorithms usually require manual adjustment of some parameters, such as thresholds, filter sizes, etc., to adapt to different images and scenes. These problems and defects make the algorithm process more complicated and cannot be automated. Summary of the invention

[0004] The purpose of the present invention is to provide a lane line automatic recognition system based on semantic segmentation in response to the defects of the prior art, including: an image acquisition module, used to use a camera to collect lane images; a lane image recognition module, used to input the lane image into a semantic segmentation model, classify the pixels in the lane image in the semantic segmentation model, and output a lane image with lane line contour information and lane line category information according to the category of the pixels, wherein the lane line categories include solid lane lines and dashed lane lines; a lane line skeleton extraction and fitting module, used to use an image processing library to extract a lane line skeleton from the lane image with lane line contour information and lane line category information, wherein the lane line skeleton includes solid lane lines and dashed lane lines. Lane line skeleton and dotted lane line skeleton, fitting the solid lane line skeleton and the dotted lane line skeleton to obtain solid lane vector lines and dotted lane vector lines respectively; lane line classification module, used to calculate the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dotted lane vector lines, and the fourth vertical distance between each solid lane vector line and the adjacent dotted lane vector line, and classify each solid lane line and each dotted lane line corresponding to each solid lane vector line and each dotted lane vector line into double solid lane lines, double dotted lane lines, left solid line and right dotted line lane lines, and right solid line and left dotted line lane lines according to the second vertical distance, the third vertical distance and the fourth vertical distance.

[0005] Furthermore, in the lane image recognition module, the specific method of inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting the lane image including the lane line contour information and the lane line category information according to the category of the pixels is as follows:

[0006] The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the category of each pixel in the lane image is output. The lane image is labeled according to the category of each pixel to obtain a lane image with lane contour information and lane category information.

[0007] Furthermore, in the lane line skeleton extraction and fitting module, the specific method of extracting the lane line skeleton from the lane image of the lane line contour information and the lane line category information using the image processing library is as follows:

[0008] The lane line contour image is binarized using the opencv library, and then the lane line skeleton is extracted from the binarized lane line contour image using the thinning function of the ximgproc module in the opencv library, wherein the thinning function adopts the ZhangSuen algorithm.

[0009] Furthermore, in the lane line skeleton extraction and fitting module, the specific method for obtaining the dotted lane line skeleton is as follows:

[0010] An image processing library is used to extract all the dotted lane line skeleton segments, and the first vertical distance between the extension line of a dotted lane line skeleton segment and the extension line of the adjacent dotted lane line skeleton segment is calculated. If the first vertical distance is less than a first threshold, the two adjacent dotted lane line skeleton segments are classified as the same dotted lane line skeleton, and then the median of the ordinates of all the dotted lane line skeleton segments belonging to the same dotted lane line skeleton are calculated respectively. All the dotted lane line skeleton segments are connected end to end in order according to the order of the median of the ordinates of each dotted lane line skeleton segment to form the dotted lane line skeleton.

[0011] Furthermore, in the lane line skeleton extraction and fitting module, the specific method of fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows:

[0012] A linear function is used to fit the straight line segment of the solid lane line skeleton or the dotted lane line skeleton to obtain a fitting curve of the straight line segment of the solid lane line skeleton or the dotted lane line skeleton, the ordinate values ​​of the pixel points of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton are sorted, and then n pixel points are selected on average within the interval between the maximum and minimum ordinate values ​​of the pixel points as the fitting curve of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton, the straight line segment of the solid lane line skeleton and the fitting curve of the curved line segment are connected to obtain a solid lane vector line, and the straight line segment of the dotted lane line skeleton and the fitting curve of the curved line segment are connected to obtain a dotted lane vector line. Furthermore, in the solid lane line skeleton or the dotted lane line skeleton, if the bending angle of a certain part of the skeleton is less than a preset angle, the part is determined to be a straight line segment, and if the bending angle of a certain part of the skeleton is greater than the preset angle, the part is determined to be a straight line segment.

[0013] Further, in the lane classification module, the specific method of calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and the adjacent dashed lane vector line, and classifying each solid lane line and each dashed lane line corresponding to each solid lane vector line and each dashed lane vector line as double solid lane lines, double dashed lane lines, left solid line and right dashed line lane lines, and right solid line and left dashed line lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance is:

[0014] Calculate a second vertical distance between two adjacent solid lane vector lines, and if the second vertical distance is less than a second threshold, classify the two adjacent solid lane lines corresponding to the two adjacent solid lane vector lines as double solid lane lines;

[0015] Calculate a third vertical distance between two adjacent dashed lane vector lines, and if the third vertical distance is less than a third threshold, classify the two adjacent dashed lane lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines;

[0016] Calculate the fourth vertical distance between a solid lane vector line and the adjacent dashed lane vector line. If the fourth vertical distance is less than the fourth threshold, calculate the median of the horizontal coordinates of the solid lane vector line and the dashed lane vector line. If the median of the horizontal coordinates of the solid lane vector line is smaller than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the dashed lane vector line are the left solid line and the right dashed line lane line. If the median of the horizontal coordinates of the solid lane vector line is larger than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the solid lane vector line and the dashed lane vector line are the right solid line and the left dashed line lane line.

[0017] A method for automatically identifying lane lines based on semantic segmentation includes: using a camera to collect a lane image; inputting the lane image into a semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting a lane image with lane line contour information and lane line category information according to the category of the pixels, wherein the lane line category includes a solid lane line and a dashed lane line; using an image processing library to extract a lane line skeleton from the lane image with the lane line contour information and the lane line category information, wherein the lane line skeleton includes a solid lane line skeleton and a dashed lane line skeleton, and performing a classification of the lane line skeleton and the lane line category information. The dotted lane line skeleton is fitted to obtain the solid lane vector line and the dotted lane vector line respectively; the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dotted lane vector lines, and the fourth vertical distance between each solid lane vector line and the adjacent dotted lane vector line are calculated; according to the second vertical distance, the third vertical distance and the fourth vertical distance, each solid lane line and each dotted lane line corresponding to each solid lane vector line and each dotted lane vector line are respectively classified as double solid lane lines, double dotted lane lines, left solid line and right dotted lane lines, and right solid line and left dotted lane lines.

[0018] A computer-readable medium having a computer program / instruction stored thereon, wherein the computer program / instruction executes the above-mentioned automatic lane line recognition method based on semantic segmentation when run.

[0019] A computer program product includes a computer program / instruction, which implements the above-mentioned automatic lane line recognition method based on semantic segmentation when executed by a processor.

[0020] The beneficial effects of the present invention are:

[0021] 1. By using the semantic segmentation model, the system can identify lane markings at the pixel level and provide more accurate lane marking extraction results. This helps ensure that the extracted lane markings are consistent with the actual road conditions and provides more reliable information for the autonomous driving system to use.

[0022] 2. By vectorizing and fitting the extracted lane markings, the system can represent the lane markings as geometric shapes, such as straight segments or curves. Such representation is more convenient for subsequent processing and analysis, such as for path planning and vehicle control.

[0023] 3. For intermittent dashed line segments, the system can connect them by merging adjacent dashed line segments to form a continuous dashed line mark. This helps provide a more accurate lane line shape and improves the autonomous driving system's understanding and perception of lanes.

[0024] 4. By classifying the extracted lane markings, the system can identify different types of lane markings, such as solid lines, dotted lines, double solid lines, double dotted lines, left solid and right dotted lines, and left dotted and right solid lines. This helps the vehicle correctly understand and respond to different types of lane markings during autonomous driving, improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a system block diagram of the present invention.

[0026] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] Example 1

[0029] refer to Figure 1 , a lane line automatic recognition system based on semantic segmentation, comprising:

[0030] An image acquisition module, used for acquiring lane images using a camera device;

[0031] A lane image recognition module is used to input the lane image into a semantic segmentation model, classify the pixels in the lane image in the semantic segmentation model, and output a lane image including lane line contour information and lane line category information according to the category of the pixels, wherein the lane line category includes a solid lane line and a dashed lane line;

[0032] A lane line skeleton extraction and fitting module is used to extract a lane line skeleton from a lane image with lane line profile information and lane line category information using an image processing library, wherein the lane line skeleton includes a solid lane line skeleton and a dotted lane line skeleton, and the solid lane line skeleton and the dotted lane line skeleton are fitted to obtain a solid lane vector line and a dotted lane vector line respectively;

[0033] The lane line classification module is used to calculate the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and the adjacent dashed lane vector line, and classify each solid lane line and each dashed lane line corresponding to each solid lane vector line and each dashed lane vector line into double solid lane lines, double dashed lane lines, left solid line and right dashed line lane lines, and right solid line and left dashed line lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance.

[0034] (1) In the lane image recognition module, the specific method of inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting the lane image including the lane line contour information and the lane line category information according to the category of the pixels is as follows:

[0035] The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the category of each pixel in the lane image is output. The lane image is labeled according to the category of each pixel to obtain a lane image with lane contour information and lane category information.

[0036] The core idea of ​​the mask2former model is to regard the pixels in the image as sequences and use position encoding to preserve the spatial relationship between pixels, and then perform attention calculation and feature fusion based on the sequence-level features to generate semantic segmentation results for each pixel.

[0037] The model training process takes the original image as input and the lane line annotation map as the target output. By adjusting the model parameters and loss function, the model can learn to accurately predict the contour and category information of the lane line, where the category information is the solid lane line and the dashed lane line.

[0038] The trained mask2former model is added to 500 test data sets for prediction to obtain the contour and category information of the lane line. The IOU of these prediction results is calculated with the lane line annotation information, and the IOU index of the lane line is 83.65%. The experimental results show that the semantic segmentation model is highly robust for lane line extraction in complex scenes and under different lighting conditions.

[0039] (2) In the lane line skeleton extraction and fitting module, the specific method of extracting the lane line skeleton from the lane image with lane line contour information and lane line category information using the image processing library is as follows:

[0040] The lane line contour image is binarized using the opencv library, and then the lane line skeleton is extracted from the binarized lane line contour image using the thinning function of the ximgproc module in the opencv library, wherein the thinning function adopts the ZhangSuen algorithm.

[0041] Since the previous step obtained the lane line contour information, it is necessary to extract its skeleton line and convert the polygonal closed area into a curve.

[0042] The ZhangSuen algorithm is a classic binary image thinning algorithm that thins the binary image and removes redundant pixels to obtain a skeleton image.

[0043] (3) In the lane line skeleton extraction and fitting module, the specific method for obtaining the dotted lane line skeleton is as follows:

[0044] An image processing library is used to extract all the dotted lane line skeleton segments, and the first vertical distance between the extension line of a dotted lane line skeleton segment and the extension line of the adjacent dotted lane line skeleton segment is calculated. If the first vertical distance is less than a first threshold, the two adjacent dotted lane line skeleton segments are classified as the same dotted lane line skeleton, and then the median of the ordinates of all the dotted lane line skeleton segments belonging to the same dotted lane line skeleton are calculated respectively. All the dotted lane line skeleton segments are connected end to end in order according to the order of the median of the ordinates of each dotted lane line skeleton segment to form the dotted lane line skeleton.

[0045] Since the dotted lane lines are in segments, relative to the solid lane lines, the obtained dotted lane line skeleton segments need to be connected to obtain a complete dotted lane line skeleton.

[0046] (4) In the lane line skeleton extraction and fitting module, the specific method of fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows:

[0047] A linear function is used to fit the straight line segment of the solid lane line skeleton or the dotted lane line skeleton to obtain a fitting curve of the straight line segment of the solid lane line skeleton or the dotted lane line skeleton, the ordinate values ​​of the pixel points of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton are sorted, and then n pixel points (generally 12 pixels) are selected on average within the interval between the maximum and minimum values ​​of the ordinate values ​​of the pixel points to form a connecting line as the fitting curve of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton, the straight line segment of the solid lane line skeleton and the fitting curve of the curved line segment are connected to obtain a solid lane vector line, and the straight line segment of the dotted lane line skeleton and the fitting curve of the curved line segment are connected to obtain a dotted lane vector line.

[0048] If the bending angle of a certain part of the skeleton is less than a preset angle, the part is determined to be a straight line segment; if the bending angle of a certain part of the skeleton is greater than the preset angle, the part is determined to be a straight line segment.

[0049] (5) In the lane classification module, the specific method of calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and an adjacent dashed lane vector line, and classifying each solid lane line and each dashed lane line corresponding to each solid lane vector line and each dashed lane vector line as a double solid lane line, a double dashed lane line, a left solid line and a right dashed line lane line according to the second vertical distance, the third vertical distance, and the fourth vertical distance, respectively, is as follows:

[0050] Calculate a second vertical distance between two adjacent solid lane vector lines, and if the second vertical distance is less than a second threshold, classify the two adjacent solid lane lines corresponding to the two adjacent solid lane vector lines as double solid lane lines;

[0051] Calculate a third vertical distance between two adjacent dashed lane vector lines, and if the third vertical distance is less than a third threshold, classify the two adjacent dashed lane lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines;

[0052] Calculate the fourth vertical distance between a solid lane vector line and the adjacent dashed lane vector line. If the fourth vertical distance is less than the fourth threshold, calculate the median of the horizontal coordinates of the solid lane vector line and the dashed lane vector line. If the median of the horizontal coordinates of the solid lane vector line is smaller than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the dashed lane vector line are the left solid line and the right dashed line lane line. If the median of the horizontal coordinates of the solid lane vector line is larger than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the solid lane vector line and the dashed lane vector line are the right solid line and the left dashed line lane line.

[0053] Example 2

[0054] refer to Figure 2 , a lane line automatic recognition method based on semantic segmentation, comprising:

[0055] Using a camera to collect lane images;

[0056] Inputting the lane image into a semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting a lane image including lane line contour information and lane line category information according to the categories of the pixels, wherein the lane line categories include solid lane lines and dashed lane lines;

[0057] Extracting a lane line skeleton from a lane image with lane line contour information and lane line category information using an image processing library, wherein the lane line skeleton includes a solid lane line skeleton and a dotted lane line skeleton, and fitting the solid lane line skeleton and the dotted lane line skeleton to obtain a solid lane vector line and a dotted lane vector line respectively;

[0058] The second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and an adjacent dashed lane vector line are calculated; and the solid lane lines and dashed lane lines corresponding to each solid lane vector line and each dashed lane vector line are respectively classified as double solid lane lines, double dashed lane lines, left solid line and right dashed lane lines, and right solid line and left dashed lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance.

[0059] (1) The specific method of inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting the lane image with lane line contour information and lane line category information according to the categories of the pixels is as follows:

[0060] The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the category of each pixel in the lane image is output. The lane image is labeled according to the category of each pixel to obtain a lane image with lane contour information and lane category information.

[0061] The core idea of ​​the mask2former model is to regard the pixels in the image as sequences and use position encoding to preserve the spatial relationship between pixels, and then perform attention calculation and feature fusion based on the sequence-level features to generate semantic segmentation results for each pixel.

[0062] The model training process takes the original image as input and the lane line annotation map as the target output. By adjusting the model parameters and loss function, the model can learn to accurately predict the contour and category information of the lane line, where the category information is the solid lane line and the dashed lane line.

[0063] The trained mask2former model is added to 500 test data sets for prediction to obtain the contour and category information of the lane line. The IOU of these prediction results is calculated with the lane line annotation information, and the IOU index of the lane line is 83.65%. The experimental results show that the semantic segmentation model is highly robust for lane line extraction in complex scenes and under different lighting conditions.

[0064] (2) The specific method of extracting the lane line skeleton from the lane image with lane line contour information and lane line category information using the image processing library is as follows:

[0065] The lane line contour image is binarized using the opencv library, and then the lane line skeleton is extracted from the binarized lane line contour image using the thinning function of the ximgproc module in the opencv library, wherein the thinning function adopts the ZhangSuen algorithm.

[0066] Since the previous step obtained the lane line contour information, it is necessary to extract its skeleton line and convert the polygonal closed area into a curve.

[0067] The ZhangSuen algorithm is a classic binary image thinning algorithm that thins the binary image and removes redundant pixels to obtain a skeleton image.

[0068] (3) The specific method for obtaining the skeleton of the dashed lane line is as follows:

[0069] An image processing library is used to extract all the dotted lane line skeleton segments, and the first vertical distance between the extension line of a dotted lane line skeleton segment and the extension line of the adjacent dotted lane line skeleton segment is calculated. If the first vertical distance is less than a first threshold, the two adjacent dotted lane line skeleton segments are classified as the same dotted lane line skeleton, and then the median of the ordinates of all the dotted lane line skeleton segments belonging to the same dotted lane line skeleton are calculated respectively. All the dotted lane line skeleton segments are connected end to end in order according to the order of the median of the ordinates of each dotted lane line skeleton segment to form the dotted lane line skeleton.

[0070] Since the dotted lane lines are in segments, relative to the solid lane lines, the obtained dotted lane line skeleton segments need to be connected to obtain a complete dotted lane line skeleton.

[0071] (4) The specific method of fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows:

[0072] A linear function is used to fit the straight line segment of the solid lane line skeleton or the dotted lane line skeleton to obtain a fitting curve of the straight line segment of the solid lane line skeleton or the dotted lane line skeleton, the ordinate values ​​of the pixel points of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton are sorted, and then n pixel points (generally 12 pixels) are selected on average within the interval between the maximum and minimum values ​​of the ordinate values ​​of the pixel points to form a connecting line as the fitting curve of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton, the straight line segment of the solid lane line skeleton and the fitting curve of the curved line segment are connected to obtain a solid lane vector line, and the straight line segment of the dotted lane line skeleton and the fitting curve of the curved line segment are connected to obtain a dotted lane vector line.

[0073] If the bending angle of a certain part of the skeleton is less than a preset angle, the part is determined to be a straight line segment; if the bending angle of a certain part of the skeleton is greater than the preset angle, the part is determined to be a straight line segment.

[0074] (5) The specific method for calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and an adjacent dashed lane vector line, and classifying each solid lane line and each dashed lane line corresponding to each solid lane vector line and each dashed lane vector line as a double solid lane line, a double dashed lane line, a left solid line and a right dashed line lane line according to the second vertical distance, the third vertical distance, and the fourth vertical distance, respectively, is:

[0075] Calculate a second vertical distance between two adjacent solid lane vector lines, and if the second vertical distance is less than a second threshold, classify the two adjacent solid lane lines corresponding to the two adjacent solid lane vector lines as double solid lane lines;

[0076] Calculate a third vertical distance between two adjacent dashed lane vector lines, and if the third vertical distance is less than a third threshold, classify the two adjacent dashed lane lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines;

[0077] Calculate the fourth vertical distance between a solid lane vector line and the adjacent dashed lane vector line. If the fourth vertical distance is less than the fourth threshold, calculate the median of the horizontal coordinates of the solid lane vector line and the dashed lane vector line. If the median of the horizontal coordinates of the solid lane vector line is smaller than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the dashed lane vector line are the left solid line and the right dashed line lane line. If the median of the horizontal coordinates of the solid lane vector line is larger than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the solid lane vector line and the dashed lane vector line are the right solid line and the left dashed line lane line.

[0078] Example 3

[0079] A computer-readable medium having a computer program stored thereon, wherein the computer program executes the automatic lane line recognition method based on semantic segmentation in Example 2 when running.

[0080] Example 4

[0081] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the automatic lane line recognition method based on semantic segmentation in Example 2.

[0082] The contents not described in detail in this specification belong to the prior art known to those skilled in the art. It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0083] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. An automatic lane recognition system based on semantic segmentation, It is characterized in that include: An image acquisition module, used for acquiring lane images using a camera device; A lane image recognition module is used to input the lane image into a semantic segmentation model, classify the pixels in the lane image in the semantic segmentation model, and output a lane image including lane line contour information and lane line category information according to the category of the pixels, wherein the lane line category includes a solid lane line and a dashed lane line; A lane line skeleton extraction and fitting module is used to extract a lane line skeleton from a lane image with lane line profile information and lane line category information using an image processing library, wherein the lane line skeleton includes a solid lane line skeleton and a dotted lane line skeleton, and the solid lane line skeleton and the dotted lane line skeleton are fitted to obtain a solid lane vector line and a dotted lane vector line respectively; The lane line classification module is used to calculate the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and the adjacent dashed lane vector line, and classify each solid lane line and each dashed lane line corresponding to each solid lane vector line and each dashed lane vector line into double solid lane lines, double dashed lane lines, left solid line and right dashed line lane lines, and right solid line and left dashed line lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance.

2. The automatic lane recognition system based on semantic segmentation according to claim 1, Features: In the lane image recognition module, the specific method of inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting the lane image with lane line contour information and lane line category information according to the category of the pixels is as follows: The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the category of each pixel in the lane image is output. The lane image is labeled according to the category of each pixel to obtain a lane image with lane contour information and lane category information.

3. The automatic lane recognition system based on semantic segmentation according to claim 1, Features: In the lane line skeleton extraction and fitting module, the specific method of using the image processing library to extract the lane line skeleton from the lane image of the lane line contour information and the lane line category information is as follows: The lane line contour image is binarized using the opencv library, and then the lane line skeleton is extracted from the binarized lane line contour image using the thinning function of the ximgproc module in the opencv library, wherein the thinning function adopts the ZhangSuen algorithm.

4. The automatic lane recognition system based on semantic segmentation according to claim 1, Features: In the lane line skeleton extraction and fitting module, the specific method of obtaining the dotted lane line skeleton is as follows: An image processing library is used to extract all the dotted lane line skeleton segments, and the first vertical distance between the extension line of a dotted lane line skeleton segment and the extension line of the adjacent dotted lane line skeleton segment is calculated. If the first vertical distance is less than a first threshold, the two adjacent dotted lane line skeleton segments are classified as the same dotted lane line skeleton, and then the median of the ordinates of all the dotted lane line skeleton segments belonging to the same dotted lane line skeleton are calculated respectively. All the dotted lane line skeleton segments are connected end to end in order according to the order of the median of the ordinates of each dotted lane line skeleton segment to form the dotted lane line skeleton.

5. The automatic lane recognition system based on semantic segmentation according to claim 1, Features: In the lane line skeleton extraction and fitting module, the specific method of fitting the solid lane line skeleton and the dotted lane line skeleton to obtain the solid lane vector line and the dotted lane vector line respectively is as follows: A linear function is used to fit the straight line segment of the solid lane line skeleton or the dotted lane line skeleton to obtain a fitting curve of the straight line segment of the solid lane line skeleton or the dotted lane line skeleton, the ordinate values ​​of the pixel points of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton are sorted, and then n pixel points are selected on average within the interval between the maximum and minimum values ​​of the ordinate values ​​of the pixel points as the fitting curve of the curved line segment of the solid lane line skeleton or the dotted lane line skeleton, the straight line segment of the solid lane line skeleton and the fitting curve of the curved line segment are connected to obtain a solid lane vector line, and the straight line segment of the dotted lane line skeleton and the fitting curve of the curved line segment are connected to obtain a dotted lane vector line.

6. The automatic lane recognition system based on semantic segmentation according to claim 5, Features In a solid lane line skeleton or a dotted lane line skeleton, if a bending angle at a certain point of the skeleton is less than a preset angle, the point is determined to be a straight line segment; if a bending angle at a certain point of the skeleton is greater than the preset angle, the point is determined to be a curve segment.

7. The automatic lane recognition system based on semantic segmentation according to claim 1, Features: In the lane classification module, the specific method of calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and the adjacent dashed lane vector line, and classifying each solid lane line and each dashed lane line corresponding to each solid lane vector line and each dashed lane vector line as double solid lane lines, double dashed lane lines, left solid line and right dashed line lane lines, and right solid line and left dashed line lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance is: Calculate a second vertical distance between two adjacent solid lane vector lines, and if the second vertical distance is less than a second threshold, classify the two adjacent solid lane lines corresponding to the two adjacent solid lane vector lines as double solid lane lines; Calculate a third vertical distance between two adjacent dashed lane vector lines, and if the third vertical distance is less than a third threshold, classify the two adjacent dashed lane lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines; Calculate the fourth vertical distance between a solid lane vector line and the adjacent dashed lane vector line. If the fourth vertical distance is less than the fourth threshold, calculate the median of the horizontal coordinates of the solid lane vector line and the dashed lane vector line. If the median of the horizontal coordinates of the solid lane vector line is smaller than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the dashed lane vector line are the left solid line and the right dashed line lane line. If the median of the horizontal coordinates of the solid lane vector line is larger than the median of the horizontal coordinates of the dashed lane vector line, the solid lane line and the dashed lane line corresponding to the solid lane vector line and the dashed lane vector line are the right solid line and the left dashed line lane line.

8. A method for automatic lane line recognition based on semantic segmentation. It is characterized in that include: Using a camera to collect lane images; Inputting the lane image into a semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting a lane image including lane line contour information and lane line category information according to the categories of the pixels, wherein the lane line categories include solid lane lines and dashed lane lines; Extracting a lane line skeleton from a lane image with lane line contour information and lane line category information using an image processing library, wherein the lane line skeleton includes a solid lane line skeleton and a dotted lane line skeleton, and fitting the solid lane line skeleton and the dotted lane line skeleton to obtain a solid lane vector line and a dotted lane vector line respectively; The second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and an adjacent dashed lane vector line are calculated; and the solid lane lines and dashed lane lines corresponding to each solid lane vector line and each dashed lane vector line are respectively classified as double solid lane lines, double dashed lane lines, left solid line and right dashed lane lines, and right solid line and left dashed lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance.

9. A computer-readable medium having a computer program / instruction stored thereon, wherein the computer program / instruction executes the method for automatic lane line recognition based on semantic segmentation as claimed in claim 8 when run.

10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instructions are executed by a processor, the method for automatic lane line recognition based on semantic segmentation in claim 8 is implemented.

Images