A lane line automatic recognition system and method based on semantic segmentation
By using semantic segmentation models and image processing techniques to classify and vectorize lane lines at the pixel level, the problem of poor lane marking recognition in complex scenes and under different lighting conditions by traditional algorithms is solved, achieving more accurate lane marking extraction and classification, and improving the recognition capability of autonomous driving systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DONGFENG MOTOR GRP
- Filing Date
- 2024-07-19
- Publication Date
- 2026-07-21
AI Technical Summary
Traditional machine learning algorithms are not effective at recognizing lane markings in complex scenarios and under different lighting conditions, which makes it difficult for autonomous driving systems to achieve automation and results in insufficient accuracy.
The semantic segmentation model mask2former is used to classify lane images at the pixel level. The lane line skeleton is extracted and fitted and classified using the image processing library OpenCV to identify the lane line type.
It achieves accurate identification and vectorization of lane markings, can identify different types of lane markings, improves the understanding and perception capabilities of autonomous driving systems, ensures that the extracted lane markings are consistent with the actual situation, and provides more reliable information for autonomous driving systems.
Smart Images
Figure CN119107618B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of road recognition technology, and specifically to an automatic lane line recognition system and method based on semantic segmentation. Background Technology
[0002] The development of autonomous driving is progressing rapidly, and the most important aspect of achieving autonomous driving is automatic road recognition.
[0003] Traditional machine learning algorithms for extracting lane markings often perform poorly in complex scenes and images under different lighting conditions. For example, in strong light, shadow, or at night, the contrast and edge information of lane markings may be weak. Traditional algorithms usually require manual adjustment of some parameters, such as threshold and filter size, to adapt to different images and scenes. These problems and shortcomings make the algorithm process relatively complex and cannot be automated. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing an automatic lane line recognition system based on semantic segmentation. The system includes: an image acquisition module for acquiring lane images using a camera; a lane image recognition module for inputting the lane images into a semantic segmentation model, classifying pixels in the lane images within the model, and outputting lane line contour information and lane line category information based on the pixel categories, wherein the lane line categories include solid lane lines and dashed lane lines; and a lane line skeleton extraction and fitting module for extracting lane line skeletons from the lane images containing lane line contour information and lane line category information using an image processing library, wherein the lane line skeleton includes solid lines. The system generates lane line skeletons and dashed lane line skeletons, and fits these skeletons to obtain solid lane vector lines and dashed lane vector lines, respectively. A lane line classification module calculates the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and an adjacent dashed lane vector line. Based on these distances, the system classifies each solid lane vector line and each dashed lane vector line into categories such as double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines.
[0005] Furthermore, in the lane image recognition module, the specific method for inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting lane line contour information and lane line category information based on the pixel category is as follows:
[0006] The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the output is the category of each pixel in the lane image. The lane image is labeled according to the category of each pixel to obtain the lane image with lane line contour information and lane line category information.
[0007] Furthermore, in the lane line skeleton extraction and fitting module, the specific method for extracting the lane line skeleton from the lane image using an image processing library based on lane line contour information and lane line category information is as follows:
[0008] The lane line contour image is binarized using the OpenCV library, and then the thinning function of the ximgproc module in the OpenCV library is used to extract the lane line skeleton from the binarized lane line contour image. The thinning function uses the ZhangSuen algorithm.
[0009] Furthermore, in the lane line skeleton extraction and fitting module, the specific method for obtaining the dashed lane line skeleton is as follows:
[0010] All dashed lane line skeleton segments are extracted using an image processing library. The first vertical distance between the extension of a certain dashed lane line skeleton segment and the extension of the adjacent dashed lane line skeleton segments is calculated. If the first vertical distance is less than a first threshold, the two adjacent dashed lane line skeleton segments are classified into the same dashed lane line skeleton. Then, the median of the ordinate of all dashed lane line skeleton segments belonging to the same dashed lane line skeleton is calculated. Based on the magnitude of the median of the ordinate of each dashed lane line skeleton segment, all dashed lane line skeleton segments are connected end to end to form the dashed lane line skeleton.
[0011] Furthermore, in the lane line skeleton extraction and fitting module, the specific method for fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows:
[0012] A linear function is used to fit the straight segments of the solid or dashed lane line skeleton to obtain the fitted curve. The ordinate values of the pixels in the curved segments of the solid or dashed lane line skeleton are sorted. Then, n pixels are selected from the range of the maximum and minimum ordinate values and connected to form the fitted curve for the curved segments. Connecting the fitted curves of the straight and curved segments of the solid lane line skeleton yields the solid lane vector line, and connecting the fitted curves of the straight and curved segments of the dashed lane line skeleton yields the dashed lane vector line. Furthermore, in the solid or dashed lane line skeleton, if the curvature angle at a certain point is less than a preset angle, that point is determined to be a straight segment; if the curvature angle at a certain point is greater than the preset angle, that point is determined to be a curved segment.
[0013] Furthermore, in the lane classification module, the specific method for calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and adjacent dashed lane vector lines, and classifying each solid lane vector line and each dashed lane vector line into double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines based on the second, third, and fourth vertical distances, is as follows:
[0014] Calculate the second vertical distance between two adjacent solid line lane vector lines. If the second vertical distance is less than the second threshold, then classify the two adjacent solid line lane lines corresponding to the two adjacent solid line lane vector lines as double solid line lane lines.
[0015] Calculate the third perpendicular distance between two adjacent dashed lane vector lines. If the third perpendicular distance is less than the third threshold, then classify the two adjacent dashed lane vector lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines.
[0016] Calculate the fourth perpendicular distance between a solid lane vector line and its adjacent dashed lane vector line. If the fourth perpendicular distance is less than the fourth threshold, calculate the median of the x-coordinates of the solid lane vector line and its adjacent dashed lane vector line. If the median of the x-coordinates of the solid lane vector line is smaller than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are left solid and right dashed lane lines. If the median of the x-coordinates of the solid lane vector line is larger than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are right solid and left dashed lane lines.
[0017] An automatic lane line recognition method based on semantic segmentation includes: acquiring lane images using a camera device; inputting the lane images into a semantic segmentation model, classifying pixels in the lane images within the semantic segmentation model, and outputting lane image information including lane line contour information and lane line category information based on the pixel categories, wherein the lane line categories include solid lane lines and dashed lane lines; extracting lane line skeletons from the lane image information using an image processing library, wherein the lane line skeletons include solid line lane line skeletons and dashed line lane line skeletons; and processing the solid line lane line skeletons and dashed line lane line skeletons... The dashed lane line skeleton is fitted to obtain solid lane vector lines and dashed lane vector lines respectively; the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and adjacent dashed lane vector lines are calculated. Based on the second vertical distance, the third vertical distance, and the fourth vertical distance, the solid lane lines and dashed lane lines corresponding to each solid lane vector line and each dashed lane vector line are classified into double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines, respectively.
[0018] A computer-readable medium storing a computer program / instructions that, when executed, perform the above-described automatic lane line recognition method based on semantic segmentation.
[0019] A computer program product includes a computer program / instructions that, when executed by a processor, implement the above-described automatic lane line recognition method based on semantic segmentation.
[0020] The beneficial effects of this invention are as follows:
[0021] 1. By using a semantic segmentation model, this system can perform pixel-level recognition of lane markings, providing more accurate lane marking extraction results. This helps ensure that the extracted lane markings are consistent with actual road conditions, providing more reliable information for use by autonomous driving systems.
[0022] 2. By vectorizing and fitting the extracted lane markings, the system can represent lane markings as geometric shapes, such as straight line segments or curves. This representation facilitates subsequent processing and analysis, such as for path planning and vehicle control.
[0023] 3. For discontinuous dashed line segments, the system can connect them by merging adjacent dashed line segments, thus forming continuous dashed line markings. This helps to provide a more accurate lane line shape and improves the autonomous driving system's understanding and perception of lanes.
[0024] 4. By classifying the extracted lane markings, the system can identify different types of lane markings, such as solid lines, dashed lines, double solid lines, double dashed lines, left solid and right dashed lines, and left dashed and right solid lines. This helps vehicles correctly understand and respond to different types of lane markings during autonomous driving, improving driving safety. Attached Figure Description
[0025] Figure 1 This is a system block diagram of the present invention.
[0026] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation
[0027] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.
[0028] Example 1
[0029] refer to Figure 1 An automatic lane line recognition system based on semantic segmentation includes:
[0030] The image acquisition module is used to acquire lane images using camera equipment;
[0031] The lane image recognition module is used to input the lane image into the semantic segmentation model, classify the pixels in the lane image in the semantic segmentation model, and output the lane image with lane line contour information and lane line category information according to the category of the pixels. The lane line category includes solid lane lines and dashed lane lines.
[0032] The lane line skeleton extraction and fitting module is used to extract lane line skeletons from lane images with lane line contour information and lane line category information using an image processing library. The lane line skeletons include solid line lane line skeletons and dashed line lane line skeletons. The solid line lane line skeletons and dashed line lane line skeletons are fitted to obtain solid line lane vector lines and dashed line lane vector lines, respectively.
[0033] The lane line classification module is used to calculate the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and adjacent dashed lane vector lines. Based on the second, third, and fourth vertical distances, the solid lane lines and dashed lane lines corresponding to each solid lane vector line and each dashed lane vector line are classified as double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines, respectively.
[0034] (1) In the lane image recognition module, the specific method for inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting lane line contour information and lane line category information according to the category of the pixels is as follows:
[0035] The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the output is the category of each pixel in the lane image. The lane image is labeled according to the category of each pixel to obtain the lane image with lane line contour information and lane line category information.
[0036] The core idea of the mask2former model is to treat pixels in an image as a sequence, use positional encoding to preserve the spatial relationships between pixels, and then perform attention calculation and feature fusion based on sequence-level features to generate semantic segmentation results for each pixel.
[0037] The model training process involves taking the original image as input and the lane line annotation map as the target output. By adjusting the model's parameters and loss function, the model can learn to accurately predict the contour and category information of lane lines, where the category information is solid lane lines and dashed lane lines.
[0038] The trained mask2former model was fed into a test dataset of 500 images to predict the contours and categories of lane lines. The IOU of these predictions with the lane line annotations was calculated, resulting in an IOU index of 83.65% for the lane lines. This experimental result shows that the semantic segmentation model is robust to lane line extraction in complex scenes and under different lighting conditions.
[0039] (2) In the lane line skeleton extraction and fitting module, the specific method for extracting the lane line skeleton from the lane image using the image processing library based on the lane line contour information and lane line category information is as follows:
[0040] The lane line contour image is binarized using the OpenCV library, and then the thinning function of the ximgproc module in the OpenCV library is used to extract the lane line skeleton from the binarized lane line contour image. The thinning function uses the ZhangSuen algorithm.
[0041] Since the previous step yielded lane line contour information, it is necessary to extract its skeleton line and convert the polygonal closed region into a curve.
[0042] The ZhangSuen algorithm is a classic binary image thinning algorithm that thins a binary image and removes redundant pixels to obtain a skeleton image.
[0043] (3) In the lane line skeleton extraction and fitting module, the specific method for obtaining the dashed lane line skeleton is as follows:
[0044] All dashed lane line skeleton segments are extracted using an image processing library. The first vertical distance between the extension of a certain dashed lane line skeleton segment and the extension of the adjacent dashed lane line skeleton segments is calculated. If the first vertical distance is less than a first threshold, the two adjacent dashed lane line skeleton segments are classified into the same dashed lane line skeleton. Then, the median of the ordinate of all dashed lane line skeleton segments belonging to the same dashed lane line skeleton is calculated. Based on the magnitude of the median of the ordinate of each dashed lane line skeleton segment, all dashed lane line skeleton segments are connected end to end to form the dashed lane line skeleton.
[0045] Since dashed lane lines are segmented, they need to be connected together to form a complete dashed lane line skeleton, unlike solid lane lines.
[0046] (4) In the lane line skeleton extraction and fitting module, the specific method for fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows:
[0047] A linear function is used to fit the straight segments of the solid or dashed lane line skeleton to obtain the fitted curve. The ordinate values of the pixels of the curved segments of the solid or dashed lane line skeleton are sorted. Then, n pixels (usually 12) are selected on average within the range of the maximum and minimum ordinate values of the pixels and connected to form the fitted curve of the curved segment of the solid or dashed lane line skeleton. The fitted curves of the straight segments and curved segments of the solid lane line skeleton are connected to obtain the solid lane vector line. The fitted curves of the straight segments and curved segments of the dashed lane line skeleton are connected to obtain the dashed lane vector line.
[0048] If the bending angle at a certain point in the skeleton is less than a preset angle, then that point is determined to be a straight line segment; if the bending angle at a certain point in the skeleton is greater than a preset angle, then that point is determined to be a straight line segment.
[0049] (5) In the lane line classification module, the specific method for calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and adjacent dashed lane vector lines, and classifying each solid lane vector line and each dashed lane vector line into double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines according to the second vertical distance, the third vertical distance, and the fourth vertical distance, is as follows:
[0050] Calculate the second vertical distance between two adjacent solid line lane vector lines. If the second vertical distance is less than the second threshold, then classify the two adjacent solid line lane lines corresponding to the two adjacent solid line lane vector lines as double solid line lane lines.
[0051] Calculate the third perpendicular distance between two adjacent dashed lane vector lines. If the third perpendicular distance is less than the third threshold, then classify the two adjacent dashed lane vector lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines.
[0052] Calculate the fourth perpendicular distance between a solid lane vector line and its adjacent dashed lane vector line. If the fourth perpendicular distance is less than the fourth threshold, calculate the median of the x-coordinates of the solid lane vector line and its adjacent dashed lane vector line. If the median of the x-coordinates of the solid lane vector line is smaller than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are left solid and right dashed lane lines. If the median of the x-coordinates of the solid lane vector line is larger than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are right solid and left dashed lane lines.
[0053] Example 2
[0054] refer to Figure 2 An automatic lane line recognition method based on semantic segmentation includes:
[0055] Use camera equipment to capture lane images;
[0056] The lane image is input into the semantic segmentation model, and the pixels in the lane image are classified in the semantic segmentation model. Based on the category of the pixels, the lane image with lane line contour information and lane line category information is output. The lane line category includes solid lane lines and dashed lane lines.
[0057] The image processing library is used to extract lane line skeletons from lane images with lane line contour information and lane line category information. The lane line skeletons include solid line lane line skeletons and dashed line lane line skeletons. The solid line lane line skeletons and dashed line lane line skeletons are fitted to obtain solid line lane vector lines and dashed line lane vector lines, respectively.
[0058] Calculate the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and adjacent dashed lane vector lines. Based on the second, third, and fourth vertical distances, classify each solid lane vector line and each dashed lane vector line into double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines, respectively.
[0059] (1) The specific method for inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting the lane line contour information and lane line category information according to the category of the pixels is as follows:
[0060] The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the output is the category of each pixel in the lane image. The lane image is labeled according to the category of each pixel to obtain the lane image with lane line contour information and lane line category information.
[0061] The core idea of the mask2former model is to treat pixels in an image as a sequence, use positional encoding to preserve the spatial relationships between pixels, and then perform attention calculation and feature fusion based on sequence-level features to generate semantic segmentation results for each pixel.
[0062] The model training process involves taking the original image as input and the lane line annotation map as the target output. By adjusting the model's parameters and loss function, the model can learn to accurately predict the contour and category information of lane lines, where the category information is solid lane lines and dashed lane lines.
[0063] The trained mask2former model was fed into a test dataset of 500 images to predict the contours and categories of lane lines. The IOU of these predictions with the lane line annotations was calculated, resulting in an IOU index of 83.65% for the lane lines. This experimental result shows that the semantic segmentation model is robust to lane line extraction in complex scenes and under different lighting conditions.
[0064] (2) The specific method for extracting the lane skeleton from the lane image using the image processing library based on the lane outline information and lane category information is as follows:
[0065] The lane line contour image is binarized using the OpenCV library, and then the thinning function of the ximgproc module in the OpenCV library is used to extract the lane line skeleton from the binarized lane line contour image. The thinning function uses the ZhangSuen algorithm.
[0066] Since the previous step yielded lane line contour information, it is necessary to extract its skeleton line and convert the polygonal closed region into a curve.
[0067] The ZhangSuen algorithm is a classic binary image thinning algorithm that thins a binary image and removes redundant pixels to obtain a skeleton image.
[0068] (3) The specific method for obtaining the dashed lane line skeleton is as follows:
[0069] All dashed lane line skeleton segments are extracted using an image processing library. The first vertical distance between the extension of a certain dashed lane line skeleton segment and the extension of the adjacent dashed lane line skeleton segments is calculated. If the first vertical distance is less than a first threshold, the two adjacent dashed lane line skeleton segments are classified into the same dashed lane line skeleton. Then, the median of the ordinate of all dashed lane line skeleton segments belonging to the same dashed lane line skeleton is calculated. Based on the magnitude of the median of the ordinate of each dashed lane line skeleton segment, all dashed lane line skeleton segments are connected end to end to form the dashed lane line skeleton.
[0070] Since dashed lane lines are segmented, they need to be connected together to form a complete dashed lane line skeleton, unlike solid lane lines.
[0071] (4) The specific method for fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows:
[0072] A linear function is used to fit the straight segments of the solid or dashed lane line skeleton to obtain the fitted curve. The ordinate values of the pixels of the curved segments of the solid or dashed lane line skeleton are sorted. Then, n pixels (usually 12) are selected on average within the range of the maximum and minimum ordinate values of the pixels and connected to form the fitted curve of the curved segment of the solid or dashed lane line skeleton. The fitted curves of the straight segments and curved segments of the solid lane line skeleton are connected to obtain the solid lane vector line. The fitted curves of the straight segments and curved segments of the dashed lane line skeleton are connected to obtain the dashed lane vector line.
[0073] If the bending angle at a certain point in the skeleton is less than a preset angle, then that point is determined to be a straight line segment; if the bending angle at a certain point in the skeleton is greater than a preset angle, then that point is determined to be a straight line segment.
[0074] (5) The specific method for calculating the second vertical distance between adjacent solid lane vector lines, the third vertical distance between adjacent dashed lane vector lines, and the fourth vertical distance between each solid lane vector line and adjacent dashed lane vector lines, and classifying each solid lane vector line and each dashed lane vector line into double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines based on the second vertical distance, the third vertical distance, and the fourth vertical distance, is as follows:
[0075] Calculate the second vertical distance between two adjacent solid line lane vector lines. If the second vertical distance is less than the second threshold, then classify the two adjacent solid line lane lines corresponding to the two adjacent solid line lane vector lines as double solid line lane lines.
[0076] Calculate the third perpendicular distance between two adjacent dashed lane vector lines. If the third perpendicular distance is less than the third threshold, then classify the two adjacent dashed lane vector lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines.
[0077] Calculate the fourth perpendicular distance between a solid lane vector line and its adjacent dashed lane vector line. If the fourth perpendicular distance is less than the fourth threshold, calculate the median of the x-coordinates of the solid lane vector line and its adjacent dashed lane vector line. If the median of the x-coordinates of the solid lane vector line is smaller than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are left solid and right dashed lane lines. If the median of the x-coordinates of the solid lane vector line is larger than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are right solid and left dashed lane lines.
[0078] Example 3
[0079] A computer-readable medium storing a computer program that, when executed, performs the automatic lane line recognition method based on semantic segmentation in Embodiment 2.
[0080] Example 4
[0081] A computer program product includes a computer program / instructions that, when executed by a processor, implement the automatic lane line recognition method based on semantic segmentation in Embodiment 2.
[0082] The contents not described in detail in this specification are prior art known to those skilled in the art. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.
Claims
1. A lane line automatic recognition system based on semantic segmentation, characterized in that, include: The image acquisition module is used to acquire lane images using camera equipment; The lane image recognition module is used to input the lane image into the semantic segmentation model, classify the pixels in the lane image in the semantic segmentation model, and output the lane image with lane line contour information and lane line category information according to the category of the pixels. The lane line category includes solid lane lines and dashed lane lines. The lane line skeleton extraction and fitting module is used to extract lane line skeletons from lane images with lane line contour information and lane line category information using an image processing library. The lane line skeletons include solid line lane line skeletons and dashed line lane line skeletons. The solid line lane line skeletons and dashed line lane line skeletons are fitted to obtain solid line lane vector lines and dashed line lane vector lines, respectively. The lane line classification module is used to calculate the second perpendicular distance between adjacent solid lane vector lines, the third perpendicular distance between adjacent dashed lane vector lines, and the fourth perpendicular distance between each solid lane vector line and an adjacent dashed lane vector line. Based on the second, third, and fourth perpendicular distances, the solid lane vector lines and dashed lane vector lines corresponding to each solid lane vector line and each dashed lane vector line are classified into double solid lane lines, double dashed lane lines, left solid and right dashed lane lines, and right solid and left dashed lane lines, respectively. The specific method is as follows: Calculate the second vertical distance between two adjacent solid line lane vector lines. If the second vertical distance is less than the second threshold, then classify the two adjacent solid line lane lines corresponding to the two adjacent solid line lane vector lines as double solid line lane lines. Calculate the third perpendicular distance between two adjacent dashed lane vector lines. If the third perpendicular distance is less than the third threshold, then classify the two adjacent dashed lane vector lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines. Calculate the fourth perpendicular distance between a solid lane vector line and its adjacent dashed lane vector line. If the fourth perpendicular distance is less than the fourth threshold, calculate the median of the x-coordinates of the solid lane vector line and its adjacent dashed lane vector line. If the median of the x-coordinates of the solid lane vector line is smaller than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are left solid and right dashed lane lines. If the median of the x-coordinates of the solid lane vector line is larger than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are right solid and left dashed lane lines.
2. The lane line automatic recognition system based on semantic segmentation according to claim 1, characterized in that: In the lane image recognition module, the specific method for inputting the lane image into the semantic segmentation model, classifying the pixels in the lane image in the semantic segmentation model, and outputting lane line contour information and lane line category information based on the pixel category is as follows: The semantic segmentation model uses the mask2former model. The lane image is input into the trained mask2former model, and the output is the category of each pixel in the lane image. The lane image is labeled according to the category of each pixel to obtain the lane image with lane line contour information and lane line category information.
3. The automatic lane line recognition system based on semantic segmentation according to claim 1, characterized in that: In the lane line skeleton extraction and fitting module, the specific method for extracting the lane line skeleton from the lane image using an image processing library based on lane line contour information and lane line category information is as follows: The lane line contour image is binarized using the OpenCV library, and then the thinning function of the ximgproc module in the OpenCV library is used to extract the lane line skeleton from the binarized lane line contour image. The thinning function uses the ZhangSuen algorithm.
4. The automatic lane line recognition system based on semantic segmentation according to claim 1, characterized in that: The specific method for obtaining the dashed lane line skeleton in the lane line skeleton extraction and fitting module is as follows: All dashed lane line skeleton segments are extracted using an image processing library. The first vertical distance between the extension of a certain dashed lane line skeleton segment and the extension of the adjacent dashed lane line skeleton segments is calculated. If the first vertical distance is less than a first threshold, the two adjacent dashed lane line skeleton segments are classified into the same dashed lane line skeleton. Then, the median of the ordinate of all dashed lane line skeleton segments belonging to the same dashed lane line skeleton is calculated. Based on the magnitude of the median of the ordinate of each dashed lane line skeleton segment, all dashed lane line skeleton segments are connected end to end to form the dashed lane line skeleton.
5. The automatic lane line recognition system based on semantic segmentation according to claim 1, characterized in that: In the lane line skeleton extraction and fitting module, the specific method for fitting the solid lane line skeleton and the dashed lane line skeleton to obtain the solid lane vector line and the dashed lane vector line respectively is as follows: A linear function is used to fit the straight segments of the solid or dashed lane line skeleton to obtain the fitted curves. The ordinate values of the pixels of the curved segments of the solid or dashed lane line skeleton are sorted, and then n pixels are selected on average within the range of the maximum and minimum ordinate values to form the fitted curves of the curved segments of the solid or dashed lane line skeleton. The fitted curves of the straight and curved segments of the solid lane line skeleton are connected to obtain the solid lane vector line, and the fitted curves of the straight and curved segments of the dashed lane line skeleton are connected to obtain the dashed lane vector line.
6. The automatic lane line recognition system based on semantic segmentation according to claim 5, characterized in that... In a solid line lane frame or a dashed line lane frame, if the curvature angle at a certain point in the frame is less than a preset angle, then that point is determined to be a straight segment; if the curvature angle at a certain point in the frame is greater than the preset angle, then that point is determined to be a curved segment.
7. A method for automatic lane line recognition based on semantic segmentation, characterized in that, include: Use camera equipment to capture lane images; The lane image is input into the semantic segmentation model, and the pixels in the lane image are classified in the semantic segmentation model. Based on the category of the pixels, the lane image with lane line contour information and lane line category information is output. The lane line category includes solid lane lines and dashed lane lines. The image processing library is used to extract lane line skeletons from lane images with lane line contour information and lane line category information. The lane line skeletons include solid line lane line skeletons and dashed line lane line skeletons. The solid line lane line skeletons and dashed line lane line skeletons are fitted to obtain solid line lane vector lines and dashed line lane vector lines, respectively. Calculate the second perpendicular distance between adjacent solid lane vector lines, the third perpendicular distance between adjacent dashed lane vector lines, and the fourth perpendicular distance between each solid lane vector line and an adjacent dashed lane vector line. Based on the second, third, and fourth perpendicular distances, classify the corresponding solid lane lines and dashed lane lines into double solid lane lines, double dashed lane lines, left solid line and right dashed line lane lines, and right solid line and left dashed line lane lines, respectively. The specific method is as follows: Calculate the second vertical distance between two adjacent solid line lane vector lines. If the second vertical distance is less than the second threshold, then classify the two adjacent solid line lane lines corresponding to the two adjacent solid line lane vector lines as double solid line lane lines. Calculate the third perpendicular distance between two adjacent dashed lane vector lines. If the third perpendicular distance is less than the third threshold, then classify the two adjacent dashed lane vector lines corresponding to the two adjacent dashed lane vector lines as double dashed lane lines. Calculate the fourth perpendicular distance between a solid lane vector line and its adjacent dashed lane vector line. If the fourth perpendicular distance is less than the fourth threshold, calculate the median of the x-coordinates of the solid lane vector line and its adjacent dashed lane vector line. If the median of the x-coordinates of the solid lane vector line is smaller than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are left solid and right dashed lane lines. If the median of the x-coordinates of the solid lane vector line is larger than the median of the x-coordinates of the dashed lane vector line, then the solid lane vector line and the dashed lane vector line corresponding to the solid lane vector line and its adjacent dashed lane vector line are right solid and left dashed lane lines.
8. A computer-readable medium storing a computer program / instructions that, when executed, perform the automatic lane line recognition method based on semantic segmentation as described in claim 7.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the automatic lane line recognition method based on semantic segmentation as described in claim 7.