Semantic segmentation-based road lane line segmentation method and system

Abstract

The invention provides a semantic segmentation-based road lane line segmentation method and system, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining an actual road image; inputting an actual road image into the trained semantic segmentation network to obtain a segmented lane line region; skeletonizing the segmented lane line area to obtain a lane line segmentation result; wherein the semantic segmentation network is an improved BEIT network, in the improved BEIT network, two different feature extractors are used for extracting features from an input image, data enhancement is carried out on the extracted features through an Align mixing strategy, and then segmentation processing is carried out on the enhanced image. According to the invention, the accuracy of road lane line segmentation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and particularly to a method and system for segmenting road lane lines based on semantic segmentation. Background Art

[0002] Lane line recognition is an important technology in advanced driver assistance and autonomous driving. Traditional road lane line detection methods use digital image processing technology to process and segment the collected images using image morphology technology. However, although digital image processing technology has certain advantages in terms of convenience, there are significant limitations in terms of accuracy. The detection and recognition effect of road lane lines is greatly affected by the illumination during image acquisition, as well as the contrast between the color of the lane lines and the color of the road. In addition, the shape, length, area, etc. of the lane lines will also affect the recognition effect. Moreover, the recognition effect is also affected by the recognition scene, and there are differences in processing performance under different scenes.

[0003] With the development of deep learning technology, the road lane lines can be segmented and recognized through deep learning technology. However, the existing technology lacks the ability to learn fine-grained features, resulting in poor segmentation and recognition of fine-grained lane lines. In particular, there are problems such as misrecognition in special scenes such as road waterlogging, partial loss of lane lines, and crack interference. Summary of the Invention

[0004] To solve the problems existing in the prior art, embodiments of the present disclosure provide a method and system for segmenting road lane lines based on semantic segmentation. The technical solutions are as follows:

[0005] In a first aspect, a method for segmenting road lane lines based on semantic segmentation is provided, including:

[0006] Obtain an actual road image;

[0007] Input the actual road image into a trained semantic segmentation network to obtain a segmented lane line area;

[0008] Perform skeletonization processing on the segmented lane line area to obtain a lane line segmentation result;

[0009] Among them, the semantic segmentation network is an improved BEIT network. In the improved BEIT network, the input image is used by two different feature extractors to extract features, and the extracted features are subjected to data augmentation through the AlignMixup strategy, and then the augmented image is subjected to segmentation processing.

[0010] In a second aspect, a system for segmenting road lane lines based on semantic segmentation is provided, including:

[0011] A data acquisition module, configured to acquire actual road images;

[0012] A semantic segmentation module, configured to input the actual road image into a trained semantic segmentation network to obtain a segmented lane line area;

[0013] An image morphological processing module, configured to perform skeletonization processing on the segmented lane line area to obtain a lane line segmentation result;

[0014] Wherein, the semantic segmentation network is an improved BEIT network. In the improved BEIT network, two different feature extractors are used to extract features from the input image, the extracted features are enhanced through the Align mixing strategy, and then the enhanced image is segmented.

[0015] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to complete the steps of the above-mentioned method for segmenting road lane lines based on semantic segmentation.

[0016] In a fourth aspect, a computer-readable storage medium is provided for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for segmenting road lane lines based on semantic segmentation are completed.

[0017] In a fifth aspect, a computer program product is provided, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the above-mentioned method for segmenting road lane lines based on semantic segmentation are implemented.

[0018] The beneficial effects brought by the technical solutions provided in the embodiments of the present disclosure are as follows: In the embodiments of the present disclosure, in view of the deficiencies in using deep learning models for road lane line segmentation and recognition, the Align mixing strategy is introduced to enhance the input image, extract the features of the fine-grained parts in the image, and learn them to achieve a better recognition effect. The Align mixing strategy extracts and mixes the feature parts in fine-grained images with different labels to generate new features. The deep learning model can perform discriminative learning on the new features, improving the accuracy of the model in cases where it actually judges false as true and true as false.

[0019] Advantages of additional aspects of the present disclosure will be partially given in the following description, partially become apparent from the following description, or be understood through the practice of the present disclosure. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 is a flowchart of a road lane line segmentation method based on semantic segmentation provided by an embodiment of the present disclosure;

[0022] Figure 2 is a schematic structural diagram of an improved BEIT network in an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of an I2T module in an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of image preprocessing in an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of a skeletonization algorithm in an embodiment of the present disclosure;

[0026] Figure 6 is a schematic diagram of the definition of pixel positions in the skeletonization algorithm in an embodiment of the present disclosure;

[0027] Figure 7 is a schematic diagram of obtaining the center line of the lane line through skeletonization processing in an embodiment of the present disclosure;

[0028] Figure 8 is a structural block diagram of a road lane line segmentation system based on semantic segmentation provided by an embodiment of the present disclosure;

[0029] Figure 9 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0030] To make the purpose, technical solutions, and advantages of the present disclosure clearer, the following will further describe the embodiments of the present disclosure in detail with reference to the drawings.

[0031] Figure 1 is a flowchart of a road lane line segmentation method based on semantic segmentation provided by an embodiment of the present disclosure. Refer to Figure 1 and the method includes:

[0032] Step 101, obtain an actual road image;

[0033] Step 102, input the actual road image into a trained semantic segmentation network to obtain a segmented lane line area;

[0034] Step 103: Perform skeletonization on the segmented lane line area to obtain the lane line segmentation result.

[0035] In step 101, the actual road image is acquired by an in-vehicle camera. The in-vehicle camera is mounted on the vehicle and is used to obtain the road image in front of the vehicle during vehicle driving, providing visual information for vehicle assisted driving or autonomous driving functions.

[0036] In step 102, the semantic segmentation network is a deep learning network used to segment the lane lines in the image, so as to further extract the center line of the lane lines subsequently, facilitating decision-making and calculation by the autonomous driving computing center.

[0037] In this embodiment, the Vision Transformer (ViT) deep learning technology is used to perform semantic segmentation on the road lane lines, which can quickly and accurately detect the road lane lines, accurately identify the positions and types of the lane lines, and then use image morphology technology to perform skeletonization on the segmented image to obtain the center line of the lane lines and perform distance calculation.

[0038] First, collect typical road lane line images and create mask files to describe the positions and categories of the lane lines in the images, obtaining a training set and a test set. Considering that there may be multiple types of road lane lines in the same image, clear descriptions are required in the mask files.

[0039] Camera calibration is an important step in detecting the distance of road lane lines. To obtain actual distance metrics using images, camera calibration is required to convert image pixel coordinates into physical dimensions in the real world.

[0040] 1. Prepare a calibration object. A calibration board with a size pattern is needed, usually a checkerboard with multiple easily detectable feature points in the pattern.

[0041] 2. Collect images. Use the camera to take multiple photos of the calibration board from multiple different angles and distances.

[0042] 3. Corner detection. Use the OpenCV library to automatically detect the corners of the checkerboard in each image, providing the positions of the corners in the image coordinate system.

[0043] 4. Camera calibration algorithm. Utilize the detected corner pixel image coordinates and their corresponding real coordinates, and run the camera calibration algorithm to solve the internal parameters (focal length, principal coordinate, distortion coefficient) and external parameters (the position and orientation of the camera relative to the world coordinate system) of the camera.

[0044] 5. Calculate the mapping relationship. Construct a mapping function through the internal and external parameters to facilitate the conversion of image pixel coordinates into the actual length and width of the defects in the image.

[0045] Secondly, a semantic segmentation network based on Vision Transformer is constructed. The core of the Vision Transformer (abbreviated as ViT) model lies in that it abandons the convolutional layers commonly used in traditional convolutional neural networks and instead uses the self-attention mechanism to process image data. This innovation brings new perspectives and efficient methods to image recognition and processing tasks. The network adopted in this embodiment is an improved BEIT network, as Figure 2 shown.

[0046] The advantages of the improved BEIT network in this embodiment compared with traditional convolutional neural networks are:

[0047] Global vision: The self-attention mechanism endows the BEIT model with a global receptive field, enabling it to directly model long-range dependencies, which is particularly beneficial for identifying complex image structures.

[0048] Flexibility: Unlike CNNs with fixed local receptive fields and shared weights, BEIT is more flexible in processing images and can theoretically better adapt to various scale and shape changes.

[0049] Scalability: Due to the simplicity of its architecture, BEIT is easier to combine with other Transformer components, facilitating model expansion and improvement, and is applicable to various vision tasks such as object detection and semantic segmentation.

[0050] Use the Align mixing strategy for data augmentation: Align mixing is a new data augmentation algorithm that can fuse image features and enhance the model's ability to learn features.

[0051] Adopt the pre-training strategy: Directly training the model without going through Align mixing is called pre-training.

[0052] Do not use the MIM strategy: There are two main considerations. First, the MIM strategy has been added during the pre-training process, and the model already has sufficient generalization ability. Second, the Align mixing strategy has been used for data augmentation. If the MIM strategy is used simultaneously, it will bring a large amount of noise and is not conducive to the data augmentation process of the algorithm.

[0053] Use the Image2Token (hereinafter referred to as I2T) layer to enhance the feature extraction ability of the Align mixing strategy: The I2T network layer can effectively extract image features, and the pooling layer in I2T can help the model reduce the learning overhead and improve the model performance.

[0054] Introduce a data augmentation module and adopt the Align hybrid strategy to perform data augmentation on the input images. Set an image x with its label y, where the label y is one-hot encoded label data. (x, y) forms a pair of image-label data. Set the encoder network as f = D(x), where f is the feature tensor after passing through the x encoder D.

[0055] First, introduce the concept of linear mixing: Given two pairs of image-label data (x, y), (x', y'). Introduce an interpolation factor μ ∈ (0, 1) whose probability distribution follows a Beta distribution Beta(α, α) to balance the weights of the labels y and y'. The mathematical expression of linear mixing is:

[0056] mix μ (y, y') = μy + (1 - μ)y'

[0057] This is called linear mixing. Based on linear mixing, corrected mixing is introduced, that is, the concept of the Align hybrid strategy adopted in this patent. Corrected mixing is divided into two parts: the alignment stage and the mixing stage.

[0058] (1) Alignment stage. Alignment means finding the geometric correspondence between image elements before interpolation. Selecting to mix the feature tensors is an ideal method because the spatial resolution of the feature tensors is very low, reducing the optimization cost and allowing semantic correspondence. Its mathematical principle is based on the optimal transport theory and the Sinkhorn distance theory. The features F = D(x), F' = D(x') are the results after extracting features from the images x and x' respectively. Here, the feature extractors are I2T and the pooling network Max pooling. Reconstruct the dimensions of the features F and F' into the c*r format, and each row f j , f′ j , j = 1, …, r are the spatial vectors of the image, representing each spatial position in the image. Let M be an r*r cost matrix, and its elements are the pairwise distances of these vectors:

[0059] m ij = ||f i - f' j || 2 , i, j ∈ {1,..., r}

[0060] At the same time, set the transport plan matrix P as an r*r matrix:

[0061] PI = P T I = 1 / r

[0062] Among them, 1 is a vector of all 1s, which means that P is non - negative and the sum of each row and each column is 1 / r, used to represent f with uniform margins. j , f′ j is the joint probability at the spatial position of. Under the entropy regularization condition, it is selected to minimize the expected pairwise distance of its features, and this pairwise distance is represented by the linear cost function <P, M>:

[0063] P* = argmin <P, M> - εH(P) P ∈ Ur

[0064] H(P) = -∑ ij p ij logp ij is the entropy of P, <,> is the trace inner product (Frobenius inner product), and ε is a regularization coefficient. The optimal solution P * is unique and can be obtained by forming an r - order similarity matrix e -M / ε , and applying the Sinkhorn - Knopp algorithm iteratively. A smaller ε leads to a sparse P, improving the vector matching degree, but making the optimization more difficult, while a larger ε leads to a denser P, resulting in a worse vector matching degree.

[0065] (2) Mixing stage: Assign the matrix R = rP * is a doubly - stochastic r - order matrix, and its element r ij represents the probability consistency of the F - row f i corresponding to the row f j ' of F'. Therefore, align f j , f′ j as follows:

[0066]

[0067] where is the determinant of c*r, and its rows are the convex combinations corresponding to the same row f i of F. By expanding the spatial dimension, reshape into a tensor of c*w*h, which represents the alignment of F and F'. Then mix with the original feature vector F:

[0068]

[0069] Similarly, there is mixing with the original feature vector F':

[0070]

[0071] These two mixing schemes are randomly selected in the algorithm to enhance the generalization ability of the model and improve the recognition accuracy of the model for lane lines.

[0072] On this basis, in this embodiment, two different feature extractors are used to generate mixed labels by mixing features of the same image, so as to further improve the generalization ability of the model, increase the data utilization rate, improve the model performance, and enhance the resolution ability of the model for fine-grained images with the help of this algorithm.

[0073] In addition, in this embodiment, an I2T module is further introduced for image processing. Image2Token (I2T) is a lightweight network structure. As Figure 3 shown, the network includes a convolutional network layer and a pooling network layer. The network layer is embedded in the data augmentation module to extract features from the image data before using data augmentation for calculation, helping the network converge faster and improving the accuracy of the network. For the input image x, the output feature x' of I2T can be expressed as:

[0074] x' = I2T(x) = MaxPool(BN(Conv(x)))

[0075] where the dimension of x' is where S is the stride in the convolutional layer and C is the number of channels of the data.

[0076] Compared with the existing BEIT network, the improved BEIT network adds a data augmentation module to perform data augmentation on the input image. The data augmentation module uses the Align mixing strategy for data augmentation, which can confuse the fine-grained parts in the image data, improve the recognition generalization ability of the model, improve the scene usability, and I2T can extract the image features to better obtain the image features.

[0077] In step 103, the lane line distance is measured based on image morphology and image skeletonization technology to provide distance information support for vehicle autonomous driving.

[0078] First, image preprocessing is performed. For the lane line area obtained by semantic segmentation, the lane line area keeps the color unchanged, and other areas are filled with black, as Figure 4 shown. Image morphology technology can better process such images.

[0079] Subsequently, the area image is skeletonized. Each time the skeletonization algorithm runs, all non-zero pixels need to be traversed. The algorithm flow chart is as Figure 5 shown. When judging whether to delete or retain each traversed pixel (P1), the values of its surrounding 8 neighbor pixels (P2, P3, P4, P5, P6, P7, P8) need to be concerned, where the order of P2 to P8 is stipulated by the algorithm. AsFigure 6 as shown for subsequent judgment.

[0080] Each run of the algorithm requires two sub - stages. In each sub - stage, all 4 requirements need to be met to set a certain pixel to zero. That is, in stage one, for P1, when 2 ≤ B(P1) ≤ 6, A(P1) = 1, P2 * P4 * P6 = 0, and P4 * P6 * P8 = 0 are simultaneously satisfied, P1 is set to 0; similarly, in stage two, when 2 ≤ B(P1) ≤ 6, A(P1) = 1, P2 * P4 * P8 = 0, and P2 * P6 * P8 = 0 are simultaneously satisfied, P1 is set to 0. Here, B(P1) represents the number of non - zero neighbors among 8 neighbors, and A(P1) refers to how many times the 8 neighbor pixels from P2 to P8 change from 0 to 1.

[0081] After skeletonization, the center line of the lane can be accurately obtained, as Figure 7 shown. Further, operations such as ranging and position judgment can be realized based on the measurement of the obtained center line.

[0082] In the embodiments of the present disclosure, through deep learning technology, the road lane lines can be classified and recognized quickly and accurately, and the distance from the lane lines can be obtained through image morphological processing. Compared with the prior art, this solution combines deep learning technology and image morphological technology to complement their advantages. It can perform accurate distance measurement on the basis of accurately segmenting the lane lines, conveniently and quickly identify the position and type of the lane lines, thereby improving the perception and recognition of road lane lines by autonomous driving technology.

[0083] Figure 8 is a structural block diagram of a road lane line segmentation system 200 based on semantic segmentation provided by the embodiments of the present disclosure, as Figure 8 shown. The system includes: a data acquisition module 201, a semantic segmentation module 202, and an image morphological processing module 203.

[0084] Among them, the data acquisition module 201 is configured to acquire actual road images;

[0085] The semantic segmentation module 202 is configured to input the actual road image into a trained semantic segmentation network to obtain the segmented lane line area; among them, the semantic segmentation network is an improved BEI T network. In the improved BEI T network, two different feature extractors are used to extract features from the input image, and the extracted features are enhanced through the Al ign mixing strategy, and then the enhanced image is segmented.

[0086] The image morphological processing module 203 is configured to perform skeletonization processing on the segmented lane line area to obtain the lane line segmentation result.

[0087] It should be noted that: For the road lane line segmentation system 200 based on semantic segmentation provided in the above embodiments, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be assigned to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the road lane line segmentation system 200 based on semantic segmentation provided in the above embodiments and the embodiments of the road lane line segmentation method based on semantic segmentation belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.

[0088] Figure 9 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. As Figure 9 shown, the electronic device 300 may be a computer. The electronic device 300 includes: a processor 301 and a memory 302.

[0089] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0090] The memory 302 may include one or more computer-readable media, which may be non-transitory. The memory 302 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable media in the memory 302 is used to store at least one computer program, and the at least one computer program is used to be executed by the processor 301 to implement a road lane line segmentation method based on semantic segmentation provided in the embodiments of the present disclosure.

[0091] Those skilled in the art can understand that Figure 3 the structure shown in does not constitute a limitation on the electronic device 300, and it may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0092] The embodiments of the present disclosure also provide a computer-readable storage medium for storing computer instructions, and when the computer instructions are executed by a processor, the steps of a road lane line segmentation method based on semantic segmentation provided in the embodiments of the present disclosure can be completed.

[0093] The embodiments of the present disclosure also provide a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of a road lane line segmentation method based on semantic segmentation provided in the embodiments of the present disclosure are implemented.

[0094] The above are only the preferred embodiments of the present disclosure and are not used to limit the present disclosure. For those skilled in the art, various changes and modifications can be made to the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A road lane segmentation method based on semantic segmentation, characterized in that: include: Acquire actual road images; Input the actual road image into the trained semantic segmentation network to obtain the segmented lane line area; The segmented lane line area is skeletonized to obtain the lane line segmentation result; The semantic segmentation network is an improved BEIT network. In the improved BEIT network, two different feature extractors are used to extract features from the input image, the extracted features are data enhanced through the Align hybrid strategy, and then the enhanced image is segmented.

2. The method for road lane segmentation based on semantic segmentation according to claim 1, characterized in that: The Align mixing strategy includes aligning two features extracted by two different feature extractors through an assignment matrix to obtain an alignment feature corresponding to each feature, and finally linearly mixing the alignment feature with the original feature.

3. The method for road lane segmentation based on semantic segmentation as claimed in claim 2, characterized in that: Linearly mix the aligned features with the original features, including randomly linearly mixing one of the two features with its corresponding aligned feature.

4. The method for road lane segmentation based on semantic segmentation according to claim 1, characterized in that: It also includes using the I2T module to process the features extracted by the feature extractor, and performing data enhancement on the processed features through the Align hybrid strategy.

5. The method for road lane segmentation based on semantic segmentation according to claim 1, characterized in that: The segmented lane line area is skeletonized, including preprocessing the image and applying a skeletonization algorithm to the preprocessed image to extract the center line of the lane line; the image preprocessing includes, for the lane line area obtained by semantic segmentation, keeping the color of the lane line area unchanged and filling other areas with black.

6. The method for road lane segmentation based on semantic segmentation according to claim 5, characterized in that: The skeletonization algorithm includes two sub-stages. In stage one, for pixel P1, when 2≤B(P1)≤6, A(P1)=1, P2*P4*P6=0, P4*P6*P8=0 are satisfied at the same time, P1 is set to 0; in stage two, when 2≤B(P1)≤6, A(P1)=1, P2*P4*P8=0, P2*P6*P8=0 are satisfied at the same time, P1 is set to 0; wherein B(P1) represents the number of non-zero neighbors among the 8 neighbors of pixel P1, and A(P1) refers to the number of times the 8 neighbor pixels from P2 to P8 change from 0 to 1.

7. A road lane segmentation system based on semantic segmentation, characterized in that: include: A data acquisition module is configured to acquire an actual road image; The semantic segmentation module is configured to input the actual road image into the trained semantic segmentation network to obtain the segmented lane line area; The image morphology processing module is configured to perform skeleton processing on the segmented lane line area to obtain a lane line segmentation result; The semantic segmentation network is an improved BEIT network. In the improved BEIT network, two different feature extractors are used to extract features from the input image, the extracted features are data enhanced through the Align hybrid strategy, and then the enhanced image is segmented.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images