Deep learning based pavement marking perception method

Abstract

The application provides a kind of road surface mark perception method based on deep learning, first the image of high-definition visible light camera is preprocessed, then high-dimensional characteristic value is obtained by shared convolutional neural network, the network task head of shared convolutional neural network is divided into road segmentation head network, lane line segmentation head network and road surface mark detection head network, high-dimensional characteristic value is input into three networks respectively, lane line classification flag is judged, road surface mark contour flag is judged, finally multi-task fusion is output, and perception result is output.The application can input RGB image photographed by camera as input through the designed fusion network, extract deep features through the network, then output road element mask and corresponding category through multi-task framework, which can avoid redundant calculation of simple tasks, and can also output complete road surface perception segmentation mask through configuration switch.

Description

Technical Field

[0001] This invention relates to the field of lane line recognition technology, specifically a road marking perception method based on deep learning. Background Technology

[0002] Currently, with the rapid development and improvement of deep learning, autonomous driving technology has made tremendous progress. Environmental perception systems based on multi-sensor fusion are also widely used, such as in autonomous vehicles, autonomous navigation and positioning robots, and unmanned logistics. Among these, visual perception modules and high-precision map positioning modules are the most important technologies in environmental perception systems. Vision can acquire a wealth of environmental information to perform various visual tasks. Visual perception modules and high-precision map positioning modules are the core technical modules in autonomous driving technology, used in applications such as drivable area segmentation, lane detection, and detection of other road traffic signs.

[0003] However, road segmentation, lane detection, and other sign recognition involve three major visual tasks: target classification, target detection, and semantic segmentation. The feature extraction module in these three tasks directly affects the real-time driving perception of the entire perception module. Finding a method to share learning across different visual tasks in the aforementioned situations—i.e., multi-task learning—is crucial for improving model performance. Furthermore, for lane lines in autonomous driving, it's not enough to simply detect the lane lines; the lane line category (e.g., single solid yellow line, double solid yellow line, single solid line, etc.) must also be known. Currently, semantic segmentation is the primary method for recognizing other road signs. However, considering the real-time performance and throughput limitations of the perception module, could a more efficient method be adopted for pixel-level recognition and classification of other road signs (turn arrows, speed bumps, etc.)?

[0004] Chinese patent application number CN202111501632.2 provides a lane line recognition method, system, medium, device and information processing terminal. The application proposes to use a self-developed multi-task visual network to perform semantic segmentation and task perception of road surface elements. However, the application lacks pixel-level segmentation of lane line classification module and other road surface markings. Summary of the Invention

[0005] To address the problems of existing technologies, this invention provides a road marking perception method based on deep learning. This method uses a designed fusion network to take RGB images captured by a camera as input, extracts depth features through the network, and then outputs road element masks and corresponding categories through a multi-task framework.

[0006] This invention provides a deep learning-based method for road marking perception, comprising the following steps:

[0007] Step one: the image captured by high-definition visible light camera is preprocessed to obtain the input format I of neural network i Then high-dimensional feature value O is obtained through shared convolutional neural network s High-dimensional feature value O is copied three times as O s ,O s1 ,O s2 ,O s3 Among them, the network task head of the shared convolutional neural network is divided into three parts, including road segmentation head network, lane line segmentation head network and road marking detection head network

[0008] Step two: high-dimensional feature value O s1 ,O s2 ,O s3 In step one is input into road segmentation head network, lane line segmentation head network and road marking detection head network respectively, and then road segmentation result o road-mask , lane line segmentation result o line-mask And road marking detection frame O bbox are output respectively

[0009] Step three: judge lane line classification flag; if it is 1, classify the lane line, separate o line-mask In step two by lane line mask, and image scale to obtain the input feature I line of lane line classification network; then through the preset lane line classification network, the lane line category is obtained, and finally the lane line category is attached to each segmentation mask to obtain O i-line-mask ; otherwise, do not classify the lane line, and directly output the lane line O line-mask ;

[0010] Step four: judge the extraction of road marking contour flag; if it is 1, the road marking contour is segmented into foreground and background, and O bbox In step two is cropped to obtain the target-based foreground and background image, and the scale is obtained to obtain the input I bbox of road marking foreground and background segmentation network; then through the preset foreground and background segmentation network, the road marking O bbox-mask is obtained; finally, each road marking is mapped to the original image to obtain O i-bbox-mask ; otherwise, do not extract the road marking contour, and directly output the road marking detection frame O bbox ;

[0011] Step five: multi-task result fusion module; if it belongs to map reconstruction or map updating work, that is, the lane line classification flag and the road marking contour flag are both 1, the output O road-mask In step two, the output O i-line-mask In step three, and the output O i-bbox-maskPixel-level fusion is performed, and the output result is a road map element output result; if it belongs to real-time perception work, i.e., the lane line classification flag and the road sign contour flag are both 0, the O road-mask output in step two line-mask , the O bbox output in step three all and the O det output in step four are fused, and the output result is a road perception result output.

[0012] The shared convolutional neural network training process in step one includes training data processing, network design optimization, shared feature extraction and feature extraction fusion;

[0013] Training data processing: first, visible light images under normal light are captured by an unmanned vehicle, and image annotation is performed, including road and lane line pixel-level annotation, road sign, and target detection frame annotation. The annotation data standardization format is coco data set format.

[0014] Network design optimization: the road and lane line in the visible light image are pixel-level segmented, and the road sign is target detected, to obtain road and lane line mask and road sign bbox.

[0015] Shared feature extraction: a resnet-34 framework is adopted, and different expansion rates of the hole convolution are adopted in the third and fourth network structures of the network.

[0016] Feature extraction fusion: different spatial pyramid pooling layers are used to extract features, and one feature map is input into different spatial pyramid pooling layers for maximum pooling processing.

[0017] In the shared convolutional neural network in step one, the road segmentation head network and the lane line segmentation head network are composed of deconvolution layers and activation layers, and finally a sigmod activation function is connected. There are 6 convolution layers and 3 deconvolution layers in total; the road sign detection head network adopts a multi-layer feature fusion structure, which fuses SPP different layer features through a concat layer, and then obtains a target detection regression result through a convolution layer and a fully connected layer network.

[0018] In the shared convolutional neural network in step one, total variation denoising is introduced in the loss function to perform image denoising.

[0019] The total loss function of network training is as follows:

[0020] L all =γ1L det +γ2L da-seg +γ3L ll-seg +γ4L tv ,

[0021] Wherein, alpha1, alpha2, alpha3, gamma1, gamma2, gamma3, gamma4 are respectively each part loss function weight, is a constant; L tv is total variation loss, L det is detection network head loss function and:

[0022] L det = alpha1L class + alpha2L obj + alpha3L box ,

[0023] Wherein, alpha t and gamma c are weight coefficients, p t is the category probability; wherein alpha j and gamma o are weight coefficients, p t is the category probability;

[0024] rho 2 (b, b gt ) represents the Euclidean distance between the center points of the predicted box and the real box, c represents the diagonal distance of the smallest closed region that can contain the predicted box and the real box, box iou is the intersection over union of the labeled box and the predicted box, and the formula of alpha and nu is as follows: is the width-height ratio of the labeled box, is the width-height ratio of the predicted box;

[0025] L da-seg is road segmentation loss function and:

[0026] L da-seg = L ce ,

[0027] L ll-seg is lane line segmentation loss function and:

[0028] L ll-seg = L ce + L IoU ,

[0029] Wherein, L ce represents the classification error of network output pixel and target pixel;

[0030] L IoU represents the intersection over union error of network output lane line region and target region;

[0031] Total variation loss is defined as follows:

[0032]

[0033] Wherein, I i,j , I i,j+1 , I i+1,j Respectively, the center point pixel, the center point under the pixel and the center point right side pixel value.

[0034] The training process of the lane line classification network in step three is as follows:

[0035] Training data processing: through the lane line mask output by the trained visual multi-task network, then according to the lane line mask, the lane line contour is obtained, and the lane line is cut from the original picture according to the lane line contour, then the image is resized to a square through image preprocessing as the network input image, and then the lane line is classified; the annotation categories in the training data processing include double dashed line, single dashed line, double yellow solid line, single white solid line, single yellow solid line, double yellow solid line; the training data includes training set, verification set and test set;

[0036] Network design and training: including neural network by convolution layer, maximum pooling layer, activation layer and full connection layer; the activation function adopts SeLU; finally, 6 channels of softmax are used as classification output, and dropout regularization method is used to avoid overfitting;

[0037] Local corresponding normalization: for the local center features of lane line classification, local corresponding normalization is adopted, and the local generalization formula is as follows:

[0038]

[0039] Wherein, alpha is a constant, which is set according to the empirical value. The data is set to five values of different sizes by the control variable method, and then the network output result precision is determined by comparison.

[0040] The training process of the road surface mark front background segmentation network in step four is as follows:

[0041] Training data processing: through the road surface mark bbox output by the trained visual multi-task network, then the foreground target detection frame region in the original picture is extracted through the bbox coordinates as the original picture of the data set, and then the matting result picture is obtained through the graph tool as the annotation data;

[0042] Network design or training: the network adopts MSIA-matte structure.

[0043] The present application has the beneficial effects of:

[0044] 1. The RGB image captured by the camera can be taken as input through the designed fusion network, the deep features are extracted through the network, and then the road element mask and the corresponding category are output through the multi-task framework.

[0045] 2. The algorithm framework can be used to end-to-end process a large number of visible light images collected to obtain road, lane line and road sign mask results.

[0046] 3. The multi-task network method based on deep learning provided in the application can configure the algorithm framework branch switch according to product requirements or scene requirements, so as to output different road perception mask results.

[0047] 4. The algorithm architecture can avoid redundant calculation of simple tasks, and can also output complete road perception segmentation mask through the configuration switch. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 is a visual multi-task algorithm framework diagram;

[0050] Figure 2 is a foreground segmentation network diagram. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0052] Firstly, the execution steps of the visual multi-task algorithm framework flowchart provided by the application are as shown in Figure 1 , and are as follows:

[0053] Step 1: Preprocess the image captured by the high-definition visible light camera to obtain the input format I of the neural network i , and then obtain high-dimensional feature values O through the shared convolutional neural network s , copy three copies of the high-dimensional feature values O s , respectively as O s1 , O s2 , and O s3 ;

[0054] Step 2: Convert the high-dimensional eigenvalues ​​O from Step 1... s1 O s2 O s3 Input the road segmentation head network, lane line segmentation head network, and pavement marking detection head network respectively, and then output the road segmentation results respectively. road-mask Lane line segmentation results line-mask and road marking detection frame O bbox ;

[0055] Step 3: Determine the lane marking classification flag. If it is 1, classify the lane marking and change the 'o' flag from Step 2. line-mask Lane line mask separation and image scaling are performed to obtain the input features I of the lane line classification network. line Then, through a pre-defined lane classification network, lane category is obtained, and finally, the lane category is appended to each segmentation mask to obtain O. i-line-mask Conversely, if lane lines are not classified, the lane line O is output directly. line-mask ;

[0056] Step 4: Determine and extract the road marking outline marker position. If it is 1, then perform foreground and background segmentation on the road marking outline, and extract the O from Step 2. bbox Each target region is cropped to obtain a target-based foreground and background image, which is then scaled to obtain the input I of the road marking foreground and background segmentation network. bbox Then, the road marking O is obtained through a preset foreground and background segmentation network. bbox-mask Finally, the road markings are mapped onto the original image to obtain O. i-bbox-mask Conversely, if the road marking outline is not extracted, the road marking detection frame O is directly output. bbox ;

[0057] Step 5: Multi-task result fusion module. If it belongs to map reconstruction or map update work, that is, the lane line classification marker and the road surface marking outline marker are both 1, then the O output in Step 2 will be fused. road-mask The O output in step three i-line-mask The O output in step four i-bbox-mask Perform pixel-level fusion, and the output result is the road surface map element output result; if it belongs to real-time perception work, that is, the lane line classification marker and the road surface marking outline marker are both 0, then the O output in step two will be... road-mask The O output in step three line-mask The O output in step four bbox Perform bounding box and mask fusion, and output the road perception result.

[0058] II. Model Training

[0059] The neural network training included in the present application includes visual multi-task network training, lane line classification network and front background segmentation network.

[0060] 1. Visual multi-task network training

[0061] Training data processing: first, visible light images under normal light are captured by an unmanned vehicle for image annotation. The annotation content includes road and lane line pixel-level annotation, road sign (left arrow, right arrow, speed bump, etc.) target detection box annotation, and the annotation data standardization format is coco dataset format. A total of 5600 labeled data are obtained, including 4480 training sets, 560 verification sets and 560 test sets.

[0062] Network design optimization: the target task of the network is to perform pixel-level segmentation of road and lane line in visible light images and target detection of road signs to obtain road and lane line mask and road sign bbox. The training network is composed of a shared feature extraction network and a road segmentation head network, a lane line segmentation head network and a target detection head network.

[0063] The network structure adopts resnet-34 in the shared feature extraction part, and adopts an empty convolution in the third and fourth layers of the network structure to alleviate the problem of high-level feature sparsification. In the field of image segmentation, image feature extraction often uses a combination of pooling layers and up-sampling layers to reduce image size and increase receptive field, and then restore to the original image size for prediction. In the process of image size reduction and increase, a lot of detail information is lost, making it impossible to reconstruct some detail information. The empty convolution can avoid the loss of detail information to some extent. However, the empty convolution has the following problems: ① When multiple empty convolutions with the same expansion rate are stacked, many pixels in the receptive field are not used, resulting in a large number of empty holes, i.e. grid effect. ② The purpose of designing the empty convolution is to obtain a larger receptive field and improve the segmentation ability of the model for large target objects, but small target objects do not need a large receptive field and are not suitable for using empty convolutions with large expansion rates. In view of the above problems, the present patent proposes an HDC (hybrid empty convolution) that can meet the detection needs of large and small targets, and different expansion rates of empty convolutions are used in the third and fourth layers of the network structure, and the step of the empty convolution is adjusted for small targets.

[0064] The network structure adopts a spatial pyramid module (SPP) to improve the extraction and fusion of context features, and at the same time, the extraction of deep features is strengthened. In real samples, there are often many different feature scales. In order to ensure that the sample has a fixed size, there are usually two processing methods: the first is to crop the original image, which will inevitably crop some relevant features, and the feature extraction will be affected to some extent. The second is to scale the original image, and the obtained image becomes distorted, which will affect the feature extraction to some extent. The SPP can make the model input an image of any scale, and also ensure that the output is a fixed size picture. The SPP network layer adopts three spatial pyramid pooling layers of different sizes (4X4)\(2X2)\(1X1) to extract features. A feature map is input into different spatial pyramid pooling layers for maximum pooling processing. The output feature map is (16+4+1)x256 in size.

[0065] The network task head is divided into three parts: road segmentation task network, lane line segmentation task network and road surface sign detection task network. Among them, the road segmentation task network and the lane line segmentation task network are the same, mainly composed of deconvolution layers and activation layers, and finally connected with a sigmod activation function. There are a total of 6 convolution layers and 3 deconvolution layers. The road surface sign detection network adopts a network structure similar to yolov5, which adopts a multi-layer feature fusion structure. Through the concat layer, the SPP features of different layers are fused, which improves the feature extraction of targets from large to small in the process of unmanned vehicle driving. Then, through the convolution layer and the fully connected layer network, the target detection regression result is obtained.

[0066] In order to improve the comprehensive performance of the network, the total loss function of network training is as follows:

[0067] L all =γ1L det +γ2L da-seg +γ3L ll-seg +γ4L tv ,

[0068] Among them, α1, α2, α3, γ1, γ2, γ3, γ4 are the weights of each part of the loss function, which are constants; L tv is the total variation loss, L det is the loss function of the detection network head, and:

[0069] L det =α1L class +α2L obj +α3L box ,

[0070] Among them, L class =-α t (1-p t )γc log(p t ), α t and γ c It is the weighting coefficient, p t Let L be the class probability; where L is the class probability. obj =-α j (1-p t )γo log(p t ), α j and γ o It is the weighting coefficient, p t For class probabilities;

[0071] ρ 2 (b,b gt The square root of the predicted bounding box (box) represents the Euclidean distance between the centers of the predicted and ground truth bounding boxes, and the square root of the predicted bounding box (box) represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes. iou Let α and ν be the intersection-union ratio of the labeled boxes and the predicted boxes, and the formulas for α and ν are as follows: The aspect ratio of the annotation box. The aspect ratio of the predicted bounding box;

[0072] L da-seg Let be the road segmentation loss function and:

[0073] L da-seg =L ce ,

[0074] L ll-seg Let the lane line segmentation loss function be:

[0075] L ll-seg =L ce +L IoU ,

[0076] Among them, L ce This represents the classification error between the network output pixel and the target pixel;

[0077] L IoU This represents the intersection-over-union ratio error between the lane line region and the target region output by the network;

[0078] The total variation loss is defined as follows:

[0079]

[0080] Among them, I i,j I i,j+1 I i+1,j These represent the pixel values ​​at the center point, the pixel below the center point, and the pixel value to the right of the center point, respectively.

[0081] 2. Lane line classification network training

[0082] Training data processing: The lane line mask output by the trained visual multi-task network is then used to obtain the lane line contour, and the lane line is cropped from the original image according to the lane line contour. The image is then resized to a square through image preprocessing as the network input image. The lane line is then labeled by class, and the labeled classes include double dashed line, single dashed line, double yellow solid line, single white solid line, single yellow solid line, and double yellow solid line. The training data includes a training set, a validation set, and a test set.

[0083] Network design and training: The network task belongs to an image-level classification task, so a simple fully convolutional neural network is used, which is composed of convolutional layers, max pooling layers, activation layers, and fully connected layers. The activation function uses SeLU. Finally, a 6-channel softmax is used as the classification output, and dropout regularization is used to avoid overfitting.

[0084] For the local center feature of lane line classification, local corresponding normalization is used, which helps to improve generalization and accuracy. The local generalization formula is as follows:

[0085]

[0086] where α is a constant, which is set according to the empirical value. The data is set to five different values by the control variable method, and then the network output result accuracy is determined by comparison.

[0087] 3. Foreground-background segmentation network training

[0088] Foreground segmentation can be considered as the inverse process of image synthesis. Foreground-background segmentation can also be called digital image matting. In addition to the obvious foreground area, there is a very rich diversity in the edge and detail part, and these traces obviously need different alpha values to express. Therefore, multi-level network feature extraction is needed.

[0089] Training data processing: The road marking bbox output by the trained visual multi-task network is then used to extract the foreground target detection frame area in the original image as the dataset original image, and then the matting result image is obtained through the matting tool as the labeled data. The dataset includes a training set, a validation set, and a test set.

[0090] Network design or training: The network uses the MSIA-matte structure, as shown in Figure 2The structure adopts ResNeXt as the backbone network to extract semantic features representing the main content of the foreground. Meanwhile, another branch, whose bottom features are also used to infer different levels of traces, including foreground details and transparency information. The adaptive combination of these two types of features in the decoding stage is used to generate multi-scale foreground feature expression and predict the aplha channel mask. The loss function adopts L1 loss and SSIM loss to train the network.

[0091] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, the above description is only the preferred embodiments of the present application, and since it is basically similar to the method embodiments, it is described more simply. The above description is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, and the changes or replacements within the technical range disclosed by the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1.A method for pavement marking perception based on deep learning, characterized in that, The method comprises the following steps: Step one: pre-process the image captured by the high-definition visible light camera to obtain the input format I of the neural network i Then obtain high-dimensional feature values O through a shared convolutional neural network s Copy three times to obtain O s ,O s1 ,O s2 ,O s3 , respectively, wherein the network task head of the shared convolutional neural network is divided into three parts, including a road segmentation head network, a lane line segmentation head network and a road marking detection head network; Step two: input the high-dimensional feature value O s1 , s2 , s3 of step one into the road segmentation head network, the lane line segmentation head network and the road surface mark detection head network respectively, and then output the road segmentation result o road-mask , the lane line segmentation result o line-mask and the road surface mark detection frame o bbox respectively; Step three: judge lane line classification flag; if it is 1, classify lane line, and o line-mask Each lane line mask is separated, and image scale is zoomed to obtain input feature I of lane line classification network line ; then, lane line category is obtained through preset lane line classification network, and finally, lane line category is attached to each segmentation mask to obtain O i-line-mask ; otherwise, lane line classification is not performed, and lane line O is directly output line-mask ; Step four: judge the extraction of road sign contour mark flag bit; if it is 1, the road sign contour is segmented into foreground and background, and O bbox Each target region is cropped to obtain a target-based foreground and background image, and is scaled to obtain the input I of the road sign foreground and background segmentation network bbox ; then the road sign O bbox-mask is obtained through the preset foreground and background segmentation network; finally, each road sign is mapped to the original image to obtain O i-bbox-mask ; otherwise, the road sign contour extraction is not performed, and the road sign detection frame O bbox is directly output. Step five: multi-task result fusion module; if it belongs to map reconstruction or map updating work, i.e. lane line classification flag and road marking contour flag are both 1, the O road-mask output in step two, the O i-line-mask output in step three and the O i-bbox-mask output in step four are fused at pixel level, and the output result is the road map element output result; if it belongs to real-time perception work, i.e. lane line classification flag and road marking contour flag are both 0, the O road-mask output in step two, the O line-mask output in step three and the O bbox output in step four are fused by bbox and mask, and the output result is the road perception result output. 2.The deep learning-based pavement marking perception method of claim 1, wherein: The shared convolutional neural network training process in step one comprises training data processing, network design optimization, shared feature extraction, and feature extraction fusion; Training data processing: first, visible light images under normal light are captured by an unmanned vehicle for image annotation, and the annotation content includes pixel-level annotation of roads and lane lines, road sign, and target detection frame annotation; the standardized format of the annotation data is coco data set format; Network design optimization: the roads and lane lines in the visible light image are pixel-level segmented, and the road sign is target detected to obtain road and lane line mask and road sign bbox; Shared feature extraction: a resnet-34 framework is adopted, and different expansion rates of the hole convolution are adopted in the network structures of the third layer and the fourth layer of the network; Feature extraction fusion: different spatial pyramid pooling layers are adopted to extract features, and one feature map is input into different spatial pyramid pooling layers for maximum pooling processing. 3.The deep learning-based pavement marking perception method according to claim 1 or 2, characterized in that: In the shared convolutional neural network in step one, the road segmentation head network and the lane line segmentation head network are composed of deconvolution layers and activation layers, and finally connected with a sigmod activation function; the convolution layer has a total of 6 layers, and the deconvolution layer has a total of 3 layers; the road sign detection head network adopts a multi-layer feature fusion structure, fuses SPP different layer features through a concat layer, and then obtains a target detection regression result through a convolution layer and a fully connected layer network. 4.The deep learning-based pavement marking perception method of claim 1, wherein: In the shared convolutional neural network in step one, total variation denoising is introduced into the loss function for image denoising; The total loss function of network training is as follows: L all = γ1L det + γ2L da-seg + γ3L ll-seg + γ4L tv , wherein, a1, a2, a3, g1, g2, g3, g4 are the loss function weights of each part, and are constants; L tv is the total variation loss, and L det is the detection network head loss function, and L det = a1L class + a2L obj + a3L box , wherein α t and γ c are weight coefficients, p t is the class probability; wherein α j and γ o are weight coefficients, p t is the class probability; The Euclidean distance of the center points of the prediction box and the real box, c represents the diagonal distance of the minimum closed region that can contain the prediction box and the real box, box iou is the intersection over union of the labeled box and the prediction box, and the formulas of α and v are as follows: is a width-height ratio of the bounding box, is a width-height ratio of the predicted box; L da-seg is the road segmentation loss function and: L da-seg = L ce , L ll-seg is the lane line segmentation loss function and: L ll-seg = L ce + L IoU , where L ce represents the classification error of the network output pixel with respect to the target pixel; L IoU intersection over union error between the network output lane line region and the target region; The total variation loss is defined as follows: where I i,j , I i,j+1 , I i+1,j are the center pixel, the pixel below the center and the pixel right of the center, respectively. 5.The deep learning-based pavement marking perception method of claim 1, wherein: The training process of the lane line classification network in step three is as follows: Training data processing: the lane line mask output by the trained visual multi-task network is used to obtain the lane line contour, the lane line is cropped from the original image according to the lane line contour, the image is resized to a square through image preprocessing as the network input image, and then the lane line is classified; Network design and training: the neural network comprises a convolution layer, a maximum pooling layer, an activation layer, and a fully connected layer; the activation function adopts SeLU; finally, a 6-channel softmax is used as the classification output, and a dropout regularization method is used to avoid overfitting; Local corresponding normalization: local corresponding normalization is adopted for the local center features of the lane line classification, and the local generalization formula is as follows: Wherein, α is a constant, which is set according to the experience value; the data is set to 5 values of different sizes through the control variable method, and then the network output result accuracy is compared to determine. 6.The deep learning-based pavement marking perception method of claim 5, wherein: The annotation categories in the training data processing include double dashed line, single dashed line, double yellow solid line, single white solid line, single yellow solid line, and double yellow solid line; the training data comprises a training set, a validation set, and a test set. 7.The deep learning-based pavement marking perception method of claim 1, wherein: The training process of the road sign front and background segmentation network in step four is as follows: Training data processing: the road surface identification bbox output by the trained visual multi-task network, and then the foreground target detection frame region in the original image is extracted as the dataset original image through the bbox coordinates, and then the matting result image is obtained through the image subtraction tool as the annotation data; Network design or training: the network adopts the MSIA-matte structure.

Images