A lane line detection method and system in rainy and snowy weather based on generative adversarial network

Abstract

The present invention discloses a method and system for lane line detection in rainy and snowy conditions based on a generative adversarial network. The method comprises: inputting a target image in rainy and snowy weather into a rain and snow trace removal model to obtain a trace-removed image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; inputting the trace-removed image into a lane line feature extraction model to obtain lane line pixels and high-dimensional features; the lane line feature extraction model is obtained by training a convolutional neural network model using a second training set; the convolutional neural network model includes an encoder, a semantic segmentation branch, and an instance segmentation branch; the instance segmentation branch includes a first convolutional layer, an excitation layer, and a second convolutional layer; a clustering algorithm is used to cluster lane line pixels based on the high-dimensional features of the lane lines to obtain lane line instances; and the lane line instances are segmentally fitted to obtain lane lines in the target image. The present invention can accurately detect lane line positions in rainy and snowy weather and is applicable to scenarios where the number of lane lines is unknown.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving and assisted driving, and in particular to a method and system for detecting lane lines in rainy and snowy weather based on a generative adversarial network. Background Art

[0002] Lane line information can be used to estimate the vehicle's offset from the center of the road and locate its own position. It is widely used in technologies such as lane departure warning, scene understanding, and vehicle self-positioning. It is one of the essential environmental information for intelligent vehicles to achieve autonomous driving in structured environments.

[0003] Current lane detection methods are primarily categorized as those based on traditional vision and those based on deep learning. Traditional methods primarily analyze underlying image features, manually designing features based on lane line color, shape, and edge characteristics. These features are then combined with Hough transforms or filters to identify lanes. For example, a method combining LDA and LSD is used to detect lanes. This method first grayscales the color image using LDA, then uses the LSD algorithm to detect lane lines within the grayscale image. The Hough transform and voting method are used to locate the road's vanishing point, extract lane lines based on color features, and design a polar angle constraint algorithm to select lanes. An inverse perspective transform is used to project the image to a bird's-eye view, followed by DBSCAN clustering of identical lanes. Finally, a random sampling consensus algorithm is used to fit lanes. Regions of interest (ROIs) are extracted using vehicle speed and stopping distance, followed by lane clustering using an algorithm that fuses color and edge marker information. Finally, curve parameters are calculated using straight lines and Lagrange interpolation polynomials.

[0004] Traditional lane detection methods place high demands on the algorithm designer's experience and skills. Manually designed features usually only achieve good results in specific situations and perform poorly in complex situations such as blurred or occluded lane lines. Lane detection algorithms based on neural networks, on the other hand, are simpler to use and have stronger generalization capabilities. For example, an image is divided into square regions of equal size, and then a neural network is used to detect the areas belonging to lane lines. The detected lane areas are then connected to generate complete lane lines. To address the problem of lane detection in complex scenarios, an end-to-end multi-task learning network, VPGNet, is proposed. While segmenting lane lines, it also predicts the location of vanishing points and uses these vanishing points to guide the detection of invisible lanes. LineNet, by scaling and re-segmenting low-confidence lane segmentation results, can effectively detect blur, occlusion, and dashed line gaps.

[0005] However, simply extracting all lane pixels from an image is not sufficient for lane fitting. To achieve this, different lanes must be distinguished. A common approach is to use a semantic segmentation model to extract lanes while treating each lane as a distinct category, distinguishing them through multi-category semantic segmentation. However, this approach can only detect a fixed number of lanes and is ineffective when the number of lanes is unknown.

[0006] In addition, due to the interference of rain and fog in rainy and snowy weather, the lane lines in the images obtained by the sensor will be blurred and obscured to a certain extent, affecting the algorithm's segmentation of the lane lines. Therefore, the existing detection method has the problem that the lane line detection effect is poor in rainy and snowy weather and can only detect a fixed number of lane lines. Summary of the Invention

[0007] Based on this, an embodiment of the present invention provides a lane line detection method and system in rainy and snowy weather based on a generative adversarial network, which can accurately detect the position of lane lines in rainy and snowy weather and is suitable for scenarios where the number of lane lines is unknown.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A method for detecting lane lines in rainy and snowy conditions based on a generative adversarial network, comprising:

[0010] Acquire target images in rainy and snowy weather;

[0011] Inputting the target image into a rain and snow trace removal model to obtain a trace-removed image of the target image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; the first training set includes lane line images with rain and snow traces added;

[0012] The trace-removed image of the target image is input into a lane line feature extraction model to obtain lane line pixels of the target image and high-dimensional features of the lane lines of the target image; the lane line feature extraction model is obtained by training a convolutional neural network model using a second training set; the second training set is an image after traces are removed from the first training set using the rain and snow trace removal model; the convolutional neural network model includes an encoder and a semantic segmentation branch and an instance segmentation branch respectively connected to the encoder; the semantic segmentation branch is used to output lane line pixels of the second training set; the instance segmentation branch is used to output high-dimensional features of the lane lines of the second training set; the instance segmentation branch includes a first convolutional layer, a second convolutional layer, and an upsampling layer connected in sequence;

[0013] Based on the high-dimensional features of the lane lines in the target image, a clustering algorithm is used to cluster the lane line pixels in the target image to obtain lane line instances;

[0014] The lane line instances are segmentedly fitted to obtain the lane lines of the target image.

[0015] Optionally, the method for determining the rain and snow trace removal model is:

[0016] Obtaining the first training set;

[0017] Constructing the generative adversarial network; the generative adversarial network includes a generative network and a discriminative network connected in sequence; the generative network includes a first convolutional block, a residual structure, and a second convolutional block connected in sequence; the first convolutional block includes a convolutional layer; the residual structure includes four residual blocks connected in series; the second convolutional block includes three convolutional layers connected in series; the discriminative network includes a third convolutional block and a fully connected layer; the third convolutional block includes five convolutional layers connected in series;

[0018] The first training set is input into the generative network, and the first training set and the trace-removed image of the first training set output by the generative network are input into the discriminative network for training to obtain a trained generative adversarial network; the trained generative adversarial network is the rain and snow trace removal model.

[0019] Optionally, the lane feature extraction model is determined by:

[0020] Obtaining the second training set;

[0021] Constructing the convolutional neural network model; the encoder is an ENet encoder; the semantic segmentation branch is an ENet decoder with an attention mechanism embedded in each upsampling process;

[0022] The second training set is input into the encoder, and the image features output by the encoder are respectively input into the semantic segmentation branch and the instance segmentation branch for training to obtain a trained convolutional neural network model; the trained convolutional neural network model is the lane line feature extraction model.

[0023] Optionally, clustering lane line pixels in the target image using a clustering algorithm based on the high-dimensional features of the lane lines in the target image to obtain lane line instances specifically includes:

[0024] Filtering lane line pixels of the target image using confidence level;

[0025] A density-based clustering method with noise is adopted to cluster the filtered lane line pixels according to the high-dimensional features of the lane lines in the target image to obtain lane line instances.

[0026] Optionally, performing segmented fitting on the lane line instance to obtain the lane line of the target image specifically includes:

[0027] Segmenting the lane line instances according to positions to obtain lane line instances in the upper half of the image and lane line instances in the lower half of the image;

[0028] A random sampling consensus algorithm is used in combination with a least squares method to fit the lane line instances in the upper half of the image and the lane line instances in the lower half of the image respectively to obtain the lane lines of the target image.

[0029] The present invention also provides a rainy and snowy lane line detection system based on a generative adversarial network, comprising:

[0030] Image acquisition module, used to acquire target images in rainy and snowy weather;

[0031] a trace removal module, configured to input the target image into a rain and snow trace removal model to obtain a trace-removed image of the target image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; the first training set includes lane line images with rain and snow traces added;

[0032] An image extraction module is configured to input the trace-removed image of the target image into a lane line feature extraction model to obtain lane line pixel points and high-dimensional lane line features of the target image; the lane line feature extraction model is obtained by training a convolutional neural network model using a second training set; the second training set is an image after traces are removed from the first training set using the rain and snow trace removal model; the convolutional neural network model includes an encoder and a semantic segmentation branch and an instance segmentation branch respectively connected to the encoder; the semantic segmentation branch is configured to output lane line pixel points of the second training set; the instance segmentation branch is configured to output high-dimensional lane line features of the second training set; the instance segmentation branch includes a first convolutional layer, a second convolutional layer, and an upsampling layer connected in sequence;

[0033] A clustering module, configured to cluster lane line pixels in the target image using a clustering algorithm based on high-dimensional features of the lane lines in the target image to obtain lane line instances;

[0034] The segmented fitting module is used to perform segmented fitting on the lane line instance to obtain the lane line of the target image.

[0035] Optionally, the rain and snow lane line detection system based on a generative adversarial network further includes: a first model determination module; the first model determination module is used to determine the rain and snow trace removal model; the first model determination module specifically includes:

[0036] A first training set acquisition unit, configured to acquire the first training set;

[0037] A generative adversarial network construction unit, configured to construct the generative adversarial network; the generative adversarial network comprises a generative network and a discriminative network connected in sequence; the generative network comprises a first convolutional block, a residual structure, and a second convolutional block connected in sequence; the first convolutional block comprises a convolutional layer; the residual structure comprises four residual blocks connected in sequence in series; the second convolutional block comprises three convolutional layers connected in sequence in series; the discriminative network comprises a third convolutional block and a fully connected layer; the third convolutional block comprises five convolutional layers connected in sequence in series;

[0038] The first model training unit is used to input the first training set into the generative network, and input the first training set and the trace-removed image of the first training set output by the generative network into the discriminant network for training to obtain a trained generative adversarial network; the trained generative adversarial network is the rain and snow trace removal model.

[0039] Optionally, the lane line detection system for rainy and snowy weather based on a generative adversarial network further includes: a second model determination module; the second model determination module is used to determine the lane line feature extraction model; the second model determination module specifically includes:

[0040] A second training set acquiring unit, configured to acquire the second training set;

[0041] A convolutional neural network model construction unit is used to construct the convolutional neural network model; the encoder is an ENet encoder; the semantic segmentation branch is an ENet decoder that embeds an attention mechanism in each upsampling process;

[0042] The second model training unit is used to input the second training set into the encoder, and input the image features output by the encoder into the semantic segmentation branch and the instance segmentation branch for training, so as to obtain a trained convolutional neural network model; the trained convolutional neural network model is the lane line feature extraction model.

[0043] Optionally, the clustering module specifically includes:

[0044] A screening unit, configured to screen lane line pixels of the target image using confidence levels;

[0045] The pixel point clustering unit is used to cluster the filtered lane line pixels according to the high-dimensional features of the lane lines of the target image using a density-based clustering method with noise to obtain lane line instances.

[0046] Optionally, the segmented fitting module specifically includes:

[0047] A segmentation unit, configured to segment the lane line instance according to position to obtain lane line instances in the upper half of the image and lane line instances in the lower half of the image;

[0048] The fitting unit is used to adopt a random sampling consensus algorithm and a least squares method to fit the lane line instances in the upper half of the image and the lane line instances in the lower half of the image respectively to obtain the lane lines of the target image.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] An embodiment of the present invention proposes a method and system for lane line detection in rainy and snowy conditions based on a generative adversarial network. This method uses a rain and snow trace removal model, obtained by training a generative adversarial network with a first training set, to remove rain and snow traces from a target image. A lane line feature extraction model, obtained by training a convolutional neural network model with a second training set, is then used to extract lane lines from the trace-removed image. An instance segmentation branch is introduced into the convolutional neural network model to prepare for subsequent clustering. Finally, a clustering algorithm is used to distinguish different lane lines based on their high-dimensional features, thereby completing lane line detection in rainy and snowy conditions. The rain and snow trace removal model of the present invention enables accurate lane line position detection in rainy and snowy conditions. The lane line feature extraction model, combined with the clustering algorithm, makes this method suitable for scenarios where the number of lane lines is unknown. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of a method for detecting lane lines in rainy and snowy conditions based on a generative adversarial network according to an embodiment of the present invention;

[0053] Figure 2 A structural diagram of a generation network provided by an embodiment of the present invention;

[0054] Figure 3 A structural diagram of a discriminant network provided by an embodiment of the present invention;

[0055] Figure 4 A structural diagram of a convolutional neural network model provided by an embodiment of the present invention;

[0056] Figure 5 A structural diagram of the Attention-Bottleneck module provided in an embodiment of the present invention;

[0057] Figure 6 A schematic diagram illustrating the principle of lane line pixel screening according to an embodiment of the present invention;

[0058] Figure 7 Flowchart of RANSAC lane fitting provided by an embodiment of the present invention;

[0059] Figure 8 This is a structural diagram of the lane line detection system in rain and snow based on a generative adversarial network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 This is a flow chart of the lane line detection method for rainy and snowy weather based on a generative adversarial network provided by an embodiment of the present invention. Figure 1 The method of this embodiment includes:

[0063] Step 101: Acquire a target image in rainy and snowy weather.

[0064] Step 102: Input the target image into a rain and snow trace removal model to obtain a trace-removed image of the target image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; the first training set includes a lane line image with rain and snow traces (rain marks and snowflakes) added.

[0065] The method for determining the rain and snow trace removal model is as follows:

[0066] 1) Obtain the first training set.

[0067] 2) Constructing the generative adversarial network; the generative adversarial network includes a generative network and a discriminative network connected in sequence. Figure 2As shown, see Figure 2 , the generation network includes a first convolution block, a residual structure and a second convolution block connected in sequence; the first convolution block includes a convolution layer (convolution layer 1); the residual structure includes four residual blocks connected in series (residual block 1, residual block 2, residual block 3 and residual block 4); the second convolution block includes three convolution layers connected in series (convolution layer 2, convolution layer 3 and convolution layer 4); the intermediate layer of the generation network uses the PRelu activation function, and the output layer uses the tanh activation function. The discriminant network is as follows Figure 3 As shown, see Figure 3 The discriminant network includes a third convolutional block and a fully connected layer; the third convolutional block includes five convolutional layers (convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5) connected in series. The discriminant network is composed of five convolutional layers and a fully connected layer connected in series for downsampling, and the output is activated by a sigmoid function.

[0068] 3) Inputting the first training set into the generative network, and inputting the first training set and the trace-removed image of the first training set output by the generative network into the discriminative network for training, thereby obtaining a trained generative adversarial network; the trained generative adversarial network serves as the rain and snow trace removal model. The specific training process is as follows: comparing the trace-removed image of the first training set output by the generative network at the current iteration with the image in the first training set corresponding to the rain trace, and determining whether the image is true or false. If the judgment result is true, the generative adversarial network corresponding to the current iteration is determined as the trained generative adversarial network. If the judgment result is false, the parameters of the generative network and the discriminative network are adjusted, and the next iteration is performed.

[0069] Step 103: Input the trace-removed image of the target image into a lane line feature extraction model to obtain lane line pixel points of the target image and high-dimensional lane line features of the target image; the lane line feature extraction model is obtained by training a convolutional neural network model using the second training set; the convolutional neural network model includes an encoder and a semantic segmentation branch and an instance segmentation branch respectively connected to the encoder.

[0070] The second training set is the image after the rain and snow trace removal model is used to remove traces from the first training set. The trace-removed image of the target image is used as the input of the encoder, and the encoder processes the trace-removed image to output image features. The semantic segmentation branch is used to process the image features and output the lane line pixel points of the second training set. The instance segmentation branch is used to process the image features and output the high-dimensional features of the lane lines of the second training set. The instance segmentation branch includes a first convolutional layer, a second convolutional layer, and an upsampling layer connected in sequence. The first and second convolutional layers are both 3×3 convolutional layers, and the excitation layer uses the ReLu activation function. The high-dimensional features of the lane lines are features that distinguish different lane lines, and the high-dimensional features of the lane lines prepare for subsequent clustering to distinguish different lane lines.

[0071] The lane feature extraction model is determined as follows:

[0072] 1) Obtain the second training set.

[0073] 2) Construct the convolutional neural network model. The structure of the convolutional neural network model is as follows: Figure 4 As shown. The encoder is an ENet encoder. The semantic segmentation branch is an ENet decoder that embeds an attention mechanism in each upsampling process. Specifically, a self-attention mechanism is connected in series after the main branch of each upsampling BottleNeck module of the ENet decoder to obtain a semantic segmentation branch. In this embodiment, the BottleNeck module embedded with the attention mechanism is called the Attention-Bottleneck module. The structure of the Attention-Bottleneck module in the downsampling process is as follows. Figure 5 shown.

[0074] 3) Inputting the second training set into the encoder, and inputting the image features output by the encoder into the semantic segmentation branch and the instance segmentation branch respectively, and training is performed with the goal of minimizing the semantic segmentation loss function and the instance segmentation loss function to obtain a trained convolutional neural network model; the trained convolutional neural network model is the lane line feature extraction model.

[0075] The specific training process is as follows: the second training set corresponding to the current number of iterations is input into the encoder, and the encoder outputs the training image features; the training image features are respectively used as the input of the semantic segmentation branch and the instance segmentation branch, the semantic segmentation branch outputs the training lane line pixel points, and the instance segmentation branch outputs the training lane line high-dimensional features, and determines whether the semantic segmentation loss function and the instance segmentation loss function are minimized at the current number of iterations. If so, the convolutional neural network model corresponding to the current number of iterations is determined as the trained convolutional neural network model. If not, the parameters of the encoder, semantic segmentation branch, and instance segmentation branch are adjusted for the next iteration. In the case of binary division, there are only two cases for the final prediction of the model. For each category, the predicted probabilities are p and 1-p. The binary cross entropy function is used as the semantic segmentation loss function to calculate the loss. The formula is as follows:

[0076]

[0077] Among them, L bin represents the semantic segmentation loss, y i Indicates the label of pixel i, 1 indicates lane line pixel, 0 indicates background pixel, N indicates the number of pixels, p i Indicates the probability that pixel i belongs to the lane line.

[0078] For instance segmentation, in order to achieve better clustering results, points belonging to the same lane need to be close to each other, and points belonging to different lanes need to be far away from each other. The instance segmentation loss function contains two terms: variance term and distance term. The variance term is used to generate a pulling force within the same cluster, which makes the points in the same cluster gather towards the center of the cluster; the distance term is used to generate a repulsive force between different clusters, which makes different clusters move away from each other. The variance term L in the clustering loss function is: var and the distance term L dist The distance between feature points is calculated using the second-order norm, and the total loss L instance It is the weighted sum of the losses generated by the variance term and the distance term, and its formula is as follows:

[0079]

[0080] In the above formula, C is the number of lanes, C A represents any lane, C B Indicates that A Different arbitrary lane lines, N c is the number of feature points in the corresponding cluster, x i is the feature vector corresponding to the feature point output by the branch network, [x] + represents max(0,x), μ cRepresents the mean vector of cluster C. δ v Indicates the range of the internal pulling force of the cluster. The distance between the feature point and the cluster mean is less than δ v When the variance term is zero, the internal pull of the cluster is also zero. d Represents the range of inter-cluster repulsion, and the inter-cluster distance is greater than δ d When the distance term is zero, the inter-cluster repulsion is zero. v is 0.5, δ d is 3. λ1 ​​and λ2 are the weights of the distance term and the variance term. In this embodiment, the values ​​of λ1 and λ2 are both 1.

[0081] In practice, the convolutional neural network model is trained with an input image size of 512×256, a batch size of 64, and a constant learning rate of 0.0005. The model is optimized using the Adam optimizer for a total of 300 epochs, with testing on the validation set every five epochs. The model with the lowest loss on the validation set is retained as the optimal model. To mitigate overfitting during training, the following adjustments are made: 1) Each input training image is cropped or horizontally rotated with a probability of 0.5; 2) A Dropout module is added after the main branch of each BottleNeck module in the model, inactivating neurons with a probability of 0.3.

[0082] Step 104: Based on the high-dimensional features of the lane lines in the target image, a clustering algorithm is used to cluster the lane line pixels in the target image to obtain lane line instances.

[0083] Step 104 specifically includes:

[0084] 1) Filter the lane line pixels of the target image using confidence to reduce the computational complexity of the subsequent lane line detection process. Specifically, a sliding window of size k×1 is used to traverse the semantic segmentation branch row by row, retaining the lane line pixels with the highest confidence built into the sliding window, and obtaining the filtered lane line pixels. The sliding window moves horizontally k steps each time, such as Figure 6 shown.

[0085] 2) Using a density-based spatial clustering of applications with noise (DBSCAN), the filtered lane line pixels are clustered according to the high-dimensional features of the lane lines in the target image to obtain lane line instances.

[0086] Step 105: performing segmented fitting on the lane line instance to obtain the lane line of the target image.

[0087] Step 105 specifically includes:

[0088] 1) Segment the lane line instances according to their positions to obtain lane line instances in the upper half of the image and lane line instances in the lower half of the image.

[0089] 2) Using the random sampling consensus (RANSAC) algorithm and combining it with the least squares method, the lane line instances in the upper half of the image and the lane line instances in the lower half of the image are fitted respectively to obtain the lane lines of the target image.

[0090] The following is a more detailed introduction to the lane line detection method in rainy and snowy weather based on generative adversarial networks.

[0091] S1. Obtain a target image (including the image to be identified of lane lines) in rainy and snowy weather.

[0092] S2. Input the target image into the rain and snow trace removal model (pre-trained generative adversarial network) (GAN network) to remove the rain and snow traces in the image.

[0093] First, rain streaks and snowflakes are added to the images containing lane lines in the lane line dataset, and the processed images and the original images are input into the network to train the network. After the training is completed, the images to be identified containing lane lines in rainy and snowy weather are input into the generator to generate the results, completing the removal of rain and snow traces in the image.

[0094] S3. Input the image after removing rain and snow traces into the lane line feature extraction model (pre-trained convolutional neural network model).

[0095] The instance segmentation method is based on semantic segmentation and improves the semantic segmentation model ENet. Compared with other semantic segmentation tasks, a notable feature of lane lines is that lane pixels account for a very small proportion of the entire image. This means that the features extracted by the neural network are mostly background rather than the lane lines themselves. To enable the network to focus more on extracting lane line features, this embodiment embeds a self-attention mechanism in each upsampling process of the ENet decoder. Specifically, an attention module is connected in series after the last convolution of the main branch of the BottleNeck module, and the output features are weighted by the attention module.

[0096] Simply extracting all the pixels belonging to the lane lines from the image cannot meet the needs of lane line fitting. In order to meet the requirements of lane line fitting, different lane lines need to be distinguished. The current mainstream instance segmentation method usually first uses a rectangular box to represent the target, and then classifies each pixel within the rectangular box to achieve instance segmentation. It is not suitable for targets such as lane lines that are slender and inclined and difficult to describe using rectangular boxes. This embodiment generates a high-dimensional feature for clustering for each pixel, and uses the above-designed semantic segmentation loss function and instance segmentation loss function to compress the high-dimensional features belonging to the same object (cluster) into a hypersphere, and then uses these high-dimensional features to cluster the pixels. The compression principle is as follows: the loss function used by the instance segmentation branch contains two terms: variance term and distance term. The role of the variance term is to generate a pulling force acting on the same cluster, which makes the points in the same cluster gather towards the center of the cluster through the pulling force; the role of the distance term is to generate a repulsive force between clusters with different scopes, which makes different clusters move away from each other through the repulsive force. When the centers of different clusters are too close, the repulsive force between clusters causes the centers of different clusters to move away from each other. When the feature points belonging to the same cluster are too far from the cluster center, the pulling force inside the cluster will pull the feature points toward the cluster center. Through the action of pulling and repulsive forces, the high-dimensional features belonging to the same object (cluster) can be compressed into a hypersphere, and then these high-dimensional features are used to cluster the pixels. The instance segmentation branch designed in this embodiment includes two convolutional layers and an upsampling layer, where the two convolutional layers use ordinary 3×3 convolutions and use ReLU as the activation function. The upsampling layer uses a bilinear interpolation method to enlarge the feature map to obtain an output equivalent to the size of the input image. There is no need to use a rectangular box to describe the target first. It is very suitable for instance segmentation of targets with special shapes such as lane lines.

[0097] S4. Screen and filter the output results of the lane feature extraction model:

[0098] In this embodiment, the lane line semantic segmentation branch outputs all the pixels in the image that belong to the lane lines. Directly using these points for subsequent steps will increase the algorithm's computational complexity. The semantic segmentation network not only outputs the segmentation results but also provides the confidence level of the segmentation results. The higher the confidence level, the greater the probability that the segmentation result is correct. This embodiment uses the confidence level to filter the pixels, while retaining as many points with high confidence as possible. To avoid affecting the final fitting result, only pixels in the same row are compared. The screening process is as follows: a k×1 window is used to cover the image during screening. Only the pixels with the highest confidence level within the window coverage are retained, and the other pixels are filtered out. The window is then slid from left to right with a step size of k.

[0099] S5. Use clustering algorithm to segment lane line instances.

[0100] Semantic segmentation networks can only separate lane lines from the background and cannot distinguish between different lane lines. To assign pixels to different lane lines, this embodiment uses DBSCAN to cluster pixels based on the high-dimensional features output by the instance segmentation branch. Compared to the traditional K-means clustering algorithm, the instance segmentation branch designed in this embodiment has two significant advantages: first, it does not require a specific number of clusters and can identify any number of lane lines; second, the clustering process can detect abnormal noise in the data.

[0101] S6. Use polynomials to segmentally fit lane lines.

[0102] To generate continuous and smooth lane lines, this embodiment fits the lane line segmentation results. Commonly used lane line representation forms are mainly polynomials and spline curves. Considering that the calculation process of spline curves is relatively complex and time-consuming, this embodiment uses polynomials to describe lane lines. The polynomial expression of the lane line curve equation is as follows:

[0103] f(x)=a0+a1x+a2x 2 +...+a n x n ;

[0104] Among them, x is the horizontal coordinate of the pixel point, n is the polynomial order, a0-a k The polynomial coefficients are obtained by fitting the selected in-place points using the least squares method to obtain the polynomial expression of the lane curve equation. This embodiment uses the RANSAC algorithm and the least squares method when fitting the lane lines, dividing the data into in-place points (sampling points selected for fitting the lane lines) and out-place points (points not used for fitting). Then, the in-place point fitting model is reselected through continuous iteration, and finally the result with the best data fitting degree in the iterative process is selected as the final model. This method of screening data makes the model have better noise resistance. The flow chart is as follows: Figure 7 As shown, it can be summarized as follows:

[0105] ① Randomly select a certain number of inliers and fit them using the least squares method to solve the polynomial expression for the polynomial coefficients. Initially, the inliers are randomly selected from all points. Through multiple iterations of optimization, the inliers with the best fit are finally determined.

[0106] ③ Determine whether other points in the point set are suitable for the currently fitted model, and add the points that are suitable for the current model into the in-game points.

[0107] ③ If the number of points in the current game is greater than a certain value, the lane lines are refitted using the new set of points in the game, and the fitting effect is compared with the best model in the past, and the model with the best fitting effect at the current moment is retained.

[0108] ④ Repeat steps 1-3 multiple times and output the optimal model (lane curve equation corresponding to the optimal polynomial coefficients) during the iteration process.

[0109] The generative adversarial network-based lane line detection method for rainy and snowy weather of this embodiment uses a generative adversarial network to remove rain and fog elements in the image, and then extracts the lane lines, which can complete the lane line detection in rainy and snowy weather. In addition, unlike the existing method in which different lane lines are regarded as different categories and different lane lines are distinguished by multi-category semantic segmentation, an instance segmentation branch is added to regard different lane lines as different targets. This can cope with scenarios with an unknown number of lane lines, solves the problem that the existing technology has poor lane line detection effect in rainy and snowy weather and can mostly only detect a fixed number of lane lines, and is more practical.

[0110] The present invention also provides a lane line detection system for rainy and snowy days based on a generative adversarial network, see Figure 8 , the system comprising:

[0111] The image acquisition module 801 is used to acquire target images in rainy and snowy weather.

[0112] a trace removal module 802 configured to input the target image into a rain and snow trace removal model to obtain a trace-removed image of the target image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; the first training set includes lane line images with rain and snow traces added;

[0113] An image extraction module 803 is configured to input the trace-removed image of the target image into a lane line feature extraction model to obtain lane line pixels and high-dimensional features of the lane lines of the target image; the lane line feature extraction model is obtained by training a convolutional neural network model using a second training set; the second training set is an image after traces are removed from the first training set using the rain and snow trace removal model; the convolutional neural network model includes an encoder and a semantic segmentation branch and an instance segmentation branch respectively connected to the encoder; the semantic segmentation branch is configured to output lane line pixels of the second training set; the instance segmentation branch is configured to output high-dimensional features of the lane lines of the second training set; the instance segmentation branch includes a first convolutional layer, a second convolutional layer, and an upsampling layer connected in sequence;

[0114] A clustering module 804 is configured to cluster lane line pixels in the target image using a clustering algorithm based on high-dimensional features of the lane lines in the target image to obtain lane line instances;

[0115] The segmented fitting module 805 is configured to perform segmented fitting on the lane line instance to obtain the lane line of the target image.

[0116] In one example, the rain and snow lane line detection system based on a generative adversarial network further includes: a first model determination module; the first model determination module is used to determine the rain and snow trace removal model. The first model determination module specifically includes:

[0117] The first training set acquiring unit is configured to acquire the first training set.

[0118] A generative adversarial network construction unit is used to construct the generative adversarial network; the generative adversarial network includes a generative network and a discriminative network connected in sequence; the generative network includes a first convolution block, a residual structure and a second convolution block connected in sequence; the first convolution block includes a convolution layer; the residual structure includes four residual blocks connected in series; the second convolution block includes three convolution layers connected in series; the discriminative network includes a third convolution block and a fully connected layer; the third convolution block includes five convolution layers connected in series.

[0119] The first model training unit is used to input the first training set into the generative network, and input the first training set and the trace-removed image of the first training set output by the generative network into the discriminant network for training to obtain a trained generative adversarial network; the trained generative adversarial network is the rain and snow trace removal model.

[0120] In one example, the lane line detection system for rainy and snowy weather based on a generative adversarial network further includes: a second model determination module; the second model determination module is used to determine the lane line feature extraction model. The second model determination module specifically includes:

[0121] The second training set acquiring unit is configured to acquire the second training set.

[0122] A convolutional neural network model construction unit is used to construct the convolutional neural network model; the encoder is an ENet encoder; and the semantic segmentation branch is an ENet decoder that embeds an attention mechanism in each upsampling process.

[0123] The second model training unit is used to input the second training set into the encoder, and input the image features output by the encoder into the semantic segmentation branch and the instance segmentation branch for training, so as to obtain a trained convolutional neural network model; the trained convolutional neural network model is the lane line feature extraction model.

[0124] In one example, the clustering module specifically includes:

[0125] A screening unit is used to screen the lane line pixels of the target image using confidence.

[0126] The pixel point clustering unit is used to cluster the filtered lane line pixels according to the high-dimensional features of the lane lines of the target image using a density-based clustering method with noise to obtain lane line instances.

[0127] In one example, the segmented fitting module specifically includes:

[0128] The segmentation unit is used to segment the lane line instance according to the position to obtain the lane line instance in the upper half of the image and the lane line instance in the lower half of the image.

[0129] The fitting unit is used to adopt a random sampling consensus algorithm and a least squares method to fit the lane line instances in the upper half of the image and the lane line instances in the lower half of the image respectively to obtain the lane lines of the target image.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0131] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A lane line detection method in rainy and snowy conditions based on a generative adversarial network, characterized in that: The lane line detection method in rainy and snowy weather based on a generative adversarial network includes: Acquire target images in rainy and snowy weather; Inputting the target image into a rain and snow trace removal model to obtain a trace-removed image of the target image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; the first training set includes lane line images with rain and snow traces added; The trace-removed image of the target image is input into a lane line feature extraction model to obtain lane line pixels of the target image and high-dimensional features of the lane lines of the target image; the lane line feature extraction model is obtained by training a convolutional neural network model using a second training set; the second training set is an image after traces are removed from the first training set using the rain and snow trace removal model; the convolutional neural network model includes an encoder and a semantic segmentation branch and an instance segmentation branch respectively connected to the encoder; the semantic segmentation branch is used to output lane line pixels of the second training set; the instance segmentation branch is used to output high-dimensional features of the lane lines of the second training set; the instance segmentation branch includes a first convolutional layer, a second convolutional layer, and an upsampling layer connected in sequence; Based on the high-dimensional features of the lane lines in the target image, a clustering algorithm is used to cluster the lane line pixels in the target image to obtain lane line instances; Performing segmented fitting on the lane line instance to obtain the lane line of the target image; The method for determining the lane feature extraction model is: Obtaining the second training set; Constructing the convolutional neural network model; the encoder is an ENet encoder; the semantic segmentation branch is an ENet decoder with an attention mechanism embedded in each upsampling process; Inputting the second training set into the encoder, and inputting the image features output by the encoder into the semantic segmentation branch and the instance segmentation branch respectively for training, to obtain a trained convolutional neural network model; the trained convolutional neural network model is the lane feature extraction model; The method of clustering lane line pixels of the target image using a clustering algorithm based on the high-dimensional features of the lane lines of the target image to obtain lane line instances specifically includes: Filtering lane line pixels of the target image using confidence level; A density-based clustering method with noise is adopted to cluster the filtered lane line pixels according to the high-dimensional features of the lane lines in the target image to obtain lane line instances.

2. The method for detecting lane lines in rainy and snowy conditions based on a generative adversarial network according to claim 1, characterized in that: The method for determining the rain and snow trace removal model is as follows: Obtaining the first training set; Constructing the generative adversarial network; the generative adversarial network includes a generative network and a discriminative network connected in sequence; the generative network includes a first convolutional block, a residual structure, and a second convolutional block connected in sequence; the first convolutional block includes a convolutional layer; the residual structure includes four residual blocks connected in series; the second convolutional block includes three convolutional layers connected in series; the discriminative network includes a third convolutional block and a fully connected layer; the third convolutional block includes five convolutional layers connected in series; The first training set is input into the generative network, and the first training set and the trace-removed image of the first training set output by the generative network are input into the discriminative network for training to obtain a trained generative adversarial network; the trained generative adversarial network is the rain and snow trace removal model.

3. The method for detecting lane lines in rainy and snowy conditions based on a generative adversarial network according to claim 1, wherein: The performing segmented fitting on the lane line instance to obtain the lane line of the target image specifically includes: Segmenting the lane line instances according to positions to obtain lane line instances in the upper half of the image and lane line instances in the lower half of the image; A random sampling consensus algorithm is used in combination with a least squares method to fit the lane line instances in the upper half of the image and the lane line instances in the lower half of the image respectively to obtain the lane lines of the target image.

4. A lane line detection system for rainy and snowy weather based on a generative adversarial network, applied to the lane line detection method for rainy and snowy weather based on a generative adversarial network according to any one of claims 1 to 3, characterized in that: The rainy and snowy lane line detection system based on the generative adversarial network includes: Image acquisition module, used to acquire target images in rainy and snowy weather; a trace removal module, configured to input the target image into a rain and snow trace removal model to obtain a trace-removed image of the target image; the rain and snow trace removal model is obtained by training a generative adversarial network using a first training set; the first training set includes lane line images with rain and snow traces added; An image extraction module is configured to input the trace-removed image of the target image into a lane line feature extraction model to obtain lane line pixel points and high-dimensional lane line features of the target image; the lane line feature extraction model is obtained by training a convolutional neural network model using a second training set; the second training set is an image after traces are removed from the first training set using the rain and snow trace removal model; the convolutional neural network model includes an encoder and a semantic segmentation branch and an instance segmentation branch respectively connected to the encoder; the semantic segmentation branch is configured to output lane line pixel points of the second training set; the instance segmentation branch is configured to output high-dimensional lane line features of the second training set; the instance segmentation branch includes a first convolutional layer, a second convolutional layer, and an upsampling layer connected in sequence; A clustering module, configured to cluster lane line pixels in the target image using a clustering algorithm based on high-dimensional features of the lane lines in the target image to obtain lane line instances; The segmented fitting module is used to perform segmented fitting on the lane line instance to obtain the lane line of the target image.

5. The rainy and snowy lane detection system based on a generative adversarial network according to claim 4 is characterized in that: Also includes: a first model determination module; The first model determination module is used to determine the rain and snow trace removal model; The first model determination module specifically includes: A first training set acquisition unit, configured to acquire the first training set; A generative adversarial network construction unit, configured to construct the generative adversarial network; the generative adversarial network comprises a generative network and a discriminative network connected in sequence; the generative network comprises a first convolutional block, a residual structure, and a second convolutional block connected in sequence; the first convolutional block comprises a convolutional layer; the residual structure comprises four residual blocks connected in sequence in series; the second convolutional block comprises three convolutional layers connected in sequence in series; the discriminative network comprises a third convolutional block and a fully connected layer; the third convolutional block comprises five convolutional layers connected in sequence in series; The first model training unit is used to input the first training set into the generative network, and input the first training set and the trace-removed image of the first training set output by the generative network into the discriminant network for training to obtain a trained generative adversarial network; the trained generative adversarial network is the rain and snow trace removal model.

6. The lane line detection system for rainy and snowy weather based on a generative adversarial network according to claim 4 is characterized in that: Also includes: a second model determination module; The second model determination module is used to determine the lane feature extraction model; The second model determination module specifically includes: A second training set acquiring unit, configured to acquire the second training set; A convolutional neural network model construction unit is used to construct the convolutional neural network model; the encoder is an ENet encoder; the semantic segmentation branch is an ENet decoder that embeds an attention mechanism in each upsampling process; The second model training unit is used to input the second training set into the encoder, and input the image features output by the encoder into the semantic segmentation branch and the instance segmentation branch for training, so as to obtain a trained convolutional neural network model; the trained convolutional neural network model is the lane line feature extraction model.

7. The rainy and snowy lane detection system based on a generative adversarial network according to claim 4 is characterized in that: The clustering module specifically includes: A screening unit, configured to screen lane line pixels of the target image using confidence levels; The pixel point clustering unit is used to cluster the filtered lane line pixels according to the high-dimensional features of the lane lines of the target image using a density-based clustering method with noise to obtain lane line instances.

8. The rainy and snowy lane detection system based on a generative adversarial network according to claim 4 is characterized in that: The segmented fitting module specifically includes: A segmentation unit, configured to segment the lane line instance according to position to obtain lane line instances in the upper half of the image and lane line instances in the lower half of the image; The fitting unit is used to adopt a random sampling consensus algorithm and a least squares method to fit the lane line instances in the upper half of the image and the lane line instances in the lower half of the image respectively to obtain the lane lines of the target image.

Images