Lane detection methods, devices, vehicles and storage media

Abstract

This application relates to the field of autonomous driving technology, and particularly to a lane line detection method, device, vehicle, and storage medium. The method includes: lane line modeling; constructing a convolutional neural network including a feature extraction module and a prediction module; extracting image features from an image dataset using the feature extraction module; selecting output feature layers from multiple convolutional layers at different locations in the feature extraction module; generating a candidate lane line set based on the pixels of the left and right lower boundaries of the output feature layer of the last convolutional layer; constructing a candidate lane line feature set based on the output feature layers of all selected convolutional layers corresponding to each lane line in the candidate lane line set; inputting the candidate lane line feature set into the prediction module; predicting the actual position of the lane line using shallow lane line features; and predicting the actual type of the lane line using deep lane line features. This solves the problems of high computational cost, poor representation ability, and low accuracy in lane line detection of related technologies.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a lane line detection method, device, vehicle, and storage medium. Background Technology

[0002] Lane markings are the most important traffic signs in road traffic, playing a crucial role in regulating and protecting vehicle movement. Whether in vehicle safety driving systems or machine vision-based intelligent vehicle navigation, lane marking detection and recognition are fundamental and essential functional modules. They not only reduce traffic accidents but also contribute to future in-depth research on intelligent transportation.

[0003] Lane detection is an environmental perception application that uses onboard cameras or LiDAR to detect lane lines. To ensure vehicle safety during driving, autonomous driving systems need to keep the car moving along lane lines on the road, which requires accurate lane line perception. Lane detection plays a crucial role in autonomous driving.

[0004] Related technologies use encoder and decoder structures to classify pixels, but this method is computationally intensive. Alternatively, lane lines can be modeled using the near and far widths of lane lines in the image, edge point pairs can be detected, and least squares straight lane line fitting can be used. However, this method has poor representation and generalization capabilities. Summary of the Invention

[0005] This application provides a lane line detection method, device, vehicle, and storage medium to solve the problems of large computational load, poor characterization ability, and low accuracy in lane line detection in related technologies.

[0006] The first aspect of this application provides a lane line detection method, comprising the following steps: acquiring an image dataset with real-world region annotations of lane lines; lane line modeling, using a small number of parameters to replace a large number of pixels to represent lane lines; constructing a convolutional neural network including a feature extraction module and a prediction module, using the feature extraction module to extract image features from the image dataset, and selecting the output feature layers of multiple convolutional layers at different positions of the feature extraction module; selecting the output feature layer of the last convolutional layer among the multiple convolutional layers, generating a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer, and determining the candidate lane line set based on all selected convolutional layers corresponding to each lane line in the candidate lane line set. The output feature layer constructs a candidate lane line feature set, which includes shallow lane line features and deep lane line features. The candidate lane line feature set is input into the prediction module. The shallow lane line features are used to predict the actual position of the lane line, and the deep lane line features are used to predict the actual type of the lane line. A training loss value is calculated based on the actual type and actual position and the true type and true position of the lane line. The parameters of the convolutional neural network are updated using the loss value. Iterative training stops when the training loss value is less than or equal to a preset value, resulting in a trained lane line prediction model. The lane line prediction model is then used to predict the type and position of the lane line.

[0007] Based on the above technical means, the embodiments of this application can perform multi-dimensional feature extraction on a dataset containing lane lines with real area annotations through a convolutional neural network, so that the extracted features have rich expressive power. By using shallow and deep features, the position and type of lane lines can be predicted, increasing the accuracy of lane detection. Furthermore, a large number of lane line pixels can be represented by a small number of parameters for lane line modeling, reducing the amount of computation.

[0008] Further, the step of generating a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer includes: identifying the height, width, and number of channels of the last feature layer; determining the target points of the left and right boundaries and the lower boundary based on the height and width of the last feature layer, and determining the number of feature dimensions contained in each target point based on the number of channels; determining candidate lane lines at multiple angles of the three boundaries based on the number of feature dimensions, and merging the candidate lane lines at multiple angles of the three boundaries into a candidate lane line set.

[0009] Based on the above technical means, the embodiments of this application can merge three candidate lane lines with multiple boundary angles into a candidate lane line set, thereby improving the accuracy of candidate lane lines and indirectly improving the accuracy of lane line detection.

[0010] Further, constructing a candidate lane line feature set based on the features of each lane line in the candidate lane line set includes: extracting features of each lane line in the candidate lane line set; concatenating features corresponding to the same candidate lane line in different dimensions to obtain a combination of shallow lane line features and deep lane line features; and constructing a candidate lane line feature set based on the combination of shallow lane line features and deep lane line features.

[0011] Based on the above technical means, the embodiments of this application can stitch together features corresponding to the same candidate lane line in different dimensions to construct a feature set of candidate lane lines, which facilitates subsequent prediction of lane lines.

[0012] Optionally, before constructing a convolutional neural network including a feature extraction module and a prediction module, and extracting image features from the image dataset using the feature extraction module, the method further includes: representing a lane line using the intersection of the lane line and the image boundary, the angle between the lane line and the horizontal direction, and the lane line category to construct a lane line model; and converting the dataset into a dataset in the lane line model format based on the lane line model.

[0013] Based on the above technical means, the embodiments of this application use the intersection of the lane line and the image boundary, the angle between the lane line and the horizontal direction, and the lane line category to represent a lane line, thereby modeling the lane line and reducing the computational load of the model.

[0014] Optionally, the dataset includes an image, the region where lane lines are located in the image, and the category of each pixel in the lane line region. The step of converting the dataset into a dataset in the lane line model format based on the lane line model includes: obtaining the midpoints of lane line edges located at the left, lower, or right boundaries of the image from the region where the lane lines are located; obtaining lane line edges located in the image, and if the ordinates of the lane line edges are the same, treating them as a pair of points; calculating the midpoint of the pair of points and adding it to the point set; fitting a straight line based on the points in the point set and the midpoints of the lane line edges to obtain the angle between the straight line and the horizontal direction; and generating a format-converted dataset based on the midpoints of the lane line edges, the angle between the straight line and the horizontal direction, and the category of each pixel in the lane line region.

[0015] Based on the above technical means, the embodiments of this application can convert the lane line dataset into a lane line model format dataset, so that the dataset can be directly used for lane line detection in the future.

[0016] A second aspect of this application provides a lane line detection method, comprising the following steps: acquiring a road image surrounding a vehicle; inputting the road image into a pre-trained prediction model, and outputting the actual type and actual position of the lane lines in the road image, wherein the prediction model is obtained by training a convolutional neural network based on an image dataset carrying real-world region annotations, extracting shallow lane line features and deep lane line features from the road image according to the convolutional neural network, predicting the actual position of the lane lines using the shallow lane line features, and predicting the actual type of the lane lines using the deep lane line features.

[0017] A third aspect of this application provides a lane line detection device, comprising: a first acquisition module for acquiring an image dataset containing real-world region annotations of lane lines; a modeling module for lane line modeling, using a small number of parameters to replace a large number of pixels to represent lane lines; a first construction module for constructing a convolutional neural network including a feature extraction module and a prediction module, using the feature extraction module to extract image features from the image dataset, and selecting the output feature layers of multiple convolutional layers at different positions of the feature extraction module; and a second construction module for selecting the output feature layer of the last convolutional layer among the multiple convolutional layers, generating a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer, and determining the lane line detection method based on each lane in the candidate lane line set. A candidate lane line feature set is constructed using the output feature layers of all selected convolutional layers corresponding to the lane line. This candidate lane line feature set includes shallow lane line features and deep lane line features. A prediction module is used to input the candidate lane line feature set into the prediction module, predict the actual position of the lane line using the shallow lane line features, predict the actual type of the lane line using the deep lane line features, calculate a training loss value based on the actual type and position and the true type and position of the lane line, update the parameters of the convolutional neural network using the loss value, and stop iterative training when the training loss value is less than or equal to a preset value, thus obtaining a trained lane line prediction model. This lane line prediction model is then used to predict the type and position of the lane line.

[0018] A fourth aspect of this application provides a lane detection device, comprising: a second acquisition module for acquiring road images surrounding a vehicle; and an output module for inputting the road images into a pre-trained prediction model and outputting the actual type and actual position of lane lines in the road images. The prediction model is obtained by training a convolutional neural network based on an image dataset carrying real-world region annotations. The convolutional neural network extracts shallow lane line features and deep lane line features from the road images, uses the shallow lane line features to predict the actual position of the lane lines, and uses the deep lane line features to predict the actual type of the lane lines.

[0019] A fifth aspect of this application provides a vehicle including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the lane detection method as described in the above embodiments.

[0020] A sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the lane line detection method as described in the above embodiments.

[0021] Therefore, this application has at least the following beneficial effects:

[0022] (1) The embodiments of this application can use a convolutional neural network to extract multi-dimensional features from a dataset containing lane lines in real regions, so that the extracted features have rich expressive power. By using shallow and deep features, the position and type of lane lines can be predicted, increasing the accuracy of lane detection. Furthermore, a large number of lane line pixels can be represented by a small number of parameters for lane line modeling, reducing the amount of computation.

[0023] (2) The embodiments of this application can merge the candidate lane lines of three boundaries at multiple angles into a candidate lane line set, which improves the accuracy of candidate lane lines and indirectly improves the accuracy of lane line detection.

[0024] (3) In this embodiment, features corresponding to the same candidate lane line in different dimensions can be spliced ​​together to construct a feature set of candidate lane lines, which facilitates the subsequent prediction of lane lines.

[0025] (4) In this embodiment, a lane line is represented by the intersection of the lane line and the image boundary, the angle between the lane line and the horizontal direction, and the lane line category. This modeling of the lane line reduces the computational load of the model.

[0026] (5) The embodiments of this application can convert the lane line dataset into a lane line model format dataset so that the dataset can be directly used to detect lane lines in the future.

[0027] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0028] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 This is a flowchart of a lane line detection method provided according to an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of lane line modeling according to an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of a candidate lane line provided according to an embodiment of this application;

[0032] Figure 4 This is a flowchart of a lane line detection method provided according to an embodiment of this application;

[0033] Figure 5 This is an example diagram of a lane line detection device provided according to an embodiment of this application;

[0034] Figure 6 This is an example diagram of a lane line detection device provided according to an embodiment of this application;

[0035] Figure 7 This is a structural schematic diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] The relevant technologies achieve lane line detection through the following methods:

[0038] (1) The trained semantic segmentation network is used to classify all pixels in the image and identify the set of pixels belonging to each lane line according to preset rules. The semantic segmentation network is a depthwise separable convolutional structure plus a fully convolutional network;

[0039] (2) Using encoder-decoder as the basic architecture, a lane line existence branch is connected to the end of the encoder, the decoder outputs the lane line pixel probability map, and the lane line existence branch obtains the lane line existence probability.

[0040] (3) Model the lane lines using the near and far widths of the lane lines in the image, detect edge point pairs, and fit the straight lane lines using the least squares method.

[0041] The lane line detection method, apparatus, vehicle, and storage medium of this application are described below with reference to the accompanying drawings. Addressing the issues mentioned in the background art, such as using encoders and decoders to classify pixels for lane line detection, which involves high computational cost, and using near and far widths of lane lines in the image to model lane lines, detect edge point pairs, and fit straight lane lines using least squares, which suffers from poor representational and generalization capabilities, this application provides a lane line detection method. In this method, a large number of lane line pixels are represented by a small number of model parameters, thereby reducing computational cost. Simultaneously, a convolutional neural network is used to extract multi-dimensional features from the image, giving the extracted features rich expressive power and increasing the accuracy of lane line detection. Thus, the problems of high computational cost, poor representational capability, and low accuracy in lane line detection of related technologies are solved.

[0042] Specifically, Figure 1 This is a schematic flowchart of a lane line detection method provided in an embodiment of this application.

[0043] like Figure 1 As shown, the lane line detection method includes the following steps:

[0044] In step S101, an image dataset with actual region annotations carrying lane lines is obtained.

[0045] The dataset contains images, the regions in the images where lane lines are located, and the category of each pixel in the lane line region.

[0046] It should be noted that this is a common lane marking method, which is relatively easy to obtain. The information in these image datasets is manually labeled and is the actual label.

[0047] In step S102, lane lines are modeled by replacing a large number of pixels with a small number of parameters to represent the lane lines.

[0048] The specific methods for lane line modeling will be described in detail in the following embodiments, and will not be repeated here.

[0049] It should be noted that the embodiments of this application use a small number of parameters to abstract lane lines represented by a large number of pixels, which can reduce the amount of computation.

[0050] In step S103, a convolutional neural network containing a feature extraction module and a prediction module is constructed. The feature extraction module is used to extract image features from the image dataset, and the output feature layers of multiple convolutional layers at different positions of the feature extraction module are selected.

[0051] It is understood that the convolutional neural network in this application includes a feature extraction module and a prediction module. The feature extraction module is used to extract image features from the image dataset, and the prediction module is used to predict the position and type of lane lines.

[0052] Specifically, this application embodiment requires first constructing a convolutional neural network to extract image features, and then selecting output feature layers {F1, ..., Fi} from several convolutional layers at different positions, where i ≥ 3, representing the number of feature layers selected. The smallest feature layer, Fi, has a size of H. i ×W i ×C i H i W indicates high. i Indicates width, C i This represents the number of channels. Finally, the lane line image is input into the convolutional neural network, and the output feature layer {F1, ..., Fi} is generated.

[0053] In this embodiment of the application, before constructing a convolutional neural network including a feature extraction module and a prediction module, and extracting image features from the image dataset using the feature extraction module, the method further includes: representing a lane line using the intersection of the lane line and the image boundary, the angle between the lane line and the horizontal direction, and the lane line category to construct a lane line model; and converting the dataset into a lane line model format dataset based on the lane line model.

[0054] It is understood that the embodiments of this application can be divided into two parts: lane line modeling and conversion dataset.

[0055] Lane line modeling involves using three parameters to represent a lane line: the focal point between the lane line and the image boundary, the angle between the lane line and the horizontal direction, and the lane line category. Figure 2 As shown in the figure ((x, y) represents the intersection of the lane line and the image boundary, the other end of the lane line is in the middle of the image, θ is the angle between the lane line and the horizontal direction. A straight line is drawn with (x, y) as the starting point and the angle between it and the horizontal direction is θ, which serves as the teaching model for the lane line.)

[0056] It should be noted that this lane line modeling method abstracts the lane lines composed of pixels, which can reduce the computational load of the model.

[0057] The process of converting the dataset into a lane line model format based on lane line modeling includes: obtaining the midpoints of lane line edges located at the left, lower, or right boundaries of the image from the region where the lane lines are located; obtaining lane line edges located in the image, and if the ordinates of the lane line edges are the same, treating them as a pair of points, calculating the midpoint of the pair of points and adding them to the point set; fitting a straight line based on the points in the point set and the midpoints of the lane line edges to obtain the angle between the straight line and the horizontal direction; and generating the format-converted dataset based on the midpoints of the lane line edges, the angle between the straight line and the horizontal direction, and the category of each pixel in the lane line region.

[0058] Specifically, the process is as follows: From the lane line region, obtain the midpoint p1 of the lane line edge located at the left, lower, or right boundary of the image (this is the intersection of the lane line and the image boundary, with the other end of the lane line in the middle of the image). Obtain the lane line edge located in the image, where points with the same y-coordinate are considered a pair. Calculate the midpoint and add it to the click point set Points2. Use the points in Points2 and p1 to fit a straight line and obtain the angle θ between the straight line and the horizontal direction. Generate a format-converted dataset based on the midpoint p1 of the lane line edge, the angle θ between the straight line and the horizontal direction, and the category of each pixel in the lane line region.

[0059] In step S104, the output feature layer of the last selected convolutional layer is used to generate a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer. A candidate lane line feature set is constructed based on the output feature layers of all selected convolutional layers corresponding to each lane line in the candidate lane line set. The construction of the candidate lane line feature set includes shallow lane line features and deep lane line features.

[0060] It should be noted that in convolutional neural networks, downsampling or convolution with a stride greater than 1 is usually used to reduce the size of the feature map in order to achieve the purpose of abstracting features. This application utilizes shallow and deep features because deep features have undergone more feature permutations and downsampling, and have more overall features, which helps to distinguish different lane lines; shallow features contain more detailed features such as edge gradients and edge orientation, which helps to find the more accurate location of lane lines. Making full use of features at different levels can improve the accuracy of lane line detection.

[0061] In this embodiment, a candidate lane line set is generated based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer, including: identifying the height, width, and number of channels of the last feature layer; determining the target points of the left and right boundaries and the lower boundary based on the height and width of the last feature layer, and determining the number of feature dimensions contained in each target point based on the number of channels; determining candidate lane lines at multiple angles of the three boundaries based on the number of feature dimensions, and merging the candidate lane lines at multiple angles of the three boundaries into a candidate lane line set.

[0062] In this context, height is represented by Hi, width by Wi, and number of channels by Ci.

[0063] Specifically, in the feature layer Fi, its height is Hi, so there are Hi points on the left and right boundaries. Each point contains Ci-dimensional features, which represent the features of c different θ (0° < θ < 90°) candidate lines mapped back to the original image with the starting coordinates as the origin. The width of Fi is Wi, so there are Wi points on the lower boundary. Each point contains Ci-dimensional features, which represent the features of c θ (0° < θ < 180°) candidate lines mapped back to the original image with the starting coordinates as the origin. The candidate lane lines of the three boundaries are merged into a candidate lane line set {L1, ..., Ln}, where n = c × (2 × Hi + Wi) represents the candidate lane line.

[0064] In this embodiment of the application, a candidate lane line feature set is constructed based on the features of each lane line in the candidate lane line set, including: extracting the features of each lane line in the candidate lane line set; splicing the features corresponding to the same candidate lane line in different dimensions to obtain a combination of shallow lane line features and deep lane line features; and constructing a candidate lane line feature set based on the combination of shallow lane line features and deep lane line features.

[0065] It is understood that, in this embodiment of the application, features of each lane line in the candidate lane line set {L1, ..., Ln} are extracted. For each feature layer from F1 to Fi, the size is H. j ×W i ×C j If 1 ≤ j ≤ i, input it into a 1×1 convolutional layer with c channels to convert the features into H. j ×W j ×c, so that the c-dimensional features of the points on the upper boundary of Fi can correspond to c lane lines. Then, the features corresponding to the same candidate lane line in different dimensions are concatenated. The concatenation dimension on the channel is c×i. The concatenated features represent the shallow and deep features of the candidate lane line. The feature set corresponding to the candidate lane line set {L1, ..., Ln} is {L1cat, ..., Lncat}, where cat represents the concatenated feature with a size of H. cat ×W cat ×Ccat .

[0066] In summary, the method described in the above embodiments can generate a set of candidate lane lines, such as... Figure 3 As shown in the figure, p represents a point on the lower boundary of feature layer Fi, the length of feature map Fi is Wi, the height is Hi, the number of channels is Ci, and p' represents the point on the original image corresponding to p. Starting from p', c different candidate lines with θ (0° < θ < 180°) are generated.

[0067] In step S105, the candidate lane line feature set is input into the prediction module. The shallow lane line features are used to predict the actual position of the lane line, and the deep lane line features are used to predict the actual type of the lane line. The training loss value is calculated based on the actual type and actual position and the true type and true position of the lane line. The parameters of the convolutional neural network are updated using the loss value. The iterative training stops when the training loss value is less than or equal to a preset value, and the trained lane line prediction model is obtained. The lane line prediction model is used to predict the type and position of the lane line.

[0068] The preset values ​​are set according to the specific situation and are not limited thereto; the types of lane lines can include solid white lines, dashed white lines, solid yellow lines, dashed yellow lines, etc., or not lane lines, etc.

[0069] Specifically, in this embodiment, the candidate lane line feature set {L1cat, ..., Lncat} is input into the prediction module. Assuming the number of lane line categories is u, the feature set {L1cat, ..., Lncat} (with dimensions n×i×c) is converted into an n×(3+u+1) output through a convolution operation. 3 represents the horizontal and vertical coordinates and angle of the lane line boundary points, u represents the probability that the lane line belongs to each of the u categories, and 1 represents the probability that it is not a lane line. The n×(3+u+1) output obtained by the convolutional neural network is compared with the true lane line parameters (i.e., the true type and true location of the lane line). The candidate lane line whose parameters are closest to the true lane line parameters is considered detected, and the difference is calculated as the loss. The derivative of the loss with respect to the convolutional neural network is used to update the convolutional neural network parameters. This process is repeated until the loss no longer decreases, indicating that the convolutional neural network has converged, resulting in a prediction model. This prediction model is then used to predict the type and location of lane lines.

[0070] In summary, the lane detection method proposed in this application can extract multi-dimensional features from a dataset containing lane lines labeled with real-world regions using a convolutional neural network. This results in rich expressive power of the extracted features. By utilizing both shallow and deep features, the method can predict the position and type of lane lines, increasing the accuracy of lane detection. Furthermore, it reduces computation by using a small number of parameters to represent a large number of lane line pixels for lane line modeling. It can merge candidate lane lines from three boundaries at multiple angles into a candidate lane line set, improving the accuracy of candidate lane lines and indirectly improving the accuracy of lane line detection. It can also concatenate features corresponding to the same candidate lane line in different dimensions to construct a feature set for candidate lane lines, facilitating subsequent lane line prediction. By using the intersection of the lane line with the image boundary, the angle between the lane line and the horizontal direction, and the lane line category to represent a lane line, it can model the lane line, reducing the computational cost of the model. Finally, it can convert the lane line dataset into a lane line model format dataset, allowing for direct use of the dataset for lane line detection.

[0071] Next, the lane line detection method proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0072] like Figure 4 As shown, the lane line detection method includes the following steps:

[0073] In step S201, images of the road surrounding the vehicle are acquired.

[0074] Road images can be acquired through various means such as vehicle-mounted cameras or radar, without specific limitations.

[0075] In step S202, the road image is input into the pre-trained prediction model, which outputs the actual type and actual position of the lane lines in the road image. The prediction model is obtained by training a convolutional neural network based on an image dataset carrying real area labels. The shallow lane line features and deep lane line features of the road image are extracted by the convolutional neural network. The actual position of the lane lines is predicted by using the shallow lane line features, and the actual type of the lane lines is predicted by using the deep lane line features.

[0076] It is understood that in this embodiment, a road image is input into a pre-trained prediction model to predict the actual type and location of lane lines, and outputs the line category probability, edge point coordinates, and angle θ with the horizontal direction of n candidate lane lines. The category may include white solid line, white dashed line, yellow solid line, yellow dashed line, etc., or not a lane line. The prediction model has been described in the above embodiments and will not be repeated here.

[0077] According to the lane detection method proposed in the embodiments of this application, shallow and deep features are utilized. Deep features have undergone more feature arrangement and downsampling, and have more overall features, which helps to distinguish different lane lines. Shallow features contain more detailed features such as edge gradient and edge orientation, which helps to find the more accurate location of the lane line. Making full use of features at different levels can improve the accuracy of lane line detection.

[0078] The lane line detection method is illustrated below with a specific embodiment. The steps are as follows:

[0079] Step 1: Obtain the lane line dataset. The dataset contains images, the regions where lane lines are located in the images, and the category of each pixel in the lane line region. This is a common lane line annotation method and is relatively easy to obtain. This information is manually annotated and contains real labels.

[0080] Step 2: Model the lane lines. For example... Figure 2 As shown, a lane line is represented by three parameters: the intersection of the lane line with the image boundary, the angle between the lane line and the horizontal direction, and the lane line category.

[0081] Step 3: Process the lane line dataset described in Step 1 into the lane line model format described in Step 2. Specifically, from the lane line region, obtain the midpoint p1 of the lane line edge located at the left, lower, or right boundary of the image (this is the intersection of the lane line and the image boundary; the other end of the lane line is in the middle of the image). Obtain the lane line edges located in the image, where points with the same y-coordinate are considered a pair. Calculate the midpoint and add it to the point set Points2. Use the points in Points2 and p1 to fit a straight line, obtaining the angle θ between the straight line and the horizontal direction. The lane line category is directly used.

[0082] Step 4: Train the lane detection model

[0083] 4.1 Construct a convolutional neural network to extract image features, where several convolutional layers at different locations are selected as output feature layers {F1, ..., Fi}, where i ≥ 3, representing the number of feature layers selected. The smallest feature layer, Fi, has a size of H. i ×W i ×C i H i W indicates high. i Indicates width, C i Indicates the number of channels.

[0084] 4.2 Input the lane line image into the convolutional neural network and extract the feature layer {F1, ..., Fi}.

[0085] 4.3 Generating Lane Candidate Lines. In the feature layer Fi, its height is Hi, so there are Hi points on the left and right boundaries. Each point contains Ci-dimensional features, which represent the features of c different θ (0° < θ < 90°) candidate lines mapped back to the original image with the starting coordinates as the origin. The width of Fi is Wi, so there are Wi points on the lower boundary. Each point contains Ci-dimensional features, which represent the features of c θ (0° < θ < 180°) candidate lines mapped back to the original image with the starting coordinates as the origin. The candidate lane lines of the three boundaries are merged into a candidate lane line set {L1, ..., Ln}, where n = c × (2 × Hi + Wi) represents the candidate lane line.

[0086] 4.4 Extract features from each lane line in the candidate lane line set {L1, ..., Ln}. For each feature layer from F1 to Fi, the size is H. j ×W j ×C j If 1 ≤ j ≤ i, input it into a 1×1 convolutional layer with c channels to convert the features into H. j ×W i ×c, so that the c-dimensional features of the points on the upper boundary of Fi can correspond to c lane lines. Then, the features corresponding to the same candidate lane line in different dimensions are concatenated. The concatenation dimension on the channel is c×i. The concatenated features represent the shallow and deep features of the candidate lane line. The feature set corresponding to the candidate lane line set {L1, ..., Ln} is {L1cat, ..., Lncat}, where cat represents the concatenated feature with a size of H. cat ×W cat ×C cat .

[0087] 4.5 Input the candidate lane line feature set {L1cat, ..., Lncat} into the prediction module. Assuming the number of lane line categories is u, the feature set {L1cat, ..., Lncat} (with size n×i×c) is converted into an output of n×(3+u+1) through a convolution operation. 3 represents the horizontal and vertical coordinates and angle of the lane line boundary point, u represents the probability that the lane line belongs to u categories, and 1 represents the probability that it is not a lane line.

[0088] 4.6 The n×(3+u+1) output obtained by the convolutional neural network is compared with the real lane line parameters (i.e., the real type and real position of the lane lines). The candidate lane lines whose parameters are closest to the real lane line parameters are considered as detected, and their difference is calculated as the loss. The parameters of the convolutional neural network are updated using the derivative of the loss with respect to the convolutional neural network.

[0089] 4.7 Repeat steps 4.1 to 4.6 until the loss no longer decreases, i.e., the convolutional neural network converges.

[0090] Step 5: Use the obtained lane detection model to predict the unknown lane image and obtain the parameters of the lane.

[0091] Next, the lane line detection device according to the embodiments of this application is described with reference to the accompanying drawings.

[0092] Figure 5 , Figure 6 This is a block diagram of a lane line detection device according to an embodiment of this application.

[0093] like Figure 5 As shown, the lane line detection device 10 includes: a first acquisition module 101, a modeling module 102, a first construction module 103, a second construction module 104, and a prediction module 105.

[0094] The system comprises the following modules: a first acquisition module 101, which acquires an image dataset containing real-world region annotations of lane lines; a modeling module 102, which models lane lines by replacing a large number of pixels with a small number of parameters to represent lane lines; a first construction module 103, which constructs a convolutional neural network including a feature extraction module and a prediction module, extracts image features from the image dataset using the feature extraction module, and selects the output feature layers of multiple convolutional layers at different locations in the feature extraction module; and a second construction module 104, which selects the output feature layer of the last convolutional layer among multiple convolutional layers, generates a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer, and then constructs a candidate lane line set based on the pixels of each lane in the candidate lane line set. The output feature layers of all selected convolutional layers corresponding to the lane line are used to construct a candidate lane line feature set, which includes shallow lane line features and deep lane line features. The prediction module 105 is used to input the candidate lane line feature set into the prediction module, use the shallow lane line features to predict the actual position of the lane line, use the deep lane line features to predict the actual type of the lane line, calculate the training loss value based on the actual type and actual position and the true type and true position of the lane line, and use the loss value to update the parameters of the convolutional neural network until the training loss value is less than or equal to the preset value, then stop the iterative training to obtain the trained lane line prediction model, and use the lane line prediction model to predict the type and position of the lane line.

[0095] The lane detection device proposed in this application can extract multi-dimensional features from a dataset containing lane lines in real-world regions using a convolutional neural network. This allows the extracted features to have rich expressive power. By utilizing shallow and deep features, it can predict the position and type of lane lines, increasing the accuracy of lane detection. Furthermore, it uses a small number of parameters to represent a large number of lane line pixels for lane line modeling, reducing computational load. It can merge candidate lane lines from three boundaries at multiple angles into a candidate lane line set, improving the accuracy of candidate lane lines and indirectly improving the accuracy of lane line detection. It can also concatenate features corresponding to the same candidate lane line in different dimensions to construct a feature set for candidate lane lines, facilitating subsequent lane line prediction. It uses the intersection of the lane line with the image boundary, the angle between the lane line and the horizontal direction, and the lane line category to represent a lane line, thus modeling the lane line and reducing the computational load of the model. Finally, it can convert the lane line dataset into a lane line model format dataset for direct use in subsequent lane line detection.

[0096] like Figure 6 As shown, the lane line detection device 20 includes a second acquisition module 201 and an output module 202.

[0097] The second acquisition module 201 is used to acquire road images around the vehicle; the output module 202 is used to input the road images into a pre-trained prediction model and output the actual type and actual position of the lane lines in the road images. The prediction model is obtained by training a convolutional neural network based on an image dataset with real area annotations. The shallow lane line features and deep lane line features of the road images are extracted by the convolutional neural network. The shallow lane line features are used to predict the actual position of the lane lines, and the deep lane line features are used to predict the actual type of the lane lines.

[0098] It should be noted that the foregoing explanation of the lane line detection method embodiment also applies to the lane line detection device of this embodiment, and will not be repeated here.

[0099] According to the lane line detection device proposed in the embodiments of this application, shallow and deep features are utilized. Deep features have more overall features due to more feature arrangement and downsampling, which helps to distinguish different lane lines. Shallow features contain more detailed features such as edge gradient and edge orientation, which helps to find the lane line more accurately. Making full use of features at different levels can improve the accuracy of lane line detection.

[0100] Figure 7 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0101] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0102] When the processor 702 executes the program, it implements the lane line detection method provided in the above embodiments.

[0103] Furthermore, the vehicle also includes:

[0104] Communication interface 703 is used for communication between memory 701 and processor 702.

[0105] The memory 701 is used to store computer programs that can run on the processor 702.

[0106] The memory 701 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0107] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an ElSA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0108] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0109] The processor 702 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0110] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the lane line detection method described above.

[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0114] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0115] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0116] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A lane line detection method, characterized in that, Includes the following steps: Obtain an image dataset with real-world region annotations containing lane lines; Lane line modeling uses a small number of parameters to represent a large number of pixels; A convolutional neural network containing a feature extraction module and a prediction module is constructed. The feature extraction module is used to extract image features from the image dataset, and the output feature layers of multiple convolutional layers at different positions of the feature extraction module are selected. Select the output feature layer of the last convolutional layer among multiple convolutional layers, generate a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer, and construct a candidate lane line feature set based on the output feature layers of all selected convolutional layers corresponding to each lane line in the candidate lane line set. The construction of the candidate lane line feature set includes shallow lane line features and deep lane line features. The candidate lane line feature set is input into the prediction module. The shallow lane line features are used to predict the actual position of the lane line, and the deep lane line features are used to predict the actual type of the lane line. The training loss value is calculated based on the actual type and actual position and the true type and true position of the lane line. The parameters of the convolutional neural network are updated using the loss value. The iterative training stops when the training loss value is less than or equal to a preset value, and the trained lane line prediction model is obtained. The lane line prediction model is used to predict the type and position of the lane line. The step of generating a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer includes: Identify the height, width, and number of channels of the last feature layer; The target points of the left and right boundaries and the bottom boundary are determined according to the height and width of the last feature layer, and the number of feature dimensions contained in each target point is determined according to the number of channels. Based on the number of feature dimensions, candidate lane lines with three boundaries and multiple angles are determined, and these candidate lane lines with three boundaries and multiple angles are merged into a candidate lane line set. Before constructing a convolutional neural network containing a feature extraction module and a prediction module, and before using the feature extraction module to extract image features from the image dataset, the method further includes: Lane line modeling is constructed by representing a lane line using the intersection of the lane line with the image boundary, the angle between the lane line and the horizontal direction, and the lane line category. The dataset is converted into a dataset in lane line model format based on the lane line modeling.

2. The method according to claim 1, characterized in that, A candidate lane line feature set is constructed based on the features of each lane line in the candidate lane line set, including: Extract features from each lane line in the candidate lane line set; Features corresponding to the same candidate lane line in different dimensions are concatenated to obtain a combination of shallow lane line features and deep lane line features. A candidate lane line feature set is then constructed based on the combination of the shallow lane line features and the deep lane line features.

3. The method according to claim 1, characterized in that, The dataset includes images, the regions containing lane lines in the images, and the category of each pixel in the lane line regions. The process of converting the dataset into the lane line model format based on the lane line model includes: From the region where the lane line is located, obtain the midpoint of the edge of the lane line located at the left, lower, or right boundary of the image; Obtain the lane line edges located in the image. If the ordinates of the lane line edges are the same, they are considered as a pair of points. Calculate the midpoint of the pair of points and add it to the point set. Fit a straight line based on the points in the point set and the midpoint of the lane line edges to obtain the angle between the straight line and the horizontal direction. A format-converted dataset is generated based on the midpoint of the lane line edge, the angle between the straight line and the horizontal direction, and the category of each pixel in the lane line region.

4. A lane line detection method, characterized in that, Includes the following steps: Acquire road images surrounding the vehicle; input the road images into a pre-trained prediction model in a lane line detection method as described in any one of claims 1-3, and output the actual type and actual position of the lane lines in the road images, wherein the prediction model is obtained by training a convolutional neural network based on an image dataset carrying real-world region annotations, extracting shallow lane line features and deep lane line features from the road images according to the convolutional neural network, predicting the actual position of the lane lines using the shallow lane line features, and predicting the actual type of the lane lines using the deep lane line features.

5. A lane line detection device, characterized in that, include: The first acquisition module is used to acquire an image dataset with real-world region annotations carrying lane lines. The modeling module is used for lane line modeling, using a small number of parameters to replace a large number of pixels to represent lane lines. The first construction module is used to construct a convolutional neural network that includes a feature extraction module and a prediction module. The feature extraction module is used to extract image features from the image dataset, and the output feature layers of multiple convolutional layers at different positions of the feature extraction module are selected. The second construction module is used to select the output feature layer of the last convolutional layer among multiple convolutional layers, generate a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer, and construct a candidate lane line feature set based on the output feature layers of all selected convolutional layers corresponding to each lane line in the candidate lane line set. The constructed candidate lane line feature set includes shallow lane line features and deep lane line features. The prediction module is used to input the candidate lane line feature set into the prediction module, predict the actual position of the lane line using the shallow lane line features, predict the actual type of the lane line using the deep lane line features, calculate the training loss value based on the actual type and actual position and the true type and true position of the lane line, update the parameters of the convolutional neural network using the loss value, and stop iterative training when the training loss value is less than or equal to a preset value to obtain a trained lane line prediction model, and use the lane line prediction model to predict the type and position of the lane line. The step of generating a candidate lane line set based on the pixels of the lower left and lower right boundaries of the output feature layer of the last convolutional layer includes: Identify the height, width, and number of channels of the last feature layer; The target points of the left and right boundaries and the bottom boundary are determined according to the height and width of the last feature layer, and the number of feature dimensions contained in each target point is determined according to the number of channels. Based on the number of feature dimensions, candidate lane lines with three boundaries and multiple angles are determined, and these candidate lane lines with three boundaries and multiple angles are merged into a candidate lane line set. Before constructing a convolutional neural network containing a feature extraction module and a prediction module, and before using the feature extraction module to extract image features from the image dataset, the method further includes: Lane line modeling is constructed by representing a lane line using the intersection of the lane line with the image boundary, the angle between the lane line and the horizontal direction, and the lane line category. The dataset is converted into a dataset in lane line model format based on the lane line modeling.

6. A lane line detection device for implementing the lane line detection method as described in any one of claims 1-3, characterized in that, include: The second acquisition module is used to acquire images of the road surrounding the vehicle; The output module is used to input the road image into a pre-trained prediction model and output the actual type and actual position of the lane lines in the road image. The prediction model is obtained by training a convolutional neural network based on an image dataset carrying real area annotations. The convolutional neural network extracts shallow lane line features and deep lane line features of the road image. The shallow lane line features are used to predict the actual position of the lane lines, and the deep lane line features are used to predict the actual type of the lane lines.

7. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the lane line detection method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the lane line detection method as described in any one of claims 1-3.

Images