Lane line detection model training method and device and lane line detection method and device

Abstract

The application discloses a lane line detection model training method and device and a lane line detection method and device. The method comprises the following steps: acquiring a to-be-trained image and label information of lane line instances in the to-be-trained image; according to the label information of the lane line instances, pixel points in the to-be-trained image are encoded by using a preset feature coding strategy to obtain feature coding information of the pixel points, wherein the feature coding information of the pixel points comprises a horizontal feature vector and a vertical feature vector of the pixel points; and the lane line detection model is trained by using the to-be-trained image, the feature coding information of the pixel points and the label information of the lane line instances, so that a trained lane line detection model is obtained. According to the application, the feature coding strategy is used to code all pixel points in the image based on the label information of the lane line instances, and the lane line detection model is trained in combination with the to-be-trained image and the label information of the lane line instances, so that the accuracy and real-time performance of the lane line detection are improved, and the lane line detection is not limited by a preset number of lane lines.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a lane line detection model training method, apparatus, and lane line detection method and apparatus. Background Technology

[0002] Lane detection is an essential function in autonomous driving. It can be applied to lane-level positioning and then to functional modules such as LDW (Lane Departure Warning) and LKA (Lane Keeping Assistance).

[0003] One existing lane detection scheme first acquires a road image as input, then uses CNN (Convolutional Neural Networks) to process the input road image, obtaining a binary semantic segmentation result of the lane line and a vector representation of each pixel in the road image; then, based on the above two information sources, a clustering algorithm is used to cluster the lane lines to obtain the instance segmentation result; finally, lane line fitting is performed on each lane line instance.

[0004] However, the above lane detection methods have at least the following problems: 1) The lane detection model can only detect a preset number of lane lines; 2) Clustering algorithms need to be run in the post-processing stage, but these clustering algorithms often have long convergence times, which is not conducive to the real-time detection of lane lines. Summary of the Invention

[0005] This application provides a lane line detection model training method and apparatus, as well as a lane line detection method and apparatus, to improve the accuracy and real-time performance of lane line detection.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a lane line detection model training method, wherein the method includes:

[0008] Obtain the annotation information of the image to be trained and the lane line instances in the image to be trained;

[0009] Based on the annotation information of the lane line instance, the pixels in the image to be trained are encoded using a preset feature encoding strategy to obtain the feature encoding information of the pixel. The feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel.

[0010] The lane detection model is trained using the feature encoding information of the image to be trained, the feature encoding information of the pixels, and the annotation information of the lane line instances, to obtain the trained lane detection model.

[0011] Optionally, the image to be trained includes multiple pixel rows and multiple pixel columns. The step of encoding the pixels in the image to be trained using a preset feature encoding strategy based on the annotation information of the lane line instance to obtain the feature encoding information of the pixel includes:

[0012] The pixel position of the pixel in the image to be trained is determined according to the method of each pixel row and each pixel column;

[0013] Based on the annotation information of the lane line instance and the pixel position of the lane line instance in the image to be trained, the horizontal feature vector and vertical feature vector of the pixel are determined.

[0014] Optionally, the pixel row includes the current pixel row and the next pixel row, and the pixel point includes the lane line pixel point corresponding to the lane line instance. The step of encoding the pixel points in the image to be trained using a preset feature encoding strategy based on the annotation information of the lane line instance to obtain the feature encoding information of the pixel points includes:

[0015] Based on the annotation information of the lane line instance, determine the position of the lane line pixel in multiple current pixel columns of the current pixel row and the position of multiple next pixel columns of the next pixel row;

[0016] Based on the position of the lane line pixel in each of the multiple current pixel columns of the current pixel row, determine the lateral feature vector of the lane line pixel in each of the current pixel columns of the current pixel row;

[0017] Based on the positions of the lane line pixels in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row, determine the vertical feature vectors of the lane line pixels in each current pixel column of the current pixel row.

[0018] The horizontal feature vectors of the lane line pixels in each current pixel column of the current pixel row and the vertical feature vectors of each current pixel column of the current pixel row are used as the feature encoding information of the lane line pixels.

[0019] Optionally, determining the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row based on the position of the lane line pixel in each of the multiple current pixel columns of the current pixel row includes:

[0020] Based on the position of the lane line pixel in the current pixel row among multiple current pixel columns, determine the center pixel column of the lane line pixel in the current pixel row;

[0021] Using the center pixel column of the lane line pixel in the current pixel row as a reference, determine the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row.

[0022] Secondly, embodiments of this application also provide a lane line detection method, wherein the method includes:

[0023] Acquire the image to be detected;

[0024] The lane line detection model is used to detect lane lines in the image to be detected, and the lane line semantic segmentation results and the feature encoding information of the pixels are obtained.

[0025] Based on the lane line semantic segmentation results and the feature encoding information of the pixels, the lane line pixels in the image to be detected are clustered.

[0026] The lane line instance segmentation results are determined based on the clustering results, and the final lane line detection results are obtained based on the lane line instance segmentation results.

[0027] The lane detection model is trained based on any of the aforementioned lane detection model training methods.

[0028] Optionally, the feature encoding information of the pixels includes the feature encoding information of lane line pixels, and the step of clustering lane line pixels in the image to be detected based on the lane line semantic segmentation result and the feature encoding information of the pixels includes:

[0029] Based on the lane line semantic segmentation results, determine the current pixel row and current pixel column of the lane line pixels in the image to be detected;

[0030] Based on the feature encoding information of the lane line pixels, determine the horizontal feature vector of the lane line pixels in the current pixel row and the current pixel column;

[0031] Based on the lateral feature vectors of the lane line pixels in the current pixel row and the current pixel column, the lane line pixels in the current pixel row are clustered to obtain multiple lane line pixel clusters in the current pixel row.

[0032] Optionally, determining the lane line instance segmentation result based on the clustering result includes:

[0033] Determine whether the current pixel row has a corresponding historical pixel row;

[0034] If it does not exist, the multiple lane line pixel clusters in the current pixel row are identified sequentially to obtain the lane line instance segmentation result;

[0035] If they exist, multiple lane line pixel clusters of the historical pixel row are obtained, and the multiple lane line pixel clusters of the historical pixel row are matched with the multiple lane line pixel clusters of the current pixel row based on the vertical feature vector of the lane line instance of the historical pixel row and the vertical feature vector of the lane line instance of the current pixel row to obtain the identifier of each lane line pixel cluster of the current pixel row.

[0036] Optionally, the lane detection result is the lane instance detection result at the current time. After determining the lane instance segmentation result based on the clustering result, and obtaining the final lane detection result based on the lane instance segmentation result, the method further includes:

[0037] Obtain the lane line instance detection results from the previous moment;

[0038] Based on the lane line instance segmentation result of the previous moment and the lane line instance segmentation result of the current moment, determine the distance between the lane line instance of the current moment and the corresponding lane line instance of the previous moment.

[0039] The distance between the current lane line instance and the corresponding lane line instance from the previous moment is used to facilitate lane line tracking by a preset lane line tracking algorithm.

[0040] Thirdly, embodiments of this application also provide a lane line detection model training device, wherein the device includes:

[0041] The first acquisition unit is used to acquire the training image and the annotation information of lane line instances in the training image;

[0042] The encoding unit is used to encode the pixels in the image to be trained according to the annotation information of the lane line instance using a preset feature encoding strategy to obtain the feature encoding information of the pixel, wherein the feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel.

[0043] The training unit is used to train a lane detection model using the feature encoding information of the image to be trained and the pixel points, as well as the annotation information of the lane line instances, to obtain the trained lane detection model.

[0044] Fourthly, embodiments of this application also provide a lane line detection device, wherein the device includes:

[0045] The second acquisition unit is used to acquire the image to be detected;

[0046] The detection unit is used to perform lane line detection on the image to be detected using a lane line detection model, and obtain lane line semantic segmentation results and feature encoding information of pixel points.

[0047] A clustering unit is used to cluster lane line pixels in the image to be detected based on the lane line semantic segmentation result and the feature encoding information of the pixel.

[0048] The first determining unit is used to determine the lane line instance segmentation result based on the clustering result, so as to obtain the final lane line detection result based on the lane line instance segmentation result.

[0049] The lane detection model is trained based on the aforementioned lane detection model training device.

[0050] Fifthly, embodiments of this application also provide an electronic device, including:

[0051] Processor; and

[0052] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned lane detection model training methods or lane detection methods.

[0053] Sixthly, embodiments of this application also provide a computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform any of the aforementioned lane line detection model training methods or lane line detection methods.

[0054] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: The lane detection model training method of this application embodiment first obtains the training image and the annotation information of lane line instances in the training image; then, based on the annotation information of the lane line instances, it encodes the pixels in the training image using a preset feature encoding strategy to obtain the feature encoding information of the pixels; finally, it trains the lane detection model using the training image and the feature encoding information of the pixels to obtain the trained lane detection model. The lane detection model training method of this application embodiment is based on the annotation information of lane line instances and uses a preset feature encoding strategy to encode the features of lane line instances, which is used as a supervision signal to train the lane detection model, improving the accuracy and real-time performance of the lane detection model, and is not limited by the preset number of lane lines, greatly expanding the application scenarios. Attached Figure Description

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1 This is a flowchart illustrating a lane line detection model training method in an embodiment of this application.

[0057] Figure 2 This is a flowchart illustrating a lane line detection method according to an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of the overall process of lane line detection in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the structure of a lane line detection model training device in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of a lane line detection device according to an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0064] This application provides a method for training a lane line detection model, such as... Figure 1 The diagram shows a flowchart of a lane detection model training method according to an embodiment of this application. The method includes at least the following steps S110 to S130:

[0065] Step S110: Obtain the annotation information of the image to be trained and the lane line instances in the image to be trained.

[0066] In this embodiment of the application, when training the lane line detection model, it is necessary to first obtain the image to be trained and the annotation information of the lane line instances in the image to be trained.

[0067] The training image mentioned above can be the original road image collected. Of course, in order to improve the training efficiency of the lane detection model, the region of interest (ROI) in the original road image can also be cropped and preprocessed before being used as the training image. Those skilled in the art can flexibly choose how to obtain the training image according to actual needs, and no specific limitation is made here.

[0068] The labeling information for the lane line instances mentioned above generally refers to the label of each pixel in the image. For example, if the pixel value of a pixel is 0, it means that the pixel is the background; if it is greater than 0, it means that the pixel belongs to the lane line instance with that value as its ID.

[0069] Step S120: Based on the annotation information of the lane line instance, the pixels in the image to be trained are encoded using a preset feature encoding strategy to obtain the feature encoding information of the pixels. The feature encoding information of the pixels includes the horizontal feature vector and the vertical feature vector of the pixels.

[0070] Based on the annotation information of the lane line instances mentioned above, it is possible to determine which pixels in the image to be trained are lane line pixels corresponding to lane line instances and which are non-lane line pixels. Although existing solutions can also directly train lane line detection models based on the annotation information of these lane line instances, the detection accuracy and real-time performance of the lane line detection models obtained by this training method still need to be improved, and are limited by the preset number of lane lines.

[0071] Based on this, in order to further improve the detection accuracy and real-time performance of the lane line detection model, this application uses a preset feature encoding strategy to encode each pixel in the training image based on the annotation information of lane line instances, thereby obtaining the feature encoding information of all pixels.

[0072] The aforementioned preset feature encoding strategy fully considers the characteristics and relationships of the horizontal and vertical distribution of lane line instances' pixels in the image, extracting the horizontal and vertical feature vectors of each pixel, which are then used as ground values ​​for training the lane line detection model.

[0073] Step S130: Train the lane line detection model using the feature encoding information of the image to be trained and the pixel points, as well as the annotation information of the lane line instance, to obtain the trained lane line detection model.

[0074] After obtaining the feature encoding information of the aforementioned pixels, the feature encoding information of the corresponding pixels in the image to be trained, together with the annotation information of the lane line instances, can be used as a supervision signal to train the lane line detection model. The loss function is constructed by the deviation between the predicted value and the true value output by the lane line detection model, and the parameters of the lane line detection model are updated accordingly. Finally, the trained lane line detection model is output.

[0075] The lane detection model trained based on the feature encoding information of the pixels in the embodiments of this application does not depend on the preset number of lanes, that is, it can detect any number of lanes. Furthermore, the lane detection results obtained based on the lane detection model in the embodiments of this application do not require complex and time-consuming clustering algorithms in the post-processing stage to quickly segment lane instances, which greatly improves the detection efficiency of lane instances and meets the accuracy and real-time requirements of lane detection.

[0076] In one embodiment of this application, the image to be trained includes multiple pixel rows and multiple pixel columns. The step of encoding the pixels in the image to be trained using a preset feature encoding strategy based on the annotation information of the lane line pixels to obtain the feature encoding information of the pixels includes: determining the pixel position of the pixel in the image to be trained according to each pixel row and each pixel column; and determining the horizontal feature vector and vertical feature vector of the pixel based on the annotation information of the lane line instance and the pixel position of the lane line instance in the image to be trained.

[0077] To facilitate feature encoding of pixels in the training image, the training image in this embodiment can be divided into multiple pixel rows and multiple pixel columns. Each pixel column of each pixel row corresponds to a specific pixel. The pixel may be a lane line pixel or a non-lane line pixel such as a background pixel. A specific lane line instance can be regarded as being composed of multiple lane line pixels distributed in multiple pixel rows and multiple pixel columns.

[0078] When encoding pixels in the training image, we can first traverse each pixel row by row based on the label information of the lane line instances, and then traverse each pixel column within each pixel row to determine the position of each lane line pixel in the training image. For example, if the current training image contains three lane line instances: lane line 01, lane line 02, and lane line 03, and the lane line pixel corresponding to lane line 01 is distributed in rows 5-20 and columns 6-10 of the training image, then the pixel region formed by these rows 5-20 and columns 6-10 can be considered as the position of lane line 01.

[0079] After obtaining the position of each lane line pixel in the image to be trained, the horizontal feature vector and vertical feature vector of each lane line pixel can be calculated based on the distribution of the lane line pixel position in the pixel row and pixel column.

[0080] Furthermore, after determining the positions of lane line pixels in the training image, the positions of non-lane line pixels in the training image are also determined. This is because non-lane line pixels are not the focus of lane line detection, but they need to be clearly distinguished from lane line pixels. Therefore, non-lane line pixels in the training image can be uniformly encoded, for example, by assigning both the horizontal and vertical feature vectors of non-lane line pixels a value of 0.

[0081] In one embodiment of this application, the pixel row includes a current pixel row and a next pixel row, and the pixel point includes a lane line pixel point corresponding to a lane line instance. The step of encoding the pixel points in the training image using a preset feature encoding strategy based on the annotation information of the lane line pixel points to obtain the feature encoding information of the pixel points includes: determining the positions of the lane line pixel points in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row based on the annotation information of the lane line pixel points; determining the horizontal feature vectors of the lane line pixel points in each current pixel column of the current pixel row based on the positions of the lane line pixel points in multiple current pixel columns of the current pixel row; determining the vertical feature vectors of the lane line pixel points in each current pixel column of the current pixel row based on the positions of the lane line pixel points in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row; and using the horizontal feature vectors and vertical feature vectors of the lane line pixel points in each current pixel column of the current pixel row as the feature encoding information of the pixel points.

[0082] In this embodiment, when encoding lane line pixels, a bottom-up, row-by-row traversal can be performed. For any currently traversed row, the position of the lane line pixel within multiple current pixel columns can be determined based on the lane line instance's annotation information. For example, if the 9th row is being traversed, the lane line pixel's annotation information indicates that it is located in columns 6-10 of the 9th row. Similarly, the pixel column containing the lane line pixel in each row can be obtained using the above method.

[0083] When determining the lateral feature vectors of each lane line pixel corresponding to a lane line instance, it is only necessary to consider the lateral characteristics and interrelationships of the multiple lane line instances distributed in the image. For example, if a road image contains three lane line instances, namely lane line 01, lane line 02, and lane line 03, then regardless of which pixel row these three lane line instances are located in, the three lane lines can be distinguished based on the lateral distribution of the lane line pixels in that row.

[0084] Therefore, in determining the lateral feature vector of each current pixel column in the current pixel row, the embodiments of this application can directly determine it based on the position of the lane line pixel in the multiple current pixel columns of the current pixel row. For example, in the 9th pixel row, the lane line pixels of lane line 01 are distributed in columns 6-10, the lane line pixels of lane line 02 are distributed in columns 20-25, and the lane line pixels of lane line 03 are distributed in columns 35-40. The distribution difference of these three in the pixel columns of the 9th pixel row is very obvious.

[0085] When determining the vertical feature vector of each lane line pixel corresponding to a lane line instance, it is necessary to consider the distribution positions of lane line pixels in the upper and lower rows. Taking a bottom-up, row-by-row traversal as an example, in addition to needing the positions of multiple current pixel columns (cols) of the lane line pixels in the current pixel row (row), it is also necessary to need the positions of multiple next pixel columns in the next pixel row. For example, if the current pixel row is row 9, then the next pixel row is row 8. Based on the positions of multiple lane line pixels in row 8, the vertical distribution relationship of lane line pixels between the upper and lower pixel rows is constructed.

[0086] It should be noted that the above encoding process can be performed separately for each lane line instance. Of course, those skilled in the art can also encode multiple lane line instances simultaneously according to the actual situation, without making specific limitations here.

[0087] In one embodiment of this application, determining the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row based on the position of the lane line pixel in each current pixel column of the current pixel row includes: determining the center pixel column of the lane line pixel in the current pixel row based on the position of the lane line pixel in each current pixel column of the current pixel row; and determining the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row based on the center pixel column of the lane line pixel in the current pixel row.

[0088] In determining the lateral feature vectors of each lane line pixel corresponding to a lane line instance, this application employs a simple and fast feature encoding method. Specifically, for any current pixel row, the pixel column containing the lane line pixels in the current pixel row is first determined. Then, the center pixel column is determined based on the relative positions of these pixel columns. Finally, using the center pixel column as a reference, the lateral feature vectors of the lane line pixels located in each pixel column are determined.

[0089] For example, based on the label information of the lane line instance, we can determine that the lane line pixel corresponding to lane line 01 is located in columns 6-10 of the current pixel row. We can then calculate the average of these columns: (6+7+8+9+10) / 5 = 8, thus determining the center pixel column as column 8. To simplify the process, we can assign a value of 0 to the horizontal feature vector of the lane line pixel in the center pixel column, assign a value of 1 to the horizontal feature vectors of the lane line pixels in columns 6 and 7 to the left of the center pixel column, and assign a value of -1 to the horizontal feature vectors of the lane line pixels in columns 9 and 10 to the right of the center pixel column. This results in the horizontal feature vector of a lane line pixel in a pixel row being represented in the form {1,1,0,-1,-1}. The same processing method applies to other lane line instances. For non-lane line pixels in the training image, a value of 0 can be directly assigned.

[0090] To facilitate understanding of the above embodiments, pseudocode for a lane line pixel encoding process is provided below:

[0091] horizontal_vector,vertical_vector=zeros(H,W,2)

[0092] For lanes / / Iterate through all lane lines in the current frame of the image:

[0093] For row in range(H, -1): / / Traverse the image row by row from bottom to top.

[0094] cols = where(label[row,:]>0) / / The lane line pixel position in the image for this lane instance (lane) and its row (row).

[0095] if cols.size = 0

[0096] Continue

[0097] ###### Calculation of horizontal_vector

[0098] Calculate its mean: mean_col = mean(cols)

[0099] For points to the left of the mean, horizontal_vector[row,c] = 1.0

[0100] For points to the right of the mean, horizontal_vector[row,c] = -1.0

[0101] For points at the mean, horizontal_vector[row,c] = 0.0

[0102] ######Calculation of vertical_vector

[0103] The horizontal vector of lane line pixels corresponding to the previous pixel row (pre_row) of the lane line instance:

[0104] If row is not the bottom row:

[0105] For c in pre_cols:

[0106] vertical_vector[pre_row,c] = normalized([mean_col-c,row-pre_row])

[0107] ####Update pre_row, pre_cols:

[0108] pre_cols = cols

[0109] pre_row = row

[0110] The calculation of horizontal_vector can be referred to the description in the previous embodiment. The calculation of vertical_vector needs to be determined by combining the distribution positions of the lane line pixels in the two pixel rows above and below. Specifically, on the one hand, the mean_col, which is the center pixel column of the lane line pixels in the current pixel row, is subtracted from each pixel column c in the pre_cols, which is the pixel column of the lane line pixels in the previous pixel row, pre_row. On the other hand, the difference between the current pixel row and the previous pixel row, pre_row, is also subtracted. Then, the difference results of the above two aspects are normalized to obtain a value distributed between 0 and 1, which is used as the vertical feature vector vertical_vector of each pixel column c of the lane line pixels in the previous pixel row, pre_row.

[0111] Additionally, it should be noted that the "previous pixel row" mentioned above is not necessarily the adjacent pixel row of the current pixel row. During the row-by-row traversal, there may be one or several rows where the pixel value is 0, that is, there are no lane line pixels. For example, the lane lines may be incompletely identified due to being occluded or worn. In this case, the pixel row can be skipped directly, and the previous pixel row containing lane line pixels can be processed directly.

[0112] Therefore, if the lane line pixels are from different lane line instances, the vertical feature vectors extracted from the lane line pixels in the upper and lower rows of two different lane line instances will obviously be different in value or magnitude from the vertical feature vectors extracted from the lane line pixels in the upper and lower rows of the same lane line instance, because the distribution of lane line pixels in different lane line instances is significantly different in the pixel column cols. Therefore, the embodiments of this application fully consider the characteristics and relationships of the horizontal and vertical distribution of lane line instance pixels in the image through the above feature encoding method, and quantify the above characteristics in a more simplified way.

[0113] This application also provides a lane line detection method, such as... Figure 2 The diagram shows a flowchart of a lane line detection method according to an embodiment of this application. The method includes at least the following steps S210 to S240:

[0114] Step S210: Obtain the image to be detected.

[0115] When performing lane detection, it is necessary to first acquire the image to be detected. The image to be detected can be the original road image that has been collected. Of course, in order to improve the detection efficiency of the lane detection model, the region of interest (ROI) in the original road image can also be cropped and preprocessed before being used as the image to be detected.

[0116] Step S220: Use the lane line detection model to detect lane lines in the image to be detected, and obtain lane line semantic segmentation results and feature encoding information of pixels.

[0117] Lane detection is performed using the lane detection model trained in the aforementioned embodiments, thereby obtaining lane semantic segmentation results (H*W) and feature encoding information of pixels. Specifically, the feature encoding information of pixels may include the horizontal feature vector (H*W) and the vertical feature vector (H*W*2) of the pixels.

[0118] Step S230: Based on the lane line semantic segmentation result and the feature encoding information of the pixel, cluster the lane line pixels in the image to be detected.

[0119] Step S240: Determine the lane line instance segmentation result based on the clustering result, so as to obtain the final lane line detection result based on the lane line instance segmentation result; wherein, the lane line detection model is trained based on any of the aforementioned lane line detection model training methods.

[0120] Based on the lane line semantic segmentation results and the feature encoding information of the pixels, the lane line pixels in the image to be detected are clustered to obtain multiple lane line pixel clusters. Each lane line pixel cluster corresponds to a lane line instance. Finally, each lane line instance is fitted to obtain the final lane line detection result.

[0121] Since the horizontal and vertical feature vectors of the lane line detection model output by the model fully consider the characteristics and relationships of the lane line pixels in the horizontal and vertical distribution of the image, lane line pixels belonging to the same lane line instance can be quickly clustered into one class based on the horizontal and vertical feature vectors of the lane line pixels during the clustering process. Compared with the existing complex and time-consuming clustering algorithms, this greatly improves the efficiency of lane line detection.

[0122] In one embodiment of this application, the feature encoding information of the pixel points includes the feature encoding information of lane line pixels. The step of clustering lane line pixels in the image to be detected based on the lane line semantic segmentation result and the feature encoding information of the pixel points includes: determining the current pixel row and current pixel column of the lane line pixels in the image to be detected based on the lane line semantic segmentation result; determining the lateral feature vector of the lane line pixels in the current pixel row and the current pixel column based on the feature encoding information of the lane line pixels; and clustering the lane line pixels in the current pixel row based on the lateral feature vector of the lane line pixels in the current pixel row and the current pixel column to obtain multiple lane line pixel clusters in the current pixel row.

[0123] The process of clustering lane line pixels in this application embodiment can be regarded as the process of decoding the feature encoding information of lane line pixels. The clustering process can be divided into horizontal clustering and vertical clustering. The horizontal clustering process can be carried out by traversing row by row.

[0124] When performing lateral clustering, we can first determine the pixel rows and columns where lane line instances are distributed based on the binary semantic segmentation results of the lane lines. During row-by-row traversal, for any given pixel row, we can determine the lateral feature vector corresponding to that current pixel row and column based on the feature encoding information output by the lane line detection model. For example, the lateral feature vector of lane line 01 in row 9 is {1,1,0,-1,-1}. This can be implemented using the following pseudocode:

[0125] For row in range(H, -1): / / Iterate through each row from bottom to top

[0126] cols = np.where(BW[row,:]>Foreground Threshold) / / Based on the binary semantic segmentation results of the lane lines, find the location of the lane line pixels.

[0127] Since the non-lane line pixels between two lane lines are assigned a value of 0, the lane line pixels can be quickly clustered based on the numerical distribution of the horizontal feature vector in this pixel row, thus obtaining a cluster of lane line pixels corresponding to multiple lane line instances.

[0128] In one embodiment of this application, determining the lane line instance segmentation result based on the clustering result includes: determining whether the current pixel row has a corresponding historical pixel row; if not, sequentially identifying multiple lane line pixel clusters of the current pixel row to obtain the lane line instance segmentation result; if so, obtaining multiple lane line pixel clusters of the historical pixel row, and matching the multiple lane line pixel clusters of the historical pixel row with the multiple lane line pixel clusters of the current pixel row based on the vertical feature vectors of the lane line pixels of the historical pixel row and the vertical feature vectors of the lane line pixels of the current pixel row to obtain the identifiers of each lane line pixel cluster of the current pixel row.

[0129] After obtaining multiple lane line pixel clusters corresponding to the current pixel row, vertical clustering can be further performed. The purpose of vertical clustering is to determine which two lane line pixel clusters between the current pixel row and previously clustered pixel rows correspond to the same lane line instance. Therefore, we can first determine whether the current pixel row has a corresponding historical pixel row. If not, it means that the current pixel row is the first traversed pixel row. Therefore, we can directly number the multiple lane line pixel clusters obtained in sequence. For example, if we obtain three lane line pixel clusters, we can number them 01, 02 and 03 in sequence.

[0130] If there is a corresponding historical pixel row, since the historical pixel row has already completed the clustering of its row, that is, there are multiple lane line pixel clusters corresponding to the historical pixel row, then we can match the multiple lane line pixel clusters of the historical pixel row with the multiple lane line pixel clusters of the current pixel row based on the vertical feature vectors of the lane line pixels of the historical pixel row and the vertical feature vectors of the lane line pixels of the current pixel row, so as to determine which two lane line pixel clusters correspond to the same lane line instance, and then number the multiple lane line pixel clusters of the current pixel row according to the matching results.

[0131] For example, suppose a lane line pixel cluster in the current pixel row best matches the lane line pixel cluster numbered 01 in the historical pixel row. Then, the lane line pixel cluster in the current pixel row can also be numbered 01. If a lane line pixel cluster in the current pixel row does not match any of the lane line pixel clusters in the historical pixel row, it can be numbered based on the numbering result of the historical pixel row, for example, numbered 04, which is the maximum ID of the historical pixel row + 1.

[0132] It should be noted that the above "historical pixel rows" include all pixel rows that have been clustered before the current pixel row. All clustering results of these historical pixel rows are uniformly maintained and managed in a database of lane line pixel clusters. When matching, they can be matched with all previously clustered lane line pixel clusters, thereby ensuring the uniqueness and accuracy of lane line instance numbers.

[0133] For example, suppose the current pixel row is row 8, and the historical pixel rows include rows 9 and 10. After matching the lane line pixel clusters in rows 9 and 10, the multiple lane line pixel clusters in row 10 are numbered 01, 02, and 03 respectively, and the multiple lane line pixel clusters in row 9 are numbered 02, 03, and 04 respectively. If only the multiple lane line pixel clusters in row 8 are matched with the multiple lane line pixel clusters in row 9, then the lane line pixel cluster numbered 01 cannot be matched. If there is a lane line pixel cluster in row 8 that matches the lane line pixel cluster numbered 01, it will cause a problem with the numbering. Therefore, it is necessary to match with all lane line pixel clusters in the historical pixel rows. Two lane line pixel clusters that match successfully and correspond to the same lane line instance can be merged into one lane line pixel cluster for maintenance.

[0134] In one embodiment of this application, the lane detection result is the lane instance detection result at the current moment. After determining the lane instance segmentation result based on the clustering result, and obtaining the final lane detection result based on the lane instance segmentation result, the method further includes: obtaining the lane instance detection result at the previous moment; determining the distance between the lane instance at the current moment and the corresponding lane instance at the previous moment based on the lane instance segmentation result at the previous moment and the lane instance segmentation result at the current moment; and using a preset lane tracking algorithm to perform lane tracking based on the distance between the lane instance at the current moment and the corresponding lane instance at the previous moment.

[0135] Based on the lane detection results obtained from the aforementioned embodiments, lane tracking can be further performed. When performing lane tracking, it is necessary to combine the lane detection results of two adjacent time points. That is, based on the lane detection results of two adjacent time points, the distance between the lane instance at the current time point and the corresponding lane instance at the previous time point is calculated. Here, "correspondence" can be determined based on the longitudinal feature vectors of the lane instances at two time points. Thus, lane tracking is achieved by combining the distance between the lane instances at two adjacent time points with a preset lane tracking algorithm.

[0136] In one embodiment of this application, determining the distance between the current lane line instance and the corresponding lane line instance from the previous moment based on the lane line instance segmentation result from the previous moment and the lane line instance segmentation result from the current moment includes: determining the relative transformation relationship between the coordinates of the lane line pixel from the previous moment and the coordinates of the lane line pixel from the current moment; transforming the coordinates of the lane line pixel from the previous moment to the vehicle coordinate system from the previous moment based on the relative transformation relationship to obtain the transformed coordinates of the lane line pixel; and determining the distance between the current lane line instance and the corresponding lane line instance from the previous moment based on the transformed coordinates of the lane line pixel.

[0137] In one embodiment of this application, determining the relative transformation relationship between the coordinates of the lane line pixels at the previous moment and the coordinates of the lane line pixels at the current moment includes: acquiring vehicle pose information at the previous moment and vehicle pose information at the current moment; and determining the relative transformation relationship between the coordinates of the lane line pixels at the previous moment and the coordinates of the lane line pixels at the current moment based on the vehicle pose information at the previous moment and the vehicle pose information at the current moment.

[0138] When determining the distance between lane line instances at two adjacent time points, the pose information of the vehicle at the two adjacent time points can be obtained first. In this way, the relative transformation relationship of the coordinates of the lane line pixels in the vehicle coordinate system can be calculated based on the pose information of the vehicle at the two adjacent time points.

[0139] Based on this relative transformation relationship, the coordinates (x, y) of the lane line pixel at the previous moment can be transformed to the vehicle coordinate system at the current moment, resulting in the transformed coordinates (x', y') of the lane line pixel. Based on the transformed coordinates (x', y') of the lane line pixel, the distance between the lane line instance at the current moment and the corresponding lane line instance at the previous moment can be further determined.

[0140] In one embodiment of this application, transforming the coordinates of the lane line pixels at the previous moment to the vehicle coordinate system at the current moment according to the relative transformation relationship to obtain the coordinates of the transformed lane line pixels includes: sampling the lane line pixels at the previous moment to obtain the sampled lane line pixels at the previous moment; and transforming the coordinates of the sampled lane line pixels at the previous moment to the vehicle coordinate system at the current moment to obtain the coordinates of the transformed lane line pixels.

[0141] To improve the efficiency of lane line tracking, when transforming the coordinates of lane line pixels from the previous moment to the vehicle coordinate system at the current moment, the lane line pixels from the previous moment can be sampled first. Taking lane line i from the previous moment and lane line j from the current moment as an example, n points are sampled on lane line i. Then, according to the relative transformation relationship in the aforementioned embodiment, the n points obtained after sampling are transformed to the current moment, thus obtaining the transformed set of lane line pixels (i1, i2, i3, ..., in).

[0142] In one embodiment of this application, determining the distance between the current lane line instance and the corresponding lane line instance at the previous moment based on the coordinates of the converted lane line pixels includes: determining the ordinate of the fitted lane line pixel based on the abscissa of the converted lane line pixel and the fitting equation of the current lane line instance; and determining the distance between the current lane line instance and the corresponding lane line instance at the previous moment based on the deviation between the ordinate of the fitted lane line pixel and the ordinate of the converted lane line pixel.

[0143] Based on the transformed horizontal coordinate x' of the lane line pixel, it is input into the lane line fitting equation corresponding to the lane line j at the current time, and the vertical coordinate y' of the fitted lane line pixel can be obtained. The vertical coordinate y' of each lane line pixel is taken as the true y value at the current time, and the mean square error is calculated with the vertical coordinate y' of each fitted lane line pixel, which is used as the final distance metric between the lane line i at the previous time and the lane line j at the current time.

[0144] For ease of understanding of the various embodiments of this application, such as Figure 3The diagram illustrates the overall process of lane detection in this embodiment. First, during the model training phase, a training image is acquired, the Region of Interest (ROI) is cropped from the image, preprocessed, and then input into the lane detection model for training, resulting in a trained lane detection model. Next, during the model inference phase, the detection image is acquired and input into the trained lane detection model to perform lane detection, obtaining the lane line binary semantic segmentation result and the horizontal and vertical feature vectors of the pixels. Then, in the post-processing phase, lane line instance segmentation is performed based on the lane line binary semantic segmentation result and the horizontal and vertical feature vectors of the pixels, resulting in a lane line instance segmentation result. Lane line fitting is then performed based on the segmented lane line instances to obtain the final lane detection result. Finally, in the lane tracking phase, lane line tracking is performed based on the lane detection results of two adjacent time points, resulting in a lane line tracking result.

[0145] This application utilizes the lateral and longitudinal feature vectors of specially designed lane line instances as encoding information to train a lane line detection model. The post-processing stage does not require running complex and time-consuming clustering algorithms, and can quickly obtain the instance segmentation results of lane lines from the lane line semantic segmentation results. It also has a fast lane line tracking function.

[0146] This application also provides a lane line detection model training device 400, such as... Figure 4 As shown, a schematic diagram of a lane line detection model training device according to an embodiment of this application is provided. The device 400 includes: a first acquisition unit 410, an encoding unit 420, and a training unit 430, wherein:

[0147] The first acquisition unit 410 is used to acquire the training image and the annotation information of lane line instances in the training image;

[0148] The encoding unit 420 is used to encode the pixels in the image to be trained according to the annotation information of the lane line instance using a preset feature encoding strategy to obtain the feature encoding information of the pixel, wherein the feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel.

[0149] The training unit 430 is used to train a lane detection model using the feature encoding information of the image to be trained and the pixel points, as well as the annotation information of the lane line instance, to obtain the trained lane detection model.

[0150] In one embodiment of this application, the image to be trained includes multiple pixel rows and multiple pixel columns, and the encoding unit 420 is specifically used to: determine the pixel position of the pixel in the image to be trained according to each pixel row and each pixel column; and determine the horizontal feature vector and vertical feature vector of the pixel according to the annotation information of the lane line pixel and the pixel position of the lane line pixel in the image to be trained.

[0151] In one embodiment of this application, the pixel row includes a current pixel row and a next pixel row, and the pixel point includes a lane line pixel point corresponding to a lane line pixel point. The encoding unit 420 is specifically used for: determining the position of the lane line pixel point in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row based on the annotation information of the lane line pixel point; determining the horizontal feature vector of the lane line pixel point in each current pixel column of the current pixel row based on the position of the lane line pixel point in multiple current pixel columns of the current pixel row; determining the vertical feature vector of the lane line pixel point in each current pixel column of the current pixel row based on the position of the lane line pixel point in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row; and using the horizontal feature vector of the lane line pixel point in each current pixel column of the current pixel row and the vertical feature vector of the lane line pixel point in each current pixel column of the current pixel row as the feature encoding information of the pixel point.

[0152] In one embodiment of this application, the encoding unit 420 is specifically used to: determine the center pixel column of the lane line pixel in the current pixel row based on the position of the lane line pixel in multiple current pixel columns of the current pixel row; and determine the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row based on the center pixel column of the lane line pixel in the current pixel row.

[0153] This application also provides a lane line detection device 500, such as... Figure 5 As shown, a schematic diagram of a lane line detection device according to an embodiment of this application is provided. The device 500 includes: a second acquisition unit 510, a detection unit 520, a clustering unit 530, and a first determination unit 540, wherein:

[0154] The second acquisition unit 510 is used to acquire the image to be detected;

[0155] Detection unit 520 is used to perform lane line detection on the image to be detected using a lane line detection model, and obtain lane line semantic segmentation results and feature encoding information of pixel points;

[0156] Clustering unit 530 is used to cluster lane line pixels in the image to be detected based on the lane line semantic segmentation result and the feature encoding information of the pixel;

[0157] The first determining unit 540 is used to determine the lane line instance segmentation result based on the clustering result, so as to obtain the final lane line detection result based on the lane line instance segmentation result.

[0158] The lane detection model is trained based on the aforementioned lane detection model training device.

[0159] In one embodiment of this application, the feature encoding information of the pixel points includes the feature encoding information of lane line pixels. The clustering unit 530 is specifically used to: determine the current pixel row and current pixel column where the lane line instance in the image to be detected is located based on the lane line semantic segmentation result; determine the lateral feature vector of the lane line pixel points in the current pixel row and the current pixel column based on the feature encoding information of the lane line pixels; and cluster the lane line pixels in the current pixel row based on the lateral feature vector of the lane line pixels in the current pixel row and the current pixel column to obtain multiple lane line pixel clusters in the current pixel row.

[0160] In one embodiment of this application, the first determining unit 540 is specifically used to: determine whether there is a corresponding historical pixel row for the current pixel row; if not, then sequentially identify the multiple lane line pixel clusters of the current pixel row to obtain the lane line instance segmentation result; if there is, then obtain the multiple lane line pixel clusters of the historical pixel row, and match the multiple lane line pixel clusters of the historical pixel row with the multiple lane line pixel clusters of the current pixel row according to the vertical feature vector of the lane line pixel of the historical pixel row and the vertical feature vector of the lane line pixel of the current pixel row to obtain the identifier of each lane line pixel cluster of the current pixel row.

[0161] In one embodiment of this application, the lane detection result is the lane instance detection result at the current moment, and the device further includes: a third acquisition unit, used to acquire the lane instance detection result at the previous moment; a second determination unit, used to determine the distance between the lane instance at the current moment and the corresponding lane instance at the previous moment based on the lane instance segmentation result at the previous moment and the lane instance segmentation result at the current moment; and a tracking unit, used to perform lane tracking based on the distance between the lane instance at the current moment and the corresponding lane instance at the previous moment using a preset lane tracking algorithm.

[0162] In one embodiment of this application, the second determining unit is specifically used to: determine the relative transformation relationship between the coordinates of the lane line pixel at the previous moment and the coordinates of the lane line pixel at the current moment; according to the relative transformation relationship, transform the coordinates of the lane line pixel at the previous moment to the vehicle coordinate system at the current moment to obtain the transformed coordinates of the lane line pixel; and according to the transformed coordinates of the lane line pixel, determine the distance between the lane line instance at the current moment and the corresponding lane line instance at the previous moment.

[0163] In one embodiment of this application, the second determining unit is specifically used to: acquire vehicle pose information at the previous moment and vehicle pose information at the current moment; and determine the relative transformation relationship between the coordinates of the lane line pixels at the previous moment and the coordinates of the lane line pixels at the current moment based on the vehicle pose information at the previous moment and the vehicle pose information at the current moment.

[0164] In one embodiment of this application, the second determining unit is specifically used to: sample the lane line pixels at the previous moment to obtain the sampled lane line pixels at the previous moment; and transform the coordinates of the sampled lane line pixels at the previous moment to the vehicle coordinate system at the current moment to obtain the transformed coordinates of the lane line pixels.

[0165] In one embodiment of this application, the second determining unit is specifically used to: determine the ordinate of the fitted lane line pixel based on the abscissa of the converted lane line pixel and the fitting equation of the lane line instance at the current moment; and determine the distance between the lane line instance at the current moment and the corresponding lane line instance at the previous moment based on the deviation between the ordinate of the fitted lane line pixel and the ordinate of the converted lane line pixel.

[0166] It is understood that the lane detection model training device described above can implement each step of the lane detection model training method provided in the foregoing embodiments. The relevant explanations of the lane detection model training method are applicable to the lane detection model training device and will not be repeated here.

[0167] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also an internal bus, network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its functions.

[0168] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0169] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0170] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a lane detection model training device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0171] Obtain the annotation information of the image to be trained and the lane line instances in the image to be trained;

[0172] Based on the annotation information of the lane line instance, the pixels in the image to be trained are encoded using a preset feature encoding strategy to obtain the feature encoding information of the pixel. The feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel.

[0173] The lane detection model is trained using the feature encoding information of the image to be trained, the feature encoding information of the pixels, and the annotation information of the lane line instances, to obtain the trained lane detection model.

[0174] The above is as stated in this application. Figure 1The method executed by the lane detection model training device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0175] The electronic device can also perform Figure 1 The method for training a lane detection model is described, and the lane detection model training device is implemented in... Figure 1 The functions of the embodiments shown are not described in detail here.

[0176] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the lane detection model training device in the illustrated embodiment is specifically used to perform the following:

[0177] Obtain the annotation information of the image to be trained and the lane line instances in the image to be trained;

[0178] Based on the annotation information of the lane line instance, the pixels in the image to be trained are encoded using a preset feature encoding strategy to obtain the feature encoding information of the pixel. The feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel.

[0179] The lane detection model is trained using the feature encoding information of the image to be trained, the feature encoding information of the pixels, and the annotation information of the lane line instances, to obtain the trained lane detection model.

[0180] It should be noted that the electronic device in this application embodiment can also be used to perform... Figure 2 The method performed by the lane line detection device disclosed in the illustrated embodiment will not be described in detail hereafter.

[0181] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0185] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0186] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0187] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0188] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0189] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for training a lane line detection model, wherein, The method includes: Obtain the annotation information of the image to be trained and the lane line instances in the image to be trained; Based on the annotation information of the lane line instance, the pixels in the image to be trained are encoded using a preset feature encoding strategy to obtain the feature encoding information of the pixel. The feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel. The lane line detection model is trained using the feature encoding information of the image to be trained, the pixel points, and the annotation information of the lane line instances, to obtain the trained lane line detection model. The image to be trained includes multiple pixel rows and multiple pixel columns. The process of encoding the pixels in the image to be trained using a preset feature encoding strategy based on the annotation information of the lane line instances, to obtain the feature encoding information of the pixels, includes: The pixel position of the pixel in the image to be trained is determined according to the method of each pixel row and each pixel column; Based on the annotation information of the lane line instance and the pixel position of the lane line instance in the image to be trained, the horizontal feature vector and vertical feature vector of the pixel are determined. The pixel row includes the current pixel row and the next pixel row, and the pixel point includes the lane line pixel point corresponding to the lane line instance. The step of encoding the pixel points in the image to be trained using a preset feature encoding strategy based on the annotation information of the lane line instance to obtain the feature encoding information of the pixel point includes: When encoding the pixels in the image to be trained, the image to be trained is traversed row by row from bottom to top; For any current pixel row being traversed, based on the annotation information of the lane line instance, determine the position of the lane line pixel in multiple current pixel columns of the current pixel row and the position of the next pixel column of the next pixel row; Based on the position of the lane line pixel in each of the multiple current pixel columns of the current pixel row, determine the lateral feature vector of the lane line pixel in each of the current pixel columns of the current pixel row; Based on the positions of the lane line pixels in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row, determine the vertical feature vectors of the lane line pixels in each current pixel column of the current pixel row. The horizontal feature vectors of the lane line pixels in each current pixel column of the current pixel row and the vertical feature vectors of each current pixel column of the current pixel row are used as the feature encoding information of the lane line pixels.

2. The lane detection model training method as described in claim 1, wherein, The step of determining the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row based on the position of the lane line pixel in each current pixel column of the current pixel row includes: Based on the position of the lane line pixel in the current pixel row among multiple current pixel columns, determine the center pixel column of the lane line pixel in the current pixel row; Using the center pixel column of the lane line pixel in the current pixel row as a reference, determine the lateral feature vector of the lane line pixel in each current pixel column of the current pixel row.

3. A lane line detection method, wherein, The method includes: Acquire the image to be detected; The lane line detection model is used to detect lane lines in the image to be detected, and the lane line semantic segmentation results and the feature encoding information of the pixels are obtained. Based on the lane line semantic segmentation results and the feature encoding information of the pixels, the lane line pixels in the image to be detected are clustered. The lane line instance segmentation results are determined based on the clustering results, and the final lane line detection results are obtained based on the lane line instance segmentation results. The lane detection model is trained based on the lane detection model training method described in any one of claims 1 to 2.

4. The lane line detection method as described in claim 3, wherein, The feature encoding information of the pixels includes the feature encoding information of lane line pixels. The step of clustering lane line pixels in the image to be detected based on the lane line semantic segmentation result and the feature encoding information of the pixels includes: Based on the lane line semantic segmentation results, determine the current pixel row and current pixel column of the lane line pixels in the image to be detected; Based on the feature encoding information of the lane line pixels, determine the horizontal feature vector of the lane line pixels in the current pixel row and the current pixel column; Based on the lateral feature vectors of the lane line pixels in the current pixel row and the current pixel column, the lane line pixels in the current pixel row are clustered to obtain multiple lane line pixel clusters in the current pixel row.

5. The lane line detection method as described in claim 4, wherein, The determination of lane line instance segmentation results based on clustering results includes: Determine whether the current pixel row has a corresponding historical pixel row; If it does not exist, the multiple lane line pixel clusters in the current pixel row are identified sequentially to obtain the lane line instance segmentation result; If they exist, multiple lane line pixel clusters of the historical pixel row are obtained, and the multiple lane line pixel clusters of the historical pixel row are matched with the multiple lane line pixel clusters of the current pixel row according to the vertical feature vectors of the lane line pixels of the historical pixel row and the vertical feature vectors of the lane line pixels of the current pixel row to obtain the identifiers of each lane line pixel cluster of the current pixel row.

6. The lane line detection method as described in claim 3, wherein, The lane detection result is the lane instance detection result at the current time. After determining the lane instance segmentation result based on the clustering result, and obtaining the final lane detection result based on the lane instance segmentation result, the method further includes: Obtain the lane line instance detection results from the previous moment; Based on the lane line instance segmentation result of the previous moment and the lane line instance segmentation result of the current moment, determine the distance between the lane line instance of the current moment and the corresponding lane line instance of the previous moment. The distance between the current lane line instance and the corresponding lane line instance from the previous moment is used to facilitate lane line tracking by a preset lane line tracking algorithm.

7. A lane line detection model training device, wherein, The device includes: The first acquisition unit is used to acquire the training image and the annotation information of lane line instances in the training image; The encoding unit is used to encode the pixels in the image to be trained according to the annotation information of the lane line instance using a preset feature encoding strategy to obtain the feature encoding information of the pixel, wherein the feature encoding information of the pixel includes the horizontal feature vector and the vertical feature vector of the pixel. The training unit is used to train the lane detection model using the feature encoding information of the image to be trained and the pixel points, as well as the annotation information of the lane line instance, to obtain the trained lane detection model. The image to be trained includes multiple pixel rows and multiple pixel columns, and the encoding unit is specifically used for: The pixel position of the pixel in the image to be trained is determined according to the method of each pixel row and each pixel column; Based on the annotation information of the lane line instance and the pixel position of the lane line instance in the image to be trained, the horizontal feature vector and vertical feature vector of the pixel are determined. The pixel row includes the current pixel row and the next pixel row, the pixel point includes the lane line pixel point corresponding to the lane line instance, and the encoding unit is specifically used for: When encoding the pixels in the image to be trained, the image to be trained is traversed row by row from bottom to top; For any current pixel row being traversed, based on the annotation information of the lane line instance, determine the position of the lane line pixel in multiple current pixel columns of the current pixel row and the position of the next pixel column of the next pixel row; Based on the position of the lane line pixel in each of the multiple current pixel columns of the current pixel row, determine the lateral feature vector of the lane line pixel in each of the current pixel columns of the current pixel row; Based on the positions of the lane line pixels in multiple current pixel columns of the current pixel row and multiple next pixel columns of the next pixel row, determine the vertical feature vectors of the lane line pixels in each current pixel column of the current pixel row. The horizontal feature vectors of the lane line pixels in each current pixel column of the current pixel row and the vertical feature vectors of each current pixel column of the current pixel row are used as the feature encoding information of the lane line pixels.

8. A lane line detection device, wherein, The device includes: The second acquisition unit is used to acquire the image to be detected; The detection unit is used to perform lane line detection on the image to be detected using a lane line detection model, and obtain lane line semantic segmentation results and feature encoding information of pixel points. A clustering unit is used to cluster lane line pixels in the image to be detected based on the lane line semantic segmentation result and the feature encoding information of the pixel. The first determining unit is used to determine the lane line instance segmentation result based on the clustering result, so as to obtain the final lane line detection result based on the lane line instance segmentation result. The lane detection model is trained based on the lane detection model training device described in claim 7.

Images