Lane line detection method and vehicle

Abstract

The present application belongs to the technical field of intelligent vehicles, and particularly relates to a lane line detection method and a vehicle. It aims to solve the problem of low efficiency, poor real-time performance and poor stability in complex environments caused by the need for complex post-processing to distinguish different lane instances in traditional lane line prediction methods. By inputting a road image into a lane line detection model, a fusion feature map is generated through a feature extraction network and a feature fusion network in the model, and a parallel multiple lane branch is used for instantiation analysis. After decoding the output of each branch, the detection result of the lane instance is obtained. It decomposes the complex multi-instance detection problem into multiple parallel single-instance prediction problems, directly outputs the instance-specific and structured detection result, and facilitates the direct use of the downstream system. It avoids the time-consuming problem caused by traditional post-processing clustering analysis, and decouples the prediction of the same instance at the feature level, effectively avoiding feature competition and confusion, and improving the detection stability in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle technology, specifically relating to a lane line detection method and vehicle. Background Technology

[0002] Lane detection is one of the most critical perception modules in autonomous driving and advanced driver assistance systems (ADAS). From basic functions like adaptive cruise control (ACC) and lane keeping assist (LCC) in Level 2 systems, to Level 3 systems allowing drivers to drive hands-free and eyes-free under specific conditions, and even future Level 4 systems achieving fully autonomous driving within preset scenarios, their realization heavily relies on the ability to accurately and robustly identify lane boundaries. With technological advancements, deep learning-based methods have become the mainstream approach in this field due to their superior performance and cost-effectiveness.

[0003] While deep learning-based lane detection methods perform well in most scenarios, they still face significant challenges when dealing with complex topologies (such as dense, intersecting, and branching lanes). Existing methods generally suffer from difficulties in instance differentiation, complex topological reasoning, and heavy post-processing computations. Specifically: 1) Methods based on row / column anchor point estimation (such as patent CN119152465 B) are prone to feature confusion in the far end of the image due to downsampling leading to competition for the same anchor point between adjacent lanes; 2) Methods based on start point and offset prediction (such as patent CN 120411911 A) may lead to missed lane detections in dual-lane scenarios due to overlapping start points; 3) Methods relying on edge detection (such as patent CN 120198880 B) still require clustering of edge data to match lane instances, increasing the computational burden; 4) Methods based on keypoint encoding and clustering (such as patent CN120298993 A) suffer from missed detections due to co-location of multiple lines, and their clustering operation itself requires more computational resources on embedded devices, easily leading to resource bottlenecks. The methods described above generally rely on complex post-processing steps to distinguish lane instances, resulting in high computational latency, low efficiency, and high resource consumption, and their stability is poor in complex scenarios.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] One objective of this invention is to address the problems of low efficiency, poor real-time performance, and poor stability in complex environments caused by the need for complex post-processing in traditional lane prediction methods to distinguish different lane instances.

[0006] To achieve the above objectives, the present invention provides a lane line detection method, comprising:

[0007] Acquire the image of the road to be detected;

[0008] The road image is input into a pre-trained lane detection model;

[0009] The road image is processed by the lane line detection model to obtain the detection results of multiple lane instances in the road image;

[0010] The lane detection model includes a feature extraction network, a feature fusion network, and multiple lane branches connected in sequence. The step of processing the road image using the lane detection model to obtain detection results for multiple lane instances in the road image includes: processing the input image through the feature extraction network and the feature fusion network to obtain a fused feature map; inputting the fused feature map to each lane branch, with each lane branch independently outputting instantiation prediction information for a corresponding lane instance; and determining the detection result for each lane instance based on the instantiation prediction information output by each lane branch.

[0011] Further, the step of determining the detection result of each lane instance based on the instantiation prediction information output by each lane branch includes: decoding multiple key point positions from the instantiation prediction information output by each lane branch; for each key point position obtained in each lane branch, retrieving the geometric offset and attribute information associated with the key point position from the instantiation prediction information; determining the lane edge position corresponding to the key point based on the key point position and the retrieved geometric offset; and retrieving the attribute information associated with the edge position from the instantiation prediction information based on the determined lane edge position and the attribute information.

[0012] Furthermore, the step of determining the detection result of each lane instance based on the instantiation prediction information output by each lane branch further includes: generating a boundary geometric representation of each lane instance based on the edge positions corresponding to all lane foreground points; performing fusion analysis based on the lane attribute prediction information corresponding to all lane foreground points to obtain the final lane attribute of each lane instance; performing fusion analysis based on the edge attribute prediction information corresponding to all lane foreground points to obtain the final edge attribute of each lane instance; and / or, the step of generating a boundary geometric representation of each lane instance based on the edge positions corresponding to all lane foreground points includes: sorting all corresponding edge positions according to the coordinates corresponding to the row of the image where they are located to form an ordered sequence of boundary points, and using the ordered boundary point representation as the geometric representation of the boundary.

[0013] Further, the instantiation prediction information includes a lane heatmap, a boundary offset map, and an attribute prediction map; the step of each lane branch independently outputting instantiation prediction information for a corresponding lane instance includes: each lane branch includes three parallel and independent prediction sub-networks, the three prediction sub-networks being used to generate the lane heatmap, the boundary offset map, and the attribute prediction map, respectively; and / or, the step of determining the detection result of each lane instance based on the instantiation prediction information output by each lane branch further includes: decoding multiple key point locations based on the lane heatmap output by each lane branch; wherein, the key point locations include lane foreground points and their coordinates in the feature map space; according to the The coordinates of the lane foreground point in the feature map space are used to index the corresponding left and right offsets from the boundary offset map output by the corresponding lane branch. Based on the horizontal coordinates of the lane foreground point in the feature map space, the left and right offsets are calculated to obtain the left and right lane edge positions corresponding to each lane foreground point. According to the coordinates of each lane foreground point in the feature map space, the corresponding lane attribute prediction information is indexed from the attribute prediction map output by the corresponding lane branch. Based on the calculated left and right lane edge positions, the corresponding left and right edge attribute prediction information is indexed from the attribute prediction map output by the corresponding lane branch.

[0014] Furthermore, the step of determining multiple lane foreground points and their coordinates in the feature map space based on the lane heatmap output by each lane branch includes: determining the pixel position with the highest built-in confidence in each row of the lane heatmap; in response to the confidence of the pixel position being greater than a preset threshold, determining it as the lane foreground point and recording its coordinates; and / or, each lane branch includes multiple predictive subnetworks that are structurally identical and parameter-independent, each predictive subnetwork being used to generate the lane heatmap, the boundary offset map, and the attribute prediction map, respectively.

[0015] Furthermore, the method further includes: acquiring a training sample set, the training sample set containing labeled sample images and lane instances; generating supervision labels for the lane instances corresponding to the sample images; the supervision labels include: multiple supervision point positions defined in the feature map space and associated with the lane instance, and a dynamic supervision quantity assigned to each supervision point position; the dynamic supervision quantity characterizes the geometric relationship between the corresponding supervision point and the edge of the lane instance in the image space; training an initial lane line detection model using the training samples as input and the constructed supervision labels as the learning target; wherein, the training employs multi-task loss supervision; the multi-task loss function includes a lane region loss for supervising the lane heatmap prediction, a margin regression loss for supervising the boundary offset prediction, and an attribute classification loss for supervising the attribute prediction map prediction.

[0016] Further, the step of generating supervised labels for the lane instances corresponding to the sample images includes: for the p-th target lane instance, p=1,2,...,P; calculating the lane centerline based on the left and right edge annotations of the p-th target lane instance, and determining Q foreground encoding points in the feature map space based on the lane centerline, where Q is a positive integer; for each of the Q foreground encoding points, calculating the geometric distance from its inverse mapping to the original image coordinates to the left and right edges of the p-th target lane instance, as the dynamic margin ground truth of the foreground encoding point; and / or, the multi-task loss function satisfies at least one of the following: the lane region loss is calculated using the focus loss function; the attribute classification loss is calculated using the cross-entropy loss function; the margin regression loss is calculated using the distributed focus loss function.

[0017] Further, the step of determining Q foreground coding points in the feature map space based on the lane centerline includes: mapping the lane centerline to the feature map space to obtain at least one centerline coding point; expanding horizontally in the feature map space starting from the centerline coding point, and determining the grid coordinates covered by the expansion as additional coding points; the centerline coding point and the additional coding points together constitute Q foreground coding points; and / or, the step of expanding horizontally in the feature map space starting from the centerline coding point includes: for each centerline coding point, determining the number of grids to expand to the left and right sides based on its distance to the left and right edges of the p-th target lane instance; expanding according to the number of grids, and the expansion range does not exceed the left and right edges of the p-th target lane instance.

[0018] Further, the step of calculating the geometric distance from each of the Q foreground encoding points to the left and right edges of the p-th target lane instance after inverse mapping to the original image coordinates includes: if the geometric distance from a foreground encoding point to an edge of the p-th target lane instance cannot be calculated or is invalid, then assigning a predetermined invalidation flag to the true value of the dynamic margin of the edge corresponding to the foreground encoding point; ignoring the item with the invalidation flag when calculating the margin regression loss during training; and / or further includes: when the ordinate obtained by inverse mapping is not an integer, interpolating based on the adjacent annotation points above and below the ordinate to obtain the edge position at the ordinate, and then calculating the geometric distance.

[0019] In other embodiments, a vehicle is provided that can perform the lane detection method described in any of the above descriptions.

[0020] Based on the foregoing description, those skilled in the art will understand that this invention acquires road images and inputs them into a lane detection model. The model then generates a fused feature map through a feature extraction network and a feature fusion network. This map is then analyzed through multiple parallel lane branches, and the output of each branch is decoded to obtain the detection results for multiple lane instances. This invention decomposes the complex multi-instance detection problem into multiple parallel, relatively simple single-instance prediction problems, simplifying the model's learning objective and output structure. It directly outputs segmented, structured detection results, facilitating direct use by downstream systems (such as vehicle control). This avoids the time-consuming problem of traditional post-processing clustering analysis. Furthermore, the prediction of the same instance is decoupled at the feature level, avoiding feature competition and confusion, thus improving the detection stability and timeliness in complex scenarios (such as lane intersections and occlusions). Attached Figure Description

[0021] The accompanying drawings, as part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation of the invention. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0022] Figure 1 This is a flowchart of a lane line detection method in some embodiments of the present invention;

[0023] Figure 2 This is a schematic diagram of the lane line detection model architecture in some embodiments of the present invention. Attached Figure Description

[0024] 001. Lane line detection model; 1. Feature extraction network; 2. Feature fusion network; 31. Lane branch; 32. Left lane branch; 33. Right lane branch; 34. Leftmost lane branch; 35. Rightmost lane branch. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0026] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0027] The following reference Figure 1 and Figure 2 The following will provide a detailed description of the lane line detection method in some embodiments of the present invention. Figure 1 This is a flowchart of a lane line detection method in some embodiments of the present invention; Figure 2 This is a schematic diagram of the lane line detection model architecture in some embodiments of the present invention.

[0028] In some embodiments of this invention, a lane detection method is provided to address the problems of low efficiency and high error rate caused by the need for complex post-processing (such as clustering) in traditional dense prediction methods to distinguish different lane instances. By acquiring road images and inputting them into a lane detection model, the method extracts multi-scale basic features of the image through a feature extraction network, fuses deep and shallow features through a feature fusion network to generate semantically rich and detailed fused feature maps, and implements parallel lane branches. The output of each branch is decoded to obtain the detection results of multiple lane instances. This decomposes the complex multi-instance detection problem into multiple parallel, relatively simple single-instance prediction problems, simplifying the model's learning objectives and output structure. It avoids time-consuming post-processing clustering, and the inference process is purely forward computation, resulting in high speed. Simultaneously, the prediction of the same instance is decoupled at the feature level, avoiding feature competition and confusion, and improving detection stability in complex scenarios (such as lane intersections and occlusions). Furthermore, it directly outputs segmented, structured detection results, facilitating direct use by downstream systems (such as vehicle control).

[0029] The lane line detection method of the present invention is applicable to any scenario that requires real-time and accurate lane perception, such as L2+ level autonomous driving (such as lane keeping, automatic lane changing), advanced driver assistance systems, high-precision map crowdsourcing updates and autonomous driving simulation testing.

[0030] like Figure 1 As shown, the lane line detection method includes at least:

[0031] Step 100: Obtain the road image to be detected. The road image mentioned here can be a road image directly uploaded by the user, or a road image uploaded by a device (such as a road image detected in real time by a detection device in a vehicle).

[0032] Step 200: Input the road image into the pre-trained lane detection model 001.

[0033] Before inputting the road image into the lane detection model 001, the image can be preprocessed. This preprocessing may include: adjusting the image size to a preset input size for the model; and normalizing the image pixel values. Those skilled in the art can use the above-described processing methods or other image preprocessing methods as needed, which will not be elaborated upon here.

[0034] Step 300: Process the road image using lane line detection model 001 to obtain the detection results of multiple lane instances in the road image.

[0035] like Figure 2 As shown, the lane detection model 001 includes a feature extraction network 1, a feature fusion network 2, and multiple lane branches set in parallel, which are connected in sequence.

[0036] Feature extraction network 1 is configured to perform multi-scale feature extraction on the input image and output multi-scale features containing at least one low-level feature map and at least one high-level feature map. The low-level feature map has high spatial resolution, while the high-level feature map has high semantic information content.

[0037] Feature fusion network 2 is connected to feature extraction network 1. Feature fusion network 2 is configured to perform cross-scale fusion of multi-scale features and output a fused feature map. The fused feature map contains both spatial detail information from the low-level feature map and semantic information from the high-level feature map.

[0038] Multiple lane branches are connected to Feature Fusion Network 2. These lane branches are configured to share a fused feature map as input and, based on the fused feature map, independently generate prediction information corresponding to a lane instance. The multiple lane branches are structurally identical, and the model parameters used to generate prediction information are independent of each other.

[0039] The multiple lane branches of this invention include at least a left lane branch 32, a secondary lane branch 31, and a right lane branch 33. The secondary lane branch 31 is a model branch used to detect the current lane of the vehicle. It is mainly responsible for identifying and tracking the lane the vehicle is traveling in, and serves as the reference for lane keeping, centering control, and determining whether the vehicle is deviating from its lane. The left lane branch 32 and the right lane branch 33 are respectively responsible for detecting the adjacent lanes to the left and right of the vehicle.

[0040] It should be noted that the parallel lane branching architecture of this invention is scalable. In multi-lane roads (e.g., four-lane, five-lane, and above), it can detect lanes beyond the current lane, the left adjacent lane, the right adjacent lane, and even further lanes, i.e., the number of parallel branches increases according to the number of lanes. For example, in a system that needs to detect five lanes, five parallel branches can be configured, corresponding to the leftmost lane branch 34, the left lane branch 32, the right lane branch 31, the right lane branch 33, and the rightmost lane branch 35, respectively. The working principle of each newly added branch is exactly the same as the aforementioned three branches, independently responsible for predicting the complete instantiation information of its corresponding lane. All branches share the output of the same feature fusion network 2, and learn the features of their respective target lanes through independent parameters, thereby achieving synchronous and decoupled detection of multiple lanes.

[0041] In some embodiments, step S300 includes steps S310 to S330, as follows:

[0042] Step S310: The input image is processed by feature extraction network 1 and feature fusion network 2 to obtain a fused feature map.

[0043] A single frame of input image is acquired and resized to the model input size. The image is then fed into the feature extraction module for feature extraction, yielding feature maps at different downsampling sizes. The feature extraction network can be a classic CNN feature extraction network, such as VGG, ResNet, Inception, or MobileNet; or an attention-based Transformer feature extraction network, such as Swin-Transformer or Vision-Transformer (ViT).

[0044] The extracted features of various sizes are fed into Feature Fusion Network 2 to fuse low-level and high-dimensional features. Generally, we consider low-level features to focus more on the texture details of the image, while high-level features focus more on the semantic information of the image. Feature Fusion Network 2 enables information interaction between low-level and high-level features, which helps increase the information content and feature representation capability of the output feature map. Feature Fusion Network 2 can use any of the following: Feature Pyramid Network (FPN), Path Aggregation Network (PANet), and Weighted Bidirectional Feature Pyramid Network (BiFPN).

[0045] Step S320: The fused feature map is input to each lane branch. Each lane branch is configured to independently output instantiation prediction information of a corresponding lane instance based on the input fused feature map.

[0046] The instantiated prediction information includes lane heatmaps, boundary offset maps, and attribute prediction maps.

[0047] Each lane branch includes three predictive subnetworks that are structurally identical and parametrically independent. These three predictive subnetworks are used to generate lane heatmaps, boundary offset maps, and attribute prediction maps, respectively.

[0048] For each lane branch, steps S321 to S324 are executed as follows:

[0049] Step S321: The fused feature map is input in parallel to multiple prediction sub-networks. Each lane branch includes multiple prediction sub-networks with independent and parallel structural parameters, and the model parameters corresponding to the multiple prediction sub-networks are independent among the multiple lane branches. In this invention, each lane branch has three prediction sub-networks: a first prediction sub-network, a second prediction sub-network, and a third prediction sub-network, which are used to predict the lane heatmap, the boundary offset map, and the attribute prediction map, respectively.

[0050] Step S322: The fused feature map is processed by the first prediction sub-network and passed through the first nonlinear activation function to generate a lane heatmap representing the confidence of each position belonging to the corresponding lane instance; the first nonlinear activation function is the Sigmoid function, which makes the value of each position in the lane heatmap between 0 and 1.

[0051] The lane heatmap is a single-channel feature map, such as [1, H, W], with the same size as the feature map output by the model. The value of each pixel in the map represents the confidence level of the model's prediction that the location belongs to the lane instance of the current branch, reflecting the approximate location of the left, right, or main lane in the image. High confidence levels can delineate the approximate region of the lane in the image. From the lane heatmap, a series of sparse, high-confidence lane foreground point coordinates (x, y) can be extracted.

[0052] Step S323: The fused feature map is processed by the second prediction sub-network to generate a boundary offset map representing the geometric offset of each position to the edge of the corresponding lane instance; the boundary offset map generated by the second prediction sub-network contains two channels, corresponding to the left boundary offset and the right boundary offset, respectively.

[0053] The boundary offset map represents the geometric offset of each location in the feature map space from the left and right edges of the corresponding lane instance. The boundary offset map is a two-channel feature map, such as [2, H, W]. Channel 0 represents the horizontal offset from that location to the left edge of the lane (usually negative or zero); channel 1 represents the horizontal offset from that location to the right edge of the lane (usually positive or zero). After obtaining a foreground point P(x, y) from the heatmap, the boundary offset map is indexed using the coordinates of the foreground point P(x, y), resulting in a pair of offsets. left offset right The precise left and right edge coordinates of the row can be calculated using simple vector addition: Left edge x-coordinate: x 左 = x + offset left ; right side along the x-coordinate 右 = x + offset right This transforms the rough lane area positioning in the lane heatmap into precise lane boundary positioning.

[0054] Step S324: The fused feature map is processed by a third prediction sub-network and then passed through a second nonlinear activation function to generate an attribute prediction map representing the attribute information corresponding to each position. The second nonlinear activation function is a Softmax function, which is applied to the output of the third prediction sub-network in groups according to attribute categories to generate the attribute prediction map, which is a multi-channel feature map.

[0055] The attribute prediction map represents the lane attributes and edge attributes corresponding to each location in the feature map space. The attribute prediction map is a multi-channel feature map (e.g., [C, H, W]). The number of channels C equals the sum of the total number of categories of all lane and edge attributes to be predicted. Lane attributes include, but are not limited to, main / auxiliary road, whether it travels in the same direction as the vehicle, whether it is parallel to the vehicle's plane, and whether it is a straight road. Edge attributes include, but are not limited to, lane type, color, whether it is a single line, and lane quality. Logically, each channel can be divided into two parts: the lane attribute part, corresponding to the global attributes of the lane instance itself, such as {main road, auxiliary road}, {same direction, opposite direction}, {straight road, curve}, etc.; and the edge attribute part, corresponding to the physical attributes of the lane edge lines, such as {solid line, dashed line, double solid line}, {white, yellow}, {single line, double line}, {clear, blurry}, etc. During the decoding process, the corresponding lane attribute can be retrieved by indexing the foreground point coordinates (x, y) obtained from the lane heatmap onto the attribute prediction map; the precise edge coordinates (x, y) calculated from the offset map are then used. 左 , y) and (x 右 The attribute prediction graph is indexed to retrieve the corresponding edge attributes, i.e., the attributes of the boundary line itself. Since a lane instance has multiple foreground points, multiple attribute prediction vectors will be retrieved. The decoder will fuse these vectors (e.g., by averaging, weighted fusion, or voting) to finally obtain the final attribute judgment representing the entire lane instance.

[0056] Step S330: Based on the instantiation prediction information output by each lane branch, determine the detection result of each lane instance.

[0057] In some specific embodiments, step S330 includes steps S331 to S335, as follows:

[0058] Step S331: Decode multiple key point locations from the instantiated prediction information output by each lane branch. Specifically, based on the lane heatmap output by each lane branch, decode multiple key point locations, including lane foreground points and their coordinates in the feature map space.

[0059] The steps for decoding multiple keypoint locations based on the lane heatmap output from each lane branch include: determining the pixel location with the highest confidence in each row of the lane heatmap; and, in response to a pixel location's confidence exceeding a preset threshold, identifying it as a lane foreground point and recording its coordinates. Specifically, for each row of the lane heatmap, the location with the highest lane foreground confidence is selected from the multiple column locations contained in that row as a candidate keypoint location for that row. For each candidate keypoint location, it is determined whether its corresponding lane foreground confidence is greater than or equal to a preset threshold; if so, the candidate keypoint location is determined as the final keypoint location. The coordinates of all high-confidence keypoint locations in the lane heatmap are recorded to form a keypoint coordinate list. The coordinates in the keypoint coordinate list are arranged in the order of their corresponding image row coordinates for subsequent steps to obtain the foreground point coordinates. The preset threshold is set according to the operator's actual needs and is not limited here.

[0060] Step S332: For each key point location obtained in each lane branch, retrieve the geometric offset and attribute information associated with the key point location from the instantiated prediction information; determine the lane edge location corresponding to the key point based on the key point location and the retrieved geometric offset; and retrieve the attribute information associated with the edge location from the instantiated prediction information based on the determined lane edge location and attribute information.

[0061] Specifically, based on the coordinates of the lane foreground point in the feature map space, the corresponding left and right offsets are indexed from the boundary offset map output by the corresponding lane branch. Based on the horizontal coordinates, left and right offsets of the lane foreground point in the feature map space, the left and right lane edge positions corresponding to each lane foreground point are calculated. Based on the coordinates of each lane foreground point in the feature map space, the corresponding lane attribute prediction information is indexed from the attribute prediction map output by the corresponding lane branch. Based on the calculated left and right lane edge positions, the corresponding left and right edge attribute prediction information is indexed from the attribute prediction map output by the corresponding lane branch, respectively.

[0062] The step of determining the detection result of each lane instance based on the instantiated prediction information output by each lane branch also includes:

[0063] Step S333: Based on the edge positions corresponding to the points of view for all lanes, namely the left edge position and the right edge position, generate the boundary geometric representation of each lane instance, namely the left boundary geometric representation and the right boundary geometric representation.

[0064] Specifically, all corresponding edge positions are sorted according to the coordinates of the row in which they are located in the image, forming an ordered sequence of boundary points, which is then used as the geometric representation of the boundary. That is, all points in the left edge position set are sorted according to their corresponding image ordinates to form an ordered sequence of left boundary points, which serves as the geometric representation of the left boundary; all points in the right edge position set are sorted according to their corresponding image ordinates to form an ordered sequence of right boundary points, which serves as the geometric representation of the right boundary.

[0065] In other embodiments, since the sorted point sequence may be sparse, non-uniform, or even incomplete, a dense, uniform, and complete lane line can be generated through interpolation. Therefore, after the step of forming an ordered point sequence at the boundary location, the method further includes interpolating the ordered point sequence to generate a continuous boundary geometric representation on a preset ordinate sequence. The interpolation method can be linear interpolation or curve fitting interpolation.

[0066] Step S334: Based on the lane attribute prediction information corresponding to all lane foreground points, a fusion analysis is performed to obtain the final lane attribute of each lane instance. Specifically, the lane attribute prediction information corresponding to all lane foreground points is aggregated to obtain an aggregated lane attribute vector; based on the aggregated lane attribute vector, the final attribute category of each type of lane attribute is determined, that is, for the sub-vectors representing the same type of lane attribute in the aggregated lane attribute vector, the category corresponding to its maximum value is determined as the final attribute category of that type of lane attribute; before determining the final attribute category, it is determined whether the maximum value of the sub-vector is greater than a preset confidence threshold; if not, the final attribute category is determined to be unknown.

[0067] Furthermore, the aggregation step may involve calculating the average or weighted average of the lane attribute prediction information, and the weight of the weighted average may be determined based on the confidence level of the corresponding lane foreground point in the lane heatmap.

[0068] The lane attributes include at least one of the following: main road or auxiliary road, same or opposite direction as the vehicle, straight or curved.

[0069] Specifically, there are M lanes leading to each attraction, and each attraction is P. i Each corresponds to a lane attribute prediction vector v i v i It is a C-dimensional vector, where C is the total number of lane attribute categories (for example, if the attributes are {main road, auxiliary road} and {straight road, curve}, then C=4, the first two dimensions correspond to road type, and the last two dimensions correspond to lane type). Vector v i From the attribute prediction map, via the foreground point coordinates (x i , y iThe index is obtained and has been processed by Softmax, etc. It can be regarded as the confidence distribution or probability value of the point on each attribute, and finally a final lane instance attribute label is obtained, such as {Road type: Main road, Line type: Curve}.

[0070] Step S335 involves performing a fusion analysis based on the edge attribute prediction information corresponding to all lane foreground points to obtain the final edge attribute for each lane instance. Specifically, the left and right edge attribute prediction information corresponding to all lane foreground points are aggregated to obtain a left edge aggregated attribute vector and a right edge aggregated attribute vector. The attribute category with the highest probability value is determined for each of the left and right edge aggregated attribute vectors, and used as the final left edge attribute and final right edge attribute. The attribute category with the highest probability value can be determined by calculating the average or weighted average of the same edge attribute prediction information. The edge attribute categories include, but are not limited to, lane type, color, whether it is a single lane, and lane quality.

[0071] Specifically, input M points in front of the lanes, each point corresponding to two edges, and the left edge attribute prediction information set, such as Attr left =[v left1 ,v left2 ,...,v leftM ]. Among them, each v lefti It is a vector representing the model's attribute predictions for the left edge of the row containing the i-th foreground point. For example, if the probability distribution for {solid line, dashed line} is [0.9, 0.1], then the line type in the attribute information is solid line. The right edge attribute prediction information set, such as Attr... right =[v right1 , v right2 , ...,v rightM The meaning is the same as above, corresponding to the right edge. After aggregation, the final left edge attribute label is obtained, such as {type: solid line, color: white} and the final right edge attribute label.

[0072] In some embodiments of the present invention, the method further includes: step S400, the process of training the lane line detection model 001. Step S400 includes:

[0073] Step S410: Obtain the training sample set, which includes labeled sample images and lane instances.

[0074] Step S420: For the lane instance corresponding to the sample image, generate a supervision label; the supervision label includes: multiple supervision point locations defined in the feature map space and associated with the lane instance, and a dynamic supervision quantity assigned to each supervision point location; the dynamic supervision quantity characterizes the geometric relationship between the point and the edge of the lane instance in the image space.

[0075] Specifically, step S420 includes: generating supervisory labels for lane instances corresponding to sample images, including: for the p-th target lane instance, p=1,2,...,P; calculating the lane centerline based on the left and right edge annotations of the p-th target lane instance, and determining Q foreground encoding points in the feature map space based on the lane centerline, where Q is a positive integer; for each of the Q foreground encoding points, calculating the geometric distance from its inverse mapping to the original image coordinates to the left and right edges of the p-th target lane instance, as the dynamic margin ground value of the foreground encoding point. If the geometric distance from a foreground encoding point to a certain edge of the p-th target lane instance cannot be calculated or is invalid, then a predetermined invalid label is assigned to the dynamic margin ground value of the edge corresponding to the foreground encoding point; during training, the terms with invalid labels are ignored when calculating the margin regression loss.

[0076] The step of determining Q foreground coding points in the feature map space based on the lane centerline includes: mapping the lane centerline to the feature map space to obtain at least one centerline coding point; expanding horizontally in the feature map space starting from the centerline coding point, and determining the grid coordinates covered by the expansion as additional coding points; the centerline coding point and the additional coding points together constitute Q foreground coding points.

[0077] Furthermore, the step of expanding horizontally in the feature map space starting from the centerline coding point includes: for each centerline coding point, determining the number of grids to expand to the left and right sides based on its distance to the left and right edges of the p-th target lane instance; expanding according to the number of grids, and the expansion range does not exceed the left and right edges of the p-th target lane instance.

[0078] For a midline encoding point in the feature map space, the total number N of grids extending to its left and right sides satisfies: , where d left and d right These represent the distances from the centerline encoding point to the left and right edges of the current lane instance in the feature map space, respectively, and α is a preset expansion coefficient. This indicates rounding down, and the expansion range is constrained to the left and right edges of the current lane instance. The true value of the dynamic margin corresponding to each expanded grid point is calculated based on the actual geometric distance from the grid point's own inverse mapping coordinates to the left and right edges of the current lane instance.

[0079] The number of expanded grids is dynamically determined based on the distance from the centerline position to the left and right edges of the current lane instance.

[0080] In some specific embodiments, for each of the Q foreground encoded points, the step of calculating the geometric distance from its inverse-mapping to the left and right edges of the p-th target lane instance, after being mapped back to the original image coordinates, includes: when the inverse-mapping obtained ordinate is not an integer, interpolation is performed based on the adjacent annotation points above and below the ordinate to obtain the edge position at the ordinate, and then the geometric distance is calculated. Specifically, in the edge annotation sequence of the lane instance, the point corresponding to the largest integer ordinate less than or equal to the inverse-mapping ordinate is located as the first reference point, and the point corresponding to the smallest integer ordinate greater than or equal to the inverse-mapping ordinate is located as the second reference point; based on the coordinates of the first reference point and the second reference point, the edge abscissa at the inverse-mapping ordinate is calculated; based on the abscissa of the inverse-mapping coordinate and the calculated edge abscissa, the geometric distance is determined.

[0081] In this embodiment, the step of calculating the edge x-coordinate at the inverse mapping ordinate includes:

[0082] Calculate the interpolation scaling factor using the formula: t = (y' - y low ) / ( y high - y low ), where y' is the inverse mapping ordinate, (x low , y low (x) represents the coordinates of the first reference point. high , y high () represents the coordinates of the second reference point; the x-coordinate of the edge at the ordinate y' is calculated based on the interpolation scaling factor, using the formula: x y' =x low + t* ( x high - x low The edge is either the left edge or the right edge. Linear interpolation is performed on the left edge and the right edge respectively to obtain the left geometric distance and the right geometric distance.

[0083] Step S430: Using training samples as input and the constructed supervised labels as learning targets, train the initial lane detection model 001; wherein, the training adopts multi-task loss supervision; the multi-task loss function includes lane region loss for supervising lane heatmap prediction, margin regression loss for supervising boundary offset prediction, and attribute classification loss for supervising attribute prediction map prediction.

[0084] Among them, the multi-task loss function satisfies at least one of the following: lane area loss is calculated using the focus loss function; attribute classification loss is calculated using the cross-entropy loss function; and margin regression loss is calculated using the distribution focus loss function.

[0085] Specifically, the distributed focal loss function calculates the margin regression loss as follows: mapping the continuous dynamic margin true value y to the discrete interval [0, R], and determining two discrete values a and b adjacent to the left and right of y in the interval, satisfying a ≤ y < b; constructing a discrete probability distribution supervision signal Q, where Q(a) = b - y, Q(b) = y - a, and for i ∉ {a, b}, Q(i) = 0; calculating the loss value between the dynamic margin probability distribution Z predicted by the model and the supervision signal Q.

[0086] In other embodiments of the present invention, a vehicle is provided that can perform the lane line detection method described in any one of the above.

[0087] In other embodiments of the present invention, a computer is further provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the related steps of the lane line detection method described above are implemented and applied to the vehicle described above.

[0088] In other embodiments of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by the processor, the related steps of the lane line detection method described above are implemented and applied to the vehicle described above.

[0089] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, prediction model, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0091] Those skilled in the art will understand that this invention acquires road images and inputs them into a lane detection model 001. The model then generates a fused feature map via a feature extraction network 1 and a feature fusion network 2. This map is then analyzed through multiple parallel lane branches. The output of each branch is decoded to obtain the detection results for multiple lane instances. This invention decomposes the complex multi-instance detection problem into multiple parallel, relatively simple single-instance prediction problems, simplifying the model's learning objective and output structure. It directly outputs segmented, structured detection results, facilitating direct use by downstream systems (such as vehicle control). This avoids the time-consuming problem of traditional post-processing clustering analysis. Furthermore, the prediction of the same instance is decoupled at the feature level, avoiding feature competition and confusion, thus improving detection stability and timeliness in complex scenarios (such as lane intersections and occlusions).

[0092] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0093] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0094] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-described technical content to create equivalent embodiments without departing from the scope of the present invention. The implementation schemes in the above embodiments can be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A lane line detection method, characterized in that, include: Acquire the image of the road to be detected; The road image is input into a pre-trained lane detection model; The road image is processed by the lane line detection model to obtain the detection results of multiple lane instances in the road image; The lane detection model includes a feature extraction network, a feature fusion network, and multiple lane branches connected in sequence. The step of processing the road image using the lane detection model to obtain detection results for multiple lane instances in the road image includes: processing the input image through the feature extraction network and the feature fusion network to obtain a fused feature map; inputting the fused feature map to each lane branch, with each lane branch independently outputting instantiation prediction information for a corresponding lane instance; and determining the detection result for each lane instance based on the instantiation prediction information output by each lane branch.

2. The method according to claim 1, characterized in that, The step of determining the detection result of each lane instance based on the instantiated prediction information output by each lane branch includes: Multiple key point locations are decoded from the instantiated prediction information output from each lane branch; For each key point location obtained in each lane branch, the geometric offset and attribute information associated with the key point location are retrieved from the instantiated prediction information; based on the key point location and the retrieved geometric offset, the lane edge location corresponding to the key point is determined; based on the determined lane edge location and the attribute information, the attribute information associated with the edge location is retrieved from the instantiated prediction information.

3. The method according to claim 2, characterized in that, The step of determining the detection result of each lane instance based on the instantiated prediction information output by each lane branch further includes: generating a boundary geometric representation of each lane instance based on the edge positions corresponding to all lane foreground points; performing fusion analysis based on the lane attribute prediction information corresponding to all lane foreground points to obtain the final lane attribute of each lane instance; performing fusion analysis based on the edge attribute prediction information corresponding to all lane foreground points to obtain the final edge attribute of each lane instance; and / or, The step of generating a boundary geometric representation for each lane instance based on the edge positions corresponding to all lane foreground points includes: sorting all corresponding edge positions according to the coordinates of the row in which they are located in the image to form an ordered sequence of boundary points, and using the ordered boundary point representation as the geometric representation of the boundary.

4. The method according to claim 2, characterized in that, The instantiated prediction information includes lane heatmaps, boundary offset maps, and attribute prediction maps. The step of each lane branch independently outputting instantiation prediction information for a corresponding lane instance includes: each lane branch includes three parallel and independent prediction sub-networks, the three prediction sub-networks being used to generate the lane heatmap, the boundary offset map, and the attribute prediction map, respectively; and / or, The step of determining the detection result of each lane instance based on the instantiated prediction information output by each lane branch further includes: decoding multiple key point locations based on the lane heatmap output by each lane branch; wherein the key point locations include lane foreground points and their coordinates in the feature map space; indexing the corresponding left offset and right offset from the boundary offset map output by the corresponding lane branch according to the coordinates of the lane foreground points in the feature map space; calculating the left lane edge position and right lane edge position corresponding to each lane foreground point based on the horizontal coordinate of the lane foreground point in the feature map space, the left offset and the right offset; indexing the corresponding lane attribute prediction information from the attribute prediction map output by the corresponding lane branch according to the coordinates of each lane foreground point in the feature map space; and indexing the corresponding left edge attribute prediction information and right edge attribute prediction information from the attribute prediction map output by the corresponding lane branch according to the calculated left lane edge position and right lane edge position.

5. The method according to claim 4, characterized in that, The step of determining multiple lane-ahead points and their coordinates in the feature map space based on the lane heatmap output by each lane branch includes: determining the pixel position with the highest built-in confidence in each row of the lane heatmap; in response to the confidence of the pixel position being greater than a preset threshold, determining it as the lane-ahead point and recording its coordinates; and / or, Each lane branch includes multiple prediction subnetworks that are structurally identical and parametrically independent. Each prediction subnetwork is used to generate the lane heatmap, the boundary offset map, and the attribute prediction map, respectively.

6. The method according to claim 4, characterized in that, Also includes: Obtain a training sample set, which includes labeled sample images and lane instances; For the lane instance corresponding to the sample image, generate a supervision label; The supervision label includes: multiple supervision point locations defined in the feature map space and associated with the lane instance, and a dynamic supervision quantity assigned to each supervision point location; the dynamic supervision quantity characterizes the geometric relationship between the corresponding supervision point and the edge of the lane instance in the image space. Using the training samples as input and the constructed supervised labels as learning targets, the initial lane detection model is trained; wherein, the training employs multi-task loss supervision; the multi-task loss function includes lane region loss for supervising lane heatmap prediction, margin regression loss for supervising boundary offset prediction, and attribute classification loss for supervising attribute prediction map prediction.

7. The method according to claim 6, characterized in that, The step of generating supervised labels for the lane instances corresponding to the sample images includes: for the p-th target lane instance, p = 1, 2, ..., P; calculating the lane centerline based on the left and right edge annotations of the p-th target lane instance, and determining Q foreground encoding points in the feature map space based on the lane centerline, where Q is a positive integer; for each of the Q foreground encoding points, calculating the geometric distance from its inverse mapping to the original image coordinates to the left and right edges of the p-th target lane instance, as the dynamic margin ground truth of the foreground encoding point; and / or, The multi-task loss function satisfies at least one of the following: the lane area loss is calculated using the focus loss function; the attribute classification loss is calculated using the cross-entropy loss function; and the margin regression loss is calculated using the distribution focus loss function.

8. The method according to claim 7, characterized in that, The step of determining Q foreground coding points in the feature map space based on the lane centerline includes: mapping the lane centerline to the feature map space to obtain at least one centerline coding point; expanding horizontally within the feature map space starting from the centerline coding point, and determining the grid coordinates covered by the expansion as additional coding points; the centerline coding point and the additional coding points together constitute the Q foreground coding points; and / or, The step of expanding horizontally in the feature map space starting from the centerline coding point includes: for each centerline coding point, determining the number of grids to expand to the left and right sides based on its distance to the left and right edges of the p-th target lane instance; expanding according to the number of grids, and the expansion range does not exceed the left and right edges of the p-th target lane instance.

9. The method according to claim 7, characterized in that, The step of calculating the geometric distance from each of the Q foreground coding points, after being inversely mapped to the original image coordinates, to the left and right edges of the p-th target lane instance includes: If the geometric distance from the foreground encoding point to an edge of the p-th target lane instance cannot be calculated or is invalid, then a predetermined invalidation flag is assigned to the true value of the dynamic margin of the edge corresponding to the foreground encoding point; during training, the terms with the invalidation flag are ignored when calculating the margin regression loss; and / or, It also includes: when the ordinate obtained by inverse mapping is not an integer, interpolation is performed based on the adjacent markers above and below the ordinate to obtain the edge position at the ordinate, and then the geometric distance is calculated.

10. A vehicle, characterized in that, It is capable of performing the lane line detection method according to any one of claims 1 to 9.

Images