Lane detection method based on lane detection model and training method for lane detection model

Abstract

The invention discloses a lane line detection method based on a lane line detection model, a training method of the lane line detection model, electronic equipment and a storage medium, and relates to the technical field of computer vision and deep learning. The lane line detection model comprises a feature extraction network and a prediction module. The lane line detection method comprises the following steps: extracting image features of a to-be-processed image through the feature extraction network; and determining an initial implicit function of a lane line in the to-be-processed image through a prediction module based on the image features, generating an elastic lane map based on the initial implicit function, and determining a predicted lane line in the to-be-processed image according to a zero horizontal contour line of the elastic lane map. The training method of the lane line detection model comprises the steps of training a neural network through an elastic interaction energy loss function, simplifying gradient calculation of the loss function through fast Fourier transform, and obtaining accurate representation of a target lane line through updated parameters of an elastic lane map. The lane line detection model can accurately identify lanes with complex geometric structures and shapes.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511559712.1 (22) Application Date 2025.10.29 (30) Priority Data 63 / 727,205 2024.12.03 US (71) Applicant Hong Kong University of Science and Technology Address Clear Water Bay, Kowloon, Hong Kong, China (72) Inventors Xiang Yang Feng Yaxin Pang Yuwen Lan Yuan (74) Patent Agency Beijing Ying Sai Jia Hua Intellectual Property Agency Co., Ltd. 11204 Patent Attorney Wang Dazuo Wang Yanchun (51) Int.Cl. G06V 20 / 56 (2022.01) G06V 10 / 52 (2022.01) G06V 10 / 80 (2022.01) G06V 10 / 44 (2022.01) G06V 10 / 82(2022.01) G06N 3 / 0464(2023.01) G06N 3 / 045(2023.01) G06N 3 / 08(2023.01) (54) Invention Title: Lane Line Detection Method Based on Lane Line Detection Model and Training Method of Lane Line Detection Model (57) Abstract: A lane line detection method based on a lane line detection model, a training method for a lane line detection model, an electronic device, and a storage medium are disclosed, relating to the fields of computer vision and deep learning technology. The lane line detection model includes a feature extraction network and a prediction module. The lane line detection method includes: extracting image features of the image to be processed through the feature extraction network; determining the initial implicit function of the lane lines in the image to be processed through the prediction module based on the image features, generating an elastic lane map based on the initial implicit function, and determining the predicted lane lines in the image to be processed based on the zero horizontal contour line of the elastic lane map. The training method for the lane detection model is as follows: a neural network is trained using an elastic interactive energy loss function, and the gradient calculation of the loss function is simplified using a fast Fourier transform, thereby obtaining an accurate representation of the target lane line using the parameters of the updated elastic lane map. The above lane detection model can accurately identify lanes with complex geometric structures and shapes. (Claims: 3 pages; Description: 26 pages; Figures: 16 pages; CN 122157185 A 2026.06.05 CN 1 22 15 71 85 A) 1. A lane detection method based on a lane detection model, wherein the lane detection model includes a feature extraction network and a prediction module, the method comprising: extracting image features of an image to be processed through the feature extraction network; determining an initial implicit function of lane lines in the image to be processed based on the image features through the prediction module, and generating an elastic lane map based on the initial implicit function; and determining predicted lane lines in the image to be processed based on the zero-level contour line of the elastic lane map.2. The lane line detection method according to claim 1, wherein the initial implicit function of the lane line is a composite function of a level set function and a smooth step function. 3. The method according to claim 2, wherein determining the predicted lane line in the image to be processed based on the zero horizontal contour line of the elastic lane map includes: sampling the elastic lane map at equal intervals along the y-axis direction to obtain the coordinates of multiple target sample points located on the zero horizontal contour line of the elastic lane map, wherein the target sample points are points on the y-coordinate row where the value of the elastic lane map is zero; determining the predicted lane line based on the coordinates of the multiple target sample points. 4. The lane line detection method according to claim 1, wherein the feature extraction network includes a feature extraction sub-network and a multi-scale feature fusion sub-network, the feature extraction sub-network includes a residual module, and extracting image features of the image to be processed through the feature extraction network includes: extracting a first feature from the image to be processed through the residual module; and performing feature fusion through the multi-scale feature fusion sub-network based on the first feature to obtain the image features. 5. The lane line detection method according to claim 1, wherein the feature extraction network includes a feature extraction sub-network and a multi-scale feature fusion sub-network, the feature extraction sub-network includes a residual module and a feature enhancement bottleneck layer, and the step of extracting image features of the image to be processed through the feature extraction network includes: extracting initial features in the image to be processed through the residual module; performing global feature enhancement on the initial features through the feature enhancement bottleneck layer to obtain a first feature; and performing feature fusion through the multi-scale feature fusion sub-network based on the first feature to obtain the image features. 6. The lane line detection method according to claim 4 or 5, wherein the multi-scale feature fusion sub-network includes a feature pyramid module and an auxiliary feature optimization sub-module, and the step of performing feature fusion through the multi-scale feature fusion sub-network based on the first feature to obtain the image features includes: generating multiple features at different levels through the feature pyramid module based on the first feature and performing feature fusion to obtain fused features; for features at at least one level among the multiple different levels, performing feature collection through the auxiliary feature optimization sub-module to obtain auxiliary fused features, and concatenating the auxiliary fused features with the fused features to obtain the image features. 7. The lane line detection method according to claim 4 or 5, wherein the lane line detection model further includes a lane presence classification subnetwork and a lane range subnetwork, and the step of generating an elastic lane map based on the initial implicit function includes: predicting whether a vehicle exists in the image to be processed based on the first feature using the lane presence classification subnetwork. (Claim 1 / 3 page 2 CN 122157185 A)The method for training a lane line detection model includes a feature extraction network and a prediction module. The method comprises: extracting image features from a sample image using the feature extraction network; determining an initial hidden function for lane lines in the sample image using the prediction module based on the image features, and generating an elastic lane map based on the initial hidden function; determining an elastic interaction energy loss function based on the hidden functions of the elastic lane map and the true lane line labels of the sample image; updating the parameters of the lane line detection model based on the elastic interaction energy loss function to obtain an updated lane line detection model. 9. The training method according to claim 8, wherein updating the parameters of the prediction module based on the elastic interaction energy loss function comprises: determining the gradient value of the elastic interaction energy loss function in Fourier space through fast Fourier transform; and updating the parameters of the lane detection model according to the determined gradient value. 10. The training method according to claim 8, wherein the feature extraction network comprises a feature extraction sub-network and a multi-scale feature fusion sub-network, the feature extraction sub-network comprises a residual module and a feature enhancement bottleneck layer, the multi-scale feature fusion sub-network comprises a feature pyramid module and an auxiliary feature optimization sub-module, and extracting image features of the sample image through the feature extraction network comprises: extracting initial features from the sample image through the residual module; performing global feature enhancement on the initial features through the feature enhancement bottleneck layer to obtain a first feature; generating multiple features at different levels through the feature pyramid module and fusing them to obtain a fused feature based on the first feature; for features at at least one level among the multiple different levels, collecting features through the auxiliary feature optimization sub-module to obtain an auxiliary fused feature, and concatenating the auxiliary fused feature with the fused feature to obtain the image feature. 11. The training method according to claim 10, wherein the method further comprises: generating an auxiliary prediction result through the prediction module based on the auxiliary fusion feature; determining an auxiliary loss function based on the auxiliary prediction result and the real lane line label, and updating the parameters of the lane line detection model based on the auxiliary loss function to obtain an updated lane line detection model; wherein the auxiliary loss function is determined based on elastic interaction energy loss.12. The training method according to claim 11, wherein the lane line detection model further includes a lane presence classification sub-network, and the method further includes: predicting the probability of existence of each lane line in the sample image through the lane presence classification sub-network based on the first feature; determining a lane presence loss function based on the focus loss function of the one-hot representation of the existence of each lane line in the sample image and the existence of the real lane line label; and updating the parameters of the lane presence classification sub-network based on the lane presence loss function. 13. The training method according to claim 12, wherein the lane line detection model further includes a lane range sub-network, and the method further includes: predicting the probability of existence of a lane line in each y-coordinate row of the sample image through the lane range sub-network based on the first feature; determining a lane range loss function based on the cross-entropy of the one-hot representation of the existence of the lane line in each y-coordinate row of the sample image and the range of the real lane line label; and updating the parameters of the lane range sub-network based on the lane range loss function. 14. The training method according to claim 13, wherein updating the parameters of the lane detection model based on the elastic interaction energy loss function includes: determining a total loss function based on the auxiliary loss function, the lane presence loss function, the lane range loss function, and the elastic interaction energy loss function; and jointly updating the parameters of the auxiliary feature optimization sub-module, the lane presence classification sub-network, the lane range sub-network, and the prediction module based on the total loss function. 15. A lane detection method based on a lane detection model, the lane detection model including a feature extraction network and a prediction module, the method including: extracting image features of an image to be processed through the feature extraction network; and determining predicted lane lines in the image to be processed through the prediction module based on the image features, wherein the predicted lane lines are represented as the derivatives of the implicit functions of lane lines in the image to be processed. 16. A training method for a lane detection model, the lane detection model comprising: a feature extraction network and a prediction module, the method comprising: extracting image features of a sample image through the feature extraction network; determining a predicted lane line in the image to be processed through the prediction module based on the image features, wherein the predicted lane line is represented as the derivative of the implicit function of the lane line in the image to be processed; determining an elastic interaction energy loss function based on the predicted lane line and the true lane line label of the sample image; updating the parameters of the lane detection model based on the elastic interaction energy loss function to obtain an updated lane detection model. 17. An electronic device, comprising: at least one processor; andA memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the lane detection method based on the lane detection model according to any one of claims 1-7 and 15, or the training method of the lane detection model according to any one of claims 8-14 and 16. 18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the lane detection method based on the lane detection model according to any one of claims 1-7 and 15, or the training method of the lane detection model according to any one of claims 8-14 and 16. Claims 3 / 3 Page 4 CN 122157185 A Lane Detection Method Based on Lane Detection Model and Training Method of Lane Detection Model Technical Field

[0001] Embodiments of this disclosure relate to the fields of computer vision and deep learning technology, and particularly to a lane detection method based on a lane detection model and a training method of the lane detection model, an electronic device, and a storage medium. Background Technology

[0002] Artificial intelligence is currently widely used in ADAS (Advanced Driving Assistance System). Lane detection is a key technology of ADAS, which helps intelligent vehicles plan driving operations.

[0003] The task of lane detection involves real-time identification of the boundaries of the driving area. In addition to the real-time requirements, the challenges faced by lane detection can be divided into two categories: weak feature lane scenarios and complex geometry lane scenarios. As shown in Figure 1a, weak feature lane scenarios refer to lane lines that are not obvious, which usually occurs in the following situations: the road is not marked, obstruction caused by vehicles and pedestrians, or insufficient light (such as low visibility caused by shadows, night, glare, etc.). As shown in Figure 1b, complex geometry lane scenarios include lanes with complex structures or shapes, such as intersections, large-angle Y-shaped lanes and merging lanes, large curvature turns, dense lanes, etc.

[0004] Traditional lane detection methods are usually based on manually designed operators, using image gradients to detect lane edges, such as Hough transform, and then performing post-processing. However, these methods perform poorly when faced with a large number of perception tasks in various real-world scenarios. Thanks to the rapid development of deep learning, many deep learning-based lane detection methods have shown breakthrough results in achieving end-to-end understanding of autonomous driving scenarios. However, most methods are designed for straight lanes without branching or lanes with low curvature, while real-world driving scenarios are diverse, and lane structures vary, such as forked lanes at intersections, resulting in a lack of robustness when these methods are applied to real-world scenarios.

[0005] For lanes with weak features and complex geometries, it is necessary to explore and develop more robust lane detection models and methods. Summary of the Invention

[0006] Embodiments of this disclosure propose a lane detection method based on a lane detection model, a training method for the lane detection model, an electronic device, and a storage medium.

[0007] In a first aspect, embodiments of this disclosure provide a lane detection method based on a lane detection model, which includes a feature extraction network and a prediction module. The method includes: extracting image features of the image to be processed through the feature extraction network; determining an initial implicit function of the actual lane lines in the image to be processed through the prediction module based on the image features, and generating an elastic lane map based on the initial implicit function; and determining the predicted lane lines in the image to be processed according to the zero-level contour line of the elastic lane map.

[0008] In a second aspect, embodiments of the present disclosure provide a training method for a lane detection model, the lane detection model including a feature extraction network and a prediction module. The method includes: extracting image features of a sample image through the feature extraction network; determining an initial implicit function of the actual lane lines in the sample image based on the image features through the prediction module, and generating an elastic lane map based on the initial implicit function; determining an elastic interaction energy loss function based on the implicit function of the elastic lane map and the actual lane line labels of the sample image, and updating the parameters of the lane detection model based on the elastic interaction energy loss function to obtain an updated lane detection model.

[0009] In a third aspect, embodiments of the present disclosure provide a lane detection method based on a lane detection model, the lane detection model including a feature extraction network and a prediction module. The method includes: extracting image features of an image to be processed through the feature extraction network; determining a predicted lane line in the image to be processed based on the image features through the prediction module, wherein the predicted lane line is represented as the derivative of the initial implicit function of the lane lines in the image to be processed.

[0010] In a fourth aspect, embodiments of the present disclosure provide a training method for a lane detection model, the lane detection model including a feature extraction network and a prediction module, the method including: extracting image features of a sample image through the feature extraction network; determining a predicted lane line in the image to be processed through the prediction module based on the image features, wherein the predicted lane line is represented as the derivative of the implicit function of the lane line in the image to be processed; determining an elastic interaction energy loss function based on the predicted lane line and the real lane line label of the sample image; updating the parameters of the lane detection model based on the elastic interaction energy loss function to obtain an updated lane detection model.

[0011] In a fifth aspect, embodiments of the present disclosure provide an electronic device, the electronic device including: at least one processing...The device; and a memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, which, when executed by at least one processor, enable the at least one processor to implement the lane detection method based on the lane detection model as described in the first or third aspect, or the training method of the lane detection model as described in the second or fourth aspect.

[0012] In a sixth aspect, embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions for enabling a computer to implement the lane detection method based on the lane detection model as described in the first or third aspect, or the training method of the lane detection model as described in the second or fourth aspect.

[0013] The lane line detection method and the training method of the lane line detection model provided in the embodiments of this disclosure extract image features from the image to be processed, generate an elastic lane map based on the initial implicit function of the actual lane line, the elastic lane map is used to implicitly represent the predicted lane line, and the predicted lane line is constructed as a zero-width horizontal contour line in the elastic lane map. The elastic lane map determined based on the initial implicit function can be applied to curves representing complex geometric structures / shapes, and has flexible adaptability to changes in geometric structures / shapes, thereby making the predicted lane line contour closer to the actual lane line contour.

[0014] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0015] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figures 1a and 1b respectively illustrate schematic diagrams of some weak lane feature scenarios and complex geometric lane scenarios in lane detection tasks; Figures 2a, 2b, 2c, and 2d respectively illustrate schematic diagrams of several main lane detection methods based on deep neural network models; Figure 3 is an exemplary system architecture diagram in which the lane detection method or lane detection model training method of the embodiments of this disclosure can be applied; Figure 4 is a flowchart of a lane detection method based on a lane detection model provided according to an embodiment of this disclosure; Figure 5 is a flowchart of a lane detection model training method provided according to an embodiment of this disclosure; Figure 6 is an exemplary block diagram of a lane detection model provided according to an embodiment of this disclosure; Figure 7 is a schematic diagram of the architecture of the lane detection network ElasticLaneNet provided according to an embodiment of this disclosure; Figure 8a illustrates a lane detection method based on a lane detection model according to another embodiment of this disclosure.Figure 8b is a schematic diagram of an elastic lane map according to an embodiment of the present disclosure; Figure 8c is a schematic diagram of obtaining specific lane line coordinates by line-by-line sampling based on an elastic lane map according to an embodiment of the present disclosure; Figure 9 is a flowchart of line-by-line sampling based on an elastic lane map provided according to an embodiment of the present disclosure; Figure 10 is a flowchart of a training method for a lane line detection model according to another embodiment of the present disclosure; Figure 11 is a schematic diagram of driving the predicted lane line to move towards the direction approaching the real lane line label under the guidance of the elastic interaction energy loss function according to an embodiment of the present disclosure; Figure 12 is a flowchart of auxiliary training of the lane line detection model by constructing an auxiliary loss function provided according to an embodiment of the present disclosure; Figure 13 is a schematic diagram comparing the prediction results of the lane line detection network ElasticLaneNet according to the present disclosure with other lane line detection models on the SDLane dataset; Figure 14 is a schematic diagram of the prediction results of the lane line detection network ElasticLaneNet according to the present disclosure in some challenging scenarios in the CULane dataset and the TuSimple dataset. Figure 15 is a schematic diagram comparing the prediction results of the lane detection network ElasticLaneNet according to this disclosure with the true lane labels in some challenging scenarios in the CULane dataset and TuSimple dataset; Figure 16 is a schematic diagram of the architecture of a variant of the lane detection model of this disclosure, ElasticLaneNetpw, used for ablation experiments; Figure 17 is a schematic diagram of the structure of an electronic device suitable for implementing the lane detection model-based lane detection method or lane detection model training method of the embodiments of this disclosure. Detailed Description

[0016] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely for explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0017] It should be noted that the collection / collection, updating, analysis, use, transmission, and storage of user personal information involved in the technical solutions of this disclosure all comply with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, are not shared, disclosed, or sold outside of these legal uses, and are subject to the supervision and management of national regulatory authorities.

[0018] At present, lane detection methods based on deep neural network models are mainly divided into the following categories: segmentation-based methods, parameter-based methods, anchor-based methods, and row-wise methods.

[0019] 1. Segmentation-based method specification 3 / 26 pages 7 CN 122157185 A

[0020] Referring to Figure 2a, the segmentation-based method requires instance-level segmentation. For example, different colors (such as blue, green, red, and yellow) in Figures 1a and 1b can represent different instances. After obtaining the pixel-level discrimination results, post-processing steps are required to determine the position coordinates of each lane. However, treating lanes as dense pixel masks is not efficient, and overlapping lane parts are difficult to separate accurately. In addition, when lane features are weak, the predicted lane lines often show breaks or missing parts.

[0021] 2. Parameter-based method

[0022] The parameter-based method uses explicit parameterized functions, such as polynomials, Bézier curves, or explicit functions based on mathematical models to characterize lanes. For example, a quadratic polynomial as shown in Figure 2b can be used, where ( ) is the coordinate of the i-th sampling point, i is a positive integer, and a, b, and c are polynomial coefficients. However, it is usually difficult to flexibly represent complex lane shapes with explicit functions. Furthermore, parameter-based methods are highly sensitive to parameter prediction errors; small errors in higher-order coefficients can lead to significant deviations in the predicted lane shape. Therefore, when lane shapes are complex and varied, parameter-based methods struggle to achieve high-precision predictions.

[0023] 3. Anchor-based methods

[0024] Referring to Figure 2c, anchor-based methods first predefine ROIs (Regions of Interest) as line anchors, select the set of line anchors that are closest to the target lane line at a specified index level, construct the lane by predicting the offsets of these anchors, and then use traversal NMS (Non-Maximum Suppression) to determine the lane line with the highest confidence as the final prediction result. However, anchor-based methods rely on accurate prior anchor predictions and cannot adapt to diverse structures and shapes, such as intersections, turns, Y-shaped intersections, or dense lanes.

[0025] 4. Row Prediction-Based Methods

[0026] Referring to Figure 2d, most row prediction-based methods are based on CGM (Coarse Grid Maps), which are similar to instance segmentation methods but are more efficient. Lane positions (e.g., x-coordinates) in the CGM are output through row expectation values ​​or grid confidence scores. Additional outputs, such as offsets, can be integrated into the CGM through post-processing steps to improve lateral accuracy. Some studies have proposed hybrid sampling directions to address the amplified local error problem that easily occurs in row prediction-based methods. However, in real-world driving scenarios, there are various lane structures, and the parts perpendicular to the sampling direction, including intersections, large-angle Y-shaped lanes, and large-curvature curves, are difficult to localize in the CGM.

[0027] In the above-mentioned lane detection methods based on deep neural network models, parameter-based methods and anchor-point-based methods often cannot adapt to lane prediction with complex shapes, while segmentation-based methods and row prediction-based methods are mostly local processing and are prone to classification imbalance problems in multi-instance classification (pixel / grid), such as missing, interrupted or blurred lane lines. Finally, high-cost post-processing steps are required to determine the lane position.

[0028] The present disclosure will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0029] The embodiments of the present disclosure provide a lane detection method based on a lane detection model and a training method for the lane detection model. The lane detection model based on a deep neural network generates an ELM (Elastic Lane Map) for implicitly representing lane lines. Compared with parameter-based methods and anchor-point-based methods, it has stronger geometric adaptability for lane recognition with weak lane features and complex geometric shapes, and can better fit the real lane line contour. Guided by the EIE (Elastic Interaction Energy) loss function, the predicted lane lines obtained from the elastic lane map are smooth, coherent curves, and can integrate global information with low-level features, thereby overcoming the challenges of weak lane features and class imbalance that are prone to occur in segmentation-based methods (e.g., missing, interrupted, or blurred lanes). Furthermore, compared to segmentation-based and line prediction-based methods, it does not rely on costly post-processing steps to determine lane positions.

[0030] Figure 3 illustrates an exemplary system architecture 300 of a lane line detection method or a lane line detection model training method to which embodiments of the present disclosure can be applied.

[0031] As shown in Figure 3, the system architecture 300 may include a terminal device 301, a network 302, and a server 303. The network 302 serves as a medium for providing a communication link between the terminal 301 and the server 303. The network 302 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0032] Terminal device 301 interacts with server 303 via network 302 to receive or send messages, etc. Various client applications may be installed on terminal device 301 to interact with server 303.

[0033] Server 303 may be a server providing various services, such as a backend server that receives requests sent by terminal device 301 with which it has established a communication connection. The backend server can receive and analyze the requests sent by the terminal device and generate processing results to feed back to the terminal device.

[0034] Terminal device 301 can, for example, capture an image to be processed including road lane markings and send the image to be processed to server 303; server 303 can, for example, process the image to be processed using a lane detection model to obtain a prediction result, which may include at least one predicted lane line, and server 303 can send the prediction result to terminal device 303.

[0035] Server 303 can be hardware or software. When server 303 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 303 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0036] Server 303 can provide various services through its built-in applications. For example, it can provide lane detection services based on a lane detection model. Server 303 can achieve the following effects: First, it extracts image features of the image to be processed through a feature extraction network; then, based on the image features, it determines the initial hidden function of the lane lines in the image to be processed through a prediction module, and generates an elastic lane map based on the initial hidden function; finally, it determines the predicted lane lines in the image to be processed based on the zero-level contour line of the elastic lane map. The lane detection model can include a feature extraction network and a prediction module. Alternatively, taking the provision of training services for a lane detection model as an example, server 303 can achieve the following effects: First, it extracts image features of the sample image through a feature extraction network; then, based on the image features, it determines the initial hidden function of the lane lines in the sample image through a prediction module, and generates an elastic lane map based on the initial hidden function; next, it determines the elastic interaction energy loss function based on the hidden function of the elastic lane map and the true lane line labels of the sample image, and updates the parameters of the lane detection model based on the elastic interaction energy loss function to obtain the updated lane detection model.

[0037] It should be noted that the method provided in this disclosure embodiment can be executed by server 303 or by terminal device 301, and this disclosure does not limit this.

[0038] It should be understood that the number of terminal devices, networks and servers in FIG3 is merely illustrative. Depending on the implementation needs, any number of terminal devices, networks and servers can be used.

[0039] Continuing to refer to FIG4, FIG4 shows a flowchart 400 of a lane line detection method based on a lane line detection model provided according to an embodiment of this disclosure. In this embodiment, the lane line detection model may include a feature extraction network and a prediction module, and the flowchart 400 may include the following steps: Step 401, extracting image features of the image to be processed through the feature extraction network.

[0040] This step is intended to be performed by the execution entity of the lane line detection method based on the lane line detection model (e.g., page 5 / 26, CN of the specification shown in FIG3).Server 303 of 122157185 A extracts image features of the image to be processed through the feature extraction network of the lane detection model.

[0041] In the embodiments of this disclosure, the image to be processed can be a captured road image. The image to be processed may include actual lane markings, which are used to mark the boundaries of the road. The image to be processed may also include other road markings, such as turn arrows, deceleration markings, etc.

[0042] In an exemplary embodiment, the above-mentioned execution subject can input the image to be processed into the feature extraction network. The feature extraction network can extract image features of the image to be processed by performing feature encoding, feature extraction, feature fusion and other operations on the image to be processed. Among them, the feature extraction network can, for example, use CNN (Convolutional Neural Network) to extract pixel values, edges, shapes, colors, textures and other features in the image, and can also perform feature fusion and other processing on these features to obtain image features.

[0043] Step 402, based on the image features, the initial implicit function of the lane lines in the image to be processed is determined by the prediction module, and an elastic lane map is generated based on the initial implicit function.

[0044] Based on step 401, this step aims to have the execution entity determine the initial implicit function of the lane lines in the image to be processed by the prediction module of the lane line detection model based on the extracted image features, and generate an elastic lane map based on the initial implicit function.

[0045] In an exemplary embodiment, the execution entity can generate an elastic lane map based on the initial implicit function, and the elastic lane map can be used to implicitly represent the contour of the predicted lane lines.

[0046] In an exemplary embodiment, the elastic lane map can be understood as an implicit function representing a curve, and the predicted lane lines are modeled as "open curves without width" in the elastic lane map, which are implicitly embedded in the elastic lane map.

[0047] In an exemplary embodiment, the contour of the predicted lane lines can be represented as a set of function values ​​under specific independent variable values.

[0048] As an exemplary embodiment, the level set function used to construct the initial implicit function is the distance from each pixel in the image to the target curve, such as the actual lane line. When a pixel is to the left of the target curve, the value of the level set function is negative; when a pixel is to the right of the target curve, the value of the level set function is positive; and the target curve corresponds to the position where the value of the level set function is 0.

[0049] Step 403: Determine the predicted lane line in the image to be processed based on the zero-level contour line of the elastic lane map.

[0050] In this step, the execution entity, based on the elastic lane map generated in step 402, determines the zero-level contour line of the elastic lane map to obtain the predicted lane line in the image to be processed.

[0051] In an exemplary embodiment, as described in step 402, the elastic lane map is based on a level set function, and the predicted lane line is represented as a zero-level contour of the elastic lane map, that is, the position of the curve where the predicted lane line is located is a zero-level set where the value of the implicit function is zero.

[0052] The lane line detection method based on the lane line detection model of the above embodiments of the present disclosure extracts image features from the image to be processed and generates an elastic lane map based on the initial implicit function of the actual lane line. The elastic lane map is used to implicitly represent the predicted lane line, and the predicted lane line is constructed as a zero-level contour with no width in the elastic lane map. The elastic lane map determined based on the implicit function can be applied to curves representing complex geometric structures / shapes and has dynamic adaptability to changes in geometric structures / shapes, thereby making the predicted lane line contour closer to the actual lane line contour.

[0053] Next, referring to FIG5, FIG5 shows a flowchart 500 of the training method of the lane line detection model provided according to the embodiments of the present disclosure. In this embodiment, the lane detection model may include a feature extraction network and a prediction module. Process 500 may include the following steps: Step 501: Extract image features of sample images through the feature extraction network.

[0054] In this step, the execution entity (e.g., server 303 in Figure 3) can perform feature encoding, feature extraction, and feature fusion on the sample images in the training dataset through the feature extraction network of the lane detection model to extract image features of the sample images.

[0055] In an exemplary embodiment, the parameters of the lane detection model are updated using the training dataset. The training dataset may include multiple sample images, and information such as pixel values, edges, shapes, colors, and textures in the sample images can be extracted as features. Each sample image may have a label, which may correspond to the expected output of the model. The label may, for example, be the ground truth (GT) of the actual lane lines.

[0056] It should be understood that the features and labels listed above are only exemplary descriptions of sample images and do not constitute a limitation of this disclosure.

[0057] Step 502: Based on image features, the initial latent function of lane lines in the sample image is determined by the prediction module, and an elastic lane map is generated based on the initial latent function.

[0058] In this step, the execution entity can determine the initial latent function of lane lines in the sample image based on the image features extracted in step 501, and generate an elastic lane map based on the initial latent function.

[0059] Step 503: The elastic interaction energy loss function is determined according to the latent functions of the elastic lane map and the real lane line labels.

[0060] In this step, the aforementioned execution entity can derive the implicit function of the actual lane line based on the real lane line label, and construct the elastic interaction energy (EIE) loss function based on the implicit function of the actual lane line and the elastic lane map. The implicit function of the actual lane line can be determined based on the level set function of the real lane line label.

[0061] Step 504: Update the parameters of the lane line detection model based on the elastic interaction energy loss function to obtain the updated lane line detection model.

[0062] In this step, the aforementioned execution entity can update the parameters of the lane line detection model based on the constructed EIE loss function to obtain the updated lane line detection model. Using the elastic lane map generated by the lane line detection model with updated parameters, a more accurate representation of the predicted lane line's location can be obtained.

[0063] In the embodiments of this disclosure, the EIE loss function based on elastic interaction energy is inspired by the physical properties of linear defects in crystals. Inspired by this elastic interaction energy, for the lane detection task, the predicted lane line and the actual lane line trajectory (such as the actual lane line label) are likened to two defect lines in a crystal dislocation. An EIE loss function is constructed based on the elastic interaction energy between the two, and the EIE loss function measures the difference between the predicted lane line and the actual lane line trajectory. During training, guided by the EIE loss function, the predicted lane line is driven to move closer to the actual lane line trajectory.

[0064] The training method of the lane detection model in the above embodiments of this disclosure constructs an EIE loss function based on the prediction result of the lane detection model and the actual lane line trajectory to update the parameters of the lane detection model. During training, guided by the EIE loss function, the predicted lane line is driven to move by minimizing the elastic interaction energy between the two curves, so that it coincides with or approaches the actual lane line, thereby making the predicted lane line contour closer to the actual lane line contour.

[0065] Referring to FIG6, FIG6 illustrates a lane line detection model 600 provided according to an embodiment of the present disclosure, which can be applied to various servers or electronic devices.

[0066] As shown in FIG6, the lane line detection model 600 provided in this embodiment may include: a feature extraction network 601 and a prediction module 602. The feature extraction network 601 can extract image features from the input image; the prediction module 602 can generate an elastic lane map for implicitly representing lane lines based on the image features extracted by the feature extraction network 601.

[0067] In an exemplary embodiment, the feature extraction network 601 may include a feature extraction sub-network 611 and a multi-scale feature fusion sub-network 612. The feature extraction sub-network 611 may, for example, include a CNN network, which passes through multiple convolutional layers.(Convolutional Layers) Convolution and downsampling are used to extract a set of features from an image.

[0068] In an exemplary embodiment, the feature extraction subnetwork 611 may include a residual module, which extracts a set of features from the input image and directly obtains a first feature based on the extracted set of features.

[0069] In some optional embodiments, the feature extraction subnetwork 611 may also include a feature enhancement bottleneck layer, which extracts initial features from the input image through the residual module and performs global feature association extraction and feature enhancement on the initial features through the feature enhancement bottleneck layer to obtain the first feature.

[0070] In some exemplary embodiments, the multi-scale feature fusion subnetwork 612 may include, for example, an FPN (Feature Pyramid Network) module, which, based on the first feature output by the feature extraction subnetwork 611, performs upsampling and feature fusion on different feature layers through the FPN module to obtain fused image features.

[0071] It should be understood that the above feature extraction submodule and feature fusion submodule are only exemplary illustrations of the feature extraction network and do not constitute a limitation of this disclosure.

[0072] In an exemplary embodiment, the prediction module 602 determines the initial implicit function of lane lines in the image based on image features, and generates an elastic lane map based on the initial implicit function. This elastic lane map can model and represent lanes with complex structures or geometric shapes. By modeling lane lines as zero-level contour lines of the elastic lane map, predicted lane lines in the image can be determined based on the elastic lane map generated by the prediction module.

[0073] The lane line detection model 600 of this embodiment can be used to implement steps 401-402 or steps 501-502 in the embodiments corresponding to FIG4 and FIG5. The specific processing flow of the relevant steps and the technical effects they bring can be referred to the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0074] The lane line detection model of the above embodiments of this disclosure provides a new perspective on lane line detection by modeling the predicted lane lines as zero-level contour lines in the elastic lane map. The generated elastic lane map has high geometric adaptability and can identify lane lines with complex and variable shapes. The predicted lane lines obtained based on the elastic lane map are smooth and coherent, and can fit the trajectory of the actual lane lines well even when the sampling points are sparse.

[0075] Referring again to FIG7, FIG7 shows a schematic diagram of the architecture of a lane detection network provided according to an embodiment of the present disclosure. This embodiment provides an end-to-end lane detection network called ElasticLaneNet.

[0076] As shown in FIG7, the network architecture of ElasticLaneNet provided in this embodiment may include the following key components: Backbone, and ELMM (Elastic Lane Detection Network).Lane Map Module (Elastic Lane Map Generation Module).

[0077] The Backbone is the backbone of ElasticLaneNet, which acts as a feature extraction network and is responsible for extracting features from the input image for subsequent processing and analysis. The Backbone can be used, for example, to extract global and local features of the image, such as edges, textures, and shapes. The Backbone may include multiple convolutional layers or other feature extraction layers to extract features at different scales or levels in the image layer by layer. For example, the output of the backbone network may be a multi-scale feature map.

[0078] The ELMM can also be called a prediction module, which is used to generate an ELM based on the image features extracted by the backbone network. The prediction module may include multiple fully connected layers or convolutional layers to map the image features to the final output space and generate the final prediction result of the model.

[0079] In this embodiment, the Backbone may further include a residual module and an FPN module. The residual module can use ResNet (Residual Network) 34 as the encoder to extract a set of features from the image and output these features as the first feature to the FPN module. The FPN module is used to perform feature fusion based on the first feature to obtain the fused image features.

[0080] In some exemplary embodiments, in order to improve performance, the backbone network is also provided with a bottleneck layer, which is located between the residual module and the FPN module. The bottleneck layer can perform global information association extraction, feature enhancement, and other processing. The bottleneck layer can use the self-attention mechanism of the Transformer to capture more global contextual information. In this paper, the bottleneck layer with Transformer is called the Transformer Bottleneck (feature enhancement, abbreviated as TB) bottleneck layer.

[0081] As an exemplary embodiment, the residual module can be used to extract a set of initial features of the image; the feature enhancement bottleneck layer is used to perform global feature association extraction and feature enhancement on the initial features extracted by the residual module to obtain the first feature; the FPN module is used to generate features at different levels based on the first feature, and to fuse all or part of the features at different levels to obtain fused image features, called fused features. The features at different levels may include, for example, high-level features and low-level features. High-level features typically include more global context information, while low-level features include more local detail information.

[0082] In some exemplary embodiments, the lane detection network may also include AFR (Auxiliary Feature Filter).Feature Refinement (AFR) submodule. The AFR submodule can perform cross-layer feature fusion on features at different levels. For example, it can collect features from one or more levels of features at different levels, such as the low-level features, to obtain auxiliary fusion features. In an exemplary embodiment, the prediction module can optimize implicit lane prediction based on the auxiliary fusion features obtained in the AFR submodule.

[0083] In an exemplary embodiment, the auxiliary fusion features obtained in the AFR submodule can be used to assist in training the lane line detection model so that the model can more accurately identify lane lines.

[0084] In some exemplary embodiments, the lane line detection network may also include a lane presence sub-network (Classification Sub-Network), which may also be called a CSN submodule. The CSN submodule can be used to predict the existence of lane lines based on a first feature, that is, the predicted probability of each lane line in the image.

[0085] In some exemplary embodiments, the lane line detection network may also include a lane range sub-network (Range Sub-Network), which may also be called an RSN submodule. The RSN submodule can be used to predict the existence and position of lane lines in a certain row based on a first feature, that is, the predicted probability of the existence of lane lines in each y-coordinate row of the image.

[0086] The above embodiments of this disclosure provide an end-to-end lane detection network, ElasticLaneNet, including a backbone network and a prediction module. The backbone network can adopt an Encoder-Transformer-FPN structure, and the prediction module outputs an ELM. The ELM output by ElasticLaneNet provides a novel lane representation method, which can implicitly represent lane lines with complex geometric structures and shapes, providing a new perspective on lane modeling. Moreover, the ELM has high geometric adaptability and can identify lane lines with complex and variable shapes. In addition, by comprehensively acquiring global spatial structure relationships and local detail information, as well as high- and low-level feature fusion, even in cases of lane line discontinuity such as occlusion or missing lines, the continuity of the predicted lane lines can be ensured and complete lane information can be included, thereby improving the accuracy of lane line detection.

[0087] To further illustrate the implementation details of the ElasticLaneNet lane detection network and lane detection method according to the embodiments of this disclosure, FIG8a shows a processing flow 800 of a lane detection method based on a lane detection model provided according to another embodiment of this disclosure.

[0088] Referring to FIG7 and FIG8a, in this embodiment, flow 800 may include the following steps: Step 801, inputting an image into the lane detection model.

[0089] In this step, the given image is input into ElasticLaneNet. The size of the given image is H×W, where H is the height and W is the width.

[0090] Step 802: Extract a set of features from the image through the residual module.

[0091] In this step, for the input H×W low-resolution image I∈RH×W×3, the downsampled image I'∈Rh×w×3 is obtained through multi-layer convolution and downsampling operations.

[0092] Taking ResNet 34 network as an example, the extracted set of features includes the output of multiple convolutional layers, where Cn is the feature map (feature map) of different resolutions generated in different stages (feature extraction stages), n=2, 3, 4. The size of the feature map Cn of each stage is 1 / 2, 1 / 4 and 1 / 8 of the original image, that is, the size of the feature maps of adjacent stages is 2 times.

[0093] Step 803: Perform global feature enhancement on a set of features through the bottleneck layer to obtain the first feature.

[0094] In this step, the extracted set of features can be further processed through the designed Transformer Bottleneck structure, such as using a self-attention mechanism for global feature association extraction and feature enhancement, thereby effectively improving the feature representation capability of the model. In addition, the bottleneck layer can also reduce the number of model parameters and improve the generalization ability of the network. In some implementations, the bottleneck layer can use the Transformer's self-attention mechanism to perform global feature extraction on feature maps in multiple channel dimensions and can suppress some unimportant channel features.

[0095] Step 804: Based on the first feature, generate features at different levels through the feature pyramid module.

[0096] In this step, the FPN module upsamples on different layers (P2 layer and P3 layer) based on the first feature to generate features at different levels. Among them, in each layer Pn, upsampling is used to increase the feature map size, expanding the feature map size to twice its original size.

[0097] For example, features at different levels may include high-level features and low-level features. High-level features have strong semantic information representation capabilities but low feature map resolution and weak geometric information representation capabilities; while low-level features have strong geometric detail information representation capabilities and high feature map resolution, but weak semantic information representation capabilities.

[0098] Step 805: Fuse features at different levels to obtain image features.

[0099] In this step, the feature map of each layer Cn can be converted to a 1:1 convolution to change the number of channels, and the resulting feature map can be added to features Pn of different scales through lateral connections. In this way, by combining features at different levels...Feature maps from all or some layers within the same level are upsampled and fused to obtain fused image features. Feature Map fusion can be achieved through the FPN module, typically using the feature map of the last layer, P1, directly as the fused feature for subsequent implicit lane representation prediction. For example, a prediction result Pred1 can be generated based on the feature map of layer P1 through a prediction module.

[0100] In some optional embodiments, the AFR submodule can also collect features from lower-level features in different layers to obtain auxiliary fusion features. For example, the AFR submodule can further collect features from layers P2 and P3 to obtain auxiliary fusion features F2 and F3. F2 and F3 are upsampled features from layers P2 and P3, respectively, and have the same size as the features of layer P1, for example, 40×100. Then, the auxiliary fusion features F2 and F3 are concatenated with features from layer P1 to obtain cross-layer fused image features for subsequent prediction. Referring to Figure 7, based on the cross-layer fused image features, the prediction module can generate the final implicit lane prediction Pred1, which has a size of N×M×w. Here, N is the number of ELMs, M is the height of the feature map, which is actually the number of sampling points along the longitudinal direction of the lane line, and w is the width of the feature map. By collecting and stitching features from different layers through the AFR submodule, more contextual information can be captured, which is beneficial for the model to better understand global contextual information and local information.

[0101] Step 806, determine the level set function of the lane line in the image based on the image features.

[0102] In this step, the level set function of the lane line in the image can be determined based on the fused image features according to the following formula:

[0103] Where, and are the left and right sides of the lane line, respectively, and d(x,y) is the distance from the point (x,y) under the same y coordinate to the lane line in the x direction.

[0104] Step 807, generate the initial implicit function based on the level set function.

[0105] Referring to Figure 8b, in this step, based on the level set function, an initial implicit function is generated according to the following formula: , where the function is a smooth step function (Heaviside Function). This initial implicit function can be regarded as the initial implicit lane representation of the prediction, which is a composite function of the level set function and the smooth step function.

[0106] In some exemplary embodiments, can be determined according to the following formula:

[0107] where is the smoothing parameter in the smooth Heaviside function, used to control the interval width of the function transitioning from 0 to 1.

[0108] In some exemplary embodiments, can be obtained through the sigmoid function or the normalization function in the neural network.The softmax function approximates the representation.

[0109] Step 808: Optimize the initial implicit function to obtain the elastic lane map.

[0110] In this step, the initial implicit function (i.e., the initial implicit lane representation) can be optimized based on the prediction results of CSN and RSN to obtain the optimized target implicit function, and the target implicit function is used as the final elastic lane map.

[0111] Referring to FIG7, in an exemplary embodiment, the output results of CSN and RSN are multiplied with the above-mentioned initial implicit lane representation to obtain the final elastic lane map.

[0112] In practical applications, the predicted initial implicit lane representation may have an extra lane line or lane end offset, etc., which are inaccurate predictions. Based on whether there is a lane line in the image predicted by the CSN submodule, the part predicted as having no lane will be discarded; based on the longitudinal extension range of the lane line in the image predicted by the RSN submodule, the part exceeding the lane range will be discarded. After optimizing the lane existence and lane range of the initial implicit lane representation based on the prediction results of CSN and RSN, the final elastic lane map can more accurately predict the existence and specific location of lane lines.

[0113] As shown in Figure 7, when there are N (N is a positive integer) lane lines in an image, N elastic lane maps can be generated, where k=1, 2, ..., N.

[0114] Step 809: Determine the predicted lane lines in the image based on the zero horizontal contour line of the elastic lane map.

[0115] In this step, the predicted lane lines in the image can be determined based on the zero horizontal contour line of the generated elastic lane map, that is, the curve =0 in the elastic lane map represents the k-th lane line.

[0116] As shown in Figures 8c and 9, in an exemplary embodiment, based on the generated elastic lane map, the coordinates of each lane line can be obtained by sampling line by line along the y direction based on =0. Specifically, it may include the following sub-steps: Sub-step 901, sampling row by row at equal intervals along the y-axis on the elastic lane map to obtain the coordinates of multiple target sample points located on the zero horizontal contour line of the elastic lane map.

[0117] As an exemplary implementation, the position of the predicted lane line can be described by a set of discretized target sample point coordinates, where M is the number of target sample points, M is a positive integer, and is the coordinate of the i-th target sample point. Here, the target sample point is the point on the y-coordinate row where the value of the elastic lane map is zero.

[0118] Exemplarily, as shown in FIG8c, before inputting the original image, the image can be preprocessed by cropping to remove redundant parts of the image dataset that do not contain lane lines.

[0119] Sub-step 902, determining the predicted lane line based on the coordinates of the multiple target sample points.

[0120] As an exemplary implementation, based on the elastic lane map, the coordinates of multiple target sample points located on the zero horizontal contour line of the elastic lane map are obtained by sampling row by row along the y-axis direction, and then the predicted lane line is determined based on the obtained coordinates of the multiple target sample points.

[0121] The lane line detection method based on the lane line detection model in the above embodiments of the present disclosure, by combining the elastic lane map ELM that implicitly represents the predicted lane line with an efficient row sampling scheme, can overcome the limitations of segmentation-based lane models (which treat lanes as long and thin objects with a finite width, typically about 30 pixels) and some row-by-row prediction methods based on coarse grid maps, solve the class imbalance problem in multi-instance classification (pixel / grid), and does not require high-cost post-processing steps to determine the lane position; in addition, compared with parameter-based and anchor-based models, ELM has stronger geometric adaptability because lane lines can have a variety of complex shapes and can bend at any angle less than 90° on the ELM.

[0122] Referring to FIG10, FIG10 shows a processing flow 1000 of a training method for a lane line detection model provided according to another embodiment of the present disclosure. In this embodiment, taking the lane detection network ElasticLaneNet shown in Figure 7 as an example, process 1000 may include the following steps: Step 1001, inputting sample images into the lane detection model.

[0123] In this step, sample images are input into the lane detection model to be trained, such as the lane detection network ElasticLaneNet shown in Figure 7.

[0124] Step 1002, the lane detection model outputs an initial hidden function.

[0125] In this step, the end-to-end lane detection model ElasticLaneNet is used, and the generated initial hidden function is output according to the input sample images.

[0126] Step 1003, optimizing the initial hidden function to obtain the optimized elastic lane map.

[0127] In this step, as an optional implementation, the initial hidden function can be optimized according to the prediction results of CSN and RSN. For example, redundant lane lines are removed by CSN, and the part of the initial ELM that exceeds the lane range is turned into zero by the RSN module, further obtaining the optimized elastic lane map ELM.

[0128] Step 1004: Determine the EIE loss function based on the implicit function of the elastic lane map and the real lane line labels.

[0129] As mentioned above, in ElasticLaneNet, the predicted lane lines are represented as the zero-level contour lines of ELM. In order to learn a more accurate elastic lane map, ElasticLaneNet is trained by constructing the EIE loss function, so that the specification page 12 / 26 16 CN 122157185 AELM can capture the order, shape and position of various complex geometric structures, thereby identifying the accurate position of lane lines.

[0130] In the lane line detection problem, the predicted lane line and the true trajectory (GT) can be compared to two defect lines in a crystal dislocation. The system elastic interaction energy of a set of curves can be defined as follows:

[0131] where the vector dl represents the line element on the curve with the tangent direction τ, i.e. dl=τdl, represents the curve with another parameter, and is the Euclidean distance between the points on the curve and the points on the curve, i.e., .

[0132] In ElasticLaneNet, the lane line is defined as, where and are the true label and predicted value of the lane line, respectively. The above elastic interaction energy of the system can be defined by the following formula (1):

[0133] (1)

[0134] In the above formula (1), the energy can be expressed as, where the self energy, i.e. the self energy of the predicted lane line and the true label, is composed of the first two terms of formula (1); while the third term of formula (1) is the interaction energy between the two curves and.

[0135] One important characteristic of the elastic interaction energy Ei between the predicted lane line and the real label is that when the predicted lane line and the real label have opposite directions, it will tend to minimize the total energy in formula (1), and this attractive interaction is long-range.

[0136] It can be seen intuitively from Figure 11 that when the predicted lane line and the real label are consistent but opposite in direction, it will be completely eliminated, that is, the minimum value of the elastic interaction energy in formula (1) is 0. In addition, the self-energy of the predicted lane line tends to make it smooth, because the non-smooth curve is longer and has a larger self-energy.

[0137] In the embodiments of this disclosure, based on the above-optimized elastic lane map ELM, the elastic interaction energy (EIE) loss function is determined by the following formula (2): (2) In the above formula (2), is the implicit function of the real lane line label (GT), , is the level set function of the real lane line label; is the predicted elastic lane map, 0.5; is the hyperparameter. Instruction manual, page 13 / 26, 17 CN 122157185 A

[0138] Here, the energy descent gradient direction of the point on the real label and the energy descent gradient direction of the point on the predicted lane line can be obtained using the formula (1). Here, represents the Delta function of the curve, and is its normal direction. The elastic interaction energy between the predicted lane line and the real label is controlled by the hyperparameters and .

[0139] It should be noted that the “ ” symbol in formula (2) ensures that the directions of the predicted lane line and the real label, represented by and respectively, are opposite.

[0140] Step 1005: Update the parameters of the lane detection model based on the EIE loss function to obtain the updated lane detection model.

[0141] In this step, the parameters of the lane detection model can be updated based on a determined EIE loss function. For example, the EIE loss function can be used as the target loss function to train the lane detection model. When the target loss function converges, the training of the lane detection model can be stopped, and the trained lane detection model can be used as the updated lane detection model.

[0142] During the training process, under the guidance of the EIE loss function, a long-range interaction will occur between the predicted lane line and the real trajectory, thereby guiding the curve implicitly represented on the ELM to be closer to the real trajectory.

[0143] As shown in Figure 11, the predicted lane line is a dynamic curve, and is the real label of the lane line. The red arrow in the figure shows the direction of the interaction force acting on the dynamic curve. represents the distance from the point on the dynamic curve to the real label in the direction (i.e., the distance to the point with the same y-coordinate). It can be seen that the elastic interaction between the two curves provides a strong attraction, and this interaction is long-range because the energy density is inversely proportional to the distance between them and decays very slowly with increasing distance. The smoothing Heaviside function plays a role in smoothing the dynamic curve.

[0144] In an exemplary embodiment, in order to reduce the gradient calculation cost, FFT (Fast Fourier Transform) is applied to efficiently calculate the gradient value of the EIE loss function:

[0145] where and are the frequencies in the Fourier space, is the Fast Fourier Transform, and is the Inverse Fourier Transform.

[0146] By adopting FFT, the calculation cost can be reduced to O(N logN) compared to the calculation cost of O(N2) in direct integration, thereby significantly reducing the gradient calculation cost.

[0147] In some optional embodiments, the lane detection model can also be trained by constructing an auxiliary loss function based on the auxiliary fusion features obtained from the AFR submodule, aiming to further improve the accuracy of lane detection through cross-layer feature fusion and deep supervision during the training process.

[0148] Referring to Figure 12, the process of assisting the training of the lane detection model by constructing an auxiliary loss function may include the following sub-steps: Sub-step 1201, collecting features through the AFR sub-module to obtain auxiliary fusion features.

[0149] The AFR sub-module can further collect some layer features from different layers to obtain auxiliary fusion features. For example, by upsampling layers P2 and P3, auxiliary fusion features F2 and F3 are obtained.

[0150] Sub-step 1202: Based on the auxiliary fusion features, generate auxiliary prediction results through the prediction module.

[0151] Please refer to Figure 7. Based on the auxiliary fusion features F2 and F3 obtained in sub-step 1201, generate auxiliary prediction results Pred2 and Pred3 respectively through the prediction module.

[0152] Sub-step 1203: Determine the auxiliary loss function based on the auxiliary prediction results and the real lane line labels.

[0153] As an exemplary implementation, the auxiliary loss function is composed of EIE loss, which is defined as follows:

[0154] Where, is a parameter, ; and and are the prediction results Pred2 and Pred3 from layers P2 and P3 in the FPN module, respectively.

[0155] Sub-step 1204: Update the parameters of the lane line detection model based on the auxiliary loss function to obtain the updated lane line detection model.

[0156] As an exemplary embodiment, Pred1, Pred2, and Pred3 are subjected to deep supervised learning using an auxiliary loss function composed of EIE loss. The Pred1 obtained after deep supervised learning is then multiplied by the prediction results of CSN and RSN to obtain the final Elastic Lane Map (ELM).

[0157] Furthermore, in an exemplary embodiment, the existence and extent (length) of each lane line in the ELM can be jointly learned by constructing corresponding loss functions based on the prediction results of the CSN and RSN sub-modules. The CSN sub-module is used to predict whether lane lines exist in the image, and the RSN sub-module is used to predict the longitudinal extension extent (along the y-axis) of lane lines in the image.

[0158] As an exemplary embodiment, based on the predicted probability of each lane line's existence predicted by the CSN sub-module, a lane existence loss function is defined, and the parameters of the CSN sub-module are updated using this lane existence loss function. The lane presence loss function can be the focal loss function, which is defined as follows:

[0159] where Yi is the one-hot representation of the presence or absence of the actual lane line label, and Yi can be 0 or 1; Pi is the predicted probability of the existence of the i-th lane line. For example, the parameters and γ can be set to 0.25 and 2, respectively.

[0160] As an exemplary implementation, based on the predicted probability of the lane line presence in each y-coordinate row of the image predicted by the RSN submodule, the parameters of the RSN submodule are updated by defining a lane range loss function. For example, the lane range loss function can be determined based on the binary cross-entropy of the predicted probability of the lane line presence in each y-coordinate row of the sample image and the one-hot representation of the actual lane line label range. By applying this lane range loss function, the existence of a specific lane in each row yi can be learned, thereby obtaining the lane range. The lane range loss function can be defined as follows:

[0161] Where yi is the unique hot true label of the lane line in the i-th row, p(yi) is the predicted positive probability in the i-th row (i.e., the probability that the i-th row contains the lane line), and M is the number of upwards along the y-axis. Specification 15 / 26 pages 19 CN 122157185 A

[0162] In the above exemplary embodiment, the CSN submodule can be trained using the lane presence loss function for the case of no lane line; for the occasional deviation of lane endpoints on the predicted ELM, the RSN submodule can replace the post-processing step to eliminate outliers, discarding the parts that exceed the lane range or are classified as no lane, thereby obtaining the optimized ELM.

[0163] In addition, as an exemplary embodiment, the total loss function can also be determined according to the above auxiliary loss function, lane presence loss function, lane range loss function and EIE loss function, and the parameters of the AFR submodule, CSN submodule, RSN submodule and prediction module can be jointly updated based on the total loss function. The total loss function can be defined as follows:

[0164] Where, , , and are the weighting coefficients of each loss term.

[0165] As described above, the training method for the lane detection model provided in this disclosure uses an energy functional (EIE) loss specifically designed for sparse lane points to achieve global context integration and ELM learning. The ELM evolves gradually based on the EIE loss during training, generating long-range interactions between lane prediction results and the actual lane trajectory, thereby guiding the implicitly represented curve on the ELM to approach the actual trajectory.

[0166] In summary, this disclosure proposes a flexible geometrically adaptive lane detection model, ElasticLaneNet, which excels at detecting lanes with complex geometries while maintaining high detection efficiency. Within the ElasticLaneNet framework, a novel implicit lane representation method, namely the zero-level contour line of the ELM, is designed, enabling the lane detection method based on the lane detection model to perform excellently in lane recognition for complex geometries such as intersections, various curves, Y-shaped forks, and lanes merging at large angles. By applying EIE loss to guide ELM learning, long-range lane information is integrated across the entire image space while maintaining attention to local features. Furthermore, various auxiliary branch structures are designed, such as bottleneck layers and AFR submodules, to enhance feature fusion capabilities; and the ELM is optimized by jointly training the CRN and RSN submodules with the prediction module. The lane detection model / method disclosed herein can accurately identify lanes with complex geometries and shapes, and provides stable lane detection results even under conditions of weak features (such as nighttime, shadows, and occlusion).

[0167] The following will describe in detail the experimental evaluation results of the lane detection method based on the lane detection model of the present disclosure on different datasets.

[0168] The present disclosure proposes a new lane detection dataset SDLane, and applies the lane detection method based on the lane detection model of the present disclosure to the following three datasets for experimentation: SDLane, CULane and TuSimple. The experimental results show that the method of the present disclosure can achieve SOTA (State of the Art) performance on the SDLane dataset, and also performs well on the other two mainstream lane detection datasets CULane and TuSimple, and has a fast inference speed, maintaining a high FPS (Frames Per Second).

[0169] Among them, SDLane is a newly proposed dataset with up to 7 lanes and various highly complex lane structures, including various intersections, Y-shaped lanes or merging lanes with different angles, dense lanes, etc. In CULane, driving scenarios can be divided into 9 categories and have up to 4 lanes, including some complex scenarios, such as dense lanes and nighttime scenarios. It is worth noting that in SDLane, any drivable lane, such as the intersecting lanes at an intersection, is considered a lane, while in CULane, intersections are considered laneless. TuSimple is a highway dataset under good weather conditions with a maximum of 5 lanes. The curve ratio of the SDLane dataset is greater than 90%, the curve ratio of the CULane dataset is about 2%, and the curve ratio of the TuSimple dataset is about 30%. Specification 16 / 26 pages 20 CN 122157185 A

[0170] Evaluation on the CULane and SDLane datasets can use the official evaluation metrics of CULane: F1 score, precision, and recall based on IoU (Intersection over Union), where: F1=2 (Precision Recall) / (Precision+Recall); Precision=TP / (TP+FP); Recall=TP / (TP+FN).

[0171] Wherein, TP represents the number of accurately predicted lane points, i.e., the number of samples where the model predicts a positive value and the actual value is positive; FP represents the number of samples where the model predicts a positive value but the actual value is negative; FN represents the number of samples where the model predicts a negative value but the actual value is positive.

[0172] The official evaluation metrics for the TuSimple dataset are accuracy (Acc), FP, and FN. Additionally, the F1 score can also be used.Used as an evaluation metric. Accuracy Acc = Npred / Ngt, where Npred is the number of accurately predicted lane points and Ngt is the number of lane points with true lane line labels.

[0173] Further, during the experiment, the values ​​of the parameters , , and in the total loss function were set to 1.0, 1.0, 0.1, and 0.2, respectively. For feature fusion without an AFR submodule, the parameters and in the auxiliary loss function were both set to 0.3; in other cases, they were set to 0. That is, the multi-scale stitching feature fusion method and this auxiliary loss function are two further feature fusion methods and were not used together.

[0174] Referring to Figure 13, the rows in Figure 13 respectively illustrate various challenging and complex scenarios of the SDLane dataset. The top row 1 shows a winding road scenario, the second row shows a right-turn branch obscured by vehicles, the third row shows a left-turn branch obscured by vehicles, the fourth row shows a three-way intersection, the fifth row shows a newly added left-hand branch lane, the sixth row shows a Y-shaped intersection lane with a large curvature left turn and a large angle, and the bottom row 7 shows an intersection lane in a wireless scenario.

[0175] The columns in Figure 13 show a comparison of the prediction results of the lane detection network ElasticLaneNet of this disclosure with other lane detection models on the structurally diverse SDLane dataset. The first column represents the input sample image, the second column represents the ground truth lane label (GT), the third to sixth columns represent the prediction results of different types of state-of-the-art (SOTA) lane detection models LaneAF, BezierLaneNet, CLRNet, and CondLaneNet, respectively, and the seventh column represents the prediction results based on ElasticLaneNet of this disclosure. Different colors (such as red, green, blue, and yellow) represent different instances.

[0176] Based on the latest SOTA lane detection model of each type, the model was trained to reach convergence, and the prediction results obtained through experiments are recorded in Table 1 below. In Table 1 and below, ElasticLaneNet represents data without a feature enhancement bottleneck layer (TB), and ElasticLaneNetT represents data with a feature enhancement bottleneck layer.

[0177] Table 1. Comparison of evaluation metrics of prediction results of ElasticLaneNet and other SOTA lane detection models in different scenarios on the SDLane dataset. (Page 17 / 26, CN 122157185 A)

[0178] Combining Figure 13 and Table 1 above, it can be seen that segmentation-based lane detection models such as LaneAF may produce incomplete prediction results or missing lane lines when lane features are weak or overlapping lanes exist (Y-shaped lanes, etc.).This reflects its heavy reliance on low-level features and complex post-processing. Although the parameter-based lane detection model BezierLaneNet can handle many challenging instances, its predictions deviate significantly from the true GT when lane shapes are diverse. CondLaneNet is an efficient line-by-line processing method and has designed an auxiliary module called RIM (Recursive Instance Module) to handle bifurcated lanes. However, when faced with large-angle bifurcations or lanes with weak features, CondLaneNet often fails to provide the accurate shape and location of the branch lanes. It is worth noting that although the anchor-based lane model CLRNet achieves the highest accuracy in Table 1, i.e., the least redundant prediction, it often misses dense lanes, Y-shaped lanes, and turns, which are crucial for autonomous vehicles. For these situations, the anchor is not flexible enough and therefore cannot fully represent the diverse lane structures.

[0179] Table 2 below summarizes and compares the FPS inference speed of the ElasticLaneNet of this disclosure and other lane detection models applied to the SDLane dataset on the same server.

[0180] Table 2. Comparison of FPS between ElasticLaneNet and other SOTA lane detection models on the SDLane dataset. (Page 18 / 26, CN 122157185 A)

[0181] According to the summary comparison in Table 2, besides the parameter-based BezierLaneNet achieving the fastest speed (109.37 FPS), ElasticLaneNet, based on this disclosure, ranks second, reaching 75.42 FPS and 66.62 FPS respectively (without / with feature enhancement bottleneck layer TB). However, combining Table 1 and Figure 13, the predictions of the parameter-based BezierLaneNet are not very accurate. The segmentation-based lane detection model LaneAF requires a relatively time-consuming post-processing step, while the anchor-based lane detection model CLRNet requires NMS to select the best prediction result. Additionally, the row prediction-based CondLaneNet requires time to integrate multiple outputs, such as confidence maps, offset maps, and starting point maps, and needs to use an RNN (Recurrent Neural Network) module in the post-processing step to handle Y-shaped lane structures. Therefore, ElasticLaneNet, based on this disclosure, achieves a better balance between accuracy and real-time efficiency.

[0182] Table 3 below shows a comparison of the prediction results of ElasticLaneNet and other mainstream state-of-the-art lane detection models based on the ResNet34 structure on the TuSimple dataset.

[0183] Table 3. Comparison of evaluation metrics between ElasticLaneNet and other state-of-the-art lane detection models on the TuSimple dataset

[0184] TuSimple is an early lane detection dataset, mainly including highway driving scenarios with good lighting conditions. By comparing ElasticLaneNet with most state-of-the-art lane detection models (most of which use the ResNet34 backbone network), it was found that ElasticLaneNet outperforms most existing models. Among them, the F1 score of 97.05 and the accuracy of 96.48% are close to the performance data of the current state-of-the-art lane detection models (F1 score of 97.82 and accuracy of 96.9%).

[0185] Table 4 below shows the comparison of prediction results of ElasticLaneNet and other state-of-the-art lane detection models on the CULane dataset. Among them, ELM-MSE indicates data using the MSE (Mean Squared Error) loss function instead of the EIE loss function of ElasticLaneNet. The MSE loss function evaluates the performance of the model by calculating the average of the squares of the differences between the predicted value and the true value.

[0186] Table 4. Comparison of evaluation metrics of ElasticLaneNet and other SOTA lane detection models on the CULane dataset. (Page 19 / 26, CN 122157185 A)

[0187] In Table 4, the total F1 score is the F1 score based on the threshold IoU=0.5. Scenes 1 to 9 in the table correspond to the following scenes in the CULane dataset: Scene 1, Normal; Scene 2, Crowded; Scene 3, Dazzle; Scene 4, Shadow; Scene 5, Noline (i.e., no lane lines); Scene 6, Arrow; Scene 7, Curve; Scene 8, Cross; Scene 9, Night.

[0188] Referring to Table 4, based on the comparison of prediction results of ElasticLaneNet and other state-of-the-art lane detection models on the CULane dataset, ElasticLaneNet based on this disclosure also performs excellently on the commonly used CULane dataset, which covers a variety of scenarios, and surpasses many state-of-the-art lane detection models.

[0189] Figure 14 shows the prediction results of ElasticLaneNet in some scenarios of the CULane dataset and some challenging scenarios of the TuSimple dataset.

[0190] Figure 14 shows the curve, dense lanes on both sides, and weak lane feature scenes under occlusion / shade conditions in the TuSimple dataset. The other rows of Figure 14 show the congested scene, night scene, wireless scene, shadow scene, glare scene, and curve scene in the CULane dataset, respectively.

[0191] Figure 15 shows the comparison between the prediction results of ElasticLaneNet in some scenes in the CULane dataset and some challenging scenes in the TuSimple dataset and the true lane label.

[0192] As shown in Figures 14 and 15, ElasticLaneNet has shown excellent performance in some challenging scenes in the TuSimple dataset (as shown in Figure 14 and the first row of Figure 15), and can achieve accurate lane prediction results in many complex scenes in the CULane dataset.

[0193] Since most lane lines in the CULane dataset are straight, ElasticLaneNet only needs 18 sampling points to obtain good prediction results while maintaining efficient inference speed. The inference FPS of the test was 105 / 90 (without / with feature enhancement bottleneck layer), which is much faster than the 36 sampling points in Table 2. Figure 14 shows the excellent prediction of ElasticLaneNet for challenging scenes in CULane. Due to the long-range characteristics guided by the EIE loss function, ElasticLaneNet is good at handling some weak lane features, such as completely blocked by cars or glare, completely dark nights without any reference lines, and congested traffic scenes with a large number of pedestrians and vehicles. Specification 20 / 26 pages 24 CN 122157185 A

[0194] On the other hand, the known state-of-the-art methods are anchor-based methods and line-by-line prediction methods based on CGM. The advantage of ElasticLaneNet on CULane is not significant, which may be because the lane structure in CULane is mostly parallel and straight, and "intersection scenes" are regarded as "laneless". However, ElasticLaneNet performs well in handling the diverse lane shapes and intersections in the SDLane dataset. Furthermore, geometrically diverse models typically have a larger degree of freedom (DOF), providing richer geometric representations, but may sacrifice some accuracy; this phenomenon also exists in some works focusing on lane shape modeling. Therefore, adjusting and optimizing the CSN submodule for the characteristics of the CULane dataset is a possible direction for future improvement.

[0195] In addition, to further verify the effectiveness of the lane detection model ElasticLaneNet disclosed herein, ablation experiments were conducted on the model's network architecture and loss function.

[0196] Table 5 below shows the overall ablation experiment of the ElasticLaneNet network architecture on the SDLane dataset.

[0197] Table 5. Ablation experiment of ElasticLaneNet on the SDLane dataset

[0198] In Table 5, the evaluation metric for the ablation experiment in the first row is ELM-MSE, which is the data that uses the MSE loss function instead of the EIE loss function of ElasticLaneNet. That is, the endpoint outliers that move beyond a predetermined threshold are simply removed through post-processing. Compared with the second row, the EIE loss brings a significant improvement to various evaluation metrics.

[0199] As shown in Table 5, by designing the CSN and RSN sub-modules to replace the post-processing process, the ELM is refined, which can improve the F1 score and accuracy, but at the same time sacrifices a certain recall rate. That is, using the CSN / RSN sub-modules will make the prediction results more conservative. If the prediction results of the CSN / RSN sub-modules are wrong, especially for some challenging scenarios (such as dense lanes on both sides), the model will lose some detection capabilities. In the AFR submodule, the cross-layer feature collection and feature fusion scheme improves the overall performance.

[0200] As shown in Tables 1, 3 and 5, the design of the bottleneck layer brings about a significant performance improvement, presumably because the self-attention mechanism improves the prediction accuracy of lane presence and lane range, both of which are very beneficial to the refinement of ELM.

[0201] In addition, in order to study a better lane representation method, a variant of the prediction module (i.e., ELMM) as the detection head was also examined in the ablation experiment. This variant is an explicit point-by-point prediction detection head that does not include the FPN module.

[0202] As shown in Figure 16, an embodiment of this disclosure provides a lane line detection model, called the explicit model ElasticLaneNetpw. The lane line detection model may include a feature extraction network and a prediction module.

[0203] Corresponding to the explicit model ElasticLaneNetpw described above, embodiments of this disclosure also provide a lane detection method based on a lane detection model that is similar to the lane detection method in the foregoing embodiments. This lane detection method may include: extracting image features of the image to be processed through a feature extraction network; and determining predicted lane lines in the image to be processed based on the image features through a prediction module, wherein the predicted lane lines are represented as the derivatives of the implicit functions of lane lines in the image to be processed.

[0204] Further, embodiments of this disclosure also provide a training method for the above-mentioned lane detection model ElasticLaneNetpw. This training method may include: extracting image features of sample images through a feature extraction network; and determining predicted lane lines in the image to be processed based on the image features through a prediction module.The prediction module determines the predicted lane lines in the image to be processed, where the predicted lane lines are represented as the derivative of the implicit function of the lane lines in the image to be processed; the elastic interaction energy loss function is determined based on the predicted lane lines and the real lane line labels of the sample images; the parameters of the lane line detection model are updated based on the elastic interaction energy loss function to obtain the updated lane line detection model.

[0205] Referring to Figure 16, for a given image I∈RH×W×3, the prediction module of ElasticLaneNetpw can directly output the predicted lane lines, where the predicted lane lines are represented by a set of x coordinates. For example, taking the output of N predicted lane lines as an example, the network output of ElasticLaneNetpw is O'∈RN×M. Wherein, the network output O' can be represented as follows:

[0206] Wherein, k=1,2,……,N, N is the number of predicted lane lines.

[0207] Compared with the implicit representation of predicted lane lines by the zero horizontal contour line based on ElasticLaneNet based on ELM, ElasticLaneNetpw can directly output the x coordinates of N predicted lane lines, that is. The coordinates of the points on the actual lane label are.

[0208] As mentioned above, the elastic interaction energy E of a pair of open curves can be defined as:

[0209] In the embodiments of this disclosure, the elastic interaction energy E between the predicted lane line and the actual lane line of the explicit model ElasticLaneNetpw can be expressed as follows:

[0210] In the above formula, is the Delta function, which is the derivative of the aforementioned Heaviside function.

[0211] Wherein, to simplify the calculation, the following regularized Delta function is used:

[0212] This is an approximation of the derivative of; where is the x-coordinate of the point on the predicted lane line, ; is the smoothing parameter.

[0213] As can be seen from the above, satisfies , therefore, the EIE loss function between the predicted lane line and the actual lane line of the explicit model ElasticLaneNetpw can also be determined using the above formula (2). Then, the parameters of the explicit model ElasticLaneNetpw can be updated based on the EIE loss function to obtain the updated lane line detection model.

[0214] The difference between the explicit model ElasticLaneNetpw and the aforementioned ElasticLaneNet is that ElasticLaneNet outputs an elastic lane map and determines the EIE loss function based on the elastic lane map; while the explicit model ElasticLaneNetpw directly outputs the predicted lane line, which is represented as the derivative of the implicit function, i.e., the Delta function. (Pages 22 / 26, CN 122157185 A)When determining the EIE loss function, the implicit function is first determined based on the Delta function, and then the EIE loss function is determined based on the implicit function.

[0215] Referring to Figure 16, in an exemplary embodiment, the explicit model ElasticLaneNetpw may include a CSN submodule and an RSN submodule. When determining the EIE loss function, the predicted lane line output by the prediction module can be multiplied with the output results of CSN and RSN to optimize the predicted lane line. Then, the initial implicit function is determined based on the optimized predicted lane line, and the EIE loss function is determined based on the implicit function. The roles and effects of the CSN submodule and the RSN submodule can be found in the description in the foregoing embodiments, and will not be repeated here.

[0216] By minimizing the EIE loss to train the explicit network mapping, the evaluation metrics of the prediction results of ElasticLaneNetpw in different datasets are shown in Table 6 below.

[0217] Table 6. Evaluation metrics of the explicit model ElasticLaneNetpw

[0218] By comparing the evaluation metrics of the explicit model ElasticLaneNetpw with point-by-point prediction in Table 6 with those of ElasticLaneNet with the ELMM detection head in Table 1, the advantage of ElasticLaneNetpw lies in its smaller model parameters and faster inference speed. ElasticLaneNetpw performs much worse than ElasticLaneNet on structurally complex datasets (such as SDLane), but not significantly worse on CULane and TuSimple, indicating the prediction potential of the explicit model ElasticLaneNetpw on simple lane structures.

[0219] Furthermore, to illustrate the effectiveness of the EIE loss, another classic loss function, the MSE loss function, was applied in the ablation experiments for supervised learning of ELM. The performance of ELM trained based on the MSE loss function and the EIE loss function on different datasets is shown in Table 7 below.

[0220] Table 7. Comparison of ELM trained based on MSE loss function and EIE loss function 23 / 26 pages 27 CN 122157185 A

[0221] According to Table 7, compared with the MSE loss function, the EIE loss function is more effective on the scene-diverse dataset CULane and the structurally complex dataset SDLane. Furthermore, in experiments on the TuSimple dataset, the prediction results trained using the EIE loss function have smaller FP and FN, and higher accuracy Acc.

[0222] In addition, the training of the explicit model ElasticLaneNetpw based on the MSE loss function was also examined. The performance of ElasticLaneNetpw is acceptable; however, without prior anchors or CGM, using only...The ElasticLaneNetpw experiment with MSE loss fails due to training crashes.

[0223] By applying the lane detection method based on the lane detection model of this disclosure to three datasets: SDLane, TuSimple, and CULane, experimental results show that the method of this disclosure outperforms existing models on the structurally diverse SDLane dataset, achieving state-of-the-art performance with a breakthrough F1 score of 89.51, a recall of 87.50, and a precision of 91.61, while maintaining a fast inference speed (this is the result without using a feature enhancement bottleneck layer). Ablation experiments demonstrate the functionality of different modules in the ElasticLaneNet network and the EIE Loss-guided training strategy, making the ELM used for implicit lane representation more effective and accurate than explicit lane representation methods when facing challenging cases.

[0224] Referring below to FIG17, a schematic diagram of the structure of a computer system 700 suitable for implementing embodiments of this disclosure (e.g., the server or terminal device in FIG3) is shown. The terminal devices in the embodiments of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The device shown in FIG7 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0225] As shown in FIG17, the computer system 700 may include a processor (e.g., CPU, central processing unit) 701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from storage portion 708 into random access memory (RAM) 703. Various programs and data required for the operation of system 700 are also stored in RAM 703. The processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0226] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A driver 710 is also connected to the I / O interface 705 as needed. Removable media 711, such as the specification page 24 / 26, 28 CN 122157185 ADisks, optical discs, magneto-optical discs, semiconductor memories, etc., are installed on drive 710 as needed so that computer programs read from them can be installed into storage portion 708 as needed.

[0227] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication portion 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the methods of embodiments of the present disclosure.

[0228] It should be noted that the computer-readable medium described in embodiments of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0229] The computer-readable medium may be included in the electronic device; or it may exist alone and not assembled into the electronic device. The computer-readable medium carries one or more programs, when one or more of the programs are transmitted...When a program is executed by the electronic device, it can perform the various methods and processes described above, such as lane detection methods based on lane detection models and training methods for lane detection models.

[0230] Computer program code for performing the operations of embodiments of the present disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0231] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks (pages 25 / 26, CN 122157185 A) may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0232] The modules described in the embodiments of this disclosure may be implemented in software or hardware. The described modules may also be located in a processor, and the names of these modules do not necessarily constitute a limitation on the module itself.

[0233] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to the technical solutions formed by specific combinations of the above-described technical features, but should also cover solutions derived from the above-described technical features or their equivalents without departing from the above-described inventive concept.Other technical solutions formed by arbitrarily combining the features. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure. Instruction Manual Page 26 / 26, 30 CN 122157185 A, Figure 1a, Figure 1b, Figure 2a, Figure 2b; Instruction Manual Drawings 1 / 16, 31 CN 122157185 A, Figure 2c, Figure 2d; Instruction Manual Drawings 2 / 16, 32 CN 122157185 A, Figure 3; Instruction Manual Drawings 3 / 16, 33 CN 122157185 A, Figure 4; Instruction Manual Drawings 4 / 16, 34 CN 122157185 A, Figure 5; Instruction Manual Drawings 5 / 16, 35 CN 122157185 A, Figure 6; Instruction Manual Drawings 6 / 16, 36 CN 122157185 A, Figure 7; Instruction Manual Drawings 7 / 16, 37 CN 122157185 A, Figure 8a; Instruction Manual Drawings 8 / 16, 38 CN 122157185 A, Figure 8b, Figure 8c; Instruction Manual Drawings 9 / 16, 39 CN 122157185 A Figure 9 Appendix to the Instruction Manual 10 / 16 Page 40 CN 122157185 A Figure 10 Appendix to the Instruction Manual 11 / 16 Page 41 CN 122157185 A Figure 11 Appendix to the Instruction Manual 12 / 16 Page 42 CN 122157185 A Figure 12 Appendix to the Instruction Manual 13 / 16 Page 43 CN 122157185 A Figure 13 Appendix to the Instruction Manual 14 / 16 Page 44 CN 122157185 A Figure 14 Figure 15 Appendix to the Instruction Manual 15 / 16 Page 45 CN 122157185 A Figure 16 Figure 17 Appendix to the Instruction Manual 16 / 16 Page 46 CN 122157185 A Abstract A lane detection method based on a Lane Detection Model, a training method for the Lane Detection Model, an electronic device, and a storage medium are disclosed, relating to the fields of Computer Vision andDeep Learning technology. The Lane Detection Model includes a feature extraction network and a prediction module. The lane detection method includes: extracting image features of the image to be processed through the feature extraction network; determining the initial implicit function of the lanes in the image to be processed through the prediction module based on the image features; generating an Elastic Lane Map based on the initial implicit function; and determining the predicted lanes in the image to be processed based on the zero-level contours of the Elastic Lane Map. The training method for the Lane Detection Model is as follows: a Neural Network is trained using an Elastic Interaction Energy Loss function, and the gradient calculation of the Loss function is simplified via Fast Fourier Transform, thereby obtaining an accurate representation of the target lane using the updated parameters of the Elastic Lane Map. The Lane Detection Model described above can accurately recognizelanes with complex geometric structures and shapes.

Claims

1. A lane line detection method based on a lane line detection model, wherein the lane line detection model includes a feature extraction network and a prediction module, and the method includes: The feature extraction network extracts image features from the image to be processed. Based on the image features, the prediction module determines the initial latent function of the lane lines in the image to be processed, and generates an elastic lane map based on the initial latent function; The predicted lane lines in the image to be processed are determined based on the zero-level contour line of the elastic lane map.

2. The lane line detection method according to claim 1, wherein the initial implicit function of the lane line is a composite function of a horizontal set function and a smooth step function.

3. The method according to claim 2, wherein, Determining the predicted lane lines in the image to be processed based on the zero-level contour line of the elastic lane map includes: On the elastic lane map, samples are taken at equal intervals along the y-axis to obtain the coordinates of multiple target sample points located on the zero horizontal contour line of the elastic lane map, wherein the target sample points are points on the y-coordinate row of the elastic lane map with a value of zero. The predicted lane line is determined based on the coordinates of the multiple target sample points.

4. The lane line detection method according to claim 1, wherein, The feature extraction network includes a feature extraction subnetwork and a multi-scale feature fusion subnetwork. The feature extraction subnetwork includes a residual module. The step of extracting image features from the image to be processed through the feature extraction network includes: The residual module extracts the first feature from the image to be processed. Based on the first feature, the image features are obtained by feature fusion through the multi-scale feature fusion sub-network.

5. The lane line detection method according to claim 1, wherein, The feature extraction network includes a feature extraction subnetwork and a multi-scale feature fusion subnetwork. The feature extraction subnetwork includes a residual module and a feature enhancement bottleneck layer. The step of extracting image features from the image to be processed through the feature extraction network includes: The initial features are extracted from the image to be processed by the residual module; the initial features are then globally enhanced by the feature enhancement bottleneck layer to obtain the first feature; based on the first feature, the image features are obtained by feature fusion through the multi-scale feature fusion sub-network.

6. The lane line detection method according to claim 4 or 5, wherein, The multi-scale feature fusion subnetwork includes a feature pyramid module and an auxiliary feature optimization submodule. The step of fusing features based on the first feature through the multi-scale feature fusion sub-network to obtain the image features includes: Based on the first feature, multiple features at different levels are generated through the feature pyramid module and fused to obtain the fused feature. For features at at least one of the multiple different levels, auxiliary fusion features are obtained by feature collection through the auxiliary feature optimization submodule, and the auxiliary fusion features are concatenated with the fusion features to obtain the image features.

7. The lane line detection method according to claim 4 or 5, wherein, The lane line detection model also includes a lane presence classification subnetwork and a lane range subnetwork. The generation of the elastic lane map based on the initial implicit function includes: Based on the first feature, the presence of lane lines in the image to be processed is predicted by the lane existence classification subnetwork, and the longitudinal extension range of lane lines in the image to be processed is predicted by the lane range subnetwork. Based on the lane existence classification subnetwork and the prediction results of the lane existence classification subnetwork, the lane existence and lane range of the initial implicit function determined by the prediction module are optimized, and the optimized target implicit function is determined as the generated elastic lane map.

8. A training method for a lane line detection model, wherein, The lane detection model includes a feature extraction network and a prediction module, and the method includes: Image features of the sample images are extracted using the feature extraction network; Based on the image features, the prediction module determines the initial latent function of the lane lines in the sample image, and generates an elastic lane map based on the initial latent function; The elastic interaction energy loss function is determined based on the implicit function of the real lane line labels in the elastic lane map and the sample image; The lane detection model is updated by updating its parameters based on the elastic interaction energy loss function, resulting in an updated lane detection model.

9. The training method according to claim 8, wherein updating the parameters of the prediction module based on the elastic interaction energy loss function comprises: The gradient value of the elastic interaction energy loss function is determined in Fourier space by using Fast Fourier Transform; The lane detection model is updated with parameters based on the determined gradient value.

10. The training method according to claim 8, wherein, The feature extraction network includes a feature extraction subnetwork and a multi-scale feature fusion subnetwork. The feature extraction subnetwork includes a residual module and a feature enhancement bottleneck layer. The multi-scale feature fusion subnetwork includes a feature pyramid module and an auxiliary feature optimization submodule. The step of extracting image features from the sample image through the feature extraction network includes: The residual module extracts the initial features from the sample image; The initial feature is enhanced globally through the feature enhancement bottleneck layer to obtain the first feature; Based on the first feature, multiple features at different levels are generated through the feature pyramid module and fused to obtain the fused feature. For features at at least one of the multiple different levels, auxiliary fusion features are obtained by feature collection through the auxiliary feature optimization submodule, and the auxiliary fusion features are concatenated with the fusion features to obtain the image features.

11. The training method according to claim 10, wherein, The method further includes: Based on the aforementioned auxiliary fusion features, the prediction module generates auxiliary prediction results. An auxiliary loss function is determined based on the auxiliary prediction results and the real lane line labels, and the parameters of the lane line detection model are updated based on the auxiliary loss function to obtain the updated lane line detection model. The auxiliary loss function is determined based on the energy loss from elastic interactions.

12. The training method according to claim 11, wherein the lane line detection model further includes a lane presence classification subnetwork, and the method further includes: Based on the first feature, the lane presence classification subnetwork is used to predict the probability of each lane line in the sample image. The lane presence loss function is determined based on the predicted probability of each lane line in the sample image and the focal loss function of the one-hot representation of the presence or absence of the actual lane line label. The parameters of the lane presence classification subnetwork are updated based on the lane presence loss function.

13. The training method according to claim 12, wherein, The lane line detection model further includes a lane range sub-network, and the method further includes: Based on the first feature, the lane range sub-network is used to predict the probability of lane lines existing in each y-coordinate row of the sample image. The lane range loss function is determined based on the predicted probability of lane lines existing in each y-coordinate row of the sample image and the cross-entropy of the one-hot representation of the actual lane line label range. The parameters of the lane range subnetwork are updated based on the lane range loss function.

14. The training method according to claim 13, wherein, The parameter update of the lane detection model based on the elastic interaction energy loss function includes: The total loss function is determined based on the auxiliary loss function, the lane existence loss function, the lane range loss function, and the elastic interaction energy loss function. Based on the total loss function, the parameters of the auxiliary feature optimization submodule, the lane existence classification subnetwork, the lane range subnetwork, and the prediction module are jointly updated.

15. A lane line detection method based on a lane line detection model, wherein the lane line detection model includes a feature extraction network and a prediction module, and the method includes: The feature extraction network extracts image features from the image to be processed. Based on the image features, the prediction module determines the predicted lane lines in the image to be processed, wherein the predicted lane lines are represented as the derivative of the implicit function of the lane lines in the image to be processed.

16. A method for training a lane detection model, the lane detection model comprising: The method includes a feature extraction network and a prediction module, comprising: Image features of the sample images are extracted using the feature extraction network; Based on the image features, the prediction module determines the predicted lane lines in the image to be processed, wherein the predicted lane lines are represented as the derivative of the implicit function of the lane lines in the image to be processed. The elastic interaction energy loss function is determined based on the predicted lane lines and the real lane line labels of the sample images; The lane detection model is updated by updating its parameters based on the elastic interaction energy loss function, resulting in an updated lane detection model.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the lane detection method based on the lane detection model according to any one of claims 1-7 and 15, or the training method of the lane detection model according to any one of claims 8-14 and 16.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the lane detection method based on the lane detection model according to any one of claims 1-7 and 15, or the training method of the lane detection model according to any one of claims 8-14 and 16.

Images