A model training method, a lane line detection method and device

Abstract

This application discloses a model training method, a lane line detection method, and an apparatus, which can be used in the field of artificial intelligence, specifically for lane line detection in the automotive field. The method includes: labeling training samples with a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set, respectively representing feature points belonging to lane lines, local orientation information of feature points on lane lines, and information about the distance between feature points and the far endpoint of lane lines. The target model is trained based on the above ground truth labels. This application decouples the lane line prediction task from the requirement of simultaneously relying heavily on accurate local information and robust global information. Using the second type of ground truth set to obtain local information makes lane line prediction more accurate; using the third type of ground truth set to obtain global information makes lane line length prediction more precise. The combination of these two methods significantly improves the performance of lane line detection and can better adapt to real-world driving scenarios.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to a method for training a model, a method for detecting lane lines, and an apparatus. Background Technology

[0002] Artificial intelligence technology is increasingly being applied to the driving field, contributing to improved driving comfort and the development of advanced driver assistance systems (ADAS) and autonomous driving systems (ADS). As a fundamental issue in autonomous driving, lane detection plays a crucial role in applications such as real-time vehicle localization, route planning, lane keeping assist, and adaptive cruise control. Therefore, accurately detecting lane markings in real-time is a critical aspect of intelligent vehicle system design, aiding in path planning, providing road departure warnings, and offering a reference for precise navigation.

[0003] Currently, common lane detection methods are generally divided into two types: regression-based and segmentation-based. Regression-based methods can only roughly predict the position of lane lines, and their accuracy is poor when predicting lane lines at the far end of curves. For example, regression-based methods tend to predict the angle bisector of a Y-shaped lane (e.g.,...). Figure 1 The angle bisector shown cannot fit well for curves with large curvature (e.g., the angle bisector). Figure 1 The non-aligned lane lines are shown in the image; while segmentation-based methods can only provide relatively accurate predictions for local lane lines. Because segmentation is point-to-point (pixel to pixel), the overall performance of the predicted lane lines is poor. For example, it can cause a high rate of lane misalignment. Figure 2 This illustration shows an example of lane misalignment caused by a segmentation method.

[0004] The aforementioned general lane detection methods can detect lane lines locally or globally, but each has its own shortcomings. In real-world applications, due to complex driving scenarios such as vehicle occlusion, blurred lanes, and complex topological structures (such as asymmetrical lanes and near-horizontal lanes), lane detection remains a very challenging problem. Summary of the Invention

[0005] This application provides a model training method, a lane line detection method, and an apparatus. During the model training phase, three different types of ground truth annotations are applied to the training samples. These annotations represent feature points belonging to lane lines, local direction information of lane lines, and the distance between feature points and the far endpoint of lane lines. The target model is then trained based on these ground truth annotations. This application decouples the lane line prediction task from the requirement of simultaneously relying heavily on accurate local information and robust global information. Using a second type of ground truth set to obtain local information makes lane line prediction more accurate; using a third type of ground truth set to obtain global information makes lane line length prediction more precise. The combination of these two methods significantly improves lane line detection performance, enabling the prediction of any number of lane lines, including complex shapes such as Y-shaped lines and curved lane lines, thus better adapting to real-world driving scenarios.

[0006] Based on this, the embodiments of this application provide the following technical solutions:

[0007] Firstly, this application provides a model training method applicable to the field of artificial intelligence, specifically to lane detection scenarios in the automotive field. The method includes: firstly, labeling each training sample in the training set with different types of ground truth values. These labeled ground truth values ​​include a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. The second type of ground truth set can also be called a transition set, and the third type of ground truth set can also be called a distance set. In this application embodiment, the ground truth values ​​in the first type of ground truth set are used to characterize feature points belonging to the lane line; these feature points can also be called key foreground points. The second type of ground truth set may include ground truth values ​​characterizing the direction information (referred to as first direction information) from the target feature point to a first point on the target lane line. The third type of ground truth set may include ground truth values ​​characterizing the distance (referred to as first distance) between the target feature point and a first endpoint on the target lane line, where the first endpoint is the endpoint of the target lane line in the forward direction, i.e., the forward endpoint. The first distance can also be called the forward distance scalar, denoted as FDS. The first point is located in the forward direction of the target lane line, which is the lane line where the target feature point is located. The target feature point can be any point within the first type of truth set. In practical applications, any feature point within the first type of truth set can be used as the target feature point. Then, using a pre-constructed loss function (which can be called the target loss function), the target model is trained on the training set until a preset training termination condition is reached, thereby obtaining the trained target model. In this embodiment, the target loss function can be obtained based on a first loss function, a second loss function, and a third loss function, where the first loss function corresponds to the first type of truth set, the second loss function corresponds to the second type of truth set, and the third loss function corresponds to the third type of truth set.

[0008] In the above embodiments of this application, during the model training phase, three different types of ground truth annotations are applied to the training samples. These annotations are used to characterize feature points belonging to lane lines, local direction information of lane lines, and information about the distance between feature points and the far endpoint of lane lines. The target model is then trained based on these ground truth annotations. This embodiment decouples the lane line prediction task from the requirement that it heavily relies on accurate local information and robust global information. Using the second type of ground truth set to obtain local information makes lane line prediction more accurate; using the third type of ground truth set to obtain global information makes lane line length prediction more precise. The combination of these two types significantly improves the performance of lane line detection, enabling the prediction of any number of lane lines, including complex shapes such as Y-shaped lines and curved lane lines, thus better adapting to real driving scenarios.

[0009] In one possible implementation, when the second type of truth set includes a transition set of the forward direction (i.e., the second type of truth set only includes truth values ​​representing the first direction information from the target feature point to the first point on the target lane line), the method of annotating the truth values ​​(which can be called the first annotation method) can be as follows: First, arbitrarily select a feature point in the first type of truth set as the target feature point, and then draw a circle with the target feature point as the center and a preset step size as the radius. The resulting circle is the target circle, and the intersection of the target circle and the center line of the target lane line in the forward direction is the first point. Then, annotate the first direction information according to the relative positional relationship between the target feature point and the first point. This first direction information belongs to an element of the transition set of the forward direction.

[0010] In the above embodiments of this application, since the lane line is elongated and flat, local prediction can finely depict the details of the lane line. Therefore, this application introduces a second type of truth set (i.e., the transition set of the forward direction) to describe the local information of the lane line in the forward direction. Since each target feature point is local, the local direction information of the lane line can more accurately depict the local information of the lane line, thereby increasing the accuracy of lane line curves. Furthermore, in some application scenarios, only the forward branch of the lane line is needed in the above embodiments of this application. In this case, during the annotation of the second type of truth set, only the transition set of the forward direction can be annotated to save training time.

[0011] In one possible implementation, each feature point in the truth set of the first type is taken as the target feature point, and the above-described annotation method is repeated until the first direction information corresponding to each feature point is obtained. This first direction information belongs to the truth set of the second type.

[0012] In the above embodiments of this application, each feature point in the truth set of the first type obtains its corresponding first direction information, thereby making the transition set of the forward direction complete, which can improve the training effect of the model.

[0013] In one possible implementation, the second type of truth set may also include truth values ​​for characterizing the direction information (which may be called the second direction information) from the target feature point to a second point on the target lane line, which is located in the subsequent direction of the target lane line.

[0014] In the above embodiments of this application, the second type of truth set may include both the transition set of the forward direction and the transition set of the subsequent direction (that is, the second type of truth set includes truth values ​​used to characterize the first direction information from the target feature point to the first point on the target lane line and truth values ​​used to label the second direction information from the target feature point to the second point on the target lane line), which is used to train the target model to predict the entire lane line, which is closer to the real driving scenario and can improve the user experience.

[0015] In one possible implementation, when the truth set of the second type includes both the transition set of the forward direction and the transition set of the subsequent direction, the method for truth labeling the elements in the transition set of the subsequent direction can be as follows: First, arbitrarily select a feature point in the truth set of the first type as the target feature point. Then, draw a circle with the target feature point as the center and a preset step size as the radius. The resulting circle is the target circle. The intersection of the target circle and the center line of the target lane line in the subsequent direction is the second point. Afterward, label the second direction information according to the relative positional relationship between the target feature point and the second point. This second direction information then belongs to an element in the transition set of the subsequent direction.

[0016] In the above embodiments of this application, a second type of truth set (i.e., the transition set of the successor direction) is further introduced to describe the local information of the lane line in the successor direction, thereby improving the model's ability to understand and predict local information.

[0017] In one possible implementation, each feature point in the truth set of the first type is taken as the target feature point, and the above-described annotation method is repeated until the second direction information corresponding to each feature point is obtained. This second direction information also belongs to the truth set of the second type.

[0018] In the above embodiments of this application, each feature point in the truth set of the first type obtains its corresponding second direction information, thereby making the transition set of the subsequent direction complete. Based on a target feature point, and with the constructed transition set of the forward direction and the transition set of the subsequent direction, an entire lane line can be effectively decoded.

[0019] In one possible implementation, when the third type of truth set includes a forward distance set (i.e., truth values ​​representing the first distance between a target feature point and the first endpoint of the target lane line), the method for labeling the truth values ​​(which can be called the second labeling method) can be: First, calculate the first distance (i.e., FDS) between the target feature point and the first endpoint of the target lane line, and take each feature point in the first type of truth set as the target feature point. Repeat the above steps until the first distance corresponding to each feature point is obtained, and the first distance corresponding to each feature point belongs to the third type of truth set. That is, traversing the forward distance scalar of each feature point in the first type of truth set can form a forward distance set FDF∈R. H×W×1 .

[0020] In the above embodiments of this application, as can be seen from the annotation process of the second type of truth set, a lane line can be effectively decoded based on a target feature point and the constructed transition set. However, the transition set is difficult to effectively predict the end of the lane line. Therefore, this application introduces a third type of truth set (i.e., a distance set) to describe the distance between the predicted lane line and the current target feature point. The entire prediction process is highly robust because it only requires coarse global information (since only distance information is needed). Furthermore, in some application scenarios, only the forward branch of the lane line is needed in the above embodiments of this application. In this case, only the forward distance set can be annotated during the annotation process of the third type of truth set to save training time.

[0021] In one possible implementation, the third type of truth set may also include truth values ​​for characterizing the distance (which may be called the second distance) between the target feature point and the second endpoint on the target lane line, which is the endpoint of the target lane line in the subsequent direction, i.e., the subsequent endpoint.

[0022] In the above embodiments of this application, the third type of truth set may include both a forward distance set and a backward distance set (that is, the third type of truth set includes truth values ​​used to characterize the first distance between the target feature point and the first endpoint on the target lane line, and also includes truth values ​​used to characterize the second distance between the target feature point and the second endpoint on the target lane line), which is used to train the target model to predict the entire lane line, which is closer to the real driving scenario and can improve the user experience.

[0023] In one possible implementation, when the third type of truth set includes both the forward distance set and the backward distance set, the method for truth labeling elements in the backward distance set can be as follows: First, calculate the second distance (i.e., BDS) between the target feature point and the second endpoint of the target lane line. Then, treat each feature point in the first type of truth set as the target feature point and repeat the above steps until the second distance corresponding to each feature point is obtained. The second distance corresponding to each feature point also belongs to the third type of truth set. That is, traversing the backward distance scalar of each feature point in the first type of truth set can form a backward distance set BDF∈R. H×W×1 .

[0024] In the above embodiments of this application, based on a target feature point and by constructing forward distance sets and backward distance sets, an entire lane line can be effectively decoded. The distance information of the endpoints improves the target model's understanding and prediction ability of global information, and can predict any number of lane lines as well as complex Y-shaped lines, curved lane lines, etc.

[0025] In one possible implementation, labeling the training samples in the training set with ground truth includes performing a first type of ground truth labeling on the training samples in the training set. This labeling method can be referred to as a third labeling method, and specifically may include: First, obtaining a lane line point sequence based on the training samples in the training set, and during the training process, obtaining at least one feature point from the lane line point sequence. This at least one feature point belongs to the first type of ground truth set. For example, semantic segmentation can be used to predict points belonging to lane lines and points not belonging to lane lines, thus obtaining a segmentation map. The main function of the segmentation map is to generate candidate starting points for lane line detection (i.e., the starting point for lane line prediction) when performing lane line prediction. The candidate starting point can be any key foreground point in the first type of ground truth set. Each selected key foreground point needs to be labeled with a second type of ground truth set (also called a transition set) and a third type of ground truth set (also called a distance set). Each key foreground point can predict a lane line. For example, assuming that the ground truth set of the first type includes 8 key foreground points, then 8 predicted lane lines can be predicted based on the transition set and distance set of each key foreground point. Finally, the final lane line prediction result is obtained based on NMS, averaging, fitting and other methods.

[0026] In the above embodiments of this application, the annotation process of the truth set of the first type is specifically described, and it is feasible.

[0027] In one possible implementation, the first loss function can be the cross-entropy loss function.

[0028] In the above embodiments of this application, a typical form of the first loss function is specifically described, which has wide applicability.

[0029] In one possible implementation, the second loss function or the third loss function can be an L1 norm loss function or an L2 norm loss function, and this application does not limit it in this regard.

[0030] In the above embodiments of this application, the second loss function and the third loss function are specifically described in terms of selectable forms, which provides flexibility.

[0031] A second aspect of this application also provides a lane line detection method. This method may include: acquiring a target image and obtaining a predicted value of the target image based on a trained target model. The predicted value includes a first type of predicted value set (i.e., a predicted lane line segmentation map), a second type of predicted value set (i.e., a predicted transition set), and a third type of predicted value set (i.e., a predicted distance set). The first type of predicted value set includes predicted feature points representing lane lines. The second type of predicted value set includes first predicted direction information corresponding to the target predicted feature points. The third type of predicted value set includes a first predicted distance corresponding to the target predicted feature points. The first predicted direction information is information pointing to the forward direction of the target lane line. The first predicted distance is the predicted distance between the endpoint of the target lane line in the forward direction and the target predicted feature point. The target predicted feature point is any one of the first type of predicted value points. The trained target model is obtained through the training process of the method described in the first aspect or any possible implementation of the first aspect. After obtaining the predicted value of the target image, the execution device then obtains at least one predicted lane line based on the predicted value.

[0032] In the above embodiments of this application, a novel lane detection method is proposed. This method generates lane instances using predicted transition sets and predicted distance sets, which can effectively handle lane detection performance in bifurcated roads and sharp turn scenarios, and can provide good guidance in real driving scenarios.

[0033] In one possible implementation, when predicting the forward branch of the lane line, the second type of prediction set includes a set of transitions in the forward direction (i.e., the second type of prediction set only includes the first prediction direction information corresponding to the target predicted feature point), and the third type of prediction set includes a set of forward predicted distances (i.e., the third type of prediction set only includes the first prediction distance corresponding to the target predicted feature point). First, based on the target predicted feature point, the first prediction direction information, and the first prediction distance in the set of transitions in the forward direction, the forward branch of the target predicted lane line corresponding to the target predicted feature point is obtained, where the target predicted lane line is one of at least one predicted lane line.

[0034] In the above embodiments of this application, it is specifically described how to predict the forward branch of the corresponding lane line based on a predicted feature point, which is feasible.

[0035] In one possible implementation, the specific process of obtaining the forward branch of the target predicted lane line corresponding to the target predicted feature point based on the target predicted feature point, the first predicted direction information, and the first predicted distance can be as follows: predict a first predicted point belonging to the target predicted lane line based on the target predicted feature point and the first predicted direction information; then, use the first predicted point as a new target predicted feature point; repeat the above steps according to the second type of predicted value set until the distance between the last obtained first predicted point and the target predicted feature point reaches the first predicted distance; finally, connect the target predicted feature point and each obtained first predicted point in sequence to obtain the forward branch of the target predicted lane line.

[0036] In the above embodiments of this application, lane line instances in the forward direction are generated using the predicted transfer set and the predicted forward distance set in the forward direction, which has good predictive ability for curved lines, near-horizontal lines and occluded lane lines.

[0037] In one possible implementation, during training, since the second type of truth set can include only the transition set in the forward direction, or it can include both the transition set in the forward direction and the transition set in the subsequent direction, in some embodiments of this application, the second type of prediction set can include only the prediction transition set in the forward direction, or it can include both the prediction transition set in the forward direction and the prediction transition set in the subsequent direction. Similarly, since the third type of truth set can include only the forward distance set, or it can include both the forward distance set and the backward distance set, in some embodiments of this application, the third type of prediction set can also include only the forward prediction distance set, or it can include both the forward prediction distance set and the backward prediction distance set.

[0038] In the above embodiments of this application, it is specifically described that the second type of prediction value set and the third type of prediction value set may also include relevant prediction values ​​for predicting the backward branches of the lane line, which can be used to generate lane line instances in both directions, thereby improving the detection performance of the lane line.

[0039] In one possible implementation, when predicting the backward branch of the lane line, the third type of prediction value set also includes a prediction transfer set for the successor direction, and the third type of prediction value set also includes a set of backward prediction distances. First, based on the target predicted feature point, the second prediction direction information, and the second prediction distance, the backward branch of the target predicted lane line corresponding to the target predicted feature point is obtained, where the target predicted lane line is one of at least one predicted lane line.

[0040] In the above embodiments of this application, it is specifically described how to predict the backward branch of the corresponding lane line based on a predicted feature point, which is feasible.

[0041] In one possible implementation, the specific process of obtaining the backward branch of the target predicted lane line corresponding to the target predicted feature point based on the target predicted feature point, the second predicted direction information, and the second predicted distance can be as follows: predict a second predicted point belonging to the target predicted lane line based on the target predicted feature point and the second predicted direction information; then, use the second predicted point as a new target predicted feature point; repeat the above steps according to the second type of predicted value set until the distance between the last obtained second predicted point and the target predicted feature point reaches the second predicted distance; finally, connect the target predicted feature point and each obtained second predicted point in sequence to obtain the backward branch of the target predicted lane line.

[0042] In the above embodiments of this application, lane lines are generated in subsequent directions using the predicted transfer set and the predicted backward distance set, which has good predictive ability for curved lines, near-horizontal lines and occluded lane lines.

[0043] A third aspect of this application provides a training device that has the function of implementing the method described in the first aspect or any possible implementation of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0044] A fourth aspect of this application provides an execution device that has the function of implementing the method of the second aspect or any possible implementation of the second aspect described above. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0045] The fifth aspect of this application provides a training device, which may include a memory, a processor, and a bus system. The memory is used to store a program, and the processor is used to call the program stored in the memory to execute the method of the first aspect of this application or any possible implementation of the first aspect.

[0046] The sixth aspect of this application provides an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to call the program stored in the memory to execute the method of the second aspect of this application or any possible implementation of the second aspect.

[0047] A seventh aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the method described in the first aspect or any possible implementation thereof, or enable the computer to perform the method described in the second aspect or any possible implementation thereof.

[0048] The eighth aspect of this application provides a computer program that, when run on a computer, causes the computer to perform the method of the first aspect or any possible implementation thereof, or causes the computer to perform the method of the second aspect or any possible implementation thereof.

[0049] A ninth aspect of this application provides a chip including at least one processor and at least one interface circuit coupled to the processor. The interface circuit performs transceiver functions and sends instructions to the at least one processor. The at least one processor runs a computer program or instructions, having the functionality to implement the method described in the first aspect or any possible implementation of the first aspect, or having the functionality to implement the method described in the second aspect or any possible implementation of the second aspect. This functionality can be implemented in hardware, software, or a combination of hardware and software, and the hardware or software includes one or more modules corresponding to the above-described functions. Furthermore, the interface circuit communicates with other modules outside the chip. For example, the interface circuit can send a trained target model obtained by the processor on the chip to various intelligent driving agents (e.g., autonomous driving, assisted driving, etc.) for motion planning (e.g., driving behavior decision-making, global path planning, etc.). Attached Figure Description

[0050] Figure 1 This is a schematic diagram of an example of lane detection results obtained using a regression-based method.

[0051] Figure 2 This is a schematic diagram of an example of lane line detection results obtained by segmentation-based method;

[0052] Figure 3 A schematic diagram of the main framework of artificial intelligence provided in the embodiments of this application;

[0053] Figure 4 A schematic diagram of the image processing system architecture provided in the embodiments of this application;

[0054] Figure 5 A schematic diagram showing the deployment location of the trained target model provided in an embodiment of this application on an autonomous vehicle;

[0055] Figure 6 A schematic flowchart illustrating the training method for the model provided in this application embodiment;

[0056] Figure 7 A schematic diagram of the structural segmentation diagram provided in an embodiment of this application;

[0057] Figure 8 A schematic diagram illustrating the second type of truth labeling provided in this application embodiment;

[0058] Figure 9 A schematic diagram of the vector regression representation of the predecessor vector provided in an embodiment of this application;

[0059] Figure 10 A schematic diagram of a vector classification representation of the predecessor vector provided in an embodiment of this application;

[0060] Figure 11 A schematic diagram of a lane line that is not a center line provided in an embodiment of this application;

[0061] Figure 12 Another schematic diagram illustrating the second type of truth labeling provided in this application embodiment;

[0062] Figure 13 A schematic diagram of the forward direction, subsequent direction, and endpoint of a lane line provided in an embodiment of this application;

[0063] Figure 14 A schematic diagram illustrating the construction of a transfer set provided in an embodiment of this application;

[0064] Figure 15 A schematic diagram illustrating the construction of a distance set provided in an embodiment of this application;

[0065] Figure 16 A schematic diagram illustrating the ground truth annotation of training samples provided in an embodiment of this application;

[0066] Figure 17 A schematic diagram illustrating the training process of the target model provided in an embodiment of this application;

[0067] Figure 18 A schematic flowchart of the lane line detection method provided in the embodiments of this application;

[0068] Figure 19 A schematic flowchart illustrating the lane line detection process provided in this application embodiment;

[0069] Figure 20 A schematic diagram illustrating an example of the lane line detection process provided in this application embodiment;

[0070] Figure 21 A schematic diagram illustrating the decoding process of a single lane line provided in an embodiment of this application;

[0071] Figure 22 A comparison chart showing the results of comparing the algorithm of this application with the SOTA algorithm on the CULane dataset provided in the embodiments of this application;

[0072] Figure 23 A comparison chart showing the results of comparing the algorithm of this application with the SOTA algorithm on the CurveLane dataset provided in the embodiments of this application;

[0073] Figure 24 A comparison chart showing the results of comparing the algorithm of this application with the state-of-the-art algorithm on the TuSimple dataset provided in the embodiments of this application;

[0074] Figure 25 A comparison chart showing the results of comparing the algorithm of this application with the SOTA algorithm on the LLAMAS dataset provided in the embodiments of this application;

[0075] Figure 26 A comparison of lane line prediction performance between the model trained in this application and other methods on the CULane dataset;

[0076] Figure 27 A comparison of the lane line prediction performance of the model trained in this application with other methods on the CurvelLane dataset;

[0077] Figure 28 This is a schematic diagram illustrating an improvement of the method in this application for curves;

[0078] Figure 29 This is a schematic diagram illustrating an improvement of the method in this application for bifurcation lines;

[0079] Figure 30 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0080] Figure 31 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0081] Figure 32 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0082] Figure 33Another structural schematic diagram of the training device provided in the embodiments of this application;

[0083] Figure 34 Another structural schematic diagram of the execution device provided in the embodiments of this application;

[0084] Figure 35 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0085] This application provides a model training method, a lane line detection method, and an apparatus. During the model training phase, three different types of ground truth annotations are applied to the training samples. These annotations represent feature points belonging to lane lines, local direction information of lane lines, and the distance between feature points and the far endpoint of lane lines. The target model is then trained based on these ground truth annotations. This application decouples the lane line prediction task from the requirement of simultaneously relying heavily on accurate local information and robust global information. Using a second type of ground truth set to obtain local information makes lane line prediction more accurate; using a third type of ground truth set to obtain global information makes lane line length prediction more precise. The combination of these two methods significantly improves lane line detection performance, enabling the prediction of any number of lane lines, including complex shapes such as Y-shaped lines and curved lane lines, thus better adapting to real-world driving scenarios.

[0086] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0087] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below. It should be understood that the explanation of the relevant concepts may be limited due to the specific circumstances of the embodiments of this application, but it does not mean that this application can only be limited to that specific situation. The specific circumstances of different embodiments may also differ, and no specific limitation is made here.

[0088] (1) Neural Network

[0089] A neural network is a model composed of neural units. Specifically, it can be understood as a neural network with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Neural networks with many hidden layers are called deep neural networks (DNNs). The function of each layer in a neural network can be expressed mathematically. To describe it physically, each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations are: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... Operation 4 is completed using "+b", and operation 5 is implemented using "a()". The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight matrix of each layer of the neural network, where each value represents the weight of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space, as described above; that is, the W of each layer of the neural network controls how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0090] (2) Model

[0091] In the embodiments of this application, the models used for lane detection are essentially neural networks, or a portion of the model's structure is a neural network. The application of the model generally includes two stages: training and inference. The training stage is used to train the model based on the training set to obtain the trained model; the inference stage is used to perform lane detection on real unlabeled instances (i.e., real target images to be processed) using the trained model. The accuracy of the lane detection results is one of the important indicators for evaluating the quality of a model's training.

[0092] It should be noted that, in the embodiments of this application, the model described may be a variation of models such as segformer and hourglassnet.

[0093] (3) Convolutional Neural Networks (CNN)

[0094] A CNN (Neural Network Convolutional Layer) is a type of neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a CNN's convolutional layer, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units within the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0095] Convolutional kernels can be initialized as matrices of random size, and during CNN training, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, thus lowering the risk of overfitting.

[0096] (4) Loss function

[0097] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight matrix of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer). For example, if the network's prediction is too high, the weight matrix is ​​adjusted to predict a lower value. This adjustment continues until the neural network can predict the target value accurately. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the neural network becomes a process of minimizing this loss.

[0098] (5) Backpropagation algorithm

[0099] During the training of a neural network, the back propagation (BP) algorithm can be used to correct the parameters in the initial neural network model, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss. By backpropagating this error loss information, the parameters in the initial neural network model are updated, thus bringing the error loss to a convergence. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0100] (6) Wheeled mobile equipment

[0101] This is a comprehensive system integrating environmental perception, dynamic decision-making and planning, behavior control and execution, and can also be called a wheeled mobile robot or wheeled intelligent agent. For example, it can be wheeled construction equipment, autonomous vehicles, assisted driving vehicles, etc. Any device with wheeled mobility is called a wheeled mobile device as described in this application. For ease of understanding, in the following embodiments of this application, the wheeled mobile device is described as an autonomous vehicle. Autonomous vehicles can be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawnmowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, and handcarts, etc. The embodiments of this application do not impose any special limitations.

[0102] (7) Lane line detection

[0103] Lane detection can be considered a special type of object detection. Due to the significant differences between general object detection and lane detection in terms of object features, the target objects in general object detection (such as cats, dogs, and human faces) occupy areas with a certain aspect ratio and are easy to identify. However, lane lines in lane detection are long and narrow, making it difficult to accumulate enough robust features to describe the entire lane line, and thus they are not easy to identify. This places stringent requirements on neural networks, making lane detection more challenging than general object detection.

[0104] In the field of intelligent driving, lane detection is crucial for the accurate positioning of autonomous vehicles and its impact on driving decisions. As one of the most important road surface indicators, lane lines effectively guide autonomous vehicles to drive within constrained road areas. Real-time detection of lane lines is a vital component of ADAS and ADS systems in intelligent driving vehicles, assisting in path planning, providing road departure warnings, and offering a reference for precise navigation.

[0105] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0106] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 3 , Figure 3 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0107] (1) Infrastructure

[0108] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips; and the basic platform includes distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0109] (2) Data

[0110] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0111] (3) Data processing

[0112] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0113] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0114] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0115] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0116] (4) General ability

[0117] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0118] (5) Smart Products and Industry Applications

[0119] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, autonomous driving, and smart cities.

[0120] The embodiments of this application can be applied to the training and optimization of models, specifically in the construction of training sample data. When labeling the training sample data with ground truth, this application uses different types of ground truth labels on the training sample data, and then uses the training samples labeled with multiple types of ground truth sets to train the model. The model optimized and trained by this application can be specifically applied to lane detection in the field of artificial intelligence.

[0121] It should be noted that, in the embodiments of this application, the specific type of the model used for lane line detection (which may be referred to as the target model) is not limited. For example, the target model described in this application may be a model based on Segformer, a model based on HourglassNet, or a model based on other architectures. The specific type is not limited here.

[0122] After determining the target model for lane detection, it needs to be trained before it can be applied to specific lane detection tasks. Specifically, it can be based on... Figure 4 The image processing system 200 shown introduces the entire training process of the target model and lane detection. Figure 4In the image processing system 200, there are execution devices 210, training devices 220, databases 230, client devices 240, data storage systems 250 and data acquisition devices 260. The execution devices 210 include a computing module 211, which is essentially the trained target model 201 provided in the embodiments of this application.

[0123] The data acquisition device 260 (e.g., a camera mounted on a wheeled mobile device) is used to acquire the large-scale open-source dataset (i.e., training set) required by the user and store the dataset in the database 230. The training device 220 trains the target model 201 based on the dataset maintained in the database 230. It is important to note that in this embodiment, the dataset maintained in database 230 requires multiple types of ground truth annotation. Specifically, each training sample used for training device 220 needs to be annotated with ground truth, which includes a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. The ground truth values ​​in the first type of ground truth set represent feature points belonging to lane lines. The second type of ground truth set includes ground truth values ​​representing the first direction information from the target feature point to a first point on the target lane line. The third type of ground truth set includes ground truth values ​​representing the first distance between the target feature point and the first endpoint of the target lane line. The target feature point is any one of the points in the first type of ground truth set, the first point is located in the forward direction of the target lane line, the first endpoint is the endpoint of the target lane line in the forward direction, and the target lane line is the lane line where the target feature point is located. Similarly, each training sample in database 230 can be annotated with the aforementioned multiple types of ground truth, and then these annotated training samples are used to train the target model 201.

[0124] The execution device 210 can access data, code, etc., in the data storage system 250, and can also store data, instructions, etc., in the data storage system 250. The data storage system 250 can be located within the execution device 210, or it can be an external memory relative to the execution device 210.

[0125] The target model 201 trained by training device 220 can be applied to different systems or devices (i.e., execution device 210). For example, execution device 210 can be various wheeled mobile devices (wheeled construction equipment, autonomous vehicles, driver-assisted vehicles, etc.). Autonomous vehicles can also be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawnmowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, and handcarts, etc.

[0126] exist Figure 4In this embodiment, the execution device 210 is equipped with an I / O interface 212 for data interaction with external devices. A "user" can input data to the I / O interface 212 through the client device 240. For example, the client device 240 can be a camera mounted on an autonomous vehicle. Images captured by this camera are input to the computing module 211 of the execution device 210. The target model 201 in the computing module 211 detects the input image and obtains a detection result, which is then output to the camera or displayed directly on the display interface of the execution device 210 (if applicable). Furthermore, in some embodiments of this application, the client device 240 can also be integrated into the execution device 210. For example, when the execution device 210 is an autonomous vehicle, it can directly capture images through the autonomous vehicle's camera or receive images sent by other devices (e.g., mobile phones). The computing module 211 within the autonomous vehicle then detects the image, obtains a detection result, and directly displays the result on the mobile phone's display interface. The product form of the execution device 210 and the client device 240 is not limited here.

[0127] It is worth noting that Figure 4 This is merely a schematic diagram of an image processing system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 4 In this context, the data storage system 250 is an external memory relative to the execution device 210; however, in other cases, the data storage system 250 may be placed within the execution device 210. Figure 4 In this context, the client device 240 is an external device relative to the execution device 210. In other cases, the client device 240 may also be integrated into the execution device 210.

[0128] It should also be noted that in some embodiments of this application, the target model 201 can be further divided into multiple sub-modules / sub-units to jointly implement the solution provided in the embodiments of this application, and no specific limitation is made here.

[0129] It should also be noted that the training of the target model 201 described in the above embodiments can all be implemented on the cloud side. For example, the training set (which includes training samples labeled with various types of ground truth) can be obtained by a cloud-side training device 220 (which can be set on one or more servers or virtual machines), and the target model can be trained based on the training samples in the training set to obtain the trained target model 201. Then, the trained target model 201 is sent to the execution device 210 for application, for example, to the relevant systems of autonomous vehicles (such as ADAS, ADS, etc.) for lane detection. Figure 4The corresponding system architecture describes a system where training device 220 trains the target model 201 as a whole, and the trained target model 201 is then sent to execution device 210 for use. The training of the target model 201 described in the above embodiments can also be implemented entirely on the terminal side. That is, the training device 220 can be located on the terminal side. For example, a wheeled mobile device (such as an autonomous vehicle or an assisted driving vehicle) can obtain a training set (the training samples in this training set have been labeled with various types of ground truth), and train the target model 201 based on the training samples in the training set to obtain the trained target model 201. This trained target model 201 can be used directly on the terminal device, or it can be sent by the terminal device to other devices for use. Specifically, this application embodiment does not limit on which device (cloud side or terminal side) the target model 201 is trained or applied on.

[0130] It should be noted that, due to Figure 4 The trained target model 201 (or Figure 4 The execution device 210 can be deployed in various wheeled mobile devices to process relevant perceived information (e.g., video or images) on the road surface captured by a camera mounted on the wheeled mobile device, and output a predicted lane line. This predicted lane line is input into the downstream module of the wheeled mobile device for further processing. The following description uses an autonomous vehicle as an example to illustrate the deployment location of the trained target model and the overall architecture of the autonomous vehicle. Please refer to [link / reference] for details. Figure 5 , Figure 5 This illustrates a top-down, layered architecture. Each system can have defined interfaces for data transmission, ensuring data real-time performance and integrity. A brief introduction to each system follows:

[0131] (1) Environmental Sensing System

[0132] Environmental perception is the most fundamental part of autonomous vehicles. Whether it is making driving behavior decisions or global path planning, it is necessary to build on environmental perception. Based on the real-time perception results of the road traffic environment, corresponding judgments, decisions and plans are made to enable the vehicle to achieve intelligent driving.

[0133] The environmental perception system mainly uses various sensors to acquire relevant environmental information, thereby completing the construction of environmental models and the representation of traffic scene knowledge. The sensors used include one or more cameras, single-line radar (SICK), four-line radar (IBEO), and three-dimensional lidar (HDL-64E), etc. Among them, the video or images acquired by the cameras are mainly responsible for traffic light detection, lane line detection, road sign detection, vehicle recognition, etc.; other lidar sensors are mainly responsible for the detection, recognition and tracking of key obstacles in both dynamic and static modes, as well as the detection and extraction of non-key obstacles such as road boundaries, bushes, and surrounding buildings. In this embodiment, road-related perception data captured by a camera, i.e., the image to be detected, is used. This image (which can be an image directly captured by the camera or an image extracted from video captured by the camera; the image to be detected is not limited here) is input into the trained target model constructed in this embodiment. After detection by the trained target model, the predicted result of the real lane lines in the image to be detected (i.e., the predicted lane lines) is output. This predicted result can be directly used as a downstream module of the autonomous vehicle (e.g., an autonomous decision-making system) for further processing, or it can be input into other models in the environmental perception system (e.g., a map construction model). Figure 5 (Not shown) This data is used to construct an occupancy grid map (OGM) that represents the road environment, together with other perceived information. Finally, the OGM is sent to the autonomous decision-making system for further decision-making and planning.

[0134] (2) Autonomous decision-making system

[0135] The autonomous decision-making system is a key component of autonomous vehicles. This system is mainly divided into two core subsystems: behavior decision-making and motion planning. The behavior decision-making subsystem obtains the globally optimal driving route by running a global planning layer to clarify the specific driving task. Then, based on the real-time road information (such as predicted lane lines) sent by the environmental perception system, and on the basis of road traffic rules and driving experience, it decides on a reasonable driving behavior and sends the driving behavior instruction to the motion planning subsystem. The motion planning subsystem, based on the received driving behavior instruction and the current local environmental perception information, plans a feasible driving trajectory based on indicators such as safety and stability, and sends it to the control system.

[0136] (3) Control System

[0137] The control system is specifically divided into two parts: the control subsystem and the execution subsystem. The control subsystem is used to convert the feasible driving trajectory generated by the autonomous decision-making system into specific execution instructions for each execution module and then transmit them to the execution subsystem. After receiving the execution instructions from the control subsystem, the execution subsystem sends them to each controlled object to perform reasonable control of the vehicle's steering, braking, throttle, gear shift, etc., so that the vehicle can drive automatically to complete the corresponding driving operation.

[0138] It should be noted that during the operation of an autonomous vehicle, the accuracy of its driving operation depends primarily on the accuracy of the specific execution instructions generated by the control system for each execution module. This accuracy, in turn, depends on the autonomous decision-making system, which faces uncertainties. These uncertainties mainly include the following aspects: 1) Uncertainties arising from the characteristics and calibration errors of various sensors in the environmental perception system. Different sensors have different perception mechanisms, perception ranges, and corresponding error modes, and the calibration errors introduced by their installation on the autonomous vehicle will ultimately be reflected in the uncertainty of the perceived information; 2) Uncertainties arising from the data processing delay of the environmental perception system. This is because the road environment is complex... The complexity and sheer volume of data in the environmental perception system result in a large computational burden. Furthermore, the constantly changing environment inevitably leads to data delays, impacting the accuracy of the autonomous decision-making system. 3) Different processing methods for perceived information also introduce uncertainty. For example, in this application's embodiment, traditional regression-based or segmentation-based lane detection methods suffer from low lane detection accuracy. Existing lane detection methods cannot meet the increasingly demanding application scenarios of autonomous vehicles. Improving lane detection accuracy can reduce the uncertainty of the autonomous decision-making system, thereby enhancing the accuracy of the specific execution instructions generated by the control system for each execution module.

[0139] It should also be noted that, Figure 5 The overall architecture of the autonomous vehicle shown is for illustrative purposes only. In actual applications, it may contain more or fewer systems / subsystems or modules, and each system / subsystem or module may include multiple components. No specific limitations are made here.

[0140] Based on the above description, embodiments of this application provide a model training method and a lane line detection method, which can be applied to the training stage and inference stage of the target model, respectively, and are described below.

[0141] I. Training Phase

[0142] In this embodiment, the training phase is as described above. Figure 4The process by which the training device 220 performs training operations on the target model 201 using training samples from the training set. Please refer to [link / reference] for details. Figure 6 , Figure 6 A flowchart illustrating a training method for the model provided in this application embodiment may specifically include:

[0143] 601. Label the training samples in the training set with ground truth values. The ground truth values ​​include a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. The ground truth values ​​in the first type of ground truth set are used to characterize feature points belonging to lane lines. The ground truth values ​​in the second type of ground truth set include ground truth values ​​used to characterize the first direction information from the target feature point to the first point on the target lane line. The ground truth values ​​in the third type of ground truth set include ground truth values ​​used to characterize the first distance between the target feature point and the first endpoint of the target lane line. The target feature point is any one of the ground truth values ​​in the first type of ground truth set. The first point is located in the forward direction of the target lane line. The first endpoint is the endpoint of the target lane line in the forward direction. The target lane line is the lane line where the target feature point is located.

[0144] First, each training sample in the training set needs to be labeled with different types of ground truth values. These labeled ground truth values ​​include three types: Type 1, Type 2, and Type 3. The following section details how to perform ground truth labeling:

[0145] (1) The labeling process of the truth set of the first type

[0146] In this embodiment of the application, the truth values ​​in the first type of truth set are used to characterize feature points belonging to the lane line, which can also be called key foreground points.

[0147] Specifically, in some embodiments of this application, the annotation method may be: first, obtaining a lane line point column based on training samples in the training set, which is used to represent the pixel points belonging to the lane line in the training samples, and obtaining feature points based on the lane line point column.

[0148] Specifically, during training, semantic segmentation can first be used to predict points belonging to lane lines and points not belonging to lane lines, thus obtaining a segmentation map, such as... Figure 7As shown. The main function of the segmentation map is to generate candidate starting points for lane line detection (i.e., the starting points for lane line prediction) during lane line prediction. For example, key foreground points can be obtained from the segmentation map output by the model based on non-maximum suppression (NMS), averaging, etc. Each key foreground point can correspond to a predicted lane line. For example, assuming that the ground truth set of the first type includes 8 key foreground points, then 8 predicted lane lines can be predicted based on the transition set and distance set of each key foreground point. Finally, the final lane line prediction result is obtained based on NMS, averaging, fitting, etc.

[0149] It should be noted that in practice, if semantic segmentation annotations for lane lines are available, these annotations can be directly used for ground truth labeling. However, since real-world datasets often only contain lane centerline annotations, training with only lane centerlines results in insufficient data for each training iteration, leading to low training efficiency and slow model convergence. Therefore, in such cases, this embodiment can adaptively widen the lane lines based on empirical parameters. In practice, this widening operation does not affect the lane line prediction results.

[0150] (2) The annotation process of the truth set of the second type

[0151] Since lane lines are elongated and flat, and the receptive field of a CNN is limited, local prediction can finely depict the details of lane lines. Therefore, this application introduces a second type of truth set (i.e., transition set) to describe local information. This local information includes local information about the lane line's forward direction and local information about the lane line's subsequent direction. The specific annotation process is described below:

[0152] A. Set of transitions in the direction of progress

[0153] In this embodiment, the second type of truth set includes truth values ​​representing the directional information (which can be called the first direction information) from the target feature point to a first point on the target lane line. This first point is located in the forward direction of the target lane line, and the target lane line is the lane line where the target feature point is located. The target feature point can be any one of the first type of truth set. In practical applications, any feature point in the first type of truth set can be used as the target feature point. For example, assuming that for segmentation map A, there are 8 feature points (i.e., key foreground points) in the first type of truth set, then each feature point corresponds to its own forward direction information. The forward direction information corresponding to all feature points on the segmentation map can be defined as the transition set of forward directions.

[0154] Specifically, in some embodiments of this application, the annotation method can be found in [reference needed]. Figure 8 As shown: First, an arbitrary feature point is selected as the target feature point S within the truth set of the first type. Then, a circle is drawn with the target feature point S as the center and a preset step size r as the radius. The resulting circle is the target circle. The intersection of this target circle and the center line K of the target lane line in the forward direction is the first point P. Afterward, the first direction information is labeled according to the relative positional relationship between the target feature point S and the first point P. This first direction information belongs to an element of the transition set of the forward direction.

[0155] It should be noted that in some embodiments of this application, the first direction information can be represented using a forward-drive vector. In this representation, this application specifies that, relative to the current target feature point, the feature point in the vehicle's forward direction is the forward-drive feature point, and the vector pointing from the current target feature point to the forward-drive feature point is the forward-drive vector. Through this forward-drive vector, one can move from the current target feature point to the next feature point on the lane centerline in the vehicle's forward direction. The forward-drive vector can be expressed using lateral / longitudinal offset, such as... Figure 9 The vector regression representation shown can also be used for direct angle prediction, such as... Figure 10 The vector classification representation shown in this application does not limit the specific form of the predecessor vector.

[0156] It should also be noted that in the above embodiments of this application, the precursor vector is the vector between the intersection of the lane centerline and the target circle in the forward direction and the target feature point. Representing the precursor vector based on the lane centerline is to enable the model to converge quickly and correct deviations promptly during training. Furthermore, since the lane line has a certain width and its edges are approximately parallel, in other embodiments of this application, the precursor vector can also be the vector between the intersection of the lane line (excluding the centerline) and the target circle in the forward direction and the target feature point, such as... Figure 11 As shown, it could be a line located at approximately 1 / 3 of the right edge of the lane markings (e.g., Figure 11 (as shown in sub-schematic in (a)), or it can be the line at the left third of the lane line (e.g., the line at the left edge of the lane line). Figure 11 (As shown in sub-schematic diagram (b)), the line can also be a line at other locations within the lane lines; this application does not limit the specific location. For ease of explanation, the following descriptions in this application will use the lane centerline as an example.

[0157] It should also be noted that in some other embodiments of this application, each feature point in the truth set of the first type is taken as the target feature point, and the above-described annotation method is repeated until the first direction information corresponding to each feature point is obtained. This first direction information constitutes the transition set of the forward direction, so that the transition set of the forward direction has completeness.

[0158] B. The set of transitions in the successor direction

[0159] In this embodiment, the second type of truth set may further include truth values ​​representing the direction information (which may be called the second direction information) from the target feature point to a second point on the target lane line. This second point is located in the subsequent direction of the target lane line, and the target lane line is the lane line where the target feature point is located. The target feature point can be any one of the first type of truth set. In practical applications, any feature point in the first type of truth set can be used as the target feature point. For example, assuming that for segmentation map A, there are 8 feature points (i.e., key foreground points) in the first type of truth set, then each feature point corresponds to its own second direction information for its forward direction. The second direction information for the forward direction corresponding to each feature point on the segmentation map can be defined as the transition set of the subsequent directions.

[0160] Specifically, in some embodiments of this application, the annotation method can be found in [reference needed]. Figure 12 As shown: First, an arbitrary feature point is selected as the target feature point S within the truth set of the first type. Then, a circle is drawn with the target feature point S as the center and a preset step size r as the radius. The resulting circle is the target circle. The intersection of this target circle and the center line K of the target lane line in the subsequent direction is the second point Q. Afterward, the second direction information is labeled according to the relative positional relationship between the target feature point S and the second point Q. This second direction information belongs to an element of the transition set of the subsequent direction.

[0161] It should be noted that in some embodiments of this application, the second direction information can be represented using a successor vector. In this representation, the application defines the direction opposite to the vehicle's driving direction as the successor direction. Relative to the current target feature point, the feature point in the vehicle's successor direction is the successor feature point, and the vector pointing from the current target feature point to the successor feature point is the successor vector. Through this successor vector, one can move from the current target feature point to the next feature point on the lane centerline in the vehicle's successor direction. The expression of this successor vector can also be represented using lateral / longitudinal offset, or it can be directly predicted by angle; for details, please refer to [reference needed]. Figure 9 , Figure 10 The vector representation of the vector is not elaborated here, and this application does not limit the specific form of the expression of the successor vector.

[0162] It should also be noted that in the above embodiments of this application, the successor vector is the vector between the intersection of the lane centerline and the target circle in the forward direction and the target feature point. Representing the successor vector based on the lane centerline is to enable the model to converge quickly and correct deviations promptly during training. Furthermore, since the lane line has a certain width and its edges are approximately parallel, in other embodiments of this application, the successor vector can also be the vector between the intersection of the lane line (excluding the centerline) and the target circle in the forward direction and the target feature point, such as... Figure 11 As shown, this application does not impose any specific limitations on this.

[0163] It should also be noted that in some other embodiments of this application, each feature point in the truth set of the first type is taken as the target feature point, and the above-described annotation method is repeated until the second direction information corresponding to each feature point is obtained. This second direction information constitutes the transition set of the subsequent direction, so that the transition set of the subsequent direction has completeness.

[0164] It should be noted that in this embodiment, the forward and subsequent directions of the lane lines are based on the road traffic direction (or the vehicle travel direction permitted by traffic regulations). Figure 13 For example, Figure 13 This diagram illustrates two parallel lanes, with traffic flowing in the following directions: Figure 13 As shown by the medium-width arrow, for Figure 13 For the target vehicle, the direction in which the front of the target vehicle is facing is the forward and backward direction of the lane line, and the direction in which the rear of the target vehicle is facing is the subsequent direction of the lane line. In the following embodiments, the forward and backward directions of the lane line are similar, and will not be repeated here.

[0165] The transition set in the forward direction and the transition set in the subsequent direction can be collectively referred to as the transition set, such as... Figure 14 As shown, Figure 14 The illustration shows a schematic diagram of the construction of the transition set (including the transition set of the forward direction and the transition set of the successor direction). Specifically, each feature point in the truth set of the first type is taken as the target feature point, and the above-described annotation method is repeated until the first direction information and the second direction information corresponding to each feature point are obtained. These first direction information and the second direction information constitute the overall transition set.

[0166] It should be noted that, in the embodiments of this application, the transition set in the forward direction is used to predict the forward branch of the lane line, and the transition set in the subsequent direction is used to predict the backward branch of the lane line. In some application scenarios, if the backward branch is not needed, then only the transition set in the forward direction can be included when labeling the second type of truth set for the training samples; in other application scenarios, both the transition set in the forward direction and the transition set in the subsequent direction need to be included when labeling the second type of truth set for the training samples. The specific situation depends on the actual application scenario, and this application does not limit it.

[0167] To facilitate understanding, the construction of predecessor and successor vectors will be introduced below using a specific example:

[0168] Suppose we have an image I∈R H×W×3 And the binary segmentation map S∈R of the image. H×W×1 (Used to distinguish pixels belonging to lane lines from pixels not belonging to lane lines), for any target feature point p on a binary segmentation map i =(x i ,y i For each lane γ ∈ S, first find its corresponding lane line γ. L Here, the shortest distance method is used to find the corresponding target lane line, i.e., the target feature point p. i The lane line closest to a target feature point is considered the corresponding lane line (other methods can also be used to find the lane line corresponding to the target feature point; this application does not limit this method). Lane line γ L The forward and backward endpoints (referring to the points at both ends of the lane line) are denoted as follows: and Here, f represents the forward direction, i.e., the direction the vehicle is traveling, and b represents the backward direction, the direction opposite to the direction the vehicle is traveling. So, for the target feature point p... i =(x i ,y i For any ∈ S, the forward distance scalar D f (p i It can be defined as shown in equation (1):

[0169]

[0170] The backward distance scalar D b (p i It can be defined as shown in equation (2) below:

[0171]

[0172] In order to generate target feature points p iTo find the predecessor and successor vectors of the target feature point, we first need to locate the target feature point on lane line γ. L The forward neighbor feature points and backward neighbor feature points (i.e., feature points adjacent to the target feature point before and after it) are denoted as follows: and It is worth noting that the distance from the target feature point to its immediate and neighboring feature points is a pre-set step size d (d can be customized, such as the length of 8 pixels), that is:

[0173]

[0174]

[0175] Therefore, for the target feature point p i Its predecessor vector It can be expressed as follows (4):

[0176]

[0177] Similarly, target feature point p i successor vector It can be expressed as follows (5):

[0178]

[0179] Similarly, for each feature point, the above-described processing is performed to complete the second type of truth set labeling for the training samples.

[0180] (3) The annotation process of the third type of truth set

[0181] As can be seen from the annotation process of the second type of truth set described above, a lane line can be effectively decoded based on a target feature point and the constructed transition set. However, the transition set is difficult to effectively predict the end of the lane line. Therefore, this application introduces a third type of truth set (i.e., distance set) to describe the distance between the predicted lane line and the current target feature point. The entire prediction process is robust because it only requires coarse global information (since only distance information is needed). In the field of vision of the target vehicle, each lane line has two endpoints. Therefore, the distance set predicted in this application can be divided into two. In the embodiment of this application, the direction of vehicle travel is considered to be forward. Starting from the current point, the farthest point of the lane line that can be seen in the direction of vehicle travel is considered to be the forward endpoint (i.e., Figure 13 The first endpoint shown); the point on the lane line that is furthest away from the current point in the direction the vehicle is reversing is considered the rear endpoint (i.e., the point on the lane line that is furthest away from the current point in the direction the vehicle is reversing). Figure 13 The second endpoint (shown in the diagram) is opposite to the forward distance set and the backward distance set. The specific annotation process is described below:

[0182] A. Forward distance set

[0183] In this embodiment, the third type of truth set includes truth values ​​representing the distance (which may be called the first distance) between the target feature point and the first endpoint on the target lane line. The first endpoint is the endpoint of the target lane line in the forward direction, i.e., the forward endpoint. The target lane line is the lane line where the target feature point is located. The target feature point can be any one of the truth values ​​in the first type. In practical applications, any feature point in the truth value set of the first type can be used as the target feature point.

[0184] Specifically, the first distance can also be called the forward distance scalar, denoted as FDS (e.g., ...). Figure 15 The FDS shown in this application, in some embodiments, can be specifically labeled as follows: First, calculate the first distance (i.e., FDS) between the target feature point and the first endpoint of the target lane line, and take each feature point in the truth set of the first type as the target feature point. Repeat the above steps until the first distance corresponding to each feature point is obtained, and the first distance corresponding to each feature point belongs to the truth set of the third type. That is, traversing the forward distance scalar of each feature point in the truth set of the first type can form a forward distance set FDF∈R. H×W×1 For example, suppose that for a segmentation map A, there are 8 feature points (i.e., key foreground points) in the ground truth set of the first type. Then, each feature point corresponds to a first distance in its own forward direction. The first distances in the forward direction corresponding to all feature points on the segmentation map can be defined as the forward distance set. For the forward distance set, each feature point corresponds to a forward distance scalar, which represents the distance from each feature point to the forward endpoint, or represents the number of iterations required to predict the forward branch of the lane line. The number of iterations is the forward distance scalar divided by the preset step size d.

[0185] B. Back distance set

[0186] In this embodiment, the third type of truth set may further include truth values ​​representing the distance (which may be called the second distance) between the target feature point and the second endpoint on the target lane line. The second endpoint is the endpoint of the target lane line in the subsequent direction, i.e., the subsequent endpoint. The target lane line is the lane line where the target feature point is located. The target feature point can be any one of the truth sets of the first type. In practical applications, any feature point in the truth set of the first type can be used as the target feature point.

[0187] Specifically, the second distance can also be called the backward distance scalar, denoted as BDS (e.g., ...). Figure 15The BDS shown in this application, in some embodiments, can be specifically labeled as follows: First, calculate the second distance (i.e., BDS) between the target feature point and the second endpoint of the target lane line, and take each feature point in the first type of truth set as the target feature point. Repeat the above steps until the second distance corresponding to each feature point is obtained. The second distance corresponding to each feature point also belongs to the third type of truth set. That is, traversing the backward distance scalar of each feature point in the first type of truth set can form a backward distance set BDF∈R. H×W×1 .

[0188] Similarly, for ease of understanding, the construction of forward and backward distance scalars will be introduced below using a specific example:

[0189] Suppose we have an image I∈R H×W×3 And the binary segmentation map S∈R of the image. H×W×1 (Used to distinguish pixels belonging to lane lines from pixels not belonging to lane lines), for any target feature point p on a binary segmentation map i =(x i ,y i If )∈S, first find its corresponding lane line γ. L Here, the shortest distance method is used to find the corresponding target lane line, i.e., the target feature point p. i The lane line closest to a target feature point is considered the corresponding lane line (other methods can also be used to find the lane line corresponding to the target feature point; this application does not limit this method). Lane line γ L The forward and backward endpoints (referring to the points at both ends of the lane line) are denoted as follows: and The specific expressions are shown in equations (1) and (2) above, and will not be elaborated here.

[0190] It should be noted that the above-mentioned process of labeling training samples with ground truth can be done manually or by machine; this application does not impose any restrictions on this.

[0191] In summary, for any training sample in the training set, we can first go through the annotation process of the first type of truth set to obtain at least one feature point belonging to the lane line. Then, for each feature point p... i The predecessor vector corresponding to the feature point is obtained. and successor vector (In some application scenarios, only the predecessor vector can be obtained) The forward distance scalar D corresponding to this feature point f (p i ) and backward distance scalar D b (pi (In some application scenarios, only the forward distance scalar D can be obtained.) f (p i For details, please refer to [link / reference]. Figure 16 By performing the above-described annotation on each feature point, the second type of truth set annotation process and the third type of truth set annotation process can be completed.

[0192] 602. Using the target loss function, the target model is trained on the training set to obtain the trained target model. The target loss function is obtained based on the first loss function, the second loss function, and the third loss function. The first loss function corresponds to the truth set of the first type, the second loss function corresponds to the truth set of the second type, and the third loss function corresponds to the truth set of the third type.

[0193] After labeling the training samples in the training set with ground truth values, the target model can be trained using the training set with the pre-constructed loss function (which can be called the target loss function) until the preset training termination condition is reached, thereby obtaining the trained target model.

[0194] It should be noted that, in the embodiments of this application, the target loss function can be obtained based on the first loss function, the second loss function, and the third loss function, wherein the first loss function corresponds to the truth set of the first type, the second loss function corresponds to the truth set of the second type, and the third loss function corresponds to the truth set of the third type.

[0195] For example, assuming the target loss function is denoted as Loss, the first loss function as Loss1, the second loss function as Loss2, and the third loss function as Loss3, then the target loss function Loss can be expressed as:

[0196] Loss=k1*Loss1+k2*Loss2+k3*Loss3 (6)

[0197] Among them, k1, k2, and k3 are preset coefficients that are greater than zero.

[0198] It should be noted that, in some embodiments of this application, the first loss function can be the cross-entropy loss function L. SEG Specifically, it can be expressed as shown in the following formula (7):

[0199]

[0200] It should also be noted that in some other embodiments of this application, the second loss function or the third loss function can be an L1 norm loss function or an L2 norm loss function; this application does not specifically limit this. Taking the second loss function as an L1 norm loss function as an example, the L1 norm loss function can be denoted as smooth. L1 (x), which can be specifically expressed as shown in the following formula (8):

[0201]

[0202] In summary, the training process of the target model can be as follows: Figure 17 As shown: First, for the training sample X, a ground truth labeling process is performed on the training sample, specifically as described in step 601 above, thereby obtaining a segmentation map (used to obtain feature points), a transition set for the forward direction, a transition set for the subsequent direction, a forward distance set, and a backward distance set (in some embodiments, only the segmentation map, the transition set for the forward direction, and the forward distance set may be obtained; this application does not limit this). Simultaneously, the training sample X is input into the target model and, after passing through different network feature heads, outputs corresponding features. Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 ,feature Figure 5 The target model is trained using the loss function corresponding to each feature map until the training termination condition is met, thus obtaining the trained target model.

[0203] It should be noted that in some embodiments of this application, there are multiple ways to determine the extent to which the target model should be trained using the target loss function. Below are some termination conditions for ending further training of the target model, including but not limited to:

[0204] (1) The target loss function reaches the preset threshold.

[0205] After configuring the target loss function, a threshold (e.g., 0.03) can be set in advance. During the iterative training of the target model, after each training session, it is determined whether the value of the target loss function obtained in the current training round reaches the threshold. If it does not reach the threshold, training continues. If the preset threshold is reached, training is terminated. Then, the network parameter values ​​of the target model determined in the current training round are used as the network parameter values ​​of the final trained target model.

[0206] (2) The target loss function begins to converge.

[0207] After configuring the target loss function, the target model can be trained iteratively. If the difference between the target loss function value obtained in the current training round and the target loss function value obtained in the previous training round is within a preset range (e.g., within 0.01), the target loss function is considered to have converged, and training can be terminated. Then, the network parameter values ​​of the target model determined in the current training round will be used as the network parameter values ​​of the finally trained target model.

[0208] (3) The training has reached the preset number of times.

[0209] In this approach, the number of iterations for training the target model can be pre-configured (e.g., 1000 times). After configuring the target loss function, the target model can be trained iteratively. After each training round, the values ​​of the network parameters of the target model for that round are stored until the number of training iterations reaches the preset number. Then, the target model obtained in each round is validated using test data, and the network parameter values ​​with the best performance are selected as the final network parameter values ​​of the target model.

[0210] It should be noted that, in some embodiments of this application, based on experience, during actual training, this application can specifically use 320*800 image sizes as input training samples, use Adamw as the optimizer, and set the learning rate to 6*e. -4 The batch size is 32, the number of training epochs is 20, and the preset step size of the transition set is set to 10 to train the final target model.

[0211] II. Reasoning Stage

[0212] In this embodiment, the reasoning stage is as described above. Figure 4 The execution device 210 performs lane line detection using a trained target model 201. For each image to be detected, this application can use this trained model to predict lane lines, thereby obtaining the final prediction result. The process is explained in detail below; please refer to [link to relevant documentation]. Figure 18 , Figure 18 A flowchart illustrating a lane line detection method provided in this application embodiment may specifically include:

[0213] 1801. Acquire the target image and obtain the predicted value of the target image based on the trained target model. The predicted value includes a first type of predicted value set, a second type of predicted value set, and a third type of predicted value set. The first type of predicted value set includes predicted feature points used to represent lane lines. The second type of predicted value set includes first predicted direction information corresponding to the target predicted feature points. The third type of predicted value set includes first predicted distance corresponding to the target predicted feature points.

[0214] The execution device acquires a target image and obtains a predicted value for the target image based on a trained target model. This predicted value includes a first type of predicted value set (i.e., a predicted lane segmentation map), a second type of predicted value set (i.e., a predicted transition set), and a third type of predicted value set (i.e., a predicted distance set). The first type of predicted value set includes predicted feature points representing lane lines. The second type of predicted value set includes first predicted direction information corresponding to the target predicted feature points. The third type of predicted value set includes first predicted distances corresponding to the target predicted feature points. The first predicted direction information points to the direction of travel of the target lane line. The first predicted distance is the predicted distance between the endpoint of the target lane line in the direction of travel and the target predicted feature point. The target predicted feature point is any one of the first type of predicted value points. The trained target model is obtained through the above training process.

[0215] It should be noted that during training, since the ground truth set of the second type may include only the transition set in the forward direction, or it may include both the transition set in the forward direction and the transition set in the subsequent direction, in some embodiments of this application, the prediction value set of the second type may include only the prediction transition set in the forward direction, or it may include both the prediction transition set in the forward direction and the prediction transition set in the subsequent direction. Similarly, since the ground truth set of the third type may include only the forward distance set, or it may include both the forward distance set and the backward distance set, in some embodiments of this application, the prediction value set of the third type may include only the forward prediction distance set, or it may include both the forward prediction distance set and the backward prediction distance set. That is, in some embodiments of this application, the prediction value set of the second type may also include second prediction direction information corresponding to the target predicted feature point, and the prediction value set of the third type may also include second prediction distance corresponding to the target predicted feature point. The second prediction direction information is information pointing to the subsequent direction of the target lane line, and the second prediction distance is the prediction distance between the endpoint of the target lane line in the subsequent direction and the target predicted feature point.

[0216] 1802. Based on this prediction value, at least one predicted lane line can be obtained.

[0217] After obtaining the predicted value of the target image, the execution device then obtains at least one predicted lane line based on the predicted value.

[0218] It's important to note that the second type of prediction set can include only the forward direction prediction transfer set, or it can include both forward and subsequent direction prediction transfer sets; similarly, the third type of prediction set can include only the forward prediction distance set, or it can include both forward and backward prediction distance sets. The forward direction prediction transfer set and the forward prediction distance set are used to predict the forward branch of the lane line, while the subsequent direction prediction transfer set and the backward prediction distance set are used to predict the backward branch of the lane line. The following section details how the forward and backward branches of the lane line are predicted:

[0219] (1) Predict the forward branch of the lane line

[0220] In this scenario, the second type of prediction set includes a set of transitions in the forward direction, and the third type of prediction set includes a set of predicted forward distances. First, based on the target predicted feature point, the first predicted direction information, and the first predicted distance in the set of transitions in the forward direction, a forward branch of the target predicted lane line corresponding to the target predicted feature point is obtained. The target predicted lane line is one of at least one predicted lane line. Specifically, the process can be as follows: predict a first predicted point belonging to the target predicted lane line based on the target predicted feature point and the first predicted direction information. Then, using the first predicted point as a new target predicted feature point, the above steps are repeated based on the second type of prediction set until the distance between the last obtained first predicted point and the target predicted feature point reaches the first predicted distance. Finally, the target predicted feature point and each obtained first predicted point are sequentially connected to obtain the forward branch of the target predicted lane line.

[0221] (2) Predicting the backward branch of the lane line

[0222] In this scenario, the second type of prediction set includes a set of transitions in the subsequent direction, and the third type of prediction set includes a set of backward prediction distances. First, based on the target predicted feature point, the second predicted direction information, and the second predicted distance, a backward branch of the target predicted lane line corresponding to the target predicted feature point is obtained. The target predicted lane line is one of at least one predicted lane line. Specifically, the process can be as follows: predict a second predicted point belonging to the target predicted lane line based on the target predicted feature point and the second predicted direction information. Then, using the second predicted point as a new target predicted feature point, the above steps are repeated based on the second type of prediction set until the distance between the last obtained second predicted point and the target predicted feature point reaches the second predicted distance. Finally, the target predicted feature point and each obtained second predicted point are sequentially connected to obtain the backward branch of the target predicted lane line.

[0223] To facilitate understanding, a specific example will be used below to illustrate the lane line prediction process described above. Please refer to [link / reference]. Figure 19 and Figure 20 , Figure 19 This is a schematic flowchart illustrating the lane line detection process provided in an embodiment of this application. Figure 20 This is a schematic diagram illustrating a lane line detection process provided in an embodiment of this application. First, a target image is input into the trained target model. The trained target model then predicts the lane line segmentation map, the prediction transfer set for the forward direction, the prediction transfer set for the subsequent direction, the prediction distance set for the forward direction, and the prediction distance set for the backward direction of the target image. Figure 21 The diagram illustrates the decoding process for a single lane line. During testing, a semantic segmentation map of the lane line is predicted based on the input target image. This semantic segmentation map is then sparsified, for example using NMS, resulting in a very sparse set of key feature points S. Each predicted feature point in S can serve as a starting point for reconstructing the entire lane line. Next, an arbitrary starting point is selected from this set of key feature points S to reconstruct the entire lane line. Assuming the starting point is a predefined target feature point p, the preceding vector corresponding to point p can be obtained based on the predicted transition set of the predicted forward direction and the predicted forward distance set. and forward distance scalar D f (p), based on the forward distance scalar D f (p) and the pre-set step size d can be used to obtain the number of steps T required to restore the forward branch of the entire lane line. f As shown in equation (9):

[0224]

[0225] As can be seen from equation (9), the first adjacent forward-predicted feature point of point p is... The position coordinates can be calculated as shown in equation (10):

[0226]

[0227] And so on, for the (j+1)th forward predicted feature point The position coordinates can be calculated as shown in equation (11):

[0228]

[0229] Through the above process, an ordered set of points can be obtained. By connecting the points in the ordered set of points in sequence, we can obtain the forward branch of the lane line starting from point p.

[0230] Similarly, through similar operations, we can also obtain the ordered set of points of the backward branches of the lane line starting from point p. By connecting the points in the ordered set of points in sequence, we can obtain the backward branches of the lane line starting from point p.

[0231] Finally, starting from point p, connect the obtained forward and backward branch curves to obtain the entire lane line. See the attached document for details. Figure 21 The schematic diagram of the decoding process for a single lane line shown is not described in detail here. This lane line can be represented as shown in the following equation (12):

[0232]

[0233] It should be noted that for each target prediction feature point in the key feature point set, a complete lane line can be recovered using the above method. Then, methods such as NMS can be used to suppress all the obtained lane lines, thereby obtaining the final prediction result as the output.

[0234] It should be noted that the execution device that deploys the above-mentioned trained target model can be various wheeled mobile devices, such as wheeled construction equipment, autonomous vehicles, assisted driving vehicles, etc. Any device that is wheeled and mobile is called a wheeled mobile device as described in this application. Autonomous vehicles can be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawnmowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, and handcarts, etc. This application does not impose any special limitations.

[0235] The methods described in this application have achieved excellent results on academically recognized datasets such as CULane, CurveLane, TuSimple, and Llamas. To provide a more intuitive understanding of the beneficial effects of these embodiments, the technical effects are further compared below on several typical datasets:

[0236] (1) Comparison results of our algorithm and the SOTA algorithm on the CULane dataset

[0237] The CULane dataset contains 133,235 labeled images, divided into 88,880 images for training, 9,675 for validation, and 34,680 for testing. The dataset includes scenarios such as nighttime, daytime, glare, and shadows, and was collected from urban, rural, and highway environments. Due to its representativeness, it has become a widely accepted dataset in the lane line domain. The algorithm significantly improves results on the CULane dataset. Compared to the current state-of-the-art method CondLaneNet, the method in this application (i.e., ...) Figure 22 The RCLaneDet algorithm improved the F1-Measure performance by 1.02 percentage points, with increases of 1.69% and 4.45% in congested road and curved road scenarios, respectively, demonstrating the good performance of the model trained using the method described in this application in predicting curved and occluded lines. Specific comparisons with other methods are shown below. Figure 22 As shown.

[0238] (2) Comparison results of our algorithm and the SOTA algorithm on the CurveLane dataset

[0239] The CurveLanes dataset is a very challenging dataset with many complex scenarios. The method in this application (i.e. Figure 23 RCLaneDet in this paper demonstrates excellent prediction performance for lane lines with complex structures. Compared to the current state-of-the-art method CondLaneNet, this method achieves a 5.33% improvement in the F1-Measure evaluation metric, reaching 91.43%. Specific comparison results with other methods are as follows... Figure 23 As shown.

[0240] (3) Comparison results of our algorithm and the SOTA algorithm on the TuSimple dataset

[0241] The model trained in this application also achieved the best F1-measure on TuSimple. Results on the TuSimple dataset are as follows: Figure 24 As shown.

[0242] (4) Comparison results of our algorithm and the SOTA algorithm on the LLAMAS dataset

[0243] The model trained in this application also achieved the best F1-measure on LLAMAS. Results on the LLAMAS dataset are as follows: Figure 25 As shown.

[0244] In terms of the quality of the predicted lane lines, the method in this application demonstrates excellent prediction performance for curved lines, partially obscured lines, and nearly horizontal lines. A comparison of the results is shown in the figure below. Figure 26 and Figure 27 As shown, Figure 26 This image shows a comparison of the lane prediction performance of the model trained in this application with other methods on the CULane dataset. Figure 27 This image shows a comparison of the lane line prediction performance of the model trained in this application with other methods on the CurvelLane dataset.

[0245] Furthermore, existing lane detection methods are unable to effectively predict curves. Regression-based methods often suffer from inaccurate prediction capabilities due to the limited predictive power of individual prediction points, leading to confusion in far-end predictions. Segmentation-based methods often result in incorrect connections between different lane lines due to instance confusion. While this application's method also predicts segmentation maps, its primary objective is to obtain lane feature points based on the segmentation results. Then, based on one starting point (i.e., the target feature point), it predicts the entire lane line according to the transition set and distance set. This reduces the reliance on segmentation results, significantly decreasing lane line misconnections caused by instance merging errors. Moreover, because each point is local, it can more accurately characterize the local information of the lane line, thereby increasing the accuracy of lane line prediction of curved lines. For details, please refer to [reference needed]. Figure 28 The diagram shown is an improvement of the method of this application for curves.

[0246] It is also important to note that existing lane detection technologies cannot detect lane forks. To differentiate between different lane lines, clustering or instance segmentation is often used. However, clustering is often ineffective for Y-shaped lines or intersections, failing to address real-world driving scenarios. This application addresses this by setting forward and subsequent steps, generating lane line instances bidirectionally based on selected candidate feature points. When encountering a fork in the road, candidate feature points on both arms of the fork can generate instances bidirectionally. At the common portion of the fork, the transfer fields converge, merging into a single lane line. This better aligns with the presupposition of forked lane lines, and experiments demonstrate the superior performance of this method in detecting forked lane lines. For details, please refer to [reference needed]. Figure 29 The diagram shown is an improvement on the bifurcation line method of this application.

[0247] Since the trained target model in this application embodiment can be used to detect lane lines in fields such as smart cities and intelligent driving, the following will introduce several application scenarios that have been implemented in products.

[0248] (1) Autonomous driving

[0249] In autonomous driving, lane detection is crucial for the accurate positioning of autonomous vehicles and its impact on driving decisions. As one of the most important road surface indicators, lane lines effectively guide intelligent vehicles within constrained road areas. Real-time lane line detection is a vital component of intelligent vehicle driver assistance systems. This technology aids in path planning, lane departure warnings, and provides a reference for precise navigation. The target model trained according to the embodiments of this application can be deployed on systems such as... Figure 5 The aforementioned autonomous vehicle is used to perform lane line detection in real time on road-related images captured by the onboard camera.

[0250] (2) Augmented Reality (AR) Navigation

[0251] With the ever-increasing number of cars, people's activity range has expanded accordingly. Relying solely on driving experience to reach one's destination is becoming less common, and people are increasingly reliant on navigation. Traditional navigation systems project the vehicle's position onto a map using Global Positioning System (GPS) signals. Navigation software plans a route based on the set destination, guiding the user through screen displays and voice prompts. This method requires drivers to connect the map guidance and voice prompts with the real world in which their vehicle is located to understand the specific meaning of the guidance. If, while driving, one encounters a fork in the road but fails to hear the navigation prompts clearly, it's easy to miss a lane change opportunity.

[0252] In AR navigation (such as...) Figure 30 By integrating navigation information and lane detection results, the system can calculate the vehicle's precise location and the optimal time to change lanes based on the vehicle's position and lane detection results. This optimizes the presentation of guidance information, reducing the user's reception and comprehension costs; what you see is what you get, and users only need to act according to the navigation information. The combination of accurate lane detection information and reasonable navigation information can effectively guide vehicles to drive within restricted road areas, improving the user experience of current in-vehicle navigation software.

[0253] It should be understood that the autonomous driving and AR navigation described above are only two specific scenarios in which the target model and lane line detection method constructed in the embodiments of this application are applied. The target model trained in the embodiments of this application is not limited to the above scenarios when it is applied, and it can be applied to any scenario that requires lane line recognition and lane line detection.

[0254] Based on the above embodiments, in order to better implement the above solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 31 , Figure 31 This is a schematic diagram of a training device provided in an embodiment of this application. The training device 3100 specifically includes an annotation module 3101 and a training module 3102. The annotation module 3101 is used to annotate the training samples in the training set with ground truth values. The annotated ground truth values ​​include a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. In this embodiment, the ground truth values ​​in the first type of ground truth set are used to characterize feature points belonging to the lane line; these feature points can also be called key foreground points. The second type of ground truth set includes ground truth values ​​used to characterize the direction information (which can be called first direction information) from the target feature point to a first point on the target lane line. The third type of ground truth set includes ground truth values ​​used to characterize the distance (which can be called first distance) between the target feature point and a first endpoint on the target lane line, where the first endpoint is the endpoint of the target lane line in the forward direction, i.e., the forward endpoint. The first point is located in the forward direction of the target lane line, which is the lane line where the target feature point is located. The target feature point is any one of the ground truth sets of the first type. In practical applications, any feature point in the ground truth set of the first type can be used as the target feature point. The training module 3202 is used to train the target model using the training set with a pre-constructed loss function (which can be called the target loss function) until a preset training termination condition is reached, thereby obtaining the trained target model. In this embodiment, the target loss function can be obtained based on the first loss function, the second loss function, and the third loss function, where the first loss function corresponds to the ground truth set of the first type, the second loss function corresponds to the ground truth set of the second type, and the third loss function corresponds to the ground truth set of the third type.

[0255] In one possible design, the annotation module 3101 is specifically used to execute a first annotation method, which includes: determining a first point on the target lane line based on the target feature point, the first point being the intersection of the target circle and the centerline of the target lane line in the forward direction, the target circle being a circle defined with the target feature point as the center and a preset step size as the radius; then, annotating the first direction information based on the relative positional relationship between the target feature point and the first point, the first direction information being an element of the truth set of the second type.

[0256] In one possible design, the annotation module 3101 is further used to: take each feature point in the truth set of the first type as the target feature point, repeat the first annotation method until the first direction information corresponding to each feature point is obtained, and the first direction information corresponding to each feature point belongs to the truth set of the second type, so that the truth set of the second type has completeness in the forward direction.

[0257] In one possible design, the second type of truth set may also include truth values ​​for characterizing the direction information (which may be called the second direction information) from the target feature point to a second point on the target lane line. The second point is located in the subsequent direction of the target lane line, and the target lane line is the lane line where the target feature point is located. The target feature point is any one of the truth sets of the first type. In practical applications, any feature point in the truth set of the first type can be used as the target feature point.

[0258] In one possible design, the annotation module 3101 is specifically used to execute a first annotation method, which may further include: determining a second point on the target lane line based on the target feature point, the second point being the intersection of the target circle and the centerline of the target lane line in the subsequent direction, the target circle being a circle defined with the target feature point as the center and a preset step size as the radius; then, annotating the second direction information based on the relative positional relationship between the target feature point and the second point, the second direction information being an element in the truth set of the second type.

[0259] In one possible design, the annotation module 3101 is further used to: take each feature point in the truth set of the first type as the target feature point, repeatedly execute the first annotation method until the second direction information corresponding to each feature point is obtained, and the second direction information corresponding to each feature point also belongs to the truth set of the second type, so that the truth set of the second type has completeness in subsequent directions.

[0260] In one possible design, the annotation module 3101 is further used to execute a second annotation method, which includes: firstly, calculating a first distance (i.e., FDS) between the target feature point and the first endpoint of the target lane line, and taking each feature point in the truth set of the first type as the target feature point, repeating the above steps until the first distance corresponding to each feature point is obtained, and the first distance corresponding to each feature point belongs to the truth set of the third type.

[0261] In one possible design, the third type of truth set may also include truth values ​​for characterizing the distance (which may be called the second distance) between the target feature point and the second endpoint on the target lane line, which is the endpoint of the target lane line in the subsequent direction, i.e., the subsequent endpoint.

[0262] In one possible design, the annotation module 3101 is further used to execute a second annotation method, which includes: firstly, calculating the second distance (i.e., BDS) between the target feature point and the second endpoint of the target lane line, and taking each feature point in the first type of truth set as the target feature point, repeating the above steps until the second distance corresponding to each feature point is obtained, and the second distance corresponding to each feature point also belongs to the third type of truth set.

[0263] In one possible design, the annotation module 3101 is further used to execute a third annotation method, which includes: first, obtaining a lane line point column based on training samples in the training set, the lane line point column being used to represent the pixel points belonging to the lane lines in the training samples; then, annotating at least one feature point (i.e., a key foreground point) based on the lane line point column, for example, at least one feature point can be annotated from the lane line point column based on NMS, averaging, etc., and the at least one feature point belongs to the ground truth set of the first type.

[0264] It should be noted that the information interaction and execution process between the modules / units in the training device 3100 are different from those in this application. Figure 6 The corresponding method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0265] This application also provides an execution device; please refer to [link to relevant documentation]. Figure 32 , Figure 32This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 3200 includes: an acquisition module 3201 and a prediction module 3202. The acquisition module 3201 is used to acquire a target image and obtain a predicted value of the target image based on a trained target model. The predicted value includes a first type of prediction value set (i.e., a predicted lane line segmentation map), a second type of prediction value set (i.e., a predicted transition set), and a third type of prediction value set (i.e., a predicted distance set). The first type of prediction value set includes predicted feature points representing lane lines. The second type of prediction value set includes first prediction direction information corresponding to the target predicted feature points. The third type of prediction value set includes a first prediction distance corresponding to the target predicted feature points. The first prediction direction information is information pointing to the forward direction of the target lane line. The first prediction distance is the predicted distance between the endpoint of the target lane line in the forward direction and the target predicted feature point. The target predicted feature point is any one of the first type of prediction value sets. The trained target model can be obtained through the above... Figure 6 The training process described in the corresponding embodiment is used to obtain the predicted value of the target image. After obtaining the predicted value, the prediction module 3202 is used to obtain at least one predicted lane line based on the predicted value.

[0266] In one possible design, the prediction module 3202 is specifically used to: obtain a forward branch of the target prediction lane line corresponding to the target prediction feature point based on the target prediction feature point, the first prediction direction information and the first prediction distance, wherein the target prediction lane line is one of the at least one prediction lane lines.

[0267] In one possible design, the prediction module 3202 is further configured to: predict a first prediction point belonging to the target prediction lane line based on the target prediction feature point and the first prediction direction information; then, use the first prediction point as a new target prediction feature point; repeat the above steps based on the second type of prediction value set until the distance between the last obtained first prediction point and the target prediction feature point reaches the first prediction distance; finally, connect the target prediction feature point and each obtained first prediction point in sequence to obtain the forward branch of the target prediction lane line.

[0268] In one possible design, during training, since the second type of truth set can include only the forward direction transition set, or it can include both the forward direction transition set and the subsequent direction transition set, in some embodiments of this application, the second type of prediction value set can include only the forward direction prediction transition set, or it can include both the forward direction prediction transition set and the subsequent direction prediction transition set. Similarly, since the third type of truth set can include only the forward distance set, or it can include both the forward distance set and the backward distance set, in some embodiments of this application, the third type of prediction value set can also include only the forward prediction distance set, or it can include both the forward prediction distance set and the backward prediction distance set. That is, in some embodiments of this application, the second type of prediction value set can also include second prediction direction information corresponding to the target predicted feature point, and the third type of prediction value set can also include second prediction distance corresponding to the target predicted feature point. The second prediction direction information is information pointing to the subsequent direction of the target lane line, and the second prediction distance is the prediction distance between the endpoint of the target lane line in the subsequent direction and the target predicted feature point.

[0269] In one possible design, the prediction module 3202 is further configured to: obtain a backward branch of the target prediction lane line corresponding to the target prediction feature point based on the target prediction feature point, the second prediction direction information and the second prediction distance, wherein the target prediction lane line is one of the at least one prediction lane lines.

[0270] In one possible design, the prediction module 3202 is further configured to: predict a second prediction point belonging to the target prediction lane line based on the target prediction feature point and the second prediction direction information; then, use the second prediction point as a new target prediction feature point; repeat the above steps according to the second type of prediction value set until the distance between the last obtained second prediction point and the target prediction feature point reaches the second prediction distance; finally, connect the target prediction feature point and each obtained second prediction point in sequence to obtain the backward branch of the target prediction lane line.

[0271] It should be noted that the information interaction and execution process between the modules / units in the execution device 3200 are different from those in this application. Figure 18 The corresponding method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0272] Next, we will introduce another training device provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 33 , Figure 33 This is a schematic diagram of a training device provided in an embodiment of this application. The training device 3300 can be equipped with... Figure 31 The training device 3100 described in the corresponding embodiment is used to implement Figure 31 In accordance with the functionality of the training device 3100 in the corresponding embodiment, specifically, the training device 3300 is implemented by one or more servers. The training device 3300 can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 3322 and memories 3332, and one or more storage media 3330 (e.g., one or more mass storage devices) for storing application programs 3342 or data 3344. The memories 3332 and storage media 3330 can be temporary or persistent storage. The program stored in the storage media 3330 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the training device 3300. Furthermore, the CPU 3322 may be configured to communicate with the storage media 3330 and execute the series of instruction operations in the storage media 3330 on the training device 3300.

[0273] The training device 3300 may also include one or more power supplies 3326, one or more wired or wireless network interfaces 3350, one or more input / output interfaces 3358, and / or one or more operating systems 3341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0274] In this embodiment, the central processing unit 3322 is used to execute... Figure 6The training method for the model executed by the training device in the corresponding embodiment. For example, the central processing unit 3322 is used to: label the training samples in the training set with ground truth values, the labeled ground truth values ​​including a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. In the embodiments of this application, the ground truth values ​​in the first type of ground truth set are used to characterize feature points belonging to the lane line, and these feature points can also be called key foreground points. The second type of ground truth set includes ground truth values ​​used to characterize the direction information (which can be called first direction information) from the target feature point to a first point on the target lane line. The third type of ground truth set includes ground truth values ​​used to characterize the distance (which can be called first distance) between the target feature point and a first endpoint on the target lane line. The first endpoint is the endpoint of the target lane line in the forward direction, i.e., the forward endpoint. The first point is located in the forward direction of the target lane line, and the target lane line is the lane line where the target feature point is located. The target feature point can be any one of the first type of ground truth set. In practical applications, any feature point in the first type of ground truth set can be used as the target feature point. Then, using a pre-constructed loss function (which can be called the target loss function), the target model is trained on the training set until a preset training termination condition is reached, thereby obtaining the trained target model. In this embodiment, the target loss function can be obtained based on a first loss function, a second loss function, and a third loss function, wherein the first loss function corresponds to a truth set of a first type, the second loss function corresponds to a truth set of a second type, and the third loss function corresponds to a truth set of a third type.

[0275] It should be noted that the specific manner in which the central processing unit 3322 executes the above steps is different from that described in this application. Figure 6 The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in the above embodiments of this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0276] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 34 , Figure 34 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 3400 can specifically be various terminal devices, such as mobile phones, tablets, laptops, smart wearable devices, monitoring data processing devices, or radar data processing devices, etc., and is not limited here. The execution device 3400 may be equipped with... Figure 32 The execution device 3200 described in the corresponding embodiment is used to implement Figure 32The corresponding embodiment executes the function of device 3200. Specifically, execution device 3400 includes: receiver 3401, transmitter 3402, processor 3403, and memory 3404 (wherein the number of processors 3403 in execution device 3400 can be one or more). Figure 34 (Taking a processor as an example), the processor 3403 may include an application processor 34031 and a communication processor 34032. In some embodiments of this application, the receiver 3401, transmitter 3402, processor 3403, and memory 3404 may be connected via a bus or other means.

[0277] Memory 3404 may include read-only memory and random access memory, and provides instructions and data to processor 3403. A portion of memory 3404 may also include non-volatile random access memory (NVRAM). Memory 3404 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0278] Processor 3403 controls the operation of execution device 3400. In specific applications, the various components of execution device 3400 are coupled together through a bus system, which may include not only data buses but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0279] The above application Figure 18 The methods disclosed in the corresponding embodiments can be applied to or implemented by processor 3403. Processor 3403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 3403 or by instructions in software form. The processor 3403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. This processor 3403 can implement or execute this application. Figure 18The corresponding embodiments disclose various methods, steps, and logic block diagrams. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 3404, and processor 3403 reads information from memory 3404 and, in conjunction with its hardware, completes the steps of the above methods.

[0280] Receiver 3401 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of execution device 3400. Transmitter 3402 can be used to output digital or character information through the first interface; transmitter 3402 can also be used to send instructions to disk group through the first interface to modify data in disk group; transmitter 3402 may also include display devices such as display screen.

[0281] In one embodiment of this application, the processor 3403 is configured to perform lane line detection on the input target image using a trained target model, and obtain the corresponding lane line detection result. The trained target model can be... Figure 6 The corresponding training method is obtained, and the details can be found in the description of the method embodiments shown above in this application, which will not be repeated here.

[0282] This application embodiment also provides a computer-readable storage medium storing a program for performing signal processing, which, when run on a computer, causes the computer to perform the aforementioned actions. Figure 6 The steps performed by the training device described in the illustrated embodiments, or causing the computer to perform the steps as described above. Figure 18 The steps performed by the execution device described in the illustrated embodiment.

[0283] The training device and execution device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuitry. The processing unit can execute computer execution instructions stored in the storage unit, causing the chip within the training device to perform the aforementioned operations. Figure 6 The steps performed by the training device described in the illustrated embodiment, or causing the chip within the execution device to perform as described above. Figure 18 The steps performed by the execution device described in the illustrated embodiment.

[0284] Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0285] For details, please refer to Figure 35 , Figure 35 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 200. The NPU 200 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0286] In some implementations, the arithmetic circuit 2003 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 2003 is a two-dimensional pulsating array. The arithmetic circuit 2003 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2003 is a general-purpose matrix processor.

[0287] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2008.

[0288] Unified memory 2006 is used to store input and output data. Weight data is directly transferred to weight memory 2002 via direct memory access controller (DMAC) 2005. Input data is also transferred to unified memory 2006 via DMAC.

[0289] The bus interface unit 2010 (BIU) is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2009.

[0290] The bus interface unit 2010 is used by the instruction fetch memory 2009 to fetch instructions from external memory, and also by the memory access controller 2005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0291] DMAC is mainly used to move input data from external memory DDR to unified memory 2006, or weight data to weight memory 2002, or input data to input memory 2001.

[0292] The vector computation unit 2007 includes multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0293] In some implementations, the vector computation unit 2007 can store the processed output vector in the unified memory 2006. For example, the vector computation unit 2007 can apply linear and / or nonlinear functions to the output of the computation circuit 2003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 2007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 2003, for example, for use in subsequent layers of the neural network.

[0294] The instruction fetch buffer 2009 connected to the controller 2004 is used to store the instructions used by the controller 2004;

[0295] Unified Memory 2006, Input Memory 2001, Weight Memory 2002, and Instruction Fetch Memory 2009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0296] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0297] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0298] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0299] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0300] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

Claims

1. A model training method applied to lane line detection, characterized in that, include: The training samples in the training set are labeled with ground truth values. The ground truth values ​​include a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. The ground truth values ​​in the first type of ground truth set are used to characterize feature points belonging to lane lines. The ground truth values ​​in the second type of ground truth set include ground truth values ​​used to characterize the first direction information from the target feature point to the first point on the target lane line. The ground truth values ​​in the third type of ground truth set include ground truth values ​​used to characterize the first distance between the target feature point and the first endpoint of the target lane line. The target feature point is any one of the ground truth values ​​in the first type of ground truth set. The first point is located in the forward direction of the target lane line. The first endpoint is the endpoint of the target lane line in the forward direction. The target lane line is the lane line where the target feature point is located. The target model is trained using the training set using the target loss function to obtain the trained target model. The target loss function is obtained based on the first loss function, the second loss function, and the third loss function. The first loss function corresponds to the truth set of the first type, the second loss function corresponds to the truth set of the second type, and the third loss function corresponds to the truth set of the third type.

2. The method according to claim 1, characterized in that, The step of labeling training samples in the training set with ground truth includes performing a second type of ground truth labeling on the training samples in the training set, wherein the first labeling method includes: The first point on the target lane line is determined based on the target feature point. The first point is the intersection of the target circle and the center line of the target lane line in the forward direction. The target circle is a circle determined with the target feature point as the center and a preset step size as the radius. The first direction information is labeled according to the relative positional relationship between the target feature point and the first point, and the first direction information belongs to the truth set of the second type.

3. The method according to claim 2, characterized in that, The method further includes: Each feature point in the truth set of the first type is taken as the target feature point, and the first annotation method is repeated until the first direction information corresponding to each feature point is obtained. The first direction information corresponding to each feature point belongs to the truth set of the second type.

4. The method according to claim 1, characterized in that, The truth set of the second type also includes: The truth value is used to characterize the second direction information from the target feature point to a second point on the target lane line, where the second point is located in the subsequent direction of the target lane line.

5. The method according to claim 4, characterized in that, The step of labeling training samples in the training set with ground truth includes performing a second type of ground truth labeling on the training samples in the training set, and the first labeling method further includes: A second point on the target lane line is determined based on the target feature point. The second point is the intersection of the target circle and the center line of the target lane line in the subsequent direction. The target circle is a circle determined with the target feature point as the center and a preset step size as the radius. The second direction information is labeled according to the relative positional relationship between the target feature point and the second point, and the second direction information belongs to the truth set of the second type.

6. The method according to claim 5, characterized in that, The method further includes: Each feature point in the truth set of the first type is taken as the target feature point, and the first annotation method is repeatedly executed until the second direction information corresponding to each feature point is obtained. The second direction information corresponding to each feature point also belongs to the truth set of the second type.

7. The method according to any one of claims 1-6, characterized in that, The step of labeling training samples in the training set with ground truth also includes performing a third type of ground truth labeling on the training samples in the training set, wherein the second labeling method includes: Calculate the first distance between the target feature point and the first endpoint of the target lane line; Each feature point in the truth set of the first type is taken as the target feature point, and the steps of "calculating the first distance between the target feature point and the first endpoint of the target lane line, and taking each feature point in the truth set of the first type as the target feature point" are repeated until the first distance corresponding to each feature point is obtained, and the first distance corresponding to each feature point belongs to the truth set of the third type.

8. The method according to any one of claims 1-6, characterized in that, The third type of truth set also includes: The second distance is used to characterize the second endpoint between the target feature point and the second endpoint of the target lane line, where the second endpoint is the endpoint of the target lane line in the subsequent direction.

9. The method according to claim 8, characterized in that, The step of labeling training samples in the training set with ground truth also includes performing a third type of ground truth labeling on the training samples in the training set, and the second labeling method further includes: Calculate the second distance between the target feature point and the second endpoint of the target lane line; Each feature point in the truth set of the first type is taken as the target feature point, and the steps of "calculating the second distance between the target feature point and the second endpoint of the target lane line, and taking each feature point in the truth set of the first type as the target feature point" are repeated until the second distance corresponding to each feature point is obtained. The second distance corresponding to each feature point also belongs to the truth set of the third type.

10. The method according to any one of claims 1-6, characterized in that, The annotation of training samples in the training set includes performing a first type of truth annotation on the training samples in the training set, and the third annotation method includes: A lane line point column is obtained based on the training samples in the training set, and the lane line point column is used to represent the pixel points that belong to the lane line in the training samples. At least one feature point is labeled according to the lane line point list, and the at least one feature point belongs to the truth set of the first type.

11. The method according to any one of claims 1-6, characterized in that, The first loss function is the cross-entropy loss function.

12. The method according to any one of claims 1-6, characterized in that, The second loss function or the third loss function includes: an L1 norm loss function, or an L2 norm loss function.

13. A method for detecting lane lines, characterized in that, include: A target image is acquired, and a predicted value of the target image is obtained based on a trained target model. The predicted value includes a first type of predicted value set, a second type of predicted value set, and a third type of predicted value set. The first type of predicted value set includes predicted feature points that represent lane lines. The second type of predicted value set includes first predicted direction information corresponding to the target predicted feature points. The third type of predicted value set includes a first predicted distance corresponding to the target predicted feature points. The first predicted direction information is information pointing to the forward direction of the target lane line. The first predicted distance is the predicted distance between the endpoint of the target lane line in the forward direction and the target predicted feature point. The target predicted feature point is any one of the first type of predicted value sets. The trained target model is trained according to the method of any one of claims 1-12. At least one predicted lane line can be obtained based on the predicted value.

14. The method according to claim 13, characterized in that, The step of obtaining at least one predicted lane line based on the predicted value includes: Based on the target predicted feature point, the first predicted direction information, and the first predicted distance, a forward branch of the target predicted lane line corresponding to the target predicted feature point is obtained, wherein the target predicted lane line is one of the at least one predicted lane lines.

15. The method according to claim 14, characterized in that, The step of obtaining the forward branch of the target predicted lane line corresponding to the target predicted feature point based on the target predicted feature point, the first predicted direction information, and the first predicted distance includes: Based on the target prediction feature points and the first prediction direction information, a first prediction point belonging to the target prediction lane line is predicted. Using the first predicted point as a new target predicted feature point, and according to the second type of predicted value set, repeatedly execute the steps of "predicting the first predicted point belonging to the target predicted lane line according to the target predicted feature point and the first predicted direction information, using the first predicted point as a new target predicted feature point, and according to the second type of predicted value set", until the distance between the last obtained first predicted point and the target predicted feature point reaches the first predicted distance. The target predicted feature points and each obtained first predicted point are sequentially and orderly connected to obtain the forward branch of the target predicted lane line.

16. The method according to any one of claims 13-15, characterized in that, The second type of prediction value set also includes second prediction direction information corresponding to the target prediction feature point, and the third type of prediction value set also includes second prediction distance corresponding to the target prediction feature point. The second prediction direction information is information pointing to the subsequent direction of the target lane line, and the second prediction distance is the prediction distance between the endpoint of the target lane line in the subsequent direction and the target prediction feature point.

17. The method according to claim 16, characterized in that, The step of obtaining at least one predicted lane line based on the predicted value also includes: Based on the target predicted feature point, the second predicted direction information, and the second predicted distance, a backward branch of the target predicted lane line corresponding to the target predicted feature point is obtained, wherein the target predicted lane line is one of the at least one predicted lane lines.

18. The method according to claim 17, characterized in that, The step of obtaining the backward branch of the target predicted lane line corresponding to the target predicted feature point based on the target predicted feature point, the second predicted direction information, and the second predicted distance includes: Based on the target prediction feature point and the second prediction direction information, predict the second prediction point belonging to the target prediction lane line; Using the second predicted point as a new target predicted feature point, and according to the second type of predicted value set, repeatedly execute the steps of "predicting a second predicted point belonging to the target predicted lane line according to the target predicted feature point and the second predicted direction information, using the second predicted point as a new target predicted feature point, and according to the second type of predicted value set", until the distance between the last obtained second predicted point and the target predicted feature point reaches the second predicted distance. The target predicted feature points and each of the obtained second predicted points are sequentially and orderly connected to obtain the backward branch of the target predicted lane line.

19. A training device, characterized in that, include: The annotation module is used to annotate the training samples in the training set with ground truth values. The ground truth values ​​include a first type of ground truth set, a second type of ground truth set, and a third type of ground truth set. The ground truth values ​​in the first type of ground truth set are used to characterize feature points belonging to lane lines. The second type of ground truth set includes ground truth values ​​used to characterize the first direction information from the target feature point to a first point on the target lane line. The third type of ground truth set includes ground truth values ​​used to characterize the first distance between the target feature point and the first endpoint of the target lane line. The target feature point is any one of the ground truth values ​​in the first type of ground truth set. The first point is located in the forward direction of the target lane line. The first endpoint is the endpoint of the target lane line in the forward direction. The target lane line is the lane line where the target feature point is located. The training module is used to train the target model using the training set with the target loss function to obtain the trained target model. The target loss function is obtained based on the first loss function, the second loss function, and the third loss function. The first loss function corresponds to the truth set of the first type, the second loss function corresponds to the truth set of the second type, and the third loss function corresponds to the truth set of the third type.

20. The training device according to claim 19, characterized in that, The annotation module is specifically used to execute a first annotation method, which includes: The first point on the target lane line is determined based on the target feature point. The first point is the intersection of the target circle and the center line of the target lane line in the forward direction. The target circle is a circle determined with the target feature point as the center and a preset step size as the radius. The first direction information is labeled according to the relative positional relationship between the target feature point and the first point, and the first direction information belongs to the truth set of the second type.

21. The training device according to claim 20, characterized in that, The annotation module is further used for: Each feature point in the truth set of the first type is taken as the target feature point, and the first annotation method is repeated until the first direction information corresponding to each feature point is obtained. The first direction information corresponding to each feature point belongs to the truth set of the second type.

22. The training device according to claim 19, characterized in that, The truth set of the second type also includes: The truth value is used to characterize the second direction information from the target feature point to a second point on the target lane line, where the second point is located in the subsequent direction of the target lane line.

23. The training device according to claim 22, characterized in that, The annotation module is specifically used to execute a first annotation method, which includes: A second point on the target lane line is determined based on the target feature point. The second point is the intersection of the target circle and the center line of the target lane line in the subsequent direction. The target circle is a circle determined with the target feature point as the center and a preset step size as the radius. The second direction information is labeled according to the relative positional relationship between the target feature point and the second point, and the second direction information belongs to the truth set of the second type.

24. The training device according to claim 23, characterized in that, The annotation module is further configured to: take each feature point in the truth set of the first type as the target feature point, repeatedly execute the first annotation method until the second direction information corresponding to each feature point is obtained, and the second direction information corresponding to each feature point also belongs to the truth set of the second type.

25. The training device according to any one of claims 19-24, characterized in that, The annotation module is specifically used to execute a second annotation method, which includes: Calculate the first distance between the target feature point and the first endpoint of the target lane line; Each feature point in the truth set of the first type is taken as the target feature point, and the steps of "calculating the first distance between the target feature point and the first endpoint of the target lane line, and taking each feature point in the truth set of the first type as the target feature point" are repeated until the first distance corresponding to each feature point is obtained, and the first distance corresponding to each feature point belongs to the truth set of the third type.

26. The training device according to any one of claims 19-24, characterized in that, The third type of truth set also includes: The second distance is used to characterize the second endpoint between the target feature point and the second endpoint of the target lane line, where the second endpoint is the endpoint of the target lane line in the subsequent direction.

27. The training device according to claim 26, characterized in that, The annotation module is specifically used to execute a second annotation method, which includes: Calculate the second distance between the target feature point and the second endpoint of the target lane line; Each feature point in the truth set of the first type is taken as the target feature point, and the steps of "calculating the second distance between the target feature point and the second endpoint of the target lane line, and taking each feature point in the truth set of the first type as the target feature point" are repeated until the second distance corresponding to each feature point is obtained. The second distance corresponding to each feature point also belongs to the truth set of the third type.

28. The training device according to any one of claims 19-24, characterized in that, The annotation module is specifically used to execute a third annotation method, which includes: A lane line point column is obtained based on the training samples in the training set, and the lane line point column is used to represent the pixel points that belong to the lane line in the training samples. At least one feature point is labeled according to the lane line point list, and the at least one feature point belongs to the truth set of the first type.

29. An execution device, characterized in that, include: An acquisition module is used to acquire a target image and obtain a predicted value of the target image based on a trained target model. The predicted value includes a first type of predicted value set, a second type of predicted value set, and a third type of predicted value set. The first type of predicted value set includes predicted feature points that represent lane lines. The second type of predicted value set includes first predicted direction information corresponding to the target predicted feature points. The third type of predicted value set includes a first predicted distance corresponding to the target predicted feature points. The first predicted direction information is information pointing to the forward direction of the target lane line. The first predicted distance is the predicted distance between the endpoint of the target lane line in the forward direction and the target predicted feature point. The target predicted feature point is any one of the first type of predicted value sets. The trained target model is trained according to the method of any one of claims 1-12. A prediction module is used to obtain at least one predicted lane line based on the predicted value.

30. The execution device according to claim 29, characterized in that, The prediction module is specifically used for: Based on the target predicted feature point, the first predicted direction information, and the first predicted distance, a forward branch of the target predicted lane line corresponding to the target predicted feature point is obtained, wherein the target predicted lane line is one of the at least one predicted lane lines.

31. The execution device according to claim 30, characterized in that, The prediction module is further used for: Based on the target prediction feature points and the first prediction direction information, a first prediction point belonging to the target prediction lane line is predicted. Using the first predicted point as a new target predicted feature point, and according to the second type of predicted value set, repeatedly execute the steps of "predicting the first predicted point belonging to the target predicted lane line according to the target predicted feature point and the first predicted direction information, using the first predicted point as a new target predicted feature point, and according to the second type of predicted value set", until the distance between the last obtained first predicted point and the target predicted feature point reaches the first predicted distance. The target predicted feature points and each obtained first predicted point are sequentially and orderly connected to obtain the forward branch of the target predicted lane line.

32. The execution device according to any one of claims 29-31, characterized in that, The second type of prediction value set also includes second prediction direction information corresponding to the target prediction feature point, and the third type of prediction value set also includes second prediction distance corresponding to the target prediction feature point. The second prediction direction information is information pointing to the subsequent direction of the target lane line, and the second prediction distance is the prediction distance between the endpoint of the target lane line in the subsequent direction and the target prediction feature point.

33. The execution device according to claim 32, characterized in that, The prediction module is further used for: Based on the target predicted feature point, the second predicted direction information, and the second predicted distance, a backward branch of the target predicted lane line corresponding to the target predicted feature point is obtained, wherein the target predicted lane line is one of the at least one predicted lane lines.

34. The execution device according to claim 33, characterized in that, The prediction module is further used for: Based on the target prediction feature point and the second prediction direction information, predict the second prediction point belonging to the target prediction lane line; Using the second predicted point as a new target predicted feature point, and according to the second type of predicted value set, repeatedly execute the steps of "predicting a second predicted point belonging to the target predicted lane line according to the target predicted feature point and the second predicted direction information, using the second predicted point as a new target predicted feature point, and according to the second type of predicted value set", until the distance between the last obtained second predicted point and the target predicted feature point reaches the second predicted distance. The target predicted feature points and each of the obtained second predicted points are sequentially and orderly connected to obtain the backward branch of the target predicted lane line.

35. A training device comprising a processor and a memory, the processor being coupled to the memory, characterized in that, The memory is used to store programs; The processor is configured to execute a program in the memory, causing the training device to perform the method as described in any one of claims 1-12.

36. An execution device comprising a processor and a memory, the processor being coupled to the memory, characterized in that, The memory is used to store programs; The processor is configured to execute a program in the memory, causing the execution device to perform the method as described in any one of claims 13-18.

37. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to perform the method as claimed in any one of claims 1-18.

38. A computer program product comprising instructions that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-18.

39. A chip, the chip comprising a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the method as described in any one of claims 1-18.

Images