Lane line detection method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a lane line detection method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a road image; wherein the road image contains at least one lane line; extracting a feature vector of a preset dimension from the road image according to a preset branch network; wherein the preset branch networks are in one-to-one correspondence with the preset dimensions, the preset dimensions comprise at least one of a detail dimension, a context dimension and a boundary dimension, and feature vectors of the preset dimensions represent image features under the preset dimensions; determining attribute information of lane lines in the road image according to the feature vectors of the preset dimensions; wherein the attribute information comprises position information and / or color information. In combination with the feature vectors of multiple dimensions, the position and color of the lane line are output together, and the detection precision and efficiency of the lane line are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for detecting lane lines. Background Technology

[0002] With the development of autonomous driving technology, high-precision maps, as the core support for vehicle beyond-line-of-sight perception and global path planning, directly affect the safety and reliability of autonomous driving systems in terms of their construction quality.

[0003] In high-precision maps, lane lines are key road elements, requiring accurate detection of their geometric position and color type. How to reduce environmental interference and improve the detection accuracy and efficiency of lane lines is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a lane line detection method, apparatus, device, storage medium, and program product to improve the detection accuracy and efficiency of lane lines.

[0005] In a first aspect, embodiments of this application provide a lane line detection method, including:

[0006] Acquire a road image; wherein the road image contains at least one lane line;

[0007] Based on a preset branch network, feature vectors of a preset dimension are extracted from the road image; wherein, the preset branch network corresponds one-to-one with the preset dimension, and the preset dimension includes at least one of detail dimension, context dimension, and boundary dimension, and the feature vector of the preset dimension represents the image characteristics under the preset dimension;

[0008] Based on the feature vectors of each preset dimension, the attribute information of the lane lines in the road image is determined; wherein, the attribute information includes position information and color information.

[0009] Secondly, embodiments of this application provide a lane line detection device, comprising:

[0010] An acquisition unit is used to acquire a road image; wherein the road image contains at least one lane line;

[0011] An extraction unit is used to extract feature vectors of a preset dimension from the road image according to a preset branch network; wherein the preset branch network corresponds one-to-one with the preset dimension, and the preset dimension includes at least one of detail dimension, context dimension, and boundary dimension, and the feature vector of the preset dimension represents the image characteristics under the preset dimension.

[0012] The detection unit is used to determine the attribute information of lane lines in the road image based on the feature vectors of each preset dimension; wherein the attribute information includes position information and color information.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the processor to perform the implementation method described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the embodiments described in the first aspect above.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the implementation methods described in the first aspect above.

[0018] This application provides a method, apparatus, device, storage medium, and program product for detecting lane lines in road images. It allows for the acquisition of road images containing lane lines, and the extraction of feature vectors of a preset dimension from the road image based on a preset branch network. Each preset branch network corresponds to a preset dimension, which may include detail dimension, context dimension, boundary dimension, etc., thus obtaining features of the road image in different dimensions. This achieves comprehensive feature extraction, avoids information omission, reduces interference from environmental factors, and improves the accuracy of subsequent lane line detection. By combining the feature vectors of each preset dimension, the position and color information of the lane lines in the road image are determined. It eliminates the need to detect lane line color information based on position information, avoiding the amplification of color detection errors by position information detection errors, further improving the accuracy and efficiency of lane line detection. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A schematic flowchart illustrating a lane line detection method provided in an embodiment of this application;

[0021] Figure 2 A schematic flowchart illustrating a lane line detection method provided in an embodiment of this application;

[0022] Figure 3 A schematic diagram illustrating the process of calculating contrastive learning loss in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of a lane line detection device provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] First, let me explain the terms used in this application:

[0028] Standard segmentation head: used to predict a category label for each pixel in the input image, such as whether it is a lane line or the color of the lane line;

[0029] Color contrast learning subhead: Learn a feature representation that makes the features of the same object as similar as possible under different changes, while the features of different objects are as different as possible;

[0030] Contrastive learning mechanism: Used in the input feature space of the color contrast learning subheading, its core is to bring similar samples closer and push dissimilar samples further away in the feature space. It uses the idea of ​​unsupervised contrastive learning to perform supervised color feature decoupling tasks. In the input feature space of the color contrast learning subheading, it implicitly strengthens the category boundaries, thereby greatly enhancing the model's ability to identify the essential features of lane line colors without significantly increasing the computational complexity of model inference.

[0031] As autonomous driving technology advances towards advanced commercialization, high-precision maps, as a core support for vehicle beyond-line-of-sight perception and global path planning, directly impact the safety and reliability of autonomous driving systems. In high-precision maps, lane lines are key road elements, and the detection accuracy of their geometric positions (such as lane line boundaries and curvature) and semantic attributes (such as color and type) is crucial. Specific application scenarios for lane line detection can include:

[0032] 1. Lane detection under complex lighting and weather conditions: such as twilight backlight, alternating light and dark in tunnels, rain and snow cover, traditional methods are prone to detection failure due to changes in lighting or occlusion;

[0033] 2. High-freshness map update requirements: Autonomous driving has extremely high requirements for the real-time performance and integrity of map data, and needs to quickly identify dynamic changes such as wear, aging or new marking of lane lines;

[0034] 3. Multi-category color and composite line type recognition: Real roads contain complex color combinations such as white, yellow, white-yellow, and yellow-white mixed lines. Fine-grained classification is required to meet the detailed annotation needs of high-precision maps.

[0035] Traditional image processing methods mainly rely on manually designed image feature extraction and heuristic rule filtering, which are extremely sensitive to changes in lighting, shadows, and road conditions, have poor robustness, and are difficult to meet the real-time requirements of autonomous driving.

[0036] Furthermore, in the two-stage deep learning-based approach, a semantic segmentation network is first used to output the lane line position mask, and then image patches are cropped within the located region before being fed into a lightweight classification network for color recognition. This makes color recognition heavily reliant on the accuracy of position detection; any positional deviation will lead to misjudgment of color.

[0037] This application provides a lane line detection method, apparatus, equipment, storage medium, and program product, which aims to solve the above-mentioned technical problems of the prior art.

[0038] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart illustrating a lane line detection method provided in an embodiment of this application. This method can be executed by a lane line detection device. Figure 1 As shown, the method includes:

[0040] S101. Obtain a road image; wherein the road image contains at least one lane line.

[0041] For example, road images can be acquired in real time or at regular intervals. For instance, a camera can be installed on a vehicle, and while the vehicle is in motion, the camera can capture images of the road in the current environment.

[0042] A road can have at least one lane line. When acquiring road images, images containing lane lines can be identified and designated as road images; that is, a road image can contain at least one lane line. For example, multiple images captured by a vehicle can be acquired, and the acquired images can be identified. If solid or dashed lines are detected in the image, it is considered to contain lane lines, and this image is the road image for which lane line detection is required. In this embodiment, the method for identifying lines in the image is not specifically limited. Alternatively, images captured by cameras with a preset orientation can all be considered as road images.

[0043] S102. Based on the preset branch network, extract feature vectors of preset dimensions from the road image; wherein, the preset branch network corresponds one-to-one with the preset dimensions, and the preset dimensions include at least one of detail dimension, context dimension, and boundary dimension, and the feature vectors of the preset dimensions represent the image characteristics under the preset dimensions.

[0044] For example, a backbone network is pre-configured, such as the PIDNet architecture. PIDNet is a high-performance convolutional neural network specifically designed for real-time semantic segmentation tasks, which optimizes different features in image segmentation by mimicking the idea of ​​a proportional-integral-differential controller.

[0045] The backbone network can contain one or more pre-defined branch networks, which are also neural network structures. These pre-defined branch networks allow for feature extraction from road images, extracting feature vectors of a pre-defined dimension. Each pre-defined branch network corresponds to a pre-defined dimension, which refers to the aspect of interest in the road image. For example, the pre-defined dimension could be a detail dimension, a context dimension, or a boundary dimension. The detail dimension refers to the dimension of high-resolution spatial details in the road image, the context dimension refers to the semantic dimension of the context in the road image, and the boundary dimension refers to the dimension of boundary features in the road image.

[0046] For each predefined branch network, feature vectors of a predefined dimension are extracted from the road image. This results in feature vectors of multiple predefined dimensions. These feature vectors characterize the features of the road image within a predefined dimension. For example, the feature vector of the detail dimension represents local spatial details in the road image, representing fine-grained features; the feature vector of the context dimension represents global features in the road image, representing coarse-grained features; and the feature vector of the boundary dimension represents boundary enhancement features in the road image.

[0047] Each preset branch network can be processed in parallel or sequentially. In this embodiment, there are no specific limitations on the feature extraction order or the network architecture of each preset branch network.

[0048] In this embodiment, the preset branch network includes a first network, a second network, and a third network. According to the preset branch network, feature vectors of preset dimensions are extracted from the road image, including: performing feature extraction processing on the road image through the first network to obtain feature vectors of detail dimension; performing feature extraction processing on the feature vectors of detail dimension through the second network to obtain feature vectors of context dimension; and performing feature extraction processing on the road image through the third network to obtain feature vectors of boundary dimension.

[0049] Specifically, three types of branch networks can be preset: the first network, the second network, and the third network. The first network is a branch network at the detail dimension, the second network is a branch network at the context dimension, and the third network is a branch network at the boundary dimension.

[0050] The first network performs feature extraction on the road image to obtain feature vectors in the detail dimension. These feature vectors are denoted as... .Right now The output of the first network preserves high-resolution spatial details. After obtaining the feature vectors in the detail dimension, the second network performs feature extraction on these vectors to obtain the feature vectors in the context dimension. These context dimension feature vectors are denoted as... .Right now, For the second network through and The fused features are rich in contextual information, and the second network follows the first network. The third network can be processed in parallel with the first network or in parallel with the second network. Through the third network, feature extraction is performed on the road image to obtain a feature vector along the boundary dimension, denoted as [the feature vector is missing from the original text]. .Right now, This is the boundary enhancement feature output by the third network.

[0051] The backbone network can be denoted as function B(), and the input road image is X. The output can be in the form of triples to represent feature vectors of a preset dimension. The output triples can be denoted as F, F = B(X) = ( ).

[0052] In this embodiment, the design of the pre-defined three-branch network is inspired by the way the human brain processes visual information, namely, processing tasks of different natures in parallel. Instead of attempting to have a large network learn all information simultaneously, such as details, semantics, and boundaries, it decomposes the complex task of semantic segmentation into three sub-tasks and designs a dedicated branch network for each sub-task.

[0053] The advantage of this setup is that by pre-setting three branch networks, the image recognition task can be simplified, allowing each branch network to perform targeted processing, improving the accuracy and efficiency of feature vector extraction, and thus improving the accuracy and efficiency of lane line detection.

[0054] S103. Determine the attribute information of lane lines in the road image based on the feature vectors of each preset dimension; wherein, the attribute information includes position information and color information.

[0055] For example, by combining feature vectors of multiple preset dimensions, lane lines in a road image are detected, and the attribute information of the lane lines is output. The attribute information may include position information and color information. The position information and color information of the lane lines can be determined and output simultaneously, without needing to determine the position information first and then determine the color information based on the position information.

[0056] For example, based on feature vectors of detail, context, and boundary dimensions, target detection can be performed on lane lines in road images to accurately segment lane line regions and obtain lane line location information; and based on feature vectors of detail, context, and boundary dimensions, environmental factors such as illumination, shadows, and wear can be identified to obtain reliable color information.

[0057] When detecting lane line colors, environmental information in the road image can be inferred through feature vectors of various preset dimensions. For example, the weather, light intensity, and shadows can be determined. Based on the inferred environmental information, the color of the road image is adjusted. For example, the color of the road image can be adjusted to warm or cool tones based on the environmental information, and the adjusted color of the lane lines is determined as the lane line color information.

[0058] In this embodiment, a lane line detection model can be pre-built and trained. This model is a neural network model used to detect the position and color information of lane lines. In this embodiment, the model structure of the lane line detection model is not specifically limited.

[0059] This application provides a lane line detection method for detecting the position and color of lane lines in a road image. It acquires a road image containing lane lines and extracts feature vectors of a preset dimension from the road image based on a preset branch network. Each preset branch network corresponds to a preset dimension, which may include detail dimension, context dimension, boundary dimension, etc., thus obtaining features of the road image in different dimensions. This achieves comprehensive feature extraction, avoids information omission, reduces interference from environmental factors, and improves the accuracy of subsequent lane line detection. By combining the feature vectors of each preset dimension, the position and color information of the lane lines in the road image are determined. It eliminates the need to detect lane line color information based on position information, avoiding the amplification of color detection errors by position information detection errors. It simultaneously completes pixel-level positioning of lane lines and recognition of color attributes, further improving lane line detection accuracy.

[0060] Figure 2 This is a flowchart illustrating a lane line detection method provided in an embodiment of this application, as shown below. Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, a lane line detection method is described in detail, the method including:

[0061] S201. Obtain a road image; wherein the road image contains at least one lane line.

[0062] For example, this step can refer to step S101 above, and will not be repeated here.

[0063] S202. Based on the preset branch network, extract feature vectors of preset dimensions from the road image; wherein, the preset branch network corresponds one-to-one with the preset dimensions, and the preset dimensions include at least one of detail dimension, context dimension, and boundary dimension, and the feature vectors of the preset dimensions represent the image characteristics under the preset dimensions.

[0064] For example, this step can refer to step S102 above, and will not be repeated here.

[0065] S203. The feature vectors of each preset dimension are fused to obtain the comprehensive features of the road image; wherein, the comprehensive features represent the characteristics of the image under all preset dimensions.

[0066] For example, after obtaining the feature vectors of each preset dimension, all the obtained feature vectors are fused, and the fused feature vector is used as the comprehensive feature of the road image. That is, the comprehensive feature can characterize all image features of the road image under each preset dimension. For example, the feature vectors of each preset dimension can be concatenated to obtain the comprehensive feature.

[0067] In this embodiment, the feature vectors of each preset dimension are fused to obtain the comprehensive features of the road image, including: inputting the feature vectors of each preset dimension into the feature pyramid network to obtain the comprehensive features of the output road image.

[0068] Specifically, FPN (Feature Pyramid Networks) are pre-constructed, and FPN is denoted as a function. The FPN receives the triplet features F output from the backbone network, i.e., it receives feature vectors of each preset dimension, performs convolutional processing such as feature extraction on F, and generates multi-scale features U, thus obtaining the comprehensive features of the road image. The comprehensive features can be represented as U = In this embodiment, the network structure of FPN is not specifically limited.

[0069] The advantage of this setup is that by using a feature pyramid network, different feature vectors can be fused, making it easier to obtain multi-scale comprehensive features and effectively improving the detection accuracy of lane lines.

[0070] S204. Based on the comprehensive features of the road image and a preset lane line detection model, determine the attribute information of the lane lines in the road image.

[0071] For example, a lane detection model is pre-built and trained. This model is a neural network model used to perform lane detection tasks and output the color and position of the lane lines. The comprehensive features of the road image are input into the pre-built lane detection model, which then outputs the position and color information of the lane lines, thus obtaining the attribute information of the lane lines.

[0072] In the lane line detection model, the segment head can be a standard segment subhead, represented as... The standard segment header receives the comprehensive feature U of the FPN and determines the attribute information. ,Right now, It is a standard subheading The output data.

[0073] In this embodiment, the method further includes: acquiring a training dataset; wherein the training dataset includes comprehensive features and label data of the training image, and the label data represents the attribute information of lane lines in the training image; and training the initial model based on the comprehensive features, label data, and a preset color contrast learning subheading of the training image to obtain a trained lane line detection model.

[0074] Specifically, an initial model is pre-built, and training this initial model yields the lane detection model. During model training, a training dataset needs to be pre-collected. This dataset includes comprehensive features and label data for the training images. The training images are those containing lane lines collected within a historical time period, and the label data represents the actual attribute information of the lane lines in the training images. In other words, training images containing lane lines are pre-collected under different environments, and the lane lines in these images are labeled to obtain the label data for the training images.

[0075] To ensure the model's generalization ability across different scenarios, data collection needs to cover diverse driving environments. In this embodiment, the training images can be RGB (Red-Green-Blue) images captured by the vehicle's forward-facing camera. The image resolution can be 1280×720. The collection scenarios cover different weather conditions, lighting conditions, and seasons, and the road types include urban roads, highways, and rural roads. The road shapes include straight roads, curves, and inclines / declines. The lane lines in the training images can include different colors such as white, yellow, and mixed colors, as well as different types such as solid lines, dashed lines, double lines, and solid-dashed lines.

[0076] In the labeling stage, refined pixel-level semantic segmentation can be employed. For example, labelers can use tools like LabelMe to delineate and classify each lane line in the training image. White can be labeled as 1, yellow as 3, and white-yellow as 4. Specifically, under twilight lighting, a lane line that is essentially white might appear orange-yellow, but the labeler, based on prior knowledge and context, ignores this optical distortion and still labels it as category 1. For composite lane lines composed of two colors, they need to be labeled as a whole, and assigned corresponding category labels according to their standard colors, in left-right order. Finally, a single-channel label map with the same resolution as the original input is generated for each training image, providing accurate supervision signals for the model.

[0077] Before model training, the labeled full training dataset can be divided into two subsets: a training set and a validation set. In this embodiment, a stratified sampling strategy can be used for partitioning to ensure that the proportion of lane lines of each category in the two subsets is basically consistent with the overall training dataset, avoiding class distribution bias caused by random partitioning, and thus more realistically evaluating the model's generalization performance. Specifically, 80% of the training images can be set as the training set and 20% as the validation set, thereby periodically evaluating model performance, performing hyperparameter tuning, and preventing overfitting during training, ensuring the fairness and reliability of the evaluation results.

[0078] The color of lane lines is affected by environmental factors and lighting. During model training, a color contrast learning subheading can be pre-built to learn the color of lane lines using a contrastive learning mechanism, thereby improving color recognition accuracy. For example, for each image to be trained, the comprehensive features of the image are determined based on a pre-defined branch network and FPN. These comprehensive features are then input into the initial model to obtain the lane line attribute information inferred by the initial model. The pre-defined color contrast learning subheading is used to compare and learn with the pre-defined label data and the inferred color information, thus training the model and obtaining a trained lane line detection model. In this embodiment, the architecture of the pre-defined color contrast learning subheading is not specifically limited.

[0079] The beneficial effect of this setup is that the pre-defined color contrast learning subheader can be used to bring similar samples closer together and push dissimilar samples further apart within the feature space. This is a lightweight operation that does not require additional self-supervised signal annotation. It utilizes the idea of ​​unsupervised contrastive learning to perform a supervised color feature decoupling task. In the input feature space of the color contrast learning subheader, class boundaries are implicitly strengthened, thereby greatly enhancing the network's ability to identify the essential features of lane line colors and improving the detection accuracy of lane line colors without significantly increasing the computational complexity of model inference.

[0080] In this embodiment, an initial model is trained based on the comprehensive features of the image to be trained, label data, and a preset color contrast learning subheading to obtain a trained lane detection model. This includes: inputting the comprehensive features of the image to be trained into the initial model to obtain model output data and intermediate feature data; wherein the model output data represents the attribute information detected by the model, and the intermediate feature data represents the feature vector output by the convolutional layer before the model output layer; inputting the comprehensive features of the image to be trained into the preset color contrast learning subheading to obtain color feature data; wherein the color feature data represents the learned color information of the image to be trained; and training the initial model based on the model output data, intermediate feature data, color feature data, and label data of the image to be trained to obtain a trained lane detection model.

[0081] Specifically, the comprehensive features of the image to be trained are input into the initial model, i.e., into the initial standard segmentation head, to obtain the model output data, which represents the attribute information inferred by the model. Intermediate feature data can also be obtained simultaneously. Intermediate feature data is the output data of the convolutional layer before the output layer in the standard segmentation head; for example, it could be the output data of the convolutional layer preceding the model output data. If the comprehensive features are represented as U, the model output data as O, and the intermediate feature data as Z, then... That is, the model output data is the output of the standard segmenter head, i.e., the segmentation result, while the intermediate feature data is the intermediate high-dimensional feature map before the segmentation result is generated.

[0082] The segmentation head for the training task consists of a standard segmentation sub-head and a color contrast learning sub-head operating in parallel. Both sub-heads can receive and process comprehensive features. The comprehensive features of the image to be trained are input into the preset color contrast learning sub-head to obtain the output color feature data. The color feature data represents the learned color information of the image to be trained.

[0083] The initial model is continuously trained using the model's output data, intermediate feature data, color feature data, and label data from the image to be trained, ultimately resulting in a trained lane detection model. For example, the model can be trained using backpropagation based on its output data and intermediate feature data, or it can be trained using both intermediate feature data and color feature data, ultimately yielding the lane detection model.

[0084] The advantage of this setup is that by training the model using two different segmentation heads, precise supervision of the model can be achieved, improving the training accuracy of the model and thus improving the detection accuracy of lane lines.

[0085] In this embodiment, the comprehensive features of the image to be trained are input into a preset color contrast learning subheader to obtain color feature data. This includes: acquiring historical output data of the image to be trained; wherein, the historical output data represents the model output data obtained in the previous iteration during model training; the historical output data includes multiple anchor points, each representing the area where lane lines are located in the image to be trained; for each anchor point in the historical output data, a positive sample set and a negative sample set are determined based on the color information at each anchor point in the historical output data; wherein, the color information of the anchor points in the positive sample set is the same as the color information of each anchor point, and the color information of the anchor points in the negative sample set is different from the color information of each anchor point; based on the positive sample set and the negative sample set of each anchor point, the comprehensive features of the image to be trained are processed by a contrast learning mechanism based on the preset color contrast learning subheader to obtain color feature data.

[0086] Specifically, during each iteration of model training, the model output data for that training session can be obtained. Each time a model output data point is obtained, it is treated as historical output data, which can be directly retrieved during the next model training iteration. In other words, historical output data represents the model output data obtained in the previous iteration during the model training process.

[0087] For each historical output data point, multiple anchor points can be included. Anchor points are not individual pixels, but rather represent regions in the image to be trained where lane lines are located; for example, they can be connected pixel regions that are not part of the background. For each anchor point in the historical output data, the color information at each anchor point is obtained, i.e., the color information of each lane line region inferred during the previous training. For each anchor point in the historical output data, based on the color information at each anchor point, a positive sample set and a negative sample set are determined for that anchor point. That is, each anchor point can correspond to one positive sample set and one negative sample set.

[0088] Each anchor point is sequentially used as a candidate anchor point. Both the positive and negative sample sets of the candidate anchor points contain multiple anchor points, excluding the candidate anchor point itself. The color information of anchor points in the positive sample set is the same as the color information of the candidate anchor point, while the color information of anchor points in the negative sample set is different from the color information of the candidate anchor point. That is, feature vectors of non-background connected pixel regions with the exact same color as the candidate anchor point are searched from historical output data, thus bringing samples of the same category closer together and ensuring that the model can learn the unique features of each specific category; feature vectors of non-background connected pixel regions with different colors are searched from historical output data, thus separating samples of all different categories, increasing the discriminative power of the features, and enhancing the model's discriminative power and robustness.

[0089] If the historical output data contains anchor points with unknown colors, these anchor points can be used as negative samples for all anchor points, thereby encouraging the model to learn the unknown color category and improving the model's detection performance.

[0090] Based on the positive and negative sample sets of each anchor point, a pre-defined color contrast learning subheading is used to process the comprehensive features of the training image using a contrastive learning mechanism to obtain color feature data. In other words, the pre-defined color contrast learning subheading allows for contrastive learning of color information in the positive and negative sample sets, enabling the network to ultimately achieve robust features with high invariance to appearance changes such as illumination, shadows, and wear. In this embodiment, the specific processing procedure of the contrastive learning mechanism is not limited.

[0091] The beneficial effect of this setup is that, in the contrastive learning mechanism, a positive sample set and a negative sample set for color are determined, so that the features of the same object under different changes are as similar as possible, while the features of different objects are as different as possible, thereby improving the training accuracy of the model and thus improving the color detection accuracy of lane lines.

[0092] In this embodiment, an initial model is trained based on the model output data, intermediate feature data, color feature data, and label data of the image to be trained to obtain a trained lane detection model. This includes: determining a first loss value based on the model output data and label data of the image to be trained, using a preset cross-entropy loss function; wherein the first loss value represents the accuracy of determining the position information of the lane lines; determining a second loss value based on the intermediate feature data and color feature data of the image to be trained, using a preset color contrast loss function; wherein the second loss value represents the accuracy of determining the color information of the lane lines; and training the initial model based on the first and second loss values ​​to obtain a trained lane detection model.

[0093] Specifically, different loss functions can be preset to comprehensively train the model and improve its training accuracy. For example, a Cross Entropy (CE) loss function can be preset. The cross-entropy loss function is used to supervise the segmentation results of the standard segmenter head; that is, loss is calculated on the model's output data and label data of the image to be trained, and the result is the first loss value. The first loss value characterizes the accuracy of determining the lane line position information. The first loss value can be expressed as... Where O represents the model output data and Y represents the label data.

[0094] Intermediate feature data is denoted as Z, and color feature data is denoted as E. Both Z and E are inputs for contrastive learning. A color contrast loss function is pre-set. Based on the pre-set color contrast loss function, the loss of contrastive learning is calculated on Z and E. The calculation result is the second loss value, which represents the accuracy of determining the color information of the lane lines. Figure 3 A flowchart illustrating the process of calculating the comparative learning loss. Figure 3 In this process, the comprehensive features of the image to be trained are input into the standard segmentation subhead and the color contrast learning subhead respectively. The standard segmentation subhead outputs Z, and the color contrast learning subhead outputs E. The contrast learning loss is calculated by combining Z and E.

[0095] In this embodiment, a second loss value can also be calculated based on the positive and negative sample sets. The second loss value can be characterized as... The formula for calculating the second loss value can be:

[0096] ;

[0097] ;

[0098] Where i represents the anchor point in the historical output data, and A represents the number of anchor points. Represents the positive sample set. Let p represent the negative sample set, n represent the anchor points in the positive sample set, and n represent the anchor points in the negative sample set. The feature vector representing the i-th anchor point is the color information of the i-th anchor point. The feature vector representing the p-th anchor point in the positive sample set. The feature vector representing the nth anchor point in the negative sample set. τ>0 is a temperature hyperparameter used to adjust the sensitivity of the loss function to hard negative samples; a smaller τ value makes the model focus more on distinguishing very similar sample pairs from very dissimilar sample pairs.

[0099] By combining the first and second loss values, a total loss value is obtained. This total loss value is then used to train the model, adjusting the model parameters until the final lane detection model is obtained. For example, the first and second loss values ​​can be weighted to calculate the total loss value.

[0100] The beneficial effect of this setup is that by using the cross-entropy loss function and the color contrast loss function to train the model comprehensively, the model's accuracy in detecting lane line positions and color is effectively improved.

[0101] In this embodiment, the initial model is trained based on the first loss value and the second loss value to obtain a trained lane detection model. This includes: obtaining the model parameters of the initial model; determining the third loss value based on the model parameters and a preset regularization loss function; wherein the third loss value represents the degree of fit of the model; and training the initial model based on the first loss value, the second loss value, and the third loss value to obtain a trained lane detection model.

[0102] Specifically, the model's total loss function can be constructed by a weighted sum of three loss values: a first loss value, a second loss value, and a third loss value. The first loss value is the lane line segmentation loss, the second loss value is the color contrast loss, and the third loss value is the regularization loss.

[0103] For the third loss value, a regularized loss function can be used to apply variance regularization to the model parameters to prevent overfitting. That is, the initial model parameters can be obtained, and based on these parameters, a pre-defined regularized loss function can be applied to determine the third loss value. The third loss value can be characterized as... .in, Characterizing model parameters, It is the set of parameters for the model. These are preset parameters.

[0104] By combining the first, second, and third loss values, a total loss value is obtained. This total loss value is then used to train the initial model, resulting in a trained lane detection model. For example, the first, second, and third loss values ​​can be weighted and summed, or the weight coefficients of any of these three loss values ​​can be preset.

[0105] The formula for calculating the total loss value can be:

[0106] ;

[0107] Where L is the total loss value, These are preset weighting coefficients for the second loss value, used to balance the learning of segmentation accuracy and feature discriminativeness.

[0108] The beneficial effect of this setup is that, through this multi-task learning framework, not only is the model guided on how to accurately segment lane line regions, but the model is also forced to become highly sensitive to intra-class consistency and inter-class differences in color attributes. This makes the model highly robust to appearance changes such as lighting, shadows, and wear, so that during inference, it can output accurate position and reliable color information simultaneously with just a single forward propagation.

[0109] This application provides a lane line detection method for detecting the position and color of lane lines in a road image. It acquires a road image containing lane lines and extracts feature vectors of a preset dimension from the road image based on a preset branch network. Each preset branch network corresponds to a preset dimension, which may include detail dimension, context dimension, boundary dimension, etc., thus obtaining features of the road image in different dimensions. This achieves comprehensive feature extraction, avoids information omission, reduces interference from environmental factors, and improves the accuracy of subsequent lane line detection. By combining the feature vectors of each preset dimension, the position and color information of the lane lines in the road image are determined. It eliminates the need to detect the color information of lane lines based on position information, avoiding the amplification of color detection errors by position information detection errors, and further improving the lane line detection accuracy.

[0110] Figure 4 This is a schematic diagram of the structure of a lane line detection device provided in an embodiment of this application, as shown below. Figure 4 As shown, the lane line detection device 40 provided in this embodiment includes:

[0111] The acquisition unit 401 is used to acquire a road image; wherein the road image contains at least one lane line;

[0112] Extraction unit 402 is used to extract feature vectors of a preset dimension from a road image according to a preset branch network; wherein, the preset branch network corresponds one-to-one with the preset dimension, and the preset dimension includes at least one of detail dimension, context dimension, and boundary dimension, and the feature vector of the preset dimension represents the image characteristics under the preset dimension.

[0113] The detection unit 403 is used to determine the attribute information of lane lines in the road image based on the feature vectors of each preset dimension; wherein the attribute information includes position information and color information.

[0114] In one possible implementation, the detection unit 403 includes:

[0115] The feature fusion module is used to fuse feature vectors from various preset dimensions to obtain comprehensive features of the road image; the comprehensive features represent the characteristics of the image across all preset dimensions.

[0116] The lane line detection module is used to determine the attribute information of lane lines in road images based on the comprehensive features of the road images and a preset lane line detection model.

[0117] In one possible implementation, the feature fusion module is specifically used for:

[0118] The feature vectors of each preset dimension are input into the feature pyramid network to obtain the comprehensive features of the output road image.

[0119] One possible implementation also includes:

[0120] The dataset acquisition unit is used to acquire the training dataset; the training dataset includes the comprehensive features and label data of the training image, and the label data represents the attribute information of the lane lines in the training image;

[0121] The model training unit is used to train the initial model based on the comprehensive features of the image to be trained, the label data, and the preset color contrast learning head, so as to obtain the trained lane line detection model.

[0122] In one possible implementation, the model training unit includes:

[0123] The model output module is used to input the comprehensive features of the image to be trained into the initial model to obtain model output data and intermediate feature data. The model output data represents the attribute information detected by the model, and the intermediate feature data represents the feature vector output by the convolutional layer located before the model output layer.

[0124] The contrast learning module is used to input the comprehensive features of the image to be trained into a preset color contrast learning subheader to obtain color feature data; wherein, the color feature data represents the color information of the image to be trained learned;

[0125] The model training module is used to train the initial model based on the model output data, intermediate feature data, color feature data, and label data of the image to be trained, so as to obtain the trained lane line detection model.

[0126] In one possible implementation, the contrastive learning module is specifically used for:

[0127] Obtain historical output data of the image to be trained; wherein, historical output data represents the model output data obtained in the previous iteration during the model training process; the historical output data includes multiple anchor points, and the anchor points represent the areas where the lane lines are located in the image to be trained;

[0128] For each anchor point in the historical output data, the positive sample set and negative sample set for each anchor point are determined based on the color information of each anchor point in the historical output data; wherein, the color information of the anchor points in the positive sample set is the same as the color information of each anchor point, and the color information of the anchor points in the negative sample set is different from the color information of each anchor point.

[0129] Based on the positive and negative sample sets of each anchor point, the color feature data is obtained by processing the comprehensive features of the training image using a contrastive learning mechanism based on a preset color contrast learning subhead.

[0130] In one possible implementation, the model training module is specifically used for:

[0131] Based on the model output data and label data of the image to be trained, a first loss value is determined based on a preset cross-entropy loss function; wherein, the first loss value represents the accuracy of determining the position information of the lane lines;

[0132] Based on the intermediate feature data and color feature data of the image to be trained, a second loss value is determined based on a preset color contrast loss function; wherein, the second loss value represents the accuracy of determining the color information of the lane lines;

[0133] The initial model is trained based on the first and second loss values ​​to obtain the trained lane detection model.

[0134] In one possible implementation, the model training module is specifically used for:

[0135] Obtain the model parameters of the initial model, and determine the third loss value based on the model parameters and a preset regularized loss function; whereby the third loss value characterizes the degree of fit of the model.

[0136] The initial model is trained based on the first loss value, the second loss value, and the third loss value to obtain the trained lane detection model.

[0137] In one possible implementation, the preset branch network includes a first network, a second network, and a third network; the extraction unit 402 includes:

[0138] The first extraction module is used to perform feature extraction processing on the road image through the first network to obtain feature vectors in detail dimension;

[0139] The second extraction module is used to perform feature extraction processing on the feature vector of the detail dimension through the second network to obtain the feature vector of the context dimension.

[0140] The third extraction module is used to perform feature extraction processing on the road image through a third network to obtain feature vectors in the boundary dimension.

[0141] The lane line detection device provided in this embodiment can perform the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0142] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0143] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0144] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0145] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0146] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0147] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0148] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0149] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0150] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0151] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0152] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0155] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0157] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for detecting lane lines, characterized in that, include: Acquire a road image; wherein the road image contains at least one lane line; Based on a preset branch network, feature vectors of a preset dimension are extracted from the road image; wherein, the preset branch network corresponds one-to-one with the preset dimension, and the preset dimension includes at least one of detail dimension, context dimension, and boundary dimension, and the feature vector of the preset dimension represents the image characteristics under the preset dimension; Based on the feature vectors of each preset dimension, the attribute information of the lane lines in the road image is determined; wherein, the attribute information includes position information and color information.

2. The method according to claim 1, characterized in that, Based on the feature vectors of each preset dimension, the attribute information of lane lines in the road image is determined, including: The feature vectors of each preset dimension are fused to obtain the comprehensive features of the road image; wherein, the comprehensive features characterize the features of the image under all preset dimensions; Based on the comprehensive features of the road image and a preset lane detection model, the attribute information of the lane lines in the road image is determined.

3. The method according to claim 2, characterized in that, The feature vectors of each preset dimension are fused to obtain the comprehensive features of the road image, including: The feature vectors of each preset dimension are input into the feature pyramid network to obtain the comprehensive features of the output road image.

4. The method according to claim 2, characterized in that, Also includes: Obtain the training dataset; wherein the training dataset includes comprehensive features and label data of the training image, and the label data represents the attribute information of lane lines in the training image; Based on the comprehensive features, label data, and preset color contrast learning subheading of the image to be trained, the initial model is trained to obtain the trained lane detection model.

5. The method according to claim 4, characterized in that, Based on the comprehensive features, label data, and preset color contrast learning subheading of the image to be trained, the initial model is trained to obtain a trained lane detection model, including: The comprehensive features of the image to be trained are input into the initial model to obtain model output data and intermediate feature data; wherein, the model output data represents the attribute information detected by the model, and the intermediate feature data represents the feature vector output by the convolutional layer located before the model output layer; The comprehensive features of the image to be trained are input into a preset color contrast learning subheader to obtain color feature data; wherein, the color feature data represents the learned color information of the image to be trained; The initial model is trained based on the model output data, intermediate feature data, color feature data, and label data of the image to be trained, to obtain a trained lane detection model.

6. The method according to claim 5, characterized in that, The comprehensive features of the image to be trained are input into a preset color contrast learning subheader to obtain color feature data, including: Obtain historical output data of the image to be trained; wherein, the historical output data represents the model output data obtained in the previous iteration during the model training process; the historical output data includes multiple anchor points, and the anchor points represent the areas where the lane lines are located in the image to be trained; For each anchor point in the historical output data, a positive sample set and a negative sample set for each anchor point are determined based on the color information at each anchor point in the historical output data; wherein, the color information of the anchor points in the positive sample set is the same as the color information of each anchor point, and the color information of the anchor points in the negative sample set is different from the color information of each anchor point. Based on the positive and negative sample sets of each anchor point, the comprehensive features of the image to be trained are processed by a contrastive learning mechanism based on the preset color contrast learning subhead to obtain the color feature data.

7. The method according to claim 5, characterized in that, Based on the model output data, intermediate feature data, color feature data, and label data of the image to be trained, the initial model is trained to obtain a trained lane detection model, including: Based on the model output data and label data of the image to be trained, a first loss value is determined based on a preset cross-entropy loss function; wherein, the first loss value represents the accuracy of determining the position information of the lane lines; Based on the intermediate feature data and color feature data of the image to be trained, a second loss value is determined based on a preset color contrast loss function; wherein, the second loss value represents the accuracy of determining the color information of the lane lines; The initial model is trained based on the first loss value and the second loss value to obtain a trained lane detection model.

8. The method according to any one of claims 1-7, characterized in that, The preset branch network includes a first network, a second network, and a third network; Based on a preset branch network, feature vectors of a preset dimension are extracted from the road image, including: The road image is processed by the first network to obtain the feature vector of the detail dimension; The second network is used to perform feature extraction processing on the feature vector of the detail dimension to obtain the feature vector of the context dimension; The third network is used to perform feature extraction processing on the road image to obtain the feature vector of the boundary dimension.

9. A lane line detection device, characterized in that, include: An acquisition unit is used to acquire a road image; wherein the road image contains at least one lane line; An extraction unit is used to extract feature vectors of a preset dimension from the road image according to a preset branch network; wherein the preset branch network corresponds one-to-one with the preset dimension, and the preset dimension includes at least one of detail dimension, context dimension, and boundary dimension, and the feature vector of the preset dimension represents the image characteristics under the preset dimension. The detection unit is used to determine the attribute information of lane lines in the road image based on the feature vectors of each preset dimension; wherein the attribute information includes position information and color information.

10. An electronic device / computer-readable storage medium / computer program product, characterized in that, The electronic device includes: a memory, a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8; and / or, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8; and / or, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images