Training method, lane line detection method, electric equipment and storage medium

Abstract

The invention discloses a training method of a lane line detection deep learning model and a lane line detection method. The training method of the lane line detection deep learning model comprises the following steps: configuring a channel attention mechanism module in a convolutional layer and / or a residual layer of a basic model to obtain a to-be-trained lane line detection deep learning model; and training the to-be-trained lane line detection deep learning model according to the lane line data training set and preset training parameters to obtain the lane line detection deep learning model. Thus, the channel attention mechanism module is configured in the basic model, the model automatically focuses on the lane line core features in the training process, interference of interference features on model judgment is reduced, and the lane line detection deep learning model obtained through training can output an accurate image lane line point set. The problem of low detection data accuracy caused by environmental interference is solved to a certain extent, and the lane line detection accuracy of the model in a complex environment is improved.

Description

Training methods, lane detection methods, electrical equipment and storage media Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a training method for a lane detection deep learning model, a lane detection method, an electrical device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Lane line data is a core perception data point supporting the precise positioning, path planning, and safety warning of autonomous vehicles, and its detection accuracy directly affects the operational reliability of the autonomous driving system. However, lane line detection is susceptible to environmental interference, leading to inaccurate detection data and failing to provide stable and reliable lane line data support for autonomous driving. Summary of the Invention

[0003] This application provides a training method for a deep learning model for lane detection, a lane detection method, an electrical device, a computer-readable storage medium, and a computer program product.

[0004] This application provides a training method for a lane detection deep learning model. The method includes: configuring a channel attention mechanism module in the convolutional layer and / or residual layer of a base model to obtain a lane detection deep learning model to be trained; and training the lane detection deep learning model to be trained according to a lane line data training set and preset training parameters to obtain the lane detection deep learning model.

[0005] Thus, by configuring a channel attention mechanism module in the basic model, the model can automatically focus on the core features of lane lines during training, enhancing the accuracy of extracting key information about lane lines. At the same time, it effectively suppresses the negative impact of environmental interference such as lighting and shadows on feature extraction, reducing the interference of interfering features on the model's judgment. This enables the deep learning model for lane line detection trained to output accurate image lane line point sets, solving to some extent the problem of low detection data accuracy caused by environmental interference. This improves the accuracy of lane line detection in complex environments and provides a foundation for lane line perception in autonomous driving systems.

[0006] In some embodiments, training the lane line detection deep learning model to be trained based on the lane line data training set and preset training parameters to obtain the lane line detection deep learning model includes: configuring the loss function of the lane line detection deep learning model to be trained, wherein the loss function includes a base loss function and / or a Dessian loss function; and training the lane line detection deep learning model to be trained based on the lane line data training set, the preset training parameters, and the loss function to obtain the lane line detection deep learning model.

[0007] Thus, by configuring a combination of classification loss function, structural loss function, auxiliary segmentation loss function, and Descein loss function, the problems of sample imbalance and insufficient boundary optimization in conventional training are solved to a certain extent. Furthermore, the synergistic optimization of multiple loss functions enables the model to comprehensively learn the classification features, morphological features, detailed features, and boundary features of lane lines during training, thereby improving the model's detection accuracy of lane line data to a certain extent. This allows the model to more accurately identify the edge contours of lane lines, enhances the model's adaptability to complex environments, and enables the model to accurately output lane line point sets in complex environments. Consequently, the reliability of lane line detection is improved to a certain extent, ensuring the safe driving of autonomous vehicles under various road conditions.

[0008] This application provides a lane line detection method, the method comprising: fusing lane line data according to a first lane line point set, a second lane line point set and navigation data information to obtain a target lane line point set, thereby obtaining the lane line detection information, wherein the first lane line point set is obtained by processing lane line image data by a lane line detection deep learning model described in some of the above embodiments, and the second lane line point set is obtained by processing lane line lidar point cloud data.

[0009] In this way, by integrating multi-source data, the limitations of single sensor data are made up for, and the superimposed effects of environmental interference are effectively suppressed to a certain extent. This makes the output target lane line point set stable and accurate, and to a certain extent, it can accurately reflect the actual state of the lane lines. This provides reliable lane line data support for the accurate positioning, path planning and safety warning of autonomous vehicles, and improves the operational safety and stability of the autonomous driving system.

[0010] In some embodiments, the method further includes: performing time alignment processing on initial lane line image data and initial lane line LiDAR point cloud data to obtain the lane line LiDAR point cloud data; performing a first filtering process on the lane line LiDAR point cloud data based on spatial information to obtain a third lane line point set; transforming the third lane line point set from the LiDAR coordinate system to the camera coordinate system to obtain a fourth lane line point set, wherein the LiDAR coordinate system is a coordinate system established by the LiDAR that acquired the initial lane line LiDAR point cloud data, and the camera coordinate system is a coordinate system established by the camera that acquired the initial lane line image data; transforming the fourth lane line point set from the camera coordinate system to the pixel coordinate system to obtain a fifth lane line point set, wherein the pixel coordinate system is the coordinate system where the initial lane line image data is located; and performing a second filtering process on the fifth lane line point set according to a preset target area to obtain a second lane line point set.

[0011] In this way, by performing a series of processes on the initial data, such as time alignment, spatial filtering, coordinate transformation, and target area filtering, the problems of asynchronous time, inconsistent coordinate systems, and data redundancy of the initial data are effectively solved to a certain extent. This ensures the timeliness and consistency of the data, eliminates invalid and redundant points in the initial data, improves the data quality, and obtains an accurate and effective set of second lane line points. This, in turn, improves the effect of multi-source data fusion to a certain extent and ensures the accuracy and reliability of the final lane line detection results.

[0012] In some embodiments, the step of fusing lane line data based on a first lane line point set, a second lane line point set, and navigation data information to obtain a target lane line point set, in order to obtain the lane line detection information, includes: performing point set matching processing on the first lane line point set and the second lane line point set to obtain a sixth lane line point set; transforming the sixth lane line point set from the pixel coordinate system to the vehicle coordinate system to obtain a seventh lane line point set, wherein the vehicle coordinate system is the coordinate system in which the vehicle is located; and fusing the navigation data and the seventh lane line point set to obtain the target lane line point set.

[0013] In this way, by matching point sets, the data deviation between the first and second lane line point sets is eliminated to a certain extent, achieving precise coordination between the two types of sensor data. At the same time, the lane line point set data is transformed into the vehicle coordinate system, making the lane line data directly related to the vehicle state, which is more in line with the decision-making needs of autonomous driving. In addition, the lane line data is processed in conjunction with navigation data, providing a global reference for the lane line data and ensuring the global consistency of lane line detection to a certain extent. As a result, the final target lane line point set can accurately reflect the positional relationship of the lane line relative to the vehicle and the global environment, improving the accuracy and practicality of lane line detection and providing reliable data support for the path planning, lane keeping and safety avoidance functions of autonomous vehicles.

[0014] In some implementations, the navigation data includes heading angle information, pitch angle information, roll angle information, and vehicle position information. The fusion processing of the navigation data and the seventh lane line point set to obtain the target lane line point set includes: constructing a rotation matrix based on the vehicle heading angle information, the pitch angle information, and the roll angle information; constructing a translation matrix based on the vehicle position information; and transforming the seventh lane line point set from the vehicle coordinate system to the map coordinate system based on the rotation matrix and the translation matrix to obtain the eighth lane line point set, wherein the map coordinate system is a coordinate system established based on the lane lines.

[0015] Thus, by constructing rotation and translation matrices using navigation data, the lane line point set can be accurately transformed from the local vehicle coordinate system to the global map coordinate system to a certain extent. This allows the resulting eighth lane line point set to accurately reflect the true position and attitude of the lane lines in the global environment, ensuring global consistency. This avoids the problem of accumulated positioning deviations that may occur in the local coordinate system to a certain extent, making the lane line detection results unaffected by changes in vehicle position and attitude. This improves the stability and accuracy of lane line detection to a certain extent, ensuring the accuracy of autonomous vehicles in long-distance driving and global path planning.

[0016] In some embodiments, the method further includes: performing a fitting process on the eighth lane line point set to obtain a fitted lane line model; determining the current lane line width and current curvature information based on the fitted lane line model; and determining the target lane line point set based on the eighth lane line point set, the deviation between the current lane line width and the historical lane line width, and the deviation between the current curvature information and the historical curvature information, wherein the historical lane line width and the historical curvature information are determined based on a historical fitted lane line model, and the historical fitted lane line model is obtained by fitting a historical eighth lane line point set.

[0017] In this way, by fitting the eighth lane line point set and comparing and analyzing each frame of data in the eighth lane line point set with the data from the historical eighth lane line point set, abnormal frames in the eighth lane line point set are screened and replaced. This effectively corrects abnormal frame data caused by sensor noise or environmental interference to a certain extent, avoids the accumulation of abnormal points caused by local errors, and thus improves the stability of lane line detection results to a certain extent. This ensures that autonomous vehicles can continuously obtain accurate lane line data, and guarantees the continuity and accuracy of path planning and safety warnings during driving.

[0018] This application also provides an electrical device, including: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement the above-described method.

[0019] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0020] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0021] The electrical equipment, computer-readable storage medium, and computer program product provided in this application, when implementing the above method, first configure a channel attention mechanism module in the convolutional layer and / or residual layer of the base model to obtain a lane detection deep learning model to be trained; then, based on the lane line data training set and preset training parameters, train the lane detection deep learning model to be trained to obtain the lane detection deep learning model. In this way, by configuring a channel attention mechanism module in the base model, the model automatically focuses on the core features of lane lines during training, enhancing the accuracy of extracting key lane line information, while effectively suppressing the negative impact of environmental interference such as lighting and shadows on feature extraction, reducing the interference of interfering features on model judgment. This allows the trained lane detection deep learning model to output an accurate set of image lane line points, solving to some extent the problem of inaccurate detection data caused by environmental interference, thereby improving the lane detection accuracy of the model in complex environments and providing a foundation for lane perception in autonomous driving systems.

[0022] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 is a flowchart illustrating one of the training methods of a lane detection deep learning model according to certain embodiments of this application; Figure 2 is a flowchart illustrating another of the training methods of a lane detection deep learning model according to certain embodiments of this application; Figure 3 is a flowchart illustrating one of the lane detection methods according to certain embodiments of this application; Figure 4 is a flowchart illustrating another of the lane detection methods according to certain embodiments of this application; Figure 5 is a flowchart illustrating a third of the lane detection methods according to certain embodiments of this application; Figure 6 is a flowchart illustrating a fourth of the lane detection methods according to certain embodiments of this application; Figure 7 is a flowchart illustrating a fifth of the lane detection methods according to certain embodiments of this application; Figure 8 is a flowchart illustrating a multi-sensor fusion lane detection method based on an improved UFLD v2 network according to certain embodiments of this application; Figure 9 is a flowchart illustrating lane detection processing of camera image data according to certain embodiments of this application; Figure 10 is a schematic diagram of the improved UFLD v2 network model structure according to certain embodiments of this application; Figure 11 is an improved backbone network ResNet according to certain embodiments of this application. Figure 14 is a schematic diagram of the structure and lane line data processing flow of 34; Figure 15 is a schematic diagram of the lane line detection processing flow of LiDAR data in some embodiments of this application; Figure 16 is a schematic diagram of the lane line detection processing flow of navigation data in some embodiments of this application; Figure 17 is a schematic diagram of the lane line detection processing flow of navigation data in some embodiments of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0025] Lane line data is a core perception data point supporting the precise positioning, path planning, and safety warning of autonomous vehicles, and its detection accuracy directly affects the operational reliability of the autonomous driving system. However, lane line detection is susceptible to environmental interference, leading to inaccurate detection data and failing to provide stable and reliable lane line data support for autonomous driving.

[0026] On the one hand, with the development of autonomous driving technology, traditional methods rely on manual feature extraction, which is insufficient in terms of robustness and accuracy in complex environments such as changes in lighting, shadow occlusion, and vehicle occlusion. Lane detection technology has gradually evolved from traditional image processing methods to deep learning methods. However, related deep learning methods still have the problem of inaccurate core feature extraction in complex environments. Environmental interference can seriously interfere with the feature extraction effect, ultimately causing deviations in detection data.

[0027] On the other hand, data collected by a single sensor has certain limitations. For example, camera data is easily affected by lighting conditions, and LiDAR data needs to be effectively fused with other data in complex scenarios. However, related technologies lack effective collaborative mechanisms for processing multi-source data, which further reduces the stability and accuracy of lane line detection due to the superposition effect of environmental interference. This not only makes it difficult to provide reliable lane line data support but also restricts the safe operation of autonomous driving systems.

[0028] Based on the above problems, please refer to Figure 1. This application provides a training method for a lane detection deep learning model. The method includes: 01: configuring a channel attention mechanism module in the convolutional layer and / or residual layer of the base model to obtain a lane detection deep learning model to be trained; 02: training the lane detection deep learning model to be trained according to the lane line data training set and preset training parameters to obtain the lane detection deep learning model.

[0029] This application provides a training apparatus for a lane detection deep learning model. The training method for the lane detection deep learning model according to this application can be implemented by the training apparatus for the lane detection deep learning model according to this application. Specifically, the training apparatus for the lane detection deep learning model includes a configuration module and a training module. The configuration module is used to configure channel attention mechanism modules in the convolutional layers and / or residual layers of the base model to obtain the lane detection deep learning model to be trained. The training module is used to train the lane detection deep learning model to be trained according to the lane line data training set and preset training parameters to obtain the lane detection deep learning model.

[0030] This application also provides a server, which includes a memory and a processor. The training method for the lane detection deep learning model of this application can be implemented by the server of this application. Specifically, the memory stores a computer program, and the processor is used to configure channel attention mechanism modules in the convolutional layers and / or residual layers of the base model to obtain the lane detection deep learning model to be trained. The processor is also used to train the lane detection deep learning model to be trained according to the lane line data training set and preset training parameters to obtain the lane detection deep learning model.

[0031] Specifically, the basic model is a deep learning network structure that has been proven in practice in the field of computer vision and has mature image feature extraction capabilities. For example, the Ultra Fast Lane Detection v2 (ufld v2) network, which typically includes layers such as convolutional layers and residual layers.

[0032] Convolutional layers are layers in the basic model used to extract basic image features. For example, the conv1 layer in Residual Network-34 (resnet 34) performs convolution operations on the input image by sliding a convolutional kernel of a preset size, transforming the image pixel information into feature maps. Convolutional layers can initially extract low-level features such as lane line edges and textures.

[0033] Residual layers are layers in the basic model designed to solve the gradient vanishing problem during deep network training. For example, residual blocks such as block1, block2, block3, and block4 in the ResNet 34 network. Residual blocks pass layer inputs directly to subsequent output layers through skip connections to ensure effective learning of deep features.

[0034] The channel attention mechanism module is a module that can adaptively adjust the weights of feature channels, such as the SE attention mechanism module. By evaluating the importance of each feature channel, the channel attention mechanism module assigns higher weights to channels including lane line features, while reducing the weights of environmental interference feature channels, thereby enhancing effective features and suppressing interfering features.

[0035] The lane line data training set is a dataset used for model training. It contains a large number of lane line images under different environmental scenarios such as sunny days, cloudy days, nights, occluded conditions, unoccluded conditions, and slopes. Each image is accompanied by accurate lane line annotation information, providing supervision for the model to learn lane line features.

[0036] Preset training parameters are key parameters set before model training, such as learning rate, optimization strategy, batch size, and number of iterations. Different training parameters directly affect the model's training speed, stability, and final performance. For example, training parameters could be: learning rate of 0.0001, optimization strategy of stochastic gradient descent (SGD), batch size of 32, number of iterations of 50, and image training pixel height of 352.

[0037] The deep learning model for lane detection to be trained is a model formed by configuring channel attention mechanism modules in the convolutional layers and / or residual layers of the base model. It has a preliminary feature extraction architecture, but the parameters have not been optimized through training and need to be further trained to improve the lane detection capability.

[0038] The deep learning model for lane detection to be trained can be either a model that adds an attention mechanism only to the convolutional layers, or a model that adds an attention mechanism only to the residual layers. Furthermore, when adding an attention mechanism to the residual layers, it can be done by adding the attention mechanism to only one residual block, or by combining four residual blocks into different combinations and adding the attention mechanism to the residual blocks according to the different combination methods, or by adding the attention mechanism to both the convolutional and residual layers. In practical applications, different methods of adding the attention mechanism can be selected according to different application scenarios.

[0039] The lane line detection deep learning model is a final model that, after being trained on a lane line data training set and with preset training parameters, possesses the ability to accurately extract lane line features and output accurate lane line data. The model is trained by inputting the lane line data training set into the model in batches, updating the parameters through backpropagation, and iterating to a preset number of times. After training, images from the test set are input into the model, and the model outputs the pixel coordinates of the detected lane lines to verify the effectiveness of the trained model.

[0040] The following example illustrates the training method of the lane detection deep learning model in this application, using the integration of SE attention mechanisms into the residual structures of blocks 2, 3, and 4 of the ResNet 34 backbone network of the ufld v2 neural network model: In the model construction phase, ufld v2 is selected as the base model, and its backbone network uses ResNet 34, including a conv1 convolutional layer and four residual blocks (blocks 1 to 4). SE attention mechanism modules are embedded in the three residual blocks (blocks 2, 3, and 4), specifically by adding an SE attention mechanism flow after the feature output of each residual block.

[0041] The SE attention mechanism first performs global average pooling on the feature channels to obtain channel statistics, then learns channel weights through fully connected layers and activation functions, and finally multiplies the learned weights with the original feature channels to strengthen the core feature channels of the lane lines and suppress interfering feature channels, ultimately forming a deep learning model for lane line detection to be trained.

[0042] During model training, the training parameters of the model are first set. For example, the learning rate is set to 0.0001, the optimization strategy is SGD, the batch size is 32, the number of iterations is 50, and the image training pixel height is adjusted to 352.

[0043] Subsequently, the lane line data training set was divided into multiple data batches according to batch size and input into the model to be trained sequentially. The model extracts the basic edge and texture features of the image through the conv1 convolutional layer. After the features are initially optimized by the block1 residual block, they are fed into the block2, block3, and block4 residual blocks embedded with the SE attention mechanism. In each residual block, the SE attention mechanism module adjusts the weights of the feature channels, focusing on strengthening the core features such as the outline and direction of the lane lines, and suppressing interfering features such as background and shadows.

[0044] After each batch of data is trained, the convolutional kernel weights, residual block parameters, and SE attention mechanism module weights are updated using the backpropagation algorithm based on the error between the model's predictions and the labeled information. This process is repeated until 50 iterations are completed, at which point the model parameters reach their optimal state, resulting in a lane detection deep learning model. After training, images from the test set are input into the model, and the model outputs the pixel coordinates of the lane detection points to verify the effectiveness of the trained model.

[0045] Thus, by configuring a channel attention mechanism module in the basic model, the model can automatically focus on the core features of lane lines during training, enhancing the accuracy of extracting key information about lane lines. At the same time, it effectively suppresses the negative impact of environmental interference such as lighting and shadows on feature extraction, reducing the interference of interfering features on the model's judgment. This enables the deep learning model for lane line detection trained to output accurate image lane line point sets, solving to some extent the problem of low detection data accuracy caused by environmental interference. This improves the accuracy of lane line detection in complex environments and provides a foundation for lane line perception in autonomous driving systems.

[0046] Please refer to Figures 2, 9, 10, and 11 together. In some embodiments, step 02 includes: 021: configuring the loss function of the lane detection deep learning model to be trained, wherein the loss function includes a base loss function and / or a Dessian loss function; 022: training the lane detection deep learning model to be trained according to the lane line data training set, preset training parameters, and loss function to obtain the lane detection deep learning model.

[0047] In some implementations, the configuration module is further configured to configure the loss function of the lane detection deep learning model to be trained, wherein the loss function includes a base loss function and / or a Dessian loss function. The training module is further configured to train the lane detection deep learning model to be trained based on the lane line data training set, preset training parameters, and the loss function, so as to obtain the lane detection deep learning model.

[0048] In some implementations, the processor is further configured to configure the loss function of the lane detection deep learning model to be trained, wherein the loss function includes a base loss function and / or a Dessell loss function. The processor is also configured to train the lane detection deep learning model to be trained based on a lane line data training set, preset training parameters, and the loss function, to obtain the lane detection deep learning model.

[0049] Specifically, the loss function is a mathematical function used in deep learning model training to measure the difference between the predicted result and the true label. The smaller the output loss value, the more accurate the model prediction. Loss functions include basic loss functions and supplementary loss functions. The basic loss functions include classification loss functions, structural loss functions, and auxiliary segmentation loss functions, while the supplementary loss function includes the Dessell loss function.

[0050] The base loss function is a commonly used loss function in lane detection that can basically reflect the prediction error; for example, the cross-entropy loss function in classification loss functions. The base loss function can provide the model with basic training supervision signals to ensure that the model initially learns the overall features of the lane lines.

[0051] The Dice loss function, also known as the Dice loss function, calculates the loss based on the overlap between the predicted result and the ground truth label. It is highly adaptable to imbalanced sample scenarios and can effectively enhance the model's attention to minority class features. At the same time, it is sensitive to differences in boundary regions and can drive the model to optimize the prediction accuracy of lane line boundaries, thereby improving the model's accuracy in identifying lane line edges.

[0052] The lane line detection deep learning model is the final model that, after loss function optimization and training on the lane line data training set, has the ability to accurately extract lane line features and output accurate lane line data.

[0053] Before model training, due to the different effects of loss functions, different combinations of loss functions can be selected when configuring loss functions for the lane detection deep learning model to be trained, depending on the training requirements.

[0054] For example, if only the problem of imbalanced samples needs to be addressed, the Desce loss function can be configured alone. If both the basic prediction error and the imbalanced sample problem need to be considered, the basic loss function and the Desce loss function can be configured simultaneously, and then combined using methods such as weighted summation to form the final loss function. In the combination of loss functions, the weight coefficients of each loss function can be determined by customization or through methods such as grid search. For example, the weight coefficients of the classification loss function, the structure loss function, the auxiliary segmentation loss function, and the Desce loss function can be customized to 0.3, 0.2, 0.2, and 0.3 respectively.

[0055] The following explanation uses the example of incorporating SE attention mechanisms into the residual structures of blocks 2, 3, and 4 of the backbone network ResNet 34 in the ufld v2 neural network model, and the combination of loss functions (cross-entropy loss, structural loss, auxiliary segmentation loss, and Descein loss) to improve the configuration of the deep learning model for lane detection under training, to illustrate the training method of the deep learning model for lane detection in this application: First, in the model construction stage, the core framework of the improved ufld v2 neural network structure is built. This structure uses the improved backbone network ResNet 34 as the core for feature extraction, and combines it with a localization branch and an existence branch to form a complete detection system. The localization branch is responsible for outputting the pixel coordinate localization results of the lane lines, and the existence branch determines whether each lane line exists or not, corresponding to the lane line prediction quantization parameters in the row direction of Nr_lane and the lane line existence judgment parameters in the column direction of Nc_lane, respectively.

[0056] The improved backbone network ResNet 34 is used as the core, with an input size of 3x590x1640. After the initial convolution operation of the conv1 layer, the output feature map size is 64x295x820. Then, it passes through four residual blocks in sequence: block1, improved block2, improved block3, and improved block4. The output sizes of each stage are 64x147x410, 128x74x205, 256x37x103, and 512x19x52, respectively. Finally, it outputs a feature vector with a dimension of 512x19x52 through the FC (fully connected) layer.

[0057] The core improvement of the ResNet 34 backbone network lies in the three improved blocks: block2, block3, and block4. An SE attention mechanism module is embedded at the output of each improved block to adaptively evaluate the importance of feature channels, thereby strengthening the core features of lane lines and suppressing interfering features.

[0058] Furthermore, the output features of each layer of the backbone network ResNet 34 are improved to be transmitted in a fixed dimension. The entire process adapts parameters such as N_row (feature map row dimension), N_col (feature map column dimension), and N_cdl (number of convolutional layer channels). For example, N_row=295, N_col=820, and N_cdl=64 for conv1 layer, N_row=147, N_col=410, and N_cdl=64 for block1 layer, N_row=74, N_col=205, and N_cdl=128 for block2 layer, N_row=37, N_col=103, and N_cdl=256 for block3 layer, and N_row=19, N_col=52, and N_cdl=512 for block4 layer.

[0059] During the feature transfer process, the outputs of each level are sequentially fed into the MPL (Multilayer Perceptron) for nonlinear transformation to improve the feature abstraction and expression capabilities. Subsequently, a classifier is used to make a preliminary prediction of the existence of lane lines and remove redundant and invalid feature information.

[0060] During the selection process, the Nr_lane and Nc_lane identifiers are combined to retain valid lane line feature data and remove invalid features from areas without lane lines, ensuring that training focuses on the core objective. The SE attention mechanism operates continuously within each improvement block, closely adapting to the parameter constraints of each dimension to guarantee the effectiveness of feature processing.

[0061] In addition, to address the issues of imbalanced samples and insufficient boundary optimization, a combination of cross-entropy loss function, structural loss function, auxiliary segmentation loss function, and Desce loss function is configured for the model to be trained.

[0062] The cross-entropy loss function optimizes the classification error between lane lines and the background, accurately distinguishing target regions in feature maps of different sizes, such as 64x295x820 and 64x147x410. The structural loss function constrains the continuity of lane line morphology, adapting to the morphological learning of intermediate feature maps such as 128x74x205 and 256x37x103, avoiding breaks or distortions. The auxiliary segmentation loss function focuses on local details, especially adapting to high-dimensional feature maps such as 512x19x52, enhancing the extraction of details of narrow and worn lane lines. The Desce loss function alleviates sample imbalance and optimizes the prediction effect of lane line boundaries by calculating the overlap between predictions and ground truth annotations. These four functions comprehensively cover the training supervision needs. The four loss functions, with custom weights (cross-entropy loss function 0.3, structural loss function 0.2, auxiliary segmentation loss function 0.2, and Desce loss function 0.3), work synergistically to provide comprehensive and accurate supervision signals for model training.

[0063] Subsequently, based on preset standards and actual training requirements, the model training parameters were set as follows: learning rate 0.0001, optimization strategy SGD, batch size 32, iterations 50, image training pixel height 352, row direction grid number 75, and number of lane line cells predicted in the row direction 210. The row direction grid number of 75 and the number of cells of 210 directly correspond to the quantization rule of Nr_lane, providing a basis for row-by-row prediction of the localization branch.

[0064] Meanwhile, based on the feature processing requirements of each level, the parameters of N_row, N_col, and N_cdl were adjusted accordingly: for the conv1 level, N_row=295, N_col=820, and N_cdl=64; for the block4 level, N_row=19, N_col=52, and N_cdl=512 were improved to ensure that the feature transfer of each level is adapted. This can avoid gradient explosion or slow convergence to a certain extent, and can also balance training speed and detection accuracy to a certain extent.

[0065] During model training, preprocessed image data is fed into the improved ufld v2 neural network in batches of 32.

[0066] The data is first processed through the conv1 layer to extract basic edge and texture features, outputting a 64x295x820 feature map, which is then optimized in block1 to obtain a 64x147x410 feature map.

[0067] Then, through the improved blocks 2, 3, and 4, which incorporate the SE attention mechanism, feature maps of 128x74x205, 256x37x103, and 512x19x52 are output respectively. At each level, the feature weights are adjusted by the SE module, and effective features are selected in conjunction with MPL transformation and classifier prediction.

[0068] After the features are output from the fully connected (FC) layer, they are fed into the localization branch and the existence branch, respectively. The localization branch outputs the lane line coordinate prediction result based on the Nr_lane quantization rule. The existence branch outputs the existence probability based on the Nc_lane feature dimension. The error between the two outputs and the ground truth annotation is calculated, and the total loss value is obtained by weighted summation using four loss functions. Then, the convolutional kernel weights, SE module coefficients, MPL weights, and classifier parameters are updated through backpropagation. This process is repeated until 50 iterations are completed. During training, the dynamic changes of Nr_lane and Nc_lane are recorded in real time.

[0069] Finally, after the model training is completed, the lane line image data from the test set is input into the trained model. After the complete process, the pixel coordinates of the lane line detection points are output.

[0070] The testing process focuses on verifying the model's performance in complex scenarios such as changes in lighting, shadow occlusion, and vehicle occlusion. By comparing the error between the prediction results and the actual annotations, the model's processing effect on feature maps of different sizes and the accuracy of Nr_lane and Nc_lane labels are evaluated. To a certain extent, this ensures that the trained lane detection deep learning model can meet the high requirements of autonomous driving for lane perception and provides reliable image lane point sets to support subsequent multi-source data fusion.

[0071] Understandably, during the training process, the model gradually optimizes its learning ability of lane line features through the collaborative constraints of multiple loss functions, resulting in a lane line detection deep learning model that can not only accurately identify lane line regions to a certain extent, but also enhance the detailed features and boundary accuracy of lane lines to a certain extent.

[0072] Thus, by configuring a combination of classification loss function, structural loss function, auxiliary segmentation loss function, and Descein loss function, the problems of sample imbalance and insufficient boundary optimization in conventional training are solved to a certain extent. Furthermore, the synergistic optimization of multiple loss functions enables the model to comprehensively learn the classification features, morphological features, detailed features, and boundary features of lane lines during training, thereby improving the model's detection accuracy of lane line data to a certain extent. This allows the model to more accurately identify the edge contours of lane lines, enhances the model's adaptability to complex environments, and enables the model to accurately output lane line point sets in complex environments. Consequently, the reliability of lane line detection is improved to a certain extent, ensuring the safe driving of autonomous vehicles under various road conditions.

[0073] Please refer to Figure 3. An embodiment of this application provides a lane line detection method, which includes: 03: fusing lane line data according to a first lane line point set, a second lane line point set, and navigation data information to obtain a target lane line point set, so as to obtain lane line detection information. The first lane line point set is obtained by processing lane line image data by a lane line detection deep learning model of some of the above embodiments, and the second lane line point set is obtained by processing lane line lidar point cloud data.

[0074] This application provides a lane line detection device. The lane line detection method of this application can be implemented by the lane line detection device of this application. Specifically, the lane line detection device includes a processing module. The processing module is used to fuse lane line data according to a first lane line point set, a second lane line point set, and navigation data information to obtain a target lane line point set, so as to obtain lane line detection information. The first lane line point set is obtained by processing lane line image data using a lane line detection deep learning model of some of the above embodiments, and the second lane line point set is obtained by processing lane line lidar point cloud data.

[0075] This application also provides a server, which includes a memory and a processor. The lane line detection method of this application can be implemented by the server of this application. Specifically, the memory stores a computer program, and the processor is used to fuse lane line data according to a first lane line point set, a second lane line point set, and navigation data information to obtain a target lane line point set, so as to obtain lane line detection information. The first lane line point set is obtained by processing lane line image data using a lane line detection deep learning model of some of the above embodiments, and the second lane line point set is obtained by processing lane line lidar point cloud data.

[0076] Specifically, the first lane line point set is the coordinate set obtained after processing lane line image data using a lane line detection deep learning model according to some of the above-described embodiments. Because this lane line detection deep learning model is configured with a channel attention mechanism module or a superposition loss function to optimize the lane line image data, it can, to a certain extent, enhance the core visual features of the lane lines, suppress environmental interference, and enable the output first lane line point set to accurately reflect the pixel positions of the lane lines in the image.

[0077] Lane line lidar point cloud data is three-dimensional spatial point data acquired by vehicle-mounted lidar by emitting laser beams and receiving reflected signals, including the three-dimensional coordinate information of each point.

[0078] The second lane line point set is formed by filtering and segmenting the lane line LiDAR point cloud data, selecting point cloud data related to the lane lines, and combining them to form a point set with spatial positioning advantages.

[0079] Navigation data refers to vehicle status data collected in real time by in-vehicle navigation systems and other navigation systems. It typically includes heading angle information (the angle between the vehicle's direction of travel and a reference direction), pitch angle information (the vehicle's longitudinal tilt angle), roll angle information (the vehicle's lateral tilt angle), and vehicle position information (the vehicle's coordinates in the global coordinate system). Navigation data provides a global reference for lane line data analysis, ensuring the consistency of detection results to a certain extent.

[0080] Fusion processing is a process of collaboratively optimizing lane line data from different sources using specific algorithms. The purpose of fusion processing is to integrate the data advantages of lane line data from different sources, eliminate data bias, and ultimately output accurate and stable lane line data.

[0081] The target lane line point set is a set of points obtained by fusing lane line data from different sources. The target lane line point set integrates the visual details of the first lane line point set, the spatial location of the second lane line point set, and global references from navigation data information, and can accurately reflect the actual shape and physical location of the lane lines to a certain extent.

[0082] Lane detection information is the detection result centered on the target lane line point set, including key information such as the position coordinates and morphological features of the lane lines, which can directly provide perception input for the autonomous driving system.

[0083] After acquiring the first lane line point set, the second lane line point set, and navigation data, the vehicle-mounted system fuses the lane line data from different sources to obtain lane line detection information. In practical applications, different fusion processing methods can be used for lane line data from different sources depending on the specific application scenario or requirements.

[0084] For example, the coordinates of the first lane line point set and the second lane line point set can be unified first to ensure that the two types of point sets can be directly compared. Then, the impact of the current vehicle attitude on the data can be evaluated by combining the heading angle, pitch angle and roll angle in the navigation data information, and the global coordinate reference can be determined by the vehicle position information in the navigation data information.

[0085] Subsequently, weights can be dynamically assigned to lane line data from different sources based on data reliability. For example, in well-lit, unobstructed scenarios, the visual details of the first lane line point set are more reliable, so a weight of 0.6 is assigned, the weight of the second lane line point set is 0.3, and the weight of navigation data information is 0.1. In dimly lit, heavily obstructed scenarios, the spatial position of the second lane line point set is more stable, so a weight of 0.6 is assigned, the weight of the first lane line point set is 0.3, and the weight of navigation data information is 0.1.

[0086] Finally, for points corresponding to the same lane line position, the coordinates are weighted and summed to obtain the fused coordinate points. At the same time, information from the navigation data is used to correct the tilt deviation of the point set caused by vehicle bumps and slopes. Finally, all the fused coordinate points are integrated to form the target lane line point set.

[0087] In this way, by integrating multi-source data, the limitations of single sensor data are made up for, and the superimposed effects of environmental interference are effectively suppressed to a certain extent. This makes the output target lane line point set stable and accurate, and to a certain extent, it can accurately reflect the actual state of the lane lines. This provides reliable lane line data support for the accurate positioning, path planning and safety warning of autonomous vehicles, and improves the operational safety and stability of the autonomous driving system.

[0088] Please refer to Figures 4, 12, and 14 together. In some embodiments, the lane line detection method includes: 04: performing time alignment processing on the initial lane line image data and the initial lane line LiDAR point cloud data to obtain lane line LiDAR point cloud data; 05: performing a first filtering process on the lane line LiDAR point cloud data based on spatial information to obtain a third lane line point set; 06: transforming the third lane line point set from the LiDAR coordinate system to the camera coordinate system to obtain a fourth lane line point set, wherein the LiDAR coordinate system is a coordinate system established by the LiDAR that acquired the initial lane line LiDAR point cloud data, and the camera coordinate system is a coordinate system established by the camera that acquired the initial lane line image data; 07: transforming the fourth lane line point set from the camera coordinate system to the pixel coordinate system to obtain a fifth lane line point set, wherein the pixel coordinate system is the coordinate system where the initial lane line image data is located; 08: performing a second filtering process on the fifth lane line point set according to a preset target area to obtain a second lane line point set.

[0089] In some embodiments, the processing module is further configured to perform time alignment processing on the initial lane line image data and the initial lane line LiDAR point cloud data to obtain lane line LiDAR point cloud data. The processing module is further configured to perform a first filtering process on the lane line LiDAR point cloud data based on spatial information to obtain a third lane line point set. The processing module is further configured to transform the third lane line point set from the LiDAR coordinate system to the camera coordinate system to obtain a fourth lane line point set, wherein the LiDAR coordinate system is a coordinate system established by the LiDAR that acquired the initial lane line LiDAR point cloud data, and the camera coordinate system is a coordinate system established by the camera that acquired the initial lane line image data. The processing module is further configured to transform the fourth lane line point set from the camera coordinate system to the pixel coordinate system to obtain a fifth lane line point set, wherein the pixel coordinate system is the coordinate system where the initial lane line image data is located. The processing module is further configured to perform a second filtering process on the fifth lane line point set according to a preset target area to obtain a second lane line point set.

[0090] In some embodiments, the processor is further configured to perform time alignment processing on the initial lane line image data and the initial lane line LiDAR point cloud data to obtain lane line LiDAR point cloud data. The processor is further configured to perform a first filtering process on the lane line LiDAR point cloud data based on spatial information to obtain a third lane line point set. The processor is further configured to transform the third lane line point set from the LiDAR coordinate system to the camera coordinate system to obtain a fourth lane line point set, wherein the LiDAR coordinate system is a coordinate system established based on the LiDAR that acquired the initial lane line LiDAR point cloud data, and the camera coordinate system is a coordinate system established based on the camera that acquired the initial lane line image data. The processor is further configured to transform the fourth lane line point set from the camera coordinate system to the pixel coordinate system to obtain a fifth lane line point set, wherein the pixel coordinate system is the coordinate system where the initial lane line image data is located. The processor is further configured to perform a second filtering process on the fifth lane line point set based on a preset target area to obtain a second lane line point set.

[0091] Specifically, the initial lane line image data is lane line scene image data directly collected by the vehicle-mounted camera without any processing, and usually includes visual information such as lane lines, background, and environmental interference.

[0092] The initial lane line lidar point cloud data is a set of three-dimensional spatial point data directly collected by the vehicle-mounted lidar without any processing, which can reflect the spatial distribution of lane lines and the surrounding environment.

[0093] Time alignment processing uses specific algorithms to synchronize data streams acquired by different sensors across time, ensuring that the data describes the scene at the same moment. In practical applications, different time alignment processing methods can be selected based on the needs of the application scenario. For example, timestamp-based matching synchronization can be chosen, adding precise timestamps to each frame of data acquired by the camera and LiDAR, and pairing image and point cloud data with the closest timestamps to achieve time synchronization. Alternatively, interpolation-based synchronization can be chosen, where when there is a slight discrepancy in the timestamps of the two types of data, interpolation calculations are performed on the lagging data to complete the missing point cloud or image information, thus achieving time alignment.

[0094] The lane line lidar point cloud data is lidar point cloud data that has been time-aligned and synchronized with the initial lane line image data.

[0095] Spatial information refers to the three-dimensional coordinates, distance, height, and other characteristic information of each point in the lidar point cloud data, which can be used to distinguish spatial locations.

[0096] The first filtering process involves removing invalid and noisy points unrelated to lane lines from the LiDAR point cloud data based on spatial information. For example, a ground height threshold can be set to perform ground segmentation, filtering out points in the LiDAR point cloud data that are higher than the threshold, and retaining points near the ground that are related to lane lines. For instance, if the collected LiDAR point cloud data includes 100,000 points, each with X, Y, and Z coordinates (Z being the height), and a ground height threshold of 0.1m is preset based on the smoothness of the road surface, each point in the LiDAR point cloud data is analyzed, filtering out non-ground points with Z greater than 0.1m, and retaining the ground point cloud data.

[0097] The third lane line point set is the set of LiDAR point cloud coordinates related to the lane lines that are retained after the first filtering process.

[0098] The lidar coordinate system is a three-dimensional coordinate system established with the lidar installation location as the origin, which can be used to describe the original spatial location of the lidar acquisition point.

[0099] The camera coordinate system is a three-dimensional coordinate system established with the camera's optical center as the origin, which can be used to connect the coordinate transformation between three-dimensional space and two-dimensional images.

[0100] The fourth lane line point set is the set of lane line point cloud coordinates after the lane line point set has been transformed from the LiDAR coordinate system to the camera coordinate system. For example, a camera coordinate system is established with the camera's optical center as the center, and a LiDAR coordinate system is established with the LiDAR installation position as the origin. The extrinsic parameter matrix, including the rotation matrix and translation matrix, is obtained. The data of the third lane line point set is multiplied by the extrinsic parameter matrix to achieve the transformation of the third lane line point set in the LiDAR coordinate system to the fourth lane line point set in the camera coordinate system.

[0101] The pixel coordinate system is a two-dimensional coordinate system established with the top left corner of the initial lane line image as the origin and the image edges as the coordinate axes. It can be used to describe the position of pixels in the image.

[0102] The fifth lane point set is the set of lane point cloud coordinates after transforming the lane point set from the camera coordinate system to the pixel coordinate system. For example, based on the camera intrinsic parameter matrix, the three-dimensional points of the fourth lane point set in the camera coordinate system can be projected to the two-dimensional pixel coordinate system using the perspective projection formula to obtain the fifth lane point set in the pixel coordinate system.

[0103] The preset target area is an effective area pre-defined based on the normal distribution range of lane lines in the image, and the processing of lane line data focuses on the range where lane lines may exist.

[0104] The second filtering process involves removing invalid points from the fifth lane line point cluster based on a pre-defined target area.

[0105] After receiving lane line data collected by multiple sensors in the vehicle system, the initial lane line image data and the initial lane line LiDAR point cloud data are first processed synchronously to obtain lane line LiDAR point cloud data and lane line image data that are consistent in time.

[0106] In addition, in some implementations, the initial navigation data is synchronized to obtain time-consistent lane line lidar point cloud data, lane line image data, and navigation data.

[0107] Subsequently, the spatial information of the lane line lidar point cloud data is analyzed. The height of each point in the lane line lidar point cloud data is compared with a preset height threshold. Points in the lane line lidar point cloud data whose height is greater than the preset height threshold are removed to obtain the third lane line point set.

[0108] Then, a camera coordinate system is established with the camera's optical center as the center, and a lidar coordinate system is established with the lidar installation position as the origin. An extrinsic parameter matrix, including rotation and translation matrices, is obtained. The data of the third lane line point set is multiplied by this extrinsic parameter matrix to transform the data of the third lane line point set in the lidar coordinate system to the fourth lane line point set in the camera coordinate system. Subsequently, the fourth lane line point set in the camera coordinate system is transformed into the fifth lane line point set in the pixel coordinate system through perspective projection using the camera intrinsic parameter matrix.

[0109] Finally, based on the pre-defined Region of Interest (ROI), invalid points in the fifth lane line point set are removed, and the LiDAR point cloud data of the lane lines within the ROI are retained to obtain the second lane line point set.

[0110] For example, based on the vehicle's field of vision, the ROI region can be set to pixel coordinates. That is, the road surface area 10-50m in front of the vehicle. After excluding invalid areas at the edge of the image, the point cloud data falling within the ROI is selected from the fifth lane line point set to obtain the second lane line point set.

[0111] In this way, by performing a series of processes on the initial data, such as time alignment, spatial filtering, coordinate transformation, and target area filtering, the problems of asynchronous time, inconsistent coordinate systems, and data redundancy of the initial data are effectively solved to a certain extent. This ensures the timeliness and consistency of the data, eliminates invalid and redundant points in the initial data, improves the data quality, and obtains an accurate and effective set of second lane line points. This, in turn, improves the effect of multi-source data fusion to a certain extent and ensures the accuracy and reliability of the final lane line detection results.

[0112] Please refer to Figures 5, 13, and 14 together. In some embodiments, step 03 includes: 031: performing point set matching processing on the first lane line point set and the second lane line point set to obtain the sixth lane line point set; 032: transforming the sixth lane line point set from the pixel coordinate system to the vehicle coordinate system to obtain the seventh lane line point set, wherein the vehicle coordinate system is the coordinate system in which the vehicle is located; 033: performing fusion processing on the navigation data and the seventh lane line point set to obtain the target lane line point set.

[0113] In some implementations, the processing module is further configured to perform point set matching processing on the first lane line point set and the second lane line point set to obtain a sixth lane line point set. The processing module is also configured to transform the sixth lane line point set from the pixel coordinate system to the vehicle coordinate system to obtain a seventh lane line point set, wherein the vehicle coordinate system is the coordinate system in which the vehicle is located. The processing module is further configured to perform fusion processing on the navigation data and the seventh lane line point set to obtain the target lane line point set.

[0114] In some implementations, the processor is further configured to perform point set matching processing on the first lane point set and the second lane point set to obtain a sixth lane point set. The processor is also configured to transform the sixth lane point set from the pixel coordinate system to the vehicle coordinate system to obtain a seventh lane point set, wherein the vehicle coordinate system is the coordinate system in which the vehicle is located. The processor is further configured to perform fusion processing on the navigation data and the seventh lane point set to obtain the target lane point set.

[0115] Specifically, point set matching involves using a specific algorithm to find matching point pairs corresponding to the same lane line position in two different point sets. The purpose of point set matching is to eliminate positional biases in multi-source data caused by sensor differences, and to achieve coordinated alignment between the two types of data.

[0116] For example, by setting a distance threshold, the Euclidean distance between each point in the first lane line point set and all points in the second lane line point set can be calculated. Point pairs with a distance less than the set threshold can be retained, while outliers with excessively large distances can be removed, thus completing the initial matching of multi-source lane line data. For instance, if the distance threshold is set to 5 pixels, the Euclidean distance between each point in the second lane line point set and points in the first lane line point set can be calculated. If the coordinates of a point in the second lane line point set are u=102, v=350, and the distance between the point and the left lane line point u=100, v=352 in the first lane line point set is 2.8 pixels, which is less than 5 pixels, then all points with a distance less than or equal to 5 pixels can be filtered out in the above manner to obtain the sixth lane line point set.

[0117] The sixth lane line point set is a unified coordinate set formed by removing outliers and retaining matching points after the first lane line point set and the second lane line point set have undergone point set matching processing.

[0118] The vehicle coordinate system is a three-dimensional coordinate system established with the vehicle itself as the reference. It is used to describe the positional relationship between the lane lines and the vehicle. It is usually based on the vehicle's center of gravity as the origin, with the X-axis along the vehicle's direction of travel, the Y-axis perpendicular to the vehicle body to the left, and the Z-axis perpendicular to the ground and upward.

[0119] The seventh lane line point set is the coordinate set of the sixth lane line point set after transformation from the pixel coordinate system to the vehicle coordinate system. By performing coordinate transformation on the data of the sixth lane line point set, the position association between lane lines and vehicles is achieved. For example, through joint calibration of the camera and the vehicle, the extrinsic parameter matrix from the pixel coordinate system to the vehicle coordinate system is obtained. Multiplying the coordinates of the sixth lane line point set by this extrinsic parameter matrix yields the seventh lane line point set in the vehicle coordinate system.

[0120] After processing the collected multi-source lane line data to obtain the first lane line point set and the second lane line point set, the Euclidean distance between each point in the first lane line point set and all points in the second lane line point set is calculated. Point pairs with a distance less than a set distance threshold are retained, and abnormal points with excessive distances are removed. This completes the initial matching of the multi-source lane line data and yields the sixth lane line point set.

[0121] Subsequently, using a pre-calibrated extrinsic matrix, the pixel coordinates of each pixel in the sixth lane line point set are converted into three-dimensional coordinates in the vehicle body coordinate system to obtain the seventh lane line point set.

[0122] Finally, the seventh lane line point set is optimized by combining navigation data information to form the target lane line point set.

[0123] In this way, by matching point sets, the data deviation between the first and second lane line point sets is eliminated to a certain extent, achieving precise coordination between the two types of sensor data. At the same time, the lane line point set data is transformed into the vehicle coordinate system, making the lane line data directly related to the vehicle state, which is more in line with the decision-making needs of autonomous driving. In addition, the lane line data is processed in conjunction with navigation data, providing a global reference for the lane line data and ensuring the global consistency of lane line detection to a certain extent. As a result, the final target lane line point set can accurately reflect the positional relationship of the lane line relative to the vehicle and the global environment, improving the accuracy and practicality of lane line detection and providing reliable data support for the path planning, lane keeping and safety avoidance functions of autonomous vehicles.

[0124] Please refer to Figures 6, 15, and 17 together. In some embodiments, the navigation data includes heading angle information, pitch angle information, roll angle information, and vehicle position information. Step 033 includes: 0331: Constructing a rotation matrix based on the vehicle heading angle information, pitch angle information, and roll angle information; 0332: Constructing a translation matrix based on the vehicle position information; 0333: Transforming the seventh lane line point set from the vehicle coordinate system to the map coordinate system based on the rotation matrix and translation matrix to obtain the eighth lane line point set, wherein the map coordinate system is a coordinate system established based on the lane lines.

[0125] In some implementations, the processing module is further configured to construct a rotation matrix based on the vehicle's heading angle, pitch angle, and roll angle information. The processing module is also configured to construct a translation matrix based on the vehicle's position information. Furthermore, the processing module is configured to transform the seventh lane line point set from the vehicle coordinate system to the map coordinate system based on the rotation and translation matrices, obtaining the eighth lane line point set, wherein the map coordinate system is a coordinate system established based on the lane lines.

[0126] In some implementations, the processor is further configured to construct a rotation matrix based on the vehicle's heading angle, pitch angle, and roll angle information. The processor is also configured to construct a translation matrix based on the vehicle's position information. Furthermore, the processor is configured to transform the seventh lane line point set from the vehicle coordinate system to the map coordinate system based on the rotation and translation matrices, obtaining the eighth lane line point set, wherein the map coordinate system is a coordinate system established based on the lane lines.

[0127] Specifically, the rotation matrix is ​​a square matrix used to describe the rotation transformation of the spatial coordinate system. It can correct the spatial offset of the point set caused by the change of vehicle posture through mathematical operations, and to a certain extent, it can keep the posture of the point set consistent with the map coordinate system.

[0128] A translation matrix is ​​a matrix used to describe the translation transformation of a spatial coordinate system. It can translate a set of points in a local coordinate system to the corresponding position in the map coordinate system and establish a global position association.

[0129] The map coordinate system is a global three-dimensional coordinate system established based on the actual road lane line distribution. The origin and coordinate axes are fixed, and it is used to describe the real position of the lane lines in the global environment.

[0130] The eighth lane line point set is the set of lane line point cloud coordinates in the map coordinate system after transformation by rotation and translation matrices.

[0131] When fusing navigation data and the seventh lane line point set, a rotation matrix is ​​first constructed based on the vehicle's heading angle, pitch angle, and roll angle information. For example, the vehicle is first rotated around the Z-axis of the map coordinate system based on the heading angle, then around the rotated Y-axis based on the pitch angle, and finally around the rotated X-axis based on the roll angle. The three single-axis rotation matrices are then multiplied sequentially to obtain the final rotation matrix.

[0132] Subsequently, based on the vehicle location information, a translation matrix is ​​constructed. For example, the vehicle location information X, Y, and Z in the navigation data are directly used as the core parameters of the translation matrix to construct a three-dimensional translation matrix, so that the set of line points of the seventh lane is translated from the origin of the vehicle coordinate system to the actual position of the vehicle in the map coordinate system.

[0133] Finally, based on the rotation and translation matrices, the coordinates are transformed from the vehicle coordinate system to the map coordinate system. For example, for a point (X1, Y1, Z1) in the seventh lane line point set, it is first multiplied by the rotation matrix to obtain the point (X2, Y2, Z2) after attitude correction. Then, the point is added to the coordinates in the translation matrix to obtain the point (X2+X, Y2+X, Z2+Z) in the map coordinate system. Following the above processing flow, each point in the seventh lane line point set is processed. First, the attitude offset is corrected by the rotation matrix, and then the coordinates are transformed to the map coordinate system by the translation matrix. All the transformed points are integrated to obtain the eighth lane line point set.

[0134] In this way, by using navigation data to construct rotation and translation matrices, the lane line point set can be accurately transformed from the local vehicle coordinate system to the global map coordinate system to a certain extent. This allows the resulting eighth lane line point set to accurately reflect the true position and attitude of the lane line in the global environment, possessing global consistency. This avoids the problem of accumulated positioning deviations that may occur in the local coordinate system to a certain extent, making the lane line detection results unaffected by changes in vehicle position and attitude. This improves the stability and accuracy of lane line detection to a certain extent, ensuring the accuracy of autonomous vehicles in long-distance driving and global path planning.

[0135] Please refer to Figures 7, 16, and 17 together. In some embodiments, the lane line detection method further includes: 09: fitting the eighth lane line point set to obtain a fitted lane line model; 010: determining the current lane line width and current curvature information based on the fitted lane line model; 011: determining the target lane line point set based on the eighth lane line point set, the deviation between the current lane line width and the historical lane line width, and the deviation between the current curvature information and the historical curvature information, wherein the historical lane line width and historical curvature information are determined based on the historical fitted lane line model, which is obtained by fitting the historical eighth lane line point set.

[0136] In some implementations, the processing module is further configured to perform fitting processing on the eighth lane line point set to obtain a fitted lane line model. The processing module is also configured to determine the current lane line width and current curvature information based on the fitted lane line model. The processing module is further configured to determine the target lane line point set based on the eighth lane line point set, the deviation between the current lane line width and the historical lane line width, and the deviation between the current curvature information and the historical curvature information. The historical lane line width and historical curvature information are determined based on a historical fitted lane line model, which is obtained by fitting a historical eighth lane line point set.

[0137] In some implementations, the processor is further configured to perform fitting processing on the eighth lane line point set to obtain a fitted lane line model. The processor is also configured to determine the current lane line width and current curvature information based on the fitted lane line model. The processor is further configured to determine a target lane line point set based on the eighth lane line point set, the deviation between the current lane line width and the historical lane line width, and the deviation between the current curvature information and the historical curvature information, wherein the historical lane line width and historical curvature information are determined based on a historical fitted lane line model, which is obtained by fitting a historical eighth lane line point set.

[0138] Specifically, the fitting process involves using mathematical modeling to approximate the distribution of discrete lane line points using continuous function curves, thus making the lane line shape more intuitive and regular. For example, this can be achieved by constructing a quadratic polynomial function. Where x and y are coordinates in the map coordinate system, the eighth lane line point set is fitted. Based on the distribution of the eighth lane line point set, key control points can be selected to construct a Bézier curve. By adjusting the position of the control points, the curve shape can be optimized to make it fit the discrete point set to the highest degree.

[0139] The fitted lane line model is a continuous function model obtained by fitting the set of lane line points. It can accurately reflect the overall direction and shape characteristics of the lane line to a certain extent.

[0140] The current lane width is the actual lateral dimension of the lane calculated based on the fitted lane line model. The current curvature information is the degree of lane line curvature calculated based on the fitted lane line model, which can reflect the lane line's direction change to some extent.

[0141] Historical lane widths are lane widths determined based on historical fitted lane line models at past times. Historical curvature information is lane curvature determined based on historical fitted lane line models at past times, which can reflect the regular variation patterns of lane direction to a certain extent. The historical fitted lane line model is a model obtained by fitting the historical eighth lane line point set.

[0142] The degree of deviation is the difference between the current lane line characteristics (lane line width and curvature) and the historical lane line characteristics, and can be used to determine whether the current data is abnormal.

[0143] After obtaining the eighth lane line point set through processing, a quadratic polynomial is used to fit the lane line data of each frame in the eighth lane line point set. The polynomial coefficients are solved by the least squares method to minimize the error between the fitted curve and the discrete point set, thus obtaining the fitted lane line model corresponding to each frame.

[0144] Subsequently, based on the fitted lane line model, the lane line width and curvature information of the current frame are calculated. For example, the lane line width of the current frame can be determined by the average lateral distance of the sampling points, and the curvature information of the current frame can be calculated by the second derivative method.

[0145] Finally, the absolute deviations of the lane line width in the current frame from those in historical frames, and the absolute deviations of the curvature in the current frame from those in historical frames, are calculated and compared with the corresponding preset width deviation thresholds and curvature deviation thresholds, respectively. If both deviations are within the threshold range, the current frame is retained; if either deviation exceeds the preset threshold, the current frame is replaced with a historical frame. Finally, all processed data frames are organized to obtain the target lane line point set.

[0146] For example, with a preset width offset threshold of 0.5m and a preset curvature threshold of 0.00005, the Y difference between the left and right lane lines in the current frame at X=50000m is 9.8m, the width of the corresponding lane line in the historical frame is 10.0m, the curvature of the left lane line in the current frame is 0.0002, and the curvature of the historical frame is 0.00021. The calculated absolute deviation between the lane line width of the current frame and the lane line width of the historical frame is 0.2m, and the absolute deviation between the curvature of the current frame and the curvature of the historical frame is 0.00001. Since neither exceeds the threshold, the current frame is determined to be a valid frame and is not filtered. If the absolute deviation between the lane line width of a certain frame and the lane line width of a historical frame is 0.6m, it is determined to be an abnormal frame, and the abnormal frame is replaced with a historical frame. This process is repeated for all data in the eighth lane line point set to finally obtain the target lane line point set.

[0147] In this way, by fitting the eighth lane line point set and comparing and analyzing each frame of data in the eighth lane line point set with the data from the historical eighth lane line point set, abnormal frames in the eighth lane line point set are screened and replaced. This effectively corrects abnormal frame data caused by sensor noise or environmental interference to a certain extent, avoids the accumulation of abnormal points caused by local errors, and thus improves the stability of lane line detection results to a certain extent. This ensures that autonomous vehicles can continuously obtain accurate lane line data, and guarantees the continuity and accuracy of path planning and safety warnings during driving.

[0148] Please refer to Figure 8. The following is a specific communication example to explain the lane line detection method of this application: During the lane line detection process, three types of sensors, namely camera, LiDAR, and integrated navigation, respectively collect lane line images, 3D point clouds, and vehicle status data. First, the acquisition delay is eliminated by time alignment processing to ensure that the three types of data describe the lane line scene at the same time, laying the foundation for data consistency for subsequent collaborative processing and solving the problem of initial data time synchronization to a certain extent.

[0149] Subsequently, the raw LiDAR point cloud undergoes multiple screening and coordinate transformation steps to achieve precise adaptation with the image data. First, by setting a ground height threshold of 0.1m, non-ground point clouds such as trees and buildings are removed, retaining only the ground point set related to lane lines. Then, using an extrinsic transformation matrix containing rotation and translation matrices, the point cloud in the LiDAR coordinate system is transformed to the camera coordinate system. Next, using an intrinsic parameter matrix containing the camera focal length and the origin position of the image coordinate system, it is transformed into a point cloud in the pixel coordinate system through perspective projection. Finally, based on the vehicle's field of view, a Region of Interest (ROI) is defined, and effective lane line point clouds within 10-50m in front of the vehicle are selected, completing the processing from the raw point cloud to the image-adapted point set and eliminating invalid and redundant data.

[0150] Meanwhile, the image data acquired by the camera is used to extract high-precision lane line point sets through an improved UFLD v2 network. This network uses ResNet 34 as its backbone, and its core improvement lies in incorporating an SE attention mechanism into the residual structures of blocks 2, 3, and 4. This mechanism adaptively evaluates feature channel weights, strengthening core features such as lane line contours and directions while suppressing interference from background and shadows. Simultaneously, the loss function is optimized by adding Dice Loss to the basic loss function to alleviate the sample imbalance problem caused by background pixels dominating the image, thus improving the accuracy of lane line boundaries.

[0151] Then, the lane line point sets from the image and LiDAR are matched and transformed to achieve local to global localization. First, by setting a distance threshold of 5 pixels, the Euclidean distance between the two types of point sets is calculated. Matching point pairs are filtered and outliers are removed to obtain a unified coordinate point set that integrates visual details and spatial positioning advantages. Then, using the joint calibration extrinsic parameters of the camera and vehicle, the point set in the pixel coordinate system is transformed to the vehicle coordinate system to establish the relative positional relationship between the lane lines and the vehicle. Next, combined with the vehicle attitude and position information provided by the integrated navigation, a rotation matrix and a translation matrix are constructed to transform the point set in the vehicle coordinate system to the map coordinate system, ensuring that the lane line detection results have global consistency and are not affected by changes in vehicle position and attitude.

[0152] Finally, the lane line point set in the map coordinate system is fitted and anomaly filtering is performed to further improve detection stability and accuracy. For complex scenarios such as curves and lane changes, a quadratic polynomial can be used to fit the discrete point set to accurately describe the lane line curvature characteristics. Then, the lane line width and curvature information of the current frame and historical frames in the lane line point set are compared. A width deviation threshold of 0.5m and a curvature threshold of 0.00005 are set to filter out abnormal frames caused by sensor noise or environmental interference, avoiding the accumulation of local errors.

[0153] The final output is the filtered lane line map coordinates, which provides reliable perception data support for the accurate positioning, path planning and safety warning of autonomous vehicles, and is fully adaptable to complex scenarios such as changes in lighting, vehicle occlusion and bumpy slopes.

[0154] This application also provides an electrical device, including: a processor: a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the methods of some of the above embodiments.

[0155] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in some of the above embodiments.

[0156] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in some of the above embodiments.

[0157] It is understood that a computer program includes computer program code. Computer program code can be in the form of source code, object code, executable files, or some intermediate form. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0158] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0159] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of executable request code comprising one or more steps for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0160] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A training method for a deep learning model for lane line detection, characterized in that, The method includes: configuring a channel attention mechanism module in the convolutional layer and / or residual layer of the base model to obtain a lane detection deep learning model to be trained; and training the lane detection deep learning model to be trained according to the lane data training set and preset training parameters to obtain the lane detection deep learning model.

2. The method according to claim 1, characterized in that, The step of training the lane line detection deep learning model to be trained based on the lane line data training set and preset training parameters to obtain the lane line detection deep learning model includes: configuring the loss function of the lane line detection deep learning model to be trained, wherein the loss function includes a base loss function and / or a Dessian loss function; and training the lane line detection deep learning model to be trained based on the lane line data training set, the preset training parameters, and the loss function to obtain the lane line detection deep learning model.

3. A lane line detection method, characterized in that, The method includes: fusing lane line data according to a first lane line point set, a second lane line point set, and navigation data information to obtain a target lane line point set, thereby obtaining the lane line detection information, wherein the first lane line point set is obtained by processing lane line image data using the lane line detection deep learning model as described in claim 1 or 2, and the second lane line point set is obtained by processing lane line lidar point cloud data.

4. The method according to claim 3, characterized in that, The method further includes: performing time alignment processing on initial lane line image data and initial lane line LiDAR point cloud data to obtain lane line LiDAR point cloud data; performing a first filtering process on the lane line LiDAR point cloud data based on spatial information to obtain a third lane line point set; transforming the third lane line point set from the LiDAR coordinate system to the camera coordinate system to obtain a fourth lane line point set, wherein the LiDAR coordinate system is a coordinate system established by the LiDAR that acquired the initial lane line LiDAR point cloud data, and the camera coordinate system is a coordinate system established by the camera that acquired the initial lane line image data; transforming the fourth lane line point set from the camera coordinate system to the pixel coordinate system to obtain a fifth lane line point set, wherein the pixel coordinate system is the coordinate system where the initial lane line image data is located; and performing a second filtering process on the fifth lane line point set according to a preset target area to obtain a second lane line point set.

5. The method according to claim 4, characterized in that, The step of fusing lane line data based on the first lane line point set, the second lane line point set, and navigation data information to obtain the target lane line point set, and thus obtaining the lane line detection information, includes: performing point set matching processing on the first lane line point set and the second lane line point set to obtain a sixth lane line point set; transforming the sixth lane line point set from the pixel coordinate system to the vehicle coordinate system to obtain a seventh lane line point set, wherein the vehicle coordinate system is the coordinate system in which the vehicle is located; and fusing the navigation data and the seventh lane line point set to obtain the target lane line point set.

6. The method according to claim 5, characterized in that, The navigation data includes heading angle information, pitch angle information, roll angle information, and vehicle position information. The process of fusing the navigation data and the seventh lane line point set to obtain the target lane line point set includes: constructing a rotation matrix based on the vehicle heading angle information, pitch angle information, and roll angle information; constructing a translation matrix based on the vehicle position information; and transforming the seventh lane line point set from the vehicle coordinate system to the map coordinate system based on the rotation matrix and the translation matrix to obtain the eighth lane line point set, wherein the map coordinate system is a coordinate system established based on the lane lines.

7. The method according to claim 6, characterized in that, The method further includes: performing fitting processing on the eighth lane line point set to obtain a fitted lane line model; determining the current lane line width and current curvature information based on the fitted lane line model; and determining the target lane line point set based on the eighth lane line point set, the deviation degree between the current lane line width and the historical lane line width, and the deviation degree between the current curvature information and the historical curvature information, wherein the historical lane line width and the historical curvature information are determined based on a historical fitted lane line model, and the historical fitted lane line model is obtained by fitting based on a historical eighth lane line point set.

8. An electrical appliance, characterized in that, include: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement the method as claimed in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images