A lane line detection method of a double-branch network fused with a multi-head attention mechanism
By integrating a dual-branch network with a multi-head attention mechanism and combining camera focal length and tilt angle to establish a parameterized model of lane lines, the problem of slow lane line detection speed and poor performance in existing technologies is solved, and efficient lane line detection in complex environments is achieved.
Patent Information
- Application Number
- CN202210738527.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-24
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2042-06-24
AI Technical Summary
Existing deep learning-based lane detection methods suffer from problems such as complex network structures, slow computation speed, and poor detection performance, especially in complex driving environments where lane lines are difficult to detect effectively.
A dual-branch network integrating multi-head attention mechanism is adopted, and a lane line parameterization model is established by combining camera focal length and tilt angle. The global-local dual-branch network is constructed through optimization with global-local dual-branch network and Hungarian fitting loss function to improve the inference speed and detection effect of the model.
It improves the robustness and real-time performance of lane line detection, effectively overcomes interference in complex driving environments, and enhances the accuracy and speed of lane line detection.
Smart Images
Figure CN115527177B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of automatic driving, in particular to a lane line detection method of a double-branch network fusing a multi-head attention mechanism. BACKGROUND
[0002] In view of the problems such as the rapid increase in the number of vehicles and the increasing complexity of traffic conditions in recent years, an intelligent driving assistance system is expected to become an important tool for people to travel safely by car. Lane line detection is a key and challenging link in the intelligent driving assistance system. The lane line detection system can collect the current road conditions, help the driver obtain more comprehensive road environment information, and reduce the driving burden of the driver. At present, the lane line detection method based on deep learning has problems such as complex network structure, slow calculation speed, and poor detection effect, which makes it difficult to deploy in the intelligent driving assistance system. SUMMARY
[0003] The purpose of the present application is to provide a lane line detection method of a double-branch network fusing a multi-head attention mechanism, which parameterizes the lane line in combination with the focal length and the tilt angle of the camera under the consideration of the interference caused by the uneven road in the real driving environment, maximally overcomes the interference caused by the uneven road on the lane line detection system, fuses the multi-head attention mechanism to establish a global-local double-branch network, improves the inference speed of the model, shortens the training time of the model, obtains more lane line detail information, and improves the detection effect of the detection system.
[0004] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a lane line detection method of a double-branch network fusing a multi-head attention mechanism, comprising the following steps:
[0005] Step one: establishing a polynomial lane line model combining the focal length and the tilt angle of the camera;
[0006] Step two: establishing a loss function based on Hungarian fitting, performing two-part matching between the predicted lane line parameters and the real lane line, and optimizing the regression loss of a specific lane using the matching result;
[0007] Step three: establishing a global-local double-branch network, which combines the advantages of CNN and multi-head attention mechanism, and is built in accordance with the expansion-compression structure, aiming to pay attention to more spatial information and cause global attention, and pay attention to more far-end lane line marker information in the field of view. The present application can detect lane line markers under complex driving environments, especially far-end lane line markers in the field of view, and without additional processing operations, each lane line is converged to a point to form a lane line vanishing point, overcoming the difficulties caused by factors such as lane line marker blur, occlusion, strong light interference, and dim light for identification.
[0008] The step one polynomial lane line model incorporating camera focal length and tilt angle includes:
[0009] The lane line cubic polynomial model is established as:
[0010] X i =α 1,i Y 3 +α 2,i Y 2 +α 3,i Y+b
[0011] where (X i ,Y) is the i-th lane line point on the ground, α 1,i , α 2,i , α 3,i and b are the coefficients of the i-th lane line polynomial function.
[0012] The lane line cubic polynomial "bird's eye view" model is established as:
[0013]
[0014]
[0015] where (x,y) is the corresponding pixel point position on the transformed image plane, f x is the width of the pixel on the focal plane divided by the focal length, f y is the height of the pixel on the focal plane divided by the focal length, and H is the height of the camera installation.
[0016] The lane line cubic polynomial model incorporating camera tilt angle is established as:
[0017]
[0018]
[0019] where θ is the camera tilt angle, f is the camera focal length, f' is the transformed camera focal length, and (x',y') represents the transformed pixel point position.
[0020] The lane line cubic polynomial parameterized model is established as:
[0021] p i = (c i , α 1,i ', α 2,i ', α 3,i ', f', b i ', b i '', β i , γ i )
[0022]
[0023]
[0024]
[0025] f′=f sinθ
[0026]
[0027]
[0028] In the formula, p i Let c be the set of polynomial parameters representing the shape of the i-th lane line. i ∈[0,1] represents the confidence level of the i-th lane line (0 represents background, 1 represents lane line marking), β i and γ i These represent the longitudinal start and end offsets of the i-th lane line, respectively, where i∈[0,…,M] and M is the total number of lane lines in the current image.
[0029] In step two, a Hungarian fitting loss is used to perform a bipartite match between the predicted lane line parameters and the actual lane lines. The matching results are then used to optimize the regression loss for a specific lane. A loss function for calculating the matching degree between the predicted lane line parameters and the actual lane lines is constructed as follows:
[0030] The loss function based on Hungarian fitting is as follows:
[0031] S1. Obtain the predicted lane line parameter set P = {p i};
[0032] S2, Obtain the real lane line marker set S = {(c i ,x i ,y i )};
[0033] S3. Taking the real lane line marker set S obtained in step 2, and searching for the final injective function: l:S→P, the bipartite matching problem between the predicted lane line parameter set and the real lane line marker set is defined as a minimum-cost problem:
[0034] Where M is the predefined number of lane lines. Let S be the set of real lane markings for the i-th lane. i With index l i Prediction parameter set The matching cost between them;
[0035] The loss function based on Hungarian fitting is defined as:
[0036]
[0037] Where L is the output value of the loss function, g(c i ) for category c i The probability, The index after two-part matching is The fitted lane line sequence set, μ1, μ2, μ3 are the coefficients of the loss function, L mae Let Z(·) be the mean absolute error, Z(·) be the index function, and β be the mean absolute error. i Let be the longitudinal starting offset of the i-th real lane line. The index after two-part matching is Predict the longitudinal starting offset of the lane line, γ i Let be the longitudinal cutoff offset of the i-th real lane line. The index after two-part matching is Predict the longitudinal cutoff offset of the lane line.
[0038] In step three, a global-local dual-branch network is established. This network combines the advantages of CNN and multi-head attention mechanisms and is built according to an expansion-compression structure. Its aim is to focus on more spatial information, attract global attention, and focus on more lane marking information at the far end of the field of view. The method is as follows:
[0039] A. Constructing a multi-head attention mechanism module: MultiHead(Q,K,V) = Concat(head1,…,head) h W O ,
[0040] head i =Attention(QW i Q ,KW i K VW i V ), Where MultiHead is
[0041] The output of the multi-head attention mechanism is defined by: Concat (combining the outputs of each head), head (the output of each head in the multi-head attention mechanism), Attention (self-attention mechanism), softmax (logistic regression function), d (dimensions of the query vector (Q) and key vector (K), V (value vector), and W. i Q W i Q W i Qrespectively represent the linear transformation matrices of Q, K and V of the i-th self-attention mechanism, h is the number of self-attention mechanisms, and W O is the weight matrix of multi-head attention. Since the standard multi-head attention mechanism does not encode position information, this limits the expressive power in processing visual tasks, so the present application divides the relative position encoding into P h and P w , respectively, to solve this problem;
[0042] B, build a global branch: this branch is a branch that pays attention to spatial details, with a large number of channels and a shallow network structure, to capture low-level spatial details and generate high-resolution feature representations. This branch contains three network layers, the first layer is a convolution module, and the last two layers are multi-head attention mechanism modules;
[0043] C, build a local branch: this branch is a semantic branch with a small number of channels and a deep network structure to obtain high-level semantic context information. The first layer of this branch is a Stem module for fast downsampling, then two Fused MBConv modules are used to quickly process the original input image, and four MBConv modules are used after the feature map size reaches a suitable size. Finally, an information embedding module is used to integrate the rich global context information extracted to improve the detection effect of the model;
[0044] D, build a fusion module: this module is used to enhance the mutual connection and fusion of the two types of feature representations. The output feature representations of the global branch and the local branch are complementary, so the fusion module is used to combine these two complementary feature representations. This module uses the global context information obtained by the local branch to guide the feature response of the global branch, and uses different scale guides to obtain different scale feature representations. The internal coding of multi-scale information of the two branches makes the fusion between the two branches more efficient. Since the local branch uses a fast downsampling strategy, the spatial dimension of the output is smaller than that of the global branch, so first an upsampling is used to make the feature map dimensions of the two branches consistent;
[0045] E, build an encoder-decoder lane line parameter fitting module: this module uses the encoder-decoder structure framework in Transformer. This module projects the feature map output by the fusion module into a two-dimensional sequence through linear operation, which is used for lane line parameter set fitting, and obtains the trained weights;
[0046] F, use the weight values obtained in the previous step to generate predicted lane points, compare them with the real lane points based on the Hungarian loss function, and obtain the error of the current training weight. According to the error, the weight is iteratively optimized.
[0047] According to the above steps, the network weight can be obtained, the lane line is parameterized, the inference speed of the network is improved, the parameter amount of the network is reduced, good real-time performance and robustness are obtained, and the application in the driving assistance system is more suitable.
[0048] The beneficial effects of the technical scheme are as follows:
[0049] 1. The polynomial lane line model combining the focal length and the inclination angle of the camera is used, the parameter set of the lane line is directly output, the unevenness of the road is considered, the bird's eye view model of the lane line cubic polynomial is established, the interference of unevenness on the detection system is solved, the robustness of the detection system is improved, and the inference speed of the model is improved.
[0050] 2. The loss function based on the Hungarian algorithm is established through the Hungarian algorithm, two-part matching is performed between the predicted lane line parameters and the real lane line, the regression loss of a specific lane is optimized by using the matching result, and the training result of the model is improved.
[0051] 3. The global-local dual branch network is established, the advantages of CNN and the multi-head attention mechanism are combined, and the network is built according to the expansion-compression structure, so as to pay attention to more spatial information, cause global attention, pay attention to more far-end lane line mark information in the field of view, improve the training speed of the model, and improve the detection effect of the detection system. BRIEF DESCRIPTION OF DRAWINGS
[0052] Figure 1 The global-local dual branch network structure diagram of the present application is shown in the figure.
[0053] Figure 2 The multi-head attention mechanism structure diagram of the present application is shown in the figure.
[0054] Figure 3 The MBConv module and the Fused MBConv module structure diagram of the present application is shown in the figure.
[0055] Figure 4 The Stem module and the information embedding module structure diagram of the present application is shown in the figure.
[0056] Figure 5 The fusion module structure diagram of the present application is shown in the figure.
[0057] Figure 6 The encoder-decoder lane line parameter fitting module structure diagram of the present application is shown in the figure. DETAILED DESCRIPTION
[0058] The technical scheme proposed by the present application will be further described and explained in combination with the drawings as follows:
[0059] The global-local dual branch network structure diagram proposed by the present application is shown in the figureFigure 1 As shown in Table 1, the network parameters are as follows. The global branch focuses on spatial details, featuring a large number of channels and a shallow network structure. The local branch is a semantic branch with fewer channels and a deeper network structure to obtain high-level semantic context information. A multi-head attention mechanism is employed in the last layer of the global branch to acquire richer lane line details, achieving global attention with a shallower network structure. The structure of the multi-head attention mechanism is shown in Table 1. Figure 2 As shown. The FusedMBConv and MBConv modules used in the local branches are as follows: Figure 3 As shown in the diagram. The first layer of the local branch uses the Stem module, a fast downsampling module that quickly reduces the input size and speeds up the network's processing. The last layer of the local branch uses an information embedding module, which integrates the extracted rich global context information to improve the model's detection performance. The structures of the Stem module and the information embedding module are shown in the diagram. Figure 4 As shown. The fusion module structure is as follows. Figure 5 As shown, this module is used to fuse the output features of global and local branches. The structure of the lane line parameter fitting module is as follows: Figure 6 As shown, this module projects the feature map output by the fusion module into a two-dimensional sequence through linear operations, which is used to fit the lane line parameter set to obtain the trained weights.
[0060] This invention is a dual-branch network lane detection method that combines a multi-head attention mechanism. The specific implementation steps are as follows:
[0061] Step 1: Establishing a polynomial lane line model that combines camera focal length and tilt angle includes:
[0062] (1) Establish a cubic polynomial model of the lane lines:
[0063] X i =α 1,i Y 3 +α 2,i Y 2 +α 3,i Y+b
[0064] In the formula, (X i Let α be a point on the ground of the i-th lane line, and Y be a point on the ground. 1,i α 2,i α 3,i b and b are the coefficients of the polynomial function for the i-th lane line.
[0065] (2) Establish a "bird's-eye view" model of the cubic polynomial of the lane lines:
[0066]
[0067]
[0068] where (x, y) is the corresponding pixel position on the transformed image plane, f x is the width of a pixel on the focal plane divided by the focal length, f y is the height of a pixel on the focal plane divided by the focal length, and H is the height of the camera setup.
[0069] (3) Establish a lane line cubic polynomial model combined with the camera tilt angle:
[0070]
[0071]
[0072] where θ is the camera tilt angle, f is the camera focal length, f' is the transformed camera focal length, and (x', y') represents the transformed pixel position.
[0073] (4) Establish a lane line cubic polynomial parameterization model:
[0074] p i = (c i , α 1,i ', α 2,i ', α 3,i ', f', b i ', b i ", β i , γ i )
[0075]
[0076]
[0077]
[0078] f' = f sin θ
[0079]
[0080]
[0081] where p i is the polynomial parameter set representing the shape of the i-th lane line, c i ∈ [0, 1] is the confidence of the i-th lane line (0 represents background and 1 represents lane line marking), β i and γ i represent the longitudinal start and end offset of the i-th lane line, respectively, i ∈ [0, …, M], and M is the total number of lane lines in the current image.
[0082] Step two: using Hungarian fitting loss, bipartite matching between predicted lane line parameters and real lane lines, using the matching results to optimize the regression loss of specific lanes, and constructing a loss function for calculating the matching degree between predicted lane line parameters and real lane lines as follows:
[0083] The loss function based on Hungarian fitting is as follows:
[0084] 1) Obtain the predicted lane line parameter set P = {p i};
[0085] 2) Obtain the real lane line label set S = {(c i ,x i ,y i )};
[0086] 3) The real lane line label set S obtained in step 2) is searched through the last injection function: l:S→P, and the bipartite matching problem between the predicted lane line parameter set and the real lane line label set is defined as a minimum cost problem:
[0087] Where M is the pre-defined number of lane lines, is the matching cost between the i-th real lane line label set S i and the predicted parameter set i indexed by l ;
[0088] 4) The loss function based on Hungarian fitting is defined as:
[0089]
[0090] Where L is the output value of the loss function, g(c i ) is the probability of class c i , is the fitted lane line sequence set indexed by l after bipartite matching, μ1, μ2, μ3 are coefficients of the loss function, L mae is the mean absolute error, Z(·) is the indicator function, β i is the longitudinal starting offset of the i-th real lane line, is the longitudinal starting offset of the predicted lane line indexed by l after bipartite matching, γ i is the longitudinal cutoff offset of the i-th real lane line, is the longitudinal cutoff offset of the predicted lane line indexed by l after bipartite matching.
[0091] In step three, a global-local dual branch network is established, which combines the advantages of CNN and multi-head attention mechanism, and is built according to the expansion-compression structure, aiming to pay attention to more spatial information and cause global attention. The method for paying attention to more lane marking information far from the field of view is:
[0092] a) Constructing a multi-head attention mechanism module: MultiHead(Q, K, V) = Concat(head1, …, head h )W O , head i = Attention(QW i Q , KW i K , VW i V ), where MultiHead is the output of the multi-head attention mechanism, Concat is to combine the output of each head, head is the output of each head in the multi-head attention mechanism, Attention is the self-attention mechanism, softmax is the logistic regression function, d is the dimension of the query vector (Q) and the key vector (K), V is the value vector, W i Q , W i Q , W i Q represent the matrices for linear transformation of Q, K and V of the i-th self-attention mechanism, respectively, h is the number of self-attention mechanisms, and W O is the weight matrix of multi-head attention. Since the standard multi-head attention mechanism does not encode position information, this limits the expression ability in processing visual tasks, so the present application divides the relative position encoding into P h and P w when the multi-head attention mechanism processes two-dimensional feature maps, which solves this problem;
[0093] b) Constructing a global branch: The branch is a branch that pays attention to spatial details, has a large number of channel numbers and a shallow network structure, so as to capture low-level spatial details and generate high-resolution feature representations. The branch includes three network layers, the first layer is a convolution module, and the last two layers are multi-head attention mechanism modules;
[0094] c) Constructing local branch: This branch is a semantic branch with fewer channel numbers and deep network structure to obtain high-level semantic context information. The first layer of this branch is a Stem module for fast down-sampling, then two Fused MBConv modules are used to quickly process the original input image, and four MBConv modules are used after the feature map size reaches an appropriate size. Finally, an information embedding module is used to integrate the rich global context information extracted to improve the detection effect of the model;
[0095] d) Constructing fusion module: This module is used to enhance the mutual connection and fusion of two types of feature representations. The output feature representations of the global branch and the local branch are complementary, so the fusion module is used to combine these two complementary feature representations. This module uses the global context information obtained by the local branch to guide the feature response of the global branch, and uses different scale guidance to obtain different scale feature representations. The multi-scale information of the two branches is internally encoded, making the fusion between the two branches more efficient. Since the local branch uses a fast down-sampling strategy, the spatial dimension of the output is smaller than that of the global branch, so first, up-sampling is used to make the feature map dimensions of the two branches consistent;
[0096] e) Constructing encoder-decoder lane line parameter fitting module: This module uses the encoder-decoder structure framework in Transformer. This module projects the feature map output by the fusion module into a two-dimensional sequence through linear operation, which is used for lane line parameter set fitting, and obtains the trained weight;
[0097] f) Using the weight value obtained in the previous step, the predicted lane points are generated, and by comparing with the real lane points based on the Hungarian loss function, the error of the current training weight is obtained, and the weight is iteratively optimized according to the error
[0098]
[0099] Table 1 Global-local dual branch network parameter table.
[0100] According to the above steps, the network weight can be obtained. The invention parameterizes the lane line, improves the inference speed of the network, reduces the parameter amount of the network, has good real-time performance and robustness, and is more suitable for application in the driving assistance system.
Claims
1.A lane line detection method of a dual-branch network with a fused multi-head attention mechanism, characterized in that, The method comprises the following steps: Step 1: establishing a polynomial lane line model combined with camera focal length and tilt angle; Step 2: establishing a loss function based on Hungarian fitting, performing two-way matching between predicted lane line parameters and real lane lines, and optimizing the regression loss of a specific lane by using the matching result; Step 3: establishing a global-local dual-branch network; The step 1 of establishing the polynomial lane line model combined with the camera focal length and the tilt angle comprises: establishing a lane line cubic polynomial model: X i = a 1,i Y 3 + a 2,i Y 2 + a 3,i Y + b where (X i ,Y) is the point on the ground of the ith lane line, α 1,i , α 2,i , α 3,i and b are the coefficients of the polynomial function of the ith lane line. establishing a "bird's eye view" model of the lane line cubic polynomial: In the formula, (x, y) is the corresponding pixel point position on the phase plane after conversion, f x is the width of the pixel on the focal plane divided by the focal length, f y is the height of the pixel on the focal plane divided by the focal length, and H is the height at which the camera is set up. establishing a lane line cubic polynomial model combined with the camera tilt angle: In the formula, θ is the camera tilt angle, f is the camera focal length, f' is the transformed camera focal length, and (x', y') represents the transformed pixel point position; establishing a lane line cubic polynomial parameterization model: p i = (c i , α 1,i ', α 2,i ', α 3,i ', f', b i ', b i ", β i , γ i ) f' = f sin θ where p i is the polynomial parameter set representing the shape of the i-th lane line, c i ∈ [0, 1] is the confidence of the i-th lane line, 0 represents background and 1 represents lane line marking, β i and γ i represent the longitudinal start and end offset of the i-th lane line, respectively, i ∈ [0, …, M], and M is the total number of lane lines in the current image. The step 3 of establishing the global-local dual-branch network, which is combined with CNN and multi-head attention mechanism, is built in a dilated-compressed structure, and the method for paying attention to more spatial information and lane line marking information in the far end is as follows: A, Constructing multi-head attention mechanism module: MultiHead(Q, K, V) = Concat(head1, …, head h )W O , head i = Attention(QW i Q , KW i K , VW i V ), Where MultiHead is the output of the multi-head attention mechanism, Concat is to combine the output of each head, head is the output of each head in the multi-head attention mechanism, Attention is the self-attention mechanism, softmax is the logistic regression function, d is the dimension of the query vector (Q) and the key vector (K), V is the value vector, W i Q , W i K , W i V Respectively represent the linear transformation matrix of Q, K and V of the i-th self-attention mechanism, h is the number of self-attention mechanisms, W O is the weight matrix of multi-head attention, since the standard multi-head attention mechanism does not encode the position information, which limits the expression ability in processing visual tasks, therefore, when the multi-head attention mechanism processes the two-dimensional feature map, the relative position encoding is divided into P h and P w , respectively as height and width to solve this problem; B, constructing a global branch: the branch is a branch that pays attention to spatial details, has the characteristics of a large number of channels and a shallow network structure, and is used to capture low-level spatial details and generate high-resolution feature representations. The branch comprises three network layers, the first layer is a convolution module, and the last two layers are multi-head attention mechanism modules; C, constructing a local branch: the branch is a semantic branch, which has a small number of channels and a deep network structure, and is used to obtain high-level semantic context information. The first layer of the branch is a Stem module, which is used for fast down-sampling. Then, two FusedMBConv modules are used to quickly process the original input image. After the size of the feature map reaches a suitable size, four MBConv modules are used. Finally, an information embedding module is used to integrate and extract rich global context information; D, constructing a fusion module: the module is used to enhance the mutual connection and fusion of two types of feature representations. The output characteristics of the global branch and the local branch are complementary. Therefore, the fusion module is used to combine the two complementary characteristics. The local branch adopts a fast down-sampling strategy, and the spatial dimension of the output is smaller than that of the global branch. Therefore, up-sampling is first used to make the feature map dimensions of the two branches consistent; E, constructing an encoder-decoder lane line parameter fitting module: the module adopts the encoder-decoder structure framework in the Transformer. The module projects the feature map output by the fusion module into a two-dimensional sequence through linear operation, fits the lane line parameter set, and obtains the trained weight; F, using the weight value obtained in the last step to generate predicted lane points, comparing the predicted lane points with real lane points based on the Hungarian loss function, obtaining the error of the current training weight, and iteratively optimizing the weight according to the error. 2.The method of claim 1, wherein, The Hungarian fitting loss is used in the second step, two-way matching is performed between the predicted lane line parameters and the true lane lines, and the matching result is used to optimize the regression loss of a specific lane, and a loss function for calculating the matching degree between the predicted lane line parameters and the true lane lines is constructed as follows: The loss function based on the Hungarian fitting is as follows: S1, obtain a predicted lane line parameter set P = {p i}; S2, obtain a real lane line mark set S = {(c i ,x i ,y i )}; S3, the real lane marking set S obtained in step S2 is searched for the last injection function: l:S→P, and a bipartite matching problem between the predicted lane line parameter set and the real lane marking set is defined as a minimum cost problem: where M is a predefined number of lane lines, L bmc (P li ,S i ) is the matching cost between the i-th real lane marking set S i and the predicted parameter set P i indexed by l li . S4, the loss function based on the Hungarian fitting is defined as: wherein L is the loss function output value, g(c i ) is the probability of the class c i , is the fitted lane line sequence set indexed by after bipartite matching, μ1, μ2, μ3 are coefficients of the loss function, L mae is the mean absolute error, Z(·) is the indicator function, β i is the longitudinal start offset of the i-th real lane line, is the longitudinal start offset of the predicted lane line indexed by after bipartite matching, γ i is the longitudinal end offset of the i-th real lane line, is the longitudinal end offset of the predicted lane line indexed by after bipartite matching.