Lane line detection and lane keeping method based on machine vision

Abstract

The invention discloses a lane line detection and lane keeping method based on machine vision. The method comprises the following steps: S1, establishing a lane line model and a feature extraction method based on a hybrid anchor; s2, designing a lane line detection network structure based on an attention mechanism; s3, designing a network prediction head based on the network structure features; s4, designing a network loss function based on a lane line prior feature form; s5, designing a network parameter optimization method based on the L1 regular term and the scaling factor; and S6, designing a vehicle tracking reference trajectory generation algorithm based on inverse perspective transformation and a least square method. According to the lane line detection and lane keeping method based on machine vision, by combining an advanced lane line detection network and a lane keeping control algorithm, the stability and safety of an unmanned vehicle in a complex road environment are remarkably improved, and the lane line detection and lane keeping method based on machine vision has clear theoretical significance and important application value.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving perception and control, and in particular to a lane line detection and lane keeping method based on machine vision. Background Art

[0002] Unmanned driving technology has attracted a lot of attention since the DARPA Challenge in early 2000, including a number of high-tech companies, universities and automobile manufacturers. Unmanned driving technology generally consists of three parts, namely environmental perception, decision-making planning, and motion control. Environmental perception and motion control are the key links of unmanned driving systems, and lane line detection and lane keeping technology based on machine vision play a core role in them.

[0003] In the existing technology, lane detection often uses edge detection or Hough transform methods, which cannot adapt well to various lighting and weather conditions, and have limited ability to handle occlusion and noise problems; PID control is often used in the lateral control of the vehicle. This method lacks consideration of vehicle dynamics constraints and future states, and cannot effectively deal with road curves and obstacles when facing multiple constraints and complex dynamic environments. Based on the above problems, how lane detection and vehicle lateral control can better adapt to complex environmental conditions and effectively handle road conditions has become a key issue.

[0004] The anchor-based method (convolutional neural network CNN) provides an efficient and accurate solution for identifying and locating lane lines. This method generates possible lane line positions through anchor boxes and uses CNN to extract image features for fine classification and positioning. Compared with traditional edge detection or Hough transform methods, the anchor-based method can show stronger robustness and accuracy in complex environments, can better adapt to various lighting and weather conditions, and effectively handle occlusion and noise problems.

[0005] Model Predictive Control (MPC) predicts the future vehicle motion through online optimization and generates the optimal control signal to ensure that the vehicle drives stably along the lane line. Compared with traditional PID control, MPC can consider vehicle dynamic constraints and future states, providing more accurate and smooth control effects. In addition, MPC has significant advantages in dealing with multiple constraints and complex dynamic models, and can effectively deal with road curves and obstacles. Summary of the invention

[0006] The purpose of the present invention is to provide a lane line detection and lane keeping method based on machine vision, which can effectively solve the problems of large recognition error and unstable lane keeping in complex environments of traditional methods, and can significantly improve the accuracy of lane line detection and the robustness of vehicle lateral control.

[0007] To achieve the above object, the present invention provides a lane line detection and lane keeping method based on machine vision, comprising the following steps:

[0008] S1. Lane line model and feature extraction method based on hybrid anchor;

[0009] S2. Design lane detection network structure based on attention mechanism;

[0010] S3, design network prediction head based on network structure characteristics;

[0011] S4. Design network loss function based on lane line prior feature morphology;

[0012] S5. Design network parameter optimization and pruning methods based on L1 regularization term and scaling factor;

[0013] S6. Design a vehicle tracking reference trajectory generation algorithm based on inverse perspective transformation and least squares method;

[0014] S7. Design a vehicle model predictive control algorithm based on the vehicle dynamics model in the Frenet coordinate system.

[0015] Preferably, in S1, the lane line model and feature extraction method are established as follows:

[0016]

[0017] i∈{1,…,N row}

[0018]

[0019] i∈{1,…,N col};

[0020] Among them, δ represents the step size of the feature map F corresponding to the original image, N row and N col are the number of row and column anchors, respectively, (x i ,y i ) and (x j ,y j ) are the coordinates of the i-th row anchor and the j-th column anchor respectively; for each anchor, use the above x i and i , extract the mapping feature vector of the row anchor on the feature map from the feature map F Using x j and j , extract the feature vector mapped by the column anchor on the feature map from the feature map F W F and H F are the width and length of the feature map F, CF is the depth of the feature map F.

[0021] Preferably, in S2, an attention module is provided in the lane detection network structure, and the attention module includes a fully connected layer L att , for the anchor with index i, its corresponding local feature and As a fully connected layer L att Input, L att The output is the weight of the anchor mapping to the global feature It is used to fuse the local features corresponding to anchors other than the i-th anchor to form global features. For the anchor with index i, the corresponding global feature tensor is as follows:

[0022]

[0023] in, are the weights of row anchors and column anchors mapped to global features,

[0024] For all anchors, matrix multiplication is used to generate feature tensors. Anchor, column anchors, then:

[0025]

[0026] in, and

[0027] Preferably, in S3, the network prediction head includes two prediction head branches, one is the prediction location branch P, and the other is the existence branch E. For the anchor with index i, its local feature Global Features Perform the splicing operation separately, and the output results are recorded as and and They are used as inputs of two parallel fully connected layers respectively, and finally learn a fixed-size target network T r and T c ;

[0028] T r and T c The positioning branch in can be written as follows:

[0029]

[0030]

[0031] in, and is the label of the coordinate mapping, is the rounding symbol, and Represents the number of sampling points on each row anchor, that is, the number of classes;

[0032] and The existence branch indicates:

[0033]

[0034] in, and The labels for the coordinates exist, and the entire network needs to learn The branch finally outputs the lane prediction through two prediction positioning branches and two existence branches. The output part of the network is expressed as:

[0035] P, E = f(flatten(F));

[0036] Among them, f is the classifier, flatten(·) is the flattening operation, which is used to flatten the input feature map into a one-dimensional tensor, and P,E contains (P r , P c , E r , E c ), P r and P c The dimensions are and E r and E c The dimensions are and N row and N col are the number of row and column anchors, respectively.

[0037] Preferably, in S4, the network loss function is:

[0038] L=L cls +αL exp +βL ext ;

[0039] Among them, L cls is the classification loss, L exp is the expected loss, L ext is the loss, α and β are proportional coefficients;

[0040] Classification loss L clx Defined as:

[0041]

[0042] Among them, L CE is the cross entropy loss, is the prediction of the i-th lane assigned to the row anchor, and for the j-th row anchor, yes The corresponding classification label, is the prediction of the mth lane assigned to the column anchor, corresponding to the nth column anchor, yes The corresponding classification label of , onehot(·) is the one-hot encoding function;

[0043] The probability that each preset point is a lane is defined as:

[0044]

[0045] in, is the probability of assigning the ith lane to the row anchor, is the probability of the mth lane assigned to the column anchor, softmax(·) is the probability normalization function, and the expected position is expressed as:

[0046]

[0047] in, is the expectation of the row anchor corresponding to the preset point, is the expectation of the column anchor corresponding to the preset point;

[0048] Expected loss L exp Defined as:

[0049]

[0050] There is a loss function L ext Defined as:

[0051]

[0052] Preferably, in S5, the network parameter optimization and pruning method includes:

[0053] L=∑ (x，y) l(f(x,W),y)+λ∑ γ∈Γ g(γ);

[0054] Where (x, y) represents the input and target of the network to be trained, W represents the training weight, and g(·) is the penalty for the scaling factor γ;

[0055] The dividing point between the part to be retained and the part to be deleted is expressed by the following formula:

[0056] P{N>N(γ n )}=α

[0057] Among them, N represents the distribution function of all scaling factors γ after sparseness, α represents the set of channels to be retained, and N(γ n ) represents the upper α quantile.

[0058] Preferably, in S6, the inverse perspective matrix in the vehicle tracking reference trajectory generation algorithm changes as follows:

[0059]

[0060] Among them, (u, v) is the pixel coordinate in the image coordinate system, α x =f / dx is the scaling factor on the u axis, α y =f / dy is the scaling factor on the v axis, f is the focal length of the camera, (u0,v0) is the position of the origin of the image coordinate system in the pixel coordinate system; R is the rotation matrix used to rotate the point in the world coordinate system to the camera coordinate system matrix, t is the translation vector used to translate the rotated point from the world coordinate system to the camera coordinate system; [X w Y w Z w 1] T is the homogeneous coordinate in the world coordinate, matrix A is the internal parameter matrix of the vehicle-mounted camera, and matrix M is the external parameter matrix of the vehicle-mounted camera;

[0061] A lane on the ground is represented by a cubic polynomial in the following form:

[0062] X=a3Z 3 +a2Z 2 +a1Z+a0;

[0063] Where (X, Z) represents a point on the ground, and a3 is not equal to 0;

[0064] For a point P on the curve i (x i ,y i ), and the corresponding lane line midpoint coordinates are:

[0065]

[0066] Among them, is the scale of the actual lane width in the screen coordinate system, and the coordinates of any point on the lane centerline are obtained, and then the reference trajectory of the control system is solved.

[0067] Preferably, in S7, the vehicle model predictive control algorithm is:

[0068]

[0069] in, and e dare respectively the heading angle error and the relative error from the vehicle centerline to the reference path, v x is the component of the velocity at the center of the vehicle's rear axle on the x-axis of the vehicle coordinate system, l fr is the vehicle wheelbase, δ f is the front wheel steering angle, κ ref is the curvature radius at the reference path point; the design optimization performance index function is as follows:

[0070]

[0071] U min ≤HΔU k +U k ≤U max ;

[0072] Among them, ΔX(k) is the error between the state quantity and the set value at time k, U(k) is the control quantity at time k, and H is the system matrix, which is used to convert the control input change ΔU(k) ​​into the representation in the system state space, ΔU k is the control increment at time k, Q(k) represents the weighted output error at time k, P(k) represents the weighted control amount at time k, and U max and U min are the maximum and minimum values ​​of the control quantity at time k, respectively. Solve this quadratic programming problem and, for each time k, obtain the control increment sequence in the control domain P from that time:

[0073] ΔU k =[Δu k ,Δu k+1 ,…,Δu k+P-1 ] T ;

[0074] After obtaining the future control increment sequence, use ΔU k The first element is used as the actual control output u n The increment of (k-1) is used as the output of the controlled vehicle. The control amount at this time becomes;

[0075] u n (k) = u n (k-1)+Δu k .

[0076] Therefore, the present invention adopts the above-mentioned lane line detection and lane keeping method based on machine vision, and the beneficial effects are as follows:

[0077] (1) The present invention applies model predictive control (MPC) to lane keeping strategy, which can take into account vehicle dynamic constraints and future states, provide accurate and smooth control effects, and enable the vehicle to effectively cope with road curves and obstacles.

[0078] (2) The present invention adopts an anchor-based method, that is, a convolutional neural network (CNN), which can finely classify and locate image features, show stronger robustness and accuracy in complex environments, can better adapt to various lighting and weather conditions, and effectively deal with occlusion and noise problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of an unmanned driving platform according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0080] Figure 2 It is an overall flow chart of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0081] Figure 3 It is a network structure diagram of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0082] Figure 4 It is a schematic diagram of the network pruning principle of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0083] Figure 5 1 is a diagram of tracking error and front wheel steering angle output in a straight road scenario of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention, wherein (a) is the lateral tracking error output of PD control, (b) is the lateral tracking error output of MPC control, (c) is the front wheel steering angle output under PD control, and (d) is the front wheel steering angle output under MPC control;

[0084] Figure 6 1 is a diagram of tracking error and front wheel steering angle output in a curved scene of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention, wherein (a) is the lateral deviation output of PD control, (b) is the lateral deviation output of MPC control, (c) is the front wheel steering angle output under PD control, and (d) is the front wheel steering angle output under MPC control;

[0085] Figure 7 It is a tracking error and front wheel steering angle output diagram in a curve speed change scenario of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention, wherein (a) is a comprehensive diagram of lateral deviation output and speed change under PD control, (b) is a comprehensive diagram of lateral deviation output and speed change under MPC control, (c) is a comprehensive diagram of front wheel steering angle output and speed change under PD control, and (d) is a comprehensive diagram of front wheel steering angle output and speed change under MPC control. DETAILED DESCRIPTION

[0086] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0087] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0088] Example

[0089] like Figure 1 As shown, the lane line detection and lane keeping control solution provided by the present invention is applied to an unmanned driving platform. The platform uses JetsonNano as the main controller, and the optical camera and UWB positioning module as sensors. The present invention mainly obtains the vehicle position through the UWB positioning module.

[0090] like Figure 2 As shown, a lane line detection and lane keeping method based on machine vision includes the following steps:

[0091] S1. Lane line model and feature extraction method based on hybrid anchor;

[0092] Model lane lines based on hybrid anchors:

[0093]

[0094] i∈{1,…,N row}

[0095]

[0096] i∈{1,…,N col};

[0097] Among them, δ represents the step size of the feature map F corresponding to the original image, N row and N col are the number of row anchors and column anchors, respectively, (x i ,y i ) and (x j ,y j ) are the coordinates of the i-th row anchor and the j-th column anchor respectively; for each anchor, use the above x i and i , extract the mapping feature vector of the row anchor on the feature map from the feature map F Using x j and j , extract the feature vector mapped by the column anchor on the feature map from the feature map F W F and H F are the width and length of the feature map F, C F is the depth of the feature map F.

[0098] S2, such as Figure 3 As shown in the figure, after the feature vector is extracted, the lane detection network structure is designed according to the attention mechanism;

[0099] The attention module consists of a fully connected layer L att , for the anchor with index i, its corresponding local feature and As a fully connected layer L att Input, L att The output is the weight of the anchor mapping to the global feature It is used to fuse the local features corresponding to anchors other than the i-th anchor to form global features. For the anchor with index i, the corresponding global feature tensor is as follows:

[0100]

[0101] in, are the weights mapped to the global features for row anchors and column anchors respectively. For all anchors, matrix multiplication can be used to quickly generate feature tensors. Anchor, column anchors, then:

[0102]

[0103]

[0104] in, and

[0105] S3, design network prediction head based on network structure characteristics;

[0106] The network prediction head includes two prediction head branches, one is the prediction location branch P, and the other is the existence branch E. For the anchor with index i, its local feature Global Features Perform the splicing operation separately, and the output results are recorded as and and They are used as inputs of two parallel fully connected layers respectively, and finally learn a fixed-size target network T r and T c ;

[0107] T r and T c The positioning branch in can be written as follows:

[0108]

[0109] in, and is the label of the coordinate mapping, is the rounding symbol, and Represents the number of sampling points on each row anchor, that is, the number of classes.

[0110] and The existence branch indicates:

[0111]

[0112] in, and The labels for the coordinates exist, and the entire network needs to learn Branches, through two prediction and positioning branches and two existence branches, the final output lane prediction, the output part of the network can be expressed as:

[0113] P, E = f(flatten(F));

[0114] Among them, f is the classifier, flatten(·) is the flattening operation, which is used to flatten the input feature map into a one-dimensional tensor, and P,E contains (P r , P c , E r , E c ), P r and P c The dimensions are and E r and E c The dimensions are and N row and N col are the number of row and column anchors, respectively.

[0115] S4. Design network loss function based on lane line prior feature morphology;

[0116] L=L cls +αL exp +βL ext ;

[0117] Among them, L cls is the classification loss, L exp is the expected loss, L ext is the loss, α and β are proportional coefficients;

[0118] Classification loss L cls Defined as:

[0119]

[0120] Among them, L CE is the cross entropy loss, is the prediction of the i-th lane assigned to the row anchor, and for the j-th row anchor, yes The corresponding classification label, is the prediction of the mth lane assigned to the column anchor, corresponding to the nth column anchor, yes The corresponding classification label of , onehot(·) is the one-hot encoding function;

[0121] The probability that each preset point is a lane is defined as:

[0122]

[0123] in, is the probability of assigning the ith lane to the row anchor, is the probability of the mth lane assigned to the column anchor, softmax(·) is the probability normalization function, and the expected position is expressed as:

[0124]

[0125] in, is the expectation of the row anchor corresponding to the preset point, is the expectation of the column anchor corresponding to the preset point;

[0126] Expected loss L exp Defined as:

[0127]

[0128] There is a loss function L ext Defined as:

[0129]

[0130] S5, such as Figure 4 As shown in the figure, after the lane detection network is designed, the network parameter optimization is designed based on the L1 regularization term and the proportional factor;

[0131] L=∑ (x，y) l(f(x,W),y)+λ∑ γ∈Γ g(γ);

[0132] Where (x, y) represents the input and target of the network to be trained, W represents the training weight, and g(·) is the penalty for the scaling factor γ;

[0133] The dividing point between the part to be retained and the part to be deleted is expressed by the following formula:

[0134] P{N>N(γ n )}=α

[0135] Among them, N represents the distribution function of all scaling factors γ after sparseness, α represents the set of channels to be retained, and N(γ n ) represents the upper α quantile.

[0136] S6. Based on the inverse perspective transformation and the least squares method, a vehicle tracking reference trajectory generation algorithm is designed. The inverse perspective matrix changes as follows:

[0137]

[0138] Among them, (u, v) is the pixel coordinate in the image coordinate system, α x =f / dx is the scaling factor on the u axis, α y =f / dy is the scaling factor on the v axis, f is the focal length of the camera, (u0,v0) is the position of the origin of the image coordinate system in the pixel coordinate system; R is the rotation matrix used to rotate the point in the world coordinate system to the camera coordinate system matrix, t is the translation vector used to translate the rotated point from the world coordinate system to the camera coordinate system; [X w Y w Z w 1] T is the homogeneous coordinate matrix in world coordinates, A is the internal parameter matrix of the onboard camera, and matrix M is the external parameter matrix of the onboard camera;

[0139] A cubic polynomial represents a lane on the ground and has the following form:

[0140] X=a3Z 3 +a2Z 2 +a1Z+a0;

[0141] Where (X, Z) represents a point on the ground, and a3 is not equal to 0;

[0142] For a point P on the curve i (x i ,y i ), and the corresponding lane line midpoint coordinates are:

[0143]

[0144] Among them, is the scale of the actual lane width in the screen coordinate system. From this, we can obtain the coordinates of any point on the centerline of the lane, and then solve the reference trajectory of the control system.

[0145] S7. Based on the tracking error model, a model predictive controller is designed. The state space expression under the vehicle tracking error model is:

[0146] The vehicle model predictive control algorithm is:

[0147]

[0148] in, and e d are respectively the heading angle error and the relative error from the vehicle centerline to the reference path, v x is the component of the velocity at the center of the vehicle's rear axle on the x-axis of the vehicle coordinate system, l fr is the vehicle wheelbase, δ f is the front wheel steering angle, κ ref is the radius of curvature at the reference path point;

[0149] The design optimization performance index function is as follows:

[0150]

[0151] U min ≤HΔU k +U k ≤U max ;

[0152] Among them, ΔX(k) is the error between the state quantity and the set value at time k, U(k) is the control quantity at time k, and H is the system matrix, which is used to convert the control input change ΔU(k) ​​into the representation in the system state space, ΔU k is the control increment at time k, Q(k) represents the weighted output error at time k, P(k) represents the weighted control amount at time k, and U max and U min are the maximum and minimum values ​​of the control quantity at time k, respectively. Solve this quadratic programming problem and, for each time k, obtain the control increment sequence in the control domain P from that time:

[0153] ΔU k =[Δu k ,Δu k+1 ,…,Δu k+P-1 ] T ;

[0154] After obtaining the future control increment sequence, use ΔU k The first element is used as the actual control output u n The increment of (k-1) is used as the output of the controlled vehicle. The control amount at this time becomes;

[0155] u n (k) = u n(k-1)+Δu k .

[0156] like Figure 5-Figure 6 As shown in the figure, the model predictive control MPC provided by the present invention and the PD (proportional-differential) control are comprehensively compared in the test. The vehicle starts from rest and the speed is set to 1m / s. Both methods can stably track the center reference line of the upper lane. It can be found that the oscillation of the model predictive controller is small and the rise time is short. At the same time, the front wheel steering angle output by the model predictive controller has small oscillation and the output is smoother.

[0157] like Figure 7 As shown, in the curve scenario, the vehicle body is controlled to travel at speeds of 0.3m / s, 0.5m / s, and 1m / s respectively during the movement on the road. It can be seen that compared with the PD controller, whenever the speed changes, the front wheel steering angle output by the MPC proposed in the present invention only changes slightly. At this time, the lateral deviation of the system can quickly enter the error band, which indicates its good robustness.

[0158] Therefore, the present invention adopts the above-mentioned lane line detection and lane keeping method based on machine vision. On the one hand, the anchor-based CNN method can more accurately identify and locate lane lines in complex environments. Compared with traditional methods, it effectively improves the robustness under lighting changes and occlusions, and facilitates accurate lane identification. On the other hand, combined with MPC for lateral control, it not only enhances the adaptability to various road conditions, but also ensures the stability and accuracy of the vehicle's lateral movement, significantly improving the practical application capabilities of the system.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A lane line detection and lane keeping method based on machine vision, characterized in that: The following steps are involved: S1. Lane line model and feature extraction method based on hybrid anchor; S2. Design lane detection network structure based on attention mechanism; S3, design network prediction head based on network structure characteristics; S4. Design network loss function based on lane line prior feature morphology; S5. Design network parameter optimization and pruning methods based on L1 regularization term and scaling factor; S6. Design a vehicle tracking reference trajectory generation algorithm based on inverse perspective transformation and least squares method; S7. Design a vehicle model predictive control algorithm based on the vehicle dynamics model in the Frenet coordinate system.

2. The lane line detection and lane keeping method based on machine vision according to claim 1, characterized in that: In S1, the lane line model and feature extraction method are established as follows: i∈{1,…,N row } i∈{1,…,N col }; Among them, δ represents the step size of the feature map F corresponding to the original image, N row and N col are the number of row anchors and column anchors, respectively, (x i ,y i ) and (x j ,y j ) are the coordinates of the i-th row anchor and the j-th column anchor respectively; for each anchor, use the above x i and i , extract the mapping feature vector of the row anchor on the feature map from the feature map F Using x j and j , extract the feature vector mapped by the column anchor on the feature map from the feature map F W F and H F are the width and length of the feature map F, C F is the depth of the feature map F.

3. The lane line detection and lane keeping method based on machine vision according to claim 2, characterized in that: In S2, the lane detection network structure is equipped with an attention module, which includes a fully connected layer L att , for the anchor with index i, its corresponding local feature and As a fully connected layer L att Input, L att The output is the weight of the anchor mapping to the global feature It is used to fuse the local features corresponding to anchors other than the i-th anchor to form global features. For the anchor with index i, the corresponding global feature tensor is as follows: in, are the weights mapped from row anchors and column anchors to global features. For all anchors, matrix multiplication is used to generate feature tensors. Anchor, column anchors, then: in, 4. The lane line detection and lane keeping method based on machine vision according to claim 3, characterized in that: In S3, the network prediction head includes two prediction head branches, one is the prediction location branch P, and the other is the existence branch E. For the anchor with index i, its local feature Global Features Perform the splicing operation separately, and the output results are recorded as and and They are used as inputs of two parallel fully connected layers respectively, and finally learn a fixed-size target network T r and T c ; T r and T c The positioning branch in can be written as follows: in, and is the label of the coordinate mapping, is the rounding symbol, and Represents the number of sampling points on each row anchor, that is, the number of classes; and The existence branch indicates: in, and The labels for the coordinates exist, and the entire network needs to learn The branch finally outputs the lane prediction through two prediction positioning branches and two existence branches. The output part of the network is expressed as: P, E = f(flatten(F)); Among them, f is the classifier, flatten(·) is the flattening operation, which is used to flatten the input feature map into a one-dimensional tensor, and P,E contains (P r , P c , E r , E c ), P r and P c The dimensions are and E r and E c The dimensions are and N row and N col are the number of row and column anchors, respectively.

5. The lane line detection and lane keeping method based on machine vision according to claim 4, characterized in that: In S4, the network loss function is: L=L cls +αL exp +βL ext ; Among them, L cls is the classification loss, L exp is the expected loss, L ext is the loss, α and β are proportional coefficients; Classification loss L cls Defined as: Among them, L CE is the cross entropy loss, is the prediction of the i-th lane assigned to the row anchor, and for the j-th row anchor, yes The corresponding classification label, is the prediction of the mth lane assigned to the column anchor, corresponding to the nth column anchor, yes The corresponding classification label of , onehot(·) is the one-hot encoding function; The probability that each preset point is a lane is defined as: in, is the probability of assigning the ith lane to the row anchor, is the probability of the mth lane assigned to the column anchor, softmax(·) is the probability normalization function, and the expected position is expressed as: in, is the expectation of the row anchor corresponding to the preset point, is the expectation of the column anchor corresponding to the preset point; Expected loss L exp Defined as: There is a loss function L ext Defined as:

6. The lane line detection and lane keeping method based on machine vision according to claim 5, characterized in that: In S5, network parameter optimization and pruning methods include: L=∑ (x，y) l(f(x,W),y)+λ∑ γ∈Γ g(γ); Where (x, y) represents the input and target of the network to be trained, W represents the training weight, and g(·) is the penalty for the scaling factor γ; The dividing point between the part to be retained and the part to be deleted is expressed by the following formula: P{N>N(γ n )}=a Among them, N represents the distribution function of all scaling factors γ after sparseness, α represents the set of channels to be retained, and N(γ n ) represents the upper α quantile.

7. The lane line detection and lane keeping method based on machine vision according to claim 6, characterized in that: In S6, the inverse perspective matrix in the vehicle tracking reference trajectory generation algorithm changes as follows: Among them, (u, v) is the pixel coordinate in the image coordinate system, α x =f / dx is the scaling factor on the u axis, α y =f / dy is the scaling factor on the v axis, f is the focal length of the camera, (u0,v0) is the position of the origin of the image coordinate system in the pixel coordinate system; R is the rotation matrix used to rotate the point in the world coordinate system to the camera coordinate system matrix, t is the translation vector used to translate the rotated point from the world coordinate system to the camera coordinate system; [X w Y w Z w 1] T is the homogeneous coordinate matrix in world coordinates, A is the internal parameter matrix of the onboard camera, and matrix M is the external parameter matrix of the onboard camera; A cubic polynomial represents a lane on the ground and has the following form: <h2 style=";text-align:left;direction:ltr">X = a3Z<h2 style=";text-align:left;direction:ltr"> 3 <h2 style=";text-align:left;direction:ltr"> +a2Z<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +a1Z+a0; Where (X, Z) represents a point on the ground, and a3 is not equal to 0; For a point P on the curve i (x i ,y i ), and the corresponding lane line midpoint coordinates are: Among them, is the scale of the actual lane width in the screen coordinate system, and the coordinates of any point on the lane centerline are obtained, and then the reference trajectory of the control system is solved.

8. The lane line detection and lane keeping method based on machine vision according to claim 7, characterized in that: In S7, the vehicle model predictive control algorithm is: in, and e d are respectively the heading angle error and the relative error from the vehicle centerline to the reference path, v x is the component of the velocity at the center of the vehicle's rear axle on the x-axis of the vehicle coordinate system, l fr is the vehicle wheelbase, δ f is the front wheel steering angle, κ ref is the curvature radius at the reference path point; the design optimization performance index function is as follows: IN min ≤HΔU k +U k ≤U max ; Among them, ΔX(k) is the error between the state quantity and the set value at time k, U(k) is the control quantity at time k, H is the system matrix, ΔU k is the control increment at time k, WQ(k) represents the weighted output error at time k, P(k) represents the weighted control amount at time k, and U max and U min are the maximum and minimum values ​​of the control quantity at time k, respectively. Solve this quadratic programming problem and, for each time k, obtain the control increment sequence in the control domain P from that time: ΔU k =[Δu k ,Δu k+1 ,…,Δu k+P-1 ] T ; After obtaining the future control increment sequence, use ΔU k The first element is used as the actual control output u n The increment of (k-1) is used as the output of the controlled vehicle. The control amount at this time becomes; u n (k)=u n (k-1)+Δu k 。

Images