A Lane Line Detection and Lane Keeping Method Based on Machine Vision

Abstract

The present invention discloses a lane line detection and lane keeping method based on machine vision, comprising the following steps: S1, establishing a model and feature extraction method for lane lines based on hybrid anchors; S2, designing a lane line detection network structure based on an attention mechanism; S3, designing a network prediction head based on the characteristics of the network structure; S4, designing a network loss function based on the prior feature form of lane lines; S5, designing a network parameter optimization method based on the L1 regularization term and a scaling factor; S6, designing a vehicle tracking reference trajectory generation algorithm based on inverse perspective transformation and the least squares method. By adopting the above-mentioned lane line detection and lane keeping method based on machine vision, the present invention significantly improves the stability and safety of driverless vehicles in complex road environments by combining an advanced lane line detection network and a lane keeping control algorithm, and has clear theoretical significance and important application value.

Description

Technical Field

[0001] The present invention relates to the field of driverless perception and control, and particularly to a lane line detection and lane keeping method based on machine vision. Background Art

[0002] Driverless technology has attracted a great deal of attention after the DARPA Challenge in the early 2000s, including a number of high-tech companies, major universities, and automotive OEMs. Driverless technology generally consists of three parts, namely, environmental perception, decision-making and planning, and motion control. Environmental perception and motion control are the key links of the driverless system, and the lane line detection and lane keeping technology based on machine vision plays a core role therein.

[0003] In the prior art, lane line detection often uses methods such as edge detection or Hough transform. These methods cannot well adapt to various lighting and weather conditions, and have limited ability to handle occlusion and noise problems. PID control is often used in the lateral control of vehicles. This method lacks consideration of vehicle dynamics constraints and future states and cannot effectively handle road curves and obstacles in the face of multiple constraints and complex dynamic environments. Based on the above problems, how to better adapt to complex environmental conditions and effectively handle road conditions in lane line detection and vehicle lateral control has become a key issue.

[0004] The anchor-based method (Convolutional Neural Network CNN) provides an efficient and accurate solution for identifying and locating lane lines. This method generates possible lane line positions through anchor boxes and uses CNN to extract image features for fine classification and positioning. Compared with the traditional edge detection or Hough transform methods, the anchor-based method can show stronger robustness and accuracy in complex environments, can better adapt to various lighting and weather conditions, and can effectively handle occlusion and noise problems.

[0005] Model Predictive Control (MPC) generates optimal control signals by online optimizing and predicting future vehicle motions to ensure that the vehicle travels stably along the lane line. Compared with the traditional PID control, MPC can consider vehicle dynamics constraints and future states and provide a more accurate and smooth control effect. In addition, MPC has significant advantages in dealing with multiple constraints and complex dynamic models and can effectively handle road curves and obstacles. Summary of the Invention

[0006] The object of the present invention is to provide a lane line detection and lane keeping method based on machine vision, which can effectively solve the problems of large recognition errors of traditional methods in complex environments and unstable lane keeping, and can significantly improve the accuracy of lane line detection and the robustness of vehicle lateral control.

[0007] To achieve the above object, the present invention provides a lane line detection and lane keeping method based on machine vision, comprising the following steps:

[0008] S1. Establish a lane line model and feature extraction method based on hybrid anchors;

[0009] S2. Design a lane line detection network structure based on the attention mechanism;

[0010] S3. Design a network prediction head based on the network structure features;

[0011] S4. Design a network loss function based on the prior feature morphology of the lane line;

[0012] S5. Design a network parameter optimization and pruning method based on the L1 regularization term and the scaling factor;

[0013] S6. Design a vehicle tracking reference trajectory generation algorithm based on inverse perspective transformation and the least square method;

[0014] S7. Design a vehicle model predictive control algorithm based on the vehicle dynamics model in the Frenet coordinate system.

[0015] Preferably, in S1, the lane line model and feature extraction method are established as follows:

[0016]

[0017] Wherein, δ represents the stride of the feature map F corresponding to the original image, N row and N col are the numbers of row anchors and column anchors respectively, (x i , y i ) and (x j , y j ) are the coordinates of the i-th row anchor and the j-th column anchor respectively; for each anchor, using the above x i and y i , the mapped feature vector of the row anchor on the feature map is extracted from the feature map F Using x j and y j , the feature vector mapped by the column anchor on the feature map is extracted from the feature map F W F and H F are the width and length of the feature map F respectively, and C F is the depth of the feature map F.

[0018] Preferably, in S2, an attention module is set in the lane line detection network structure, and the attention module includes a fully connected layer L att , for the anchor with index i, its corresponding local feature and As the input of the fully connected layer L att The output of L att is the weight for mapping the anchor to the global feature used to fuse the local features corresponding to other anchors except the i-th anchor, so as to form the global feature. For the anchor with index i, its corresponding global feature tensor is as follows:

[0019]

[0020]

[0021] where are the weights for mapping the row anchor and the column anchor to the global feature respectively. For all anchors, matrix multiplication is used to generate the feature tensor. For row anchors, column anchors, then there is:

[0022]

[0023] where and

[0024] Preferably, in S3, the network prediction head includes two prediction head branches, one is the prediction localization branch P, and the other is the existence branch E. For the anchor with index i, its local feature global feature are respectively concatenated, and the output results are respectively denoted as and and are respectively used as the inputs of 2 parallel fully connected layers, and finally the target network T r and T c ;

[0025] T r and T c The localization branch in can be written in the following form:

[0026]

[0027] where and are the labels for coordinate mapping, is the rounding symbol, and represent the number of upsampling points on each row anchor, that is, the number of classes;

[0028] and The existence branch of represents:

[0029]

[0030] Among them, and are labels for the existence of coordinates, and the entire network needs to learn branches. Through two prediction and localization branches and two existence branches, the prediction of the lane is finally output. The output part of the network is expressed as:

[0031] P, E = f(flatten(F));

[0032] Among them, f is the classifier, and flatten(·) is the flattening operation used to flatten the input feature map into a one-dimensional tensor. P and E contain (P r , P c , E r , E c ), and the dimensions of P r and P c are and respectively, and the dimensions of E r and E c are and N row and N col are the numbers of row anchors and column anchors respectively.

[0033] Preferably, in S4, the network loss function:

[0034] L = L cls + αL exp + βL ext ;

[0035] Among them, L cls is the classification loss, L exp is the expectation loss, L ext is the existence loss, and α and β are proportionality coefficients;

[0036] The classification loss L cls is defined as:

[0037]

[0038] Among them, L CE is the cross-entropy loss, is the prediction of the i-th lane assigned to the row anchor. For the j-th row anchor, is 's corresponding classification label, is the prediction of the m-th lane assigned to the column anchor. Corresponding to the n-th column anchor, is The corresponding classification label of , onehot(·) is the one-hot encoding function;

[0039] The probability that each preset point is a lane is defined as:

[0040]

[0041] in, is the probability of assigning the ith lane to the row anchor, is the probability of the mth lane assigned to the column anchor, softmax(·) is the probability normalization function, and the expected position is expressed as:

[0042]

[0043] in, is the expectation of the row anchor corresponding to the preset point, is the expectation of the corresponding column anchor of the preset point;

[0044] Expected loss L exp Defined as:

[0045]

[0046] There is a loss function L ext Defined as:

[0047]

[0048] Preferably, in S5, the network parameter optimization and pruning method includes:

[0049] L=∑ (x，y) l(f(x,W),y)+λ∑ γ∈Γ g(γ);

[0050] Where (x, y) represents the input and target of the network to be trained, W represents the training weight, and g(·) is the penalty for the scaling factor γ;

[0051] The dividing point between the part to be retained and the part to be deleted is expressed by the following formula:

[0052] P{N>N(γ n )}=α

[0053] Among them, N represents the distribution function of all scaling factors γ after sparseness, α represents the set of channels to be retained, and N(γ n ) represents the upper α quantile.

[0054] Preferably, in S6, the inverse perspective matrix in the vehicle tracking reference trajectory generation algorithm changes as follows:

[0055]

[0056] Among them, (u, v) are the pixel coordinates in the image coordinate system, and α x = f / dx is the scaling factor on the u-axis, and α y = f / dy is the scaling factor on the v-axis. f is the focal length of the camera, and (u0, v0) is the position of the origin of the image coordinate system in the pixel coordinate system; R is the rotation matrix used to rotate the points in the world coordinate system to the camera coordinate system matrix, and t is the translation vector used to translate the rotated points from the world coordinate system to the camera coordinate system; [X w Y w Z w 1] T is the homogeneous coordinate in the world coordinate, matrix A is the internal parameter matrix of the vehicle-mounted camera, and matrix M is the external parameter matrix of the vehicle-mounted camera;

[0057] A cubic polynomial is used to represent a lane on the ground, and the form is as follows:

[0058] X = a3Z 3 + a2Z 2 + a1Z + a0;

[0059] Among them, (X, Z) represent the points on the ground, and a3 is not equal to 0;

[0060] For the point P i (x i , y i ) on the curve, the corresponding midpoint coordinates of the lane line are:

[0061]

[0062] Among them, L is the scale of the actual lane width in the picture coordinate system. The coordinates of any point on the center line of the lane are obtained, and then the reference trajectory of the control system is solved.

[0063] Preferably, in S7, the vehicle model predictive control algorithm is:

[0064]

[0065] Among them, and e d are the heading angle error and the relative error between the vehicle center line and the reference path respectively, v x is the component of the speed at the center of the rear axle of the vehicle on the x-axis of the vehicle body coordinate system, l fr is the wheelbase of the vehicle, δ f is the front wheel steering angle, κ ref is the radius of curvature at the reference path point; The design optimization performance index function is as follows:

[0066]

[0067] U min ≤HΔU k +U k ≤U max ;

[0068] where ΔX(k) is the error between the state quantity at time k and the set value, U(k) is the control quantity at time k, H is the system matrix used to convert the change in control input ΔU(k) into a representation in the system state space, and ΔU k is the control increment at time k, Q(k) represents the output error weighting at time k, P(k) represents the control quantity weighting at time k, and U max and U min are the maximum and minimum values of the control quantity at time k respectively. Solving this quadratic programming problem, for each time k, a sequence of control increments within the control time domain P starting from that time is obtained:

[0069] ΔU k =[Δu k ,Δu k+1 ,…,Δu k+P-1 T ;

[0070] After obtaining the future sequence of control increments, use the first element of ΔU k as the increment of the actual control output u n (k - 1), and use this increment as the output of the vehicle to be controlled. At this time, the control quantity becomes;

[0071] u n (k)=u n (k - 1)+Δu k .

[0072] Therefore, the present invention adopts the above-mentioned lane line detection and lane keeping method based on machine vision, and the beneficial effects are as follows:

[0073] (1) The present invention applies model predictive control (MPC) to the lane keeping strategy, which can consider vehicle dynamics constraints and future states, provide accurate and smooth control effects, and enable the vehicle to effectively handle road curves and obstacles.

[0074] (2) The present invention adopts a method based on an anchor, that is, using a convolutional neural network (CNN), which can perform fine classification and localization of image features, show stronger robustness and accuracy in complex environments, be better adapted to various lighting and weather conditions, and effectively handle occlusion and noise problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 ​It is a schematic diagram of an unmanned driving platform according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0076] Figure 2 It is an overall flowchart of an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0077] Figure 3 It is a schematic diagram of a network structure according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0078] Figure 4 It is a schematic diagram of the network pruning principle according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention;

[0079] Figure 5 It is a diagram of tracking error and front wheel steering angle output in a straight road scenario according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention. Among them, (a) is the lateral tracking error output under PD control, (b) is the lateral tracking error output under MPC control, (c) is the front wheel steering angle output under PD control, and (d) is the front wheel steering angle output under MPC control;

[0080] Figure 6 It is a diagram of tracking error and front wheel steering angle output in a curved road scenario according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention. Among them, (a) is the lateral deviation output under PD control, (b) is the lateral deviation output under MPC control, (c) is the front wheel steering angle output under PD control, and (d) is the front wheel steering angle output under MPC control;

[0081] Figure 7 It is a diagram of tracking error and front wheel steering angle output in a curved road variable speed scenario according to an embodiment of a lane line detection and lane keeping method based on machine vision of the present invention. Among them, (a) is a comprehensive diagram of the lateral deviation output and speed change under PD control, (b) is a comprehensive diagram of the lateral deviation output and speed change under MPC control, (c) is a comprehensive diagram of the front wheel steering angle output and speed change under PD control, and (d) is a comprehensive diagram of the front wheel steering angle output and speed change under MPC control. Detailed implementation manner

[0082] The technical solutions of the present invention will be further described below with reference to the drawings and embodiments.

[0083] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meanings understood by those of ordinary skill in the art to which the present invention belongs.

[0084] Embodiment

[0085] Such as Figure 1As shown in the figure, the lane line detection and lane keeping control scheme provided by the present invention is applied to an unmanned driving platform. The platform uses Jetson Nano as the main controller, and an optical camera and a UWB positioning module as sensors. The present invention mainly obtains the vehicle position through the UWB positioning module.

[0086] As Figure 2 shown, a lane line detection and lane keeping method based on machine vision includes the following steps:

[0087] S1. Establish a model and feature extraction method for lane lines based on hybrid anchors;

[0088] Establish a model for lane lines based on hybrid anchors:

[0089]

[0090] Among them, δ represents the stride of the feature map F corresponding to the original image, N row and N col are the numbers of row anchors and column anchors respectively, (x i , y i ) and (x j , y j ) are the coordinates of the i-th row anchor and the j-th column anchor respectively; for each anchor, using the above x i and y i , extract the mapped feature vector of the row anchor on the feature map from the feature map F Use x j and y j , extract the feature vector mapped by the column anchor on the feature map from the feature map F W F and H F are the width and length of the feature map F respectively, and C F is the depth of the feature map F.

[0091] S2. As Figure 3 shown, on the basis of the extraction of feature vectors, design a lane line detection network structure according to the attention mechanism;

[0092] The attention module includes a fully connected layer L att , for the anchor with index i, its corresponding local features and are used as the input of the fully connected layer L att , and the output of L att is the weight of the anchor mapped to the global feature used to fuse the local features corresponding to other anchors except the i-th anchor, so as to form a global feature. For the anchor with index i, its corresponding global feature tensor is as follows:

[0093]

[0094] Among them, are the weights of the row anchor and column anchor mapped to the global features respectively. For all anchors, the generation of the feature tensor can be quickly implemented using matrix multiplication. For row anchors, column anchors, then there is:

[0095]

[0096] Among them, and

[0097] S3. Design the network prediction head based on the network structure features;

[0098] The network prediction head includes two prediction head branches. One is the prediction localization branch P, and the other is the existence branch E. For the anchor with index i, its local feature global feature are respectively concatenated, and the output results are respectively denoted as and and are respectively used as the inputs of two parallel fully connected layers, and finally the target network T with a fixed size is learned r and T c ;

[0099] T r and T c The localization branch in can be written in the following form:

[0100]

[0101] Among them, and are the labels of the coordinate mapping, is the rounding symbol, and represent the number of upsampling points on each row anchor, that is, the number of classes.

[0102] and The existence branch of represents:

[0103]

[0104] Among them, and are the labels of the coordinate existence. The entire network needs to learn Branch, through two prediction location branches and two existence branches, finally outputs the prediction of the lane. The output part of the network can be expressed as:

[0105] P, E = f(flatten(F));

[0106] where f is the classifier, and flatten(·) is the flattening operation used to flatten the input feature map into a one-dimensional tensor. P and E contain (P r , P c , E r , E c ), and the dimensions of P r and P c are respectively and The dimensions of E r and E c are respectively and N row and N col are the numbers of row anchors and column anchors respectively.

[0107] S4. Design the network loss function based on the prior feature morphology of the lane line;

[0108] L = L cls + αL exp + βL ext ;

[0109] where L cls is the classification loss, L exp is the expectation loss, L ext is the existence loss, and α and β are proportionality coefficients;

[0110] The classification loss L cls is defined as:

[0111]

[0112] where L CE is the cross-entropy loss, is the prediction of the i-th lane assigned to the row anchor. For the j-th row anchor, is 's corresponding classification label, is the prediction of the m-th lane assigned to the column anchor. Corresponding to the n-th column anchor, is 's corresponding classification label, and onehot(·) is the one-hot encoding function;

[0113] The probability that each preset point is a lane is defined as:

[0114]

[0115] Among them, is the probability of the i-th lane assigned to the row anchor, is the probability of the m-th lane assigned to the column anchor, softmax(·) is the probability normalization function, and the expected value of the position is expressed as:

[0116]

[0117] Among them, is the expectation of the preset point corresponding to the row anchor, is the expectation of the preset point corresponding to the column anchor;

[0118] The expected loss L exp is defined as:

[0119]

[0120] There is a loss function L ext defined as:

[0121]

[0122] S5. As Figure 4 shown, after designing the lane line detection network, the network parameter optimization is designed based on the L1 regularization term and the scale factor;

[0123] L = ∑ (x，y) l(f(x, W), y) + λ∑ γ∈Γ g(γ);

[0124] Among them, (x, y) represents the input and target of the network to be trained, W represents the training weight, and g(·) is the penalty for the scaling factor γ;

[0125] The demarcation point between the part to be retained and the part to be deleted is represented by the following formula:

[0126] P{N > N(γ n )} = α

[0127] Among them, N represents the distribution function of all scaling factors γ after sparsification, α represents the set of channels of the part to be retained, and N(γ n ) represents the upper α quantile.

[0128] S6. Based on the inverse perspective transformation and the least square method, a vehicle tracking reference trajectory generation algorithm is designed. The inverse perspective matrix changes as follows:

[0129]

[0130] Among them, (u, v) are the pixel coordinates in the image coordinate system, α x = f / dx is the scaling factor on the u-axis, αy = f / dy is the scaling factor on the v-axis, f is the focal length of the camera, and (u0, v0) is the position of the origin of the image coordinate system in the pixel coordinate system; R is the rotation matrix used to rotate a point in the world coordinate system to the camera coordinate system matrix, and t is the translation vector used to translate the rotated point from the world coordinate system to the camera coordinate system; [X w Y w Z w 1] T is the homogeneous coordinate matrix in the world coordinates. Matrix A is the internal parameter matrix of the vehicle-mounted camera, and matrix M is the external parameter matrix of the vehicle-mounted camera;

[0131] The cubic polynomial represents a lane on the ground and has the following form:

[0132] X = a3Z 3 + a2Z 2 + a1Z + a0;

[0133] Among them, (X, Z) represents a point on the ground, and a3 is not equal to 0;

[0134] For a point P i (x i , y i ) on the curve, the corresponding midpoint coordinates of the lane line are:

[0135]

[0136] Among them, L is the scale of the actual lane width in the image coordinate system. Thus, we can obtain the coordinates of any point on the center line of the lane and further solve the reference trajectory of the control system.

[0137] S7. Based on the tracking error model, design a model predictive controller. The state space expression under the vehicle tracking error model is:

[0138] The vehicle model predictive control algorithm is:

[0139]

[0140] Among them, and e d are the heading angle error and the relative error between the vehicle center line and the reference path respectively, v x is the component of the speed at the center of the rear axle of the vehicle on the x-axis of the vehicle body coordinate system, l fr is the wheelbase of the vehicle, δ f is the front wheel steering angle, and κ ref is the radius of curvature at the reference path point;

[0141] Design the optimization performance index function as follows:

[0142]

[0143] U min ≤HΔU k +U k ≤U max ;

[0144] Where ΔX(k) is the error between the state quantity at time k and the set value, U(k) is the control quantity at time k, H is the system matrix used to convert the change in control input ΔU(k) into the representation in the system state space, and ΔU k is the control increment at time k, Q(k) represents the output error weighting at time k, P(k) represents the control quantity weighting at time k, and U max and U min are the maximum and minimum values of the control quantity at time k respectively. Solving this quadratic programming problem, for each time k, the control increment sequence within the control time domain P starting from this time is obtained:

[0145] ΔU k =[Δu k ,Δu k+1 ,…,Δu k+P-1 T ;

[0146] After obtaining the future control increment sequence, use the first element of ΔU k as the increment of the actual control output u n (k - 1), and take this increment as the output of the controlled vehicle. At this time, the control quantity becomes;

[0147] u n (k)=u n (k - 1)+Δu k .

[0148] As Figures 5 - 6 shown, in the test, the model predictive control MPC provided by the present invention was comprehensively compared with PD (Proportional - Derivative) control. The vehicle starts from a standstill and the speed is set to 1 m / s. Both methods can stably track the center reference line of the lane. It can be found that the oscillation of the model predictive controller is smaller and the rise time is shorter. At the same time, the oscillation of the front wheel steering angle output by the model predictive controller is smaller and the output is smoother.

[0149] As Figure 7 shown, in the curved road scenario, when the vehicle is moving on this road at speeds of 0.3 m / s, 0.5 m / s, and 1 m / s respectively, it can be seen that compared with the PD controller, whenever the speed changes, the front wheel steering angle output by the MPC proposed by the present invention only changes by a very small amount, and at this time the lateral deviation of the system can quickly enter the error band, which indicates its good robustness.​

[0150] Therefore, the present invention adopts the above-mentioned lane line detection and lane keeping method based on machine vision. On the one hand, the anchor-based CNN method can more accurately identify and locate lane lines in complex environments. Compared with traditional methods, it effectively improves the robustness under light changes and occlusions, facilitating accurate lane recognition. On the other hand, combined with MPC for lateral control, it not only enhances the adaptability to diverse road conditions, but also ensures the smoothness and accuracy of the vehicle's lateral movement, significantly improving the practical application ability of the system.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements do not enable the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A lane line detection and lane keeping method based on machine vision, characterized in that, Including the following steps: S1. Establish a lane line model and feature extraction method based on hybrid anchors; S2. Design a lane line detection network structure based on the attention mechanism; S3. Design a network prediction head based on the network structure features; S4. Design a network loss function based on the prior feature morphology of the lane line; S5. Design a network parameter optimization and pruning method based on the L1 regularization term and the scaling factor; S6. Design a vehicle tracking reference trajectory generation algorithm based on inverse perspective transformation and the least squares method; S7. Design a vehicle model predictive control algorithm based on the vehicle dynamics model in the Frenet coordinate system; In S2, an attention module is provided in the lane line detection network structure, and the attention module includes a fully connected layer L att , for the anchor with index i, its corresponding local feature and serve as the inputs of the fully connected layer L att , and the output of L att is the weight of the anchor mapped to the global feature used to fuse the local features corresponding to the other anchors except the i-th anchor, so as to form the global feature. For the anchor with index i, its corresponding global feature tensor is as follows: Among them, are the weights of the row anchor and the column anchor mapped to the global feature respectively. For all anchors, matrix multiplication is used to generate the feature tensor. For row anchors, column anchors, there is: Among them, In S3, the network prediction head includes two prediction head branches. One is the prediction localization branch P, and the other is the existence branch E. For the anchor with index i, its local feature global feature perform concatenation operations respectively, and the output results are denoted as and and serve as the inputs of two parallel fully connected layers respectively, and finally learn the target network T of a fixed size r and T c ; T r and T c The positioning branches in Among them, and are labels for coordinate mapping, is the rounding symbol, and represent the number of upsampling points on each row anchor, that is, the number of classes; and The existence of a branch indicates that: Among them, and are labels for the existence of coordinates, and the entire network needs to learn branches. Through two prediction and localization branches and two existence branches, the prediction of the lane is finally output. The output part of the network is expressed as: P, E = f(flatten(F)); Among them, f is the classifier, and flatten(·) is the flattening operation used to flatten the input feature map into a one-dimensional tensor. P and E contain (P r , P c , E r , E c ), and the dimensions of P r and P c are and respectively. The dimensions of E r and E c are and respectively. N row and N col are the numbers of row anchors and column anchors respectively.

2. The lane line detection and lane keeping method based on machine vision according to claim 1, characterized in that: In S1, the lane line model and feature extraction method are established as follows: Among them, δ represents the stride of the feature map F corresponding to the original image, N row and N col are the numbers of row anchors and column anchors respectively. (x i , y i ) and (x j , y j ) are the coordinates of the i-th row anchor and the j-th column anchor respectively. For each anchor, using the above x i and y i , the mapped feature vector of the row anchor on the feature map is extracted from the feature map F Using x j and y j , the feature vector mapped by the column anchor on the feature map is extracted from the feature map F W F and H F are the width and length of the feature map F respectively, and C F is the depth of the feature map F.

3. A lane line detection and lane keeping method based on machine vision according to claim 2, characterized in that: In S4, the network loss function: L = L cls + αL exp + βL ext ; Among them, L cls is the classification loss, L exp is the expected loss, L ext is the existence loss, and α and β are proportionality coefficients; Classification loss L cls is defined as: Among them, L CE is the cross-entropy loss, is the prediction of the i-th lane assigned to the row anchor. For the j-th row anchor, is the corresponding classification label of, is the prediction of the m-th lane assigned to the column anchor, corresponding to the n-th column anchor, is the corresponding classification label of, and onehot(·) is the one-hot encoding function; The probability that each preset point is a lane is defined as: where, is the probability of the i-th lane assigned to the row anchor, is the probability of the m-th lane assigned to the column anchor, softmax(·) is the probability normalization function, and the expected value of the position is expressed as: Among them, is the expectation of the row anchor corresponding to the preset point, is the expectation of the column anchor corresponding to the preset point; Expected loss L exp is defined as: There is a loss function L ext which is defined as:

4. A lane line detection and lane keeping method based on machine vision according to claim 3, characterized in that: In S5, the network parameter optimization and pruning method includes: L = ∑ (x，y) l(f(x, W), y) + λ∑ γ∈Γ g(γ); where (x, y) represents the input and target for training the network, W represents the training weight, and g(·) is the penalty on the scaling factor γ; The demarcation point between the part to be retained and the part to be deleted is represented by the following formula: P{N > N(γ n )} = α Among them, N represents the distribution function of all scaling factors γ after sparsification, α represents the set of channels to be retained, and N(γ n ) represents the upper α quantile.

5. A lane line detection and lane keeping method based on machine vision according to claim 4, characterized in that: In S6, the change of the inverse perspective matrix in the vehicle tracking reference trajectory generation algorithm is as follows: where (u, v) are pixel coordinates in the image coordinate system, and α x = f / dx is the scaling factor on the u-axis, and α y = f / dy is the scaling factor on the v-axis, f is the focal length of the camera, and (u0, v0) is the position of the origin of the image coordinate system in the pixel coordinate system; R is the rotation matrix used to rotate a point in the world coordinate system to the camera coordinate system matrix, and t is the translation vector used to translate the rotated point from the world coordinate system to the camera coordinate system; [X w Y w Z w 1] T is the homogeneous coordinate matrix in the world coordinates. Matrix A is the internal parameter matrix of the vehicle-mounted camera, and matrix M is the external parameter matrix of the vehicle-mounted camera; A cubic polynomial represents a lane on the ground, in the form as follows: X = a3Z 3 + a2Z 2 + a1Z + a0; where (X, Z) represents the point on the ground and a3 is not equal to 0; For point P on the curve i (x i , y i ), the corresponding midpoint coordinates of the lane line are: where L is the scale of the actual lane width in the image coordinate system, the coordinates of any point on the center line of the lane are obtained, and then the reference trajectory of the control system is solved.

6. A lane line detection and lane keeping method based on machine vision according to claim 5, characterized in that: In S7, the vehicle model predictive control algorithm is: wherein, and e d are the heading angle error and the relative error from the vehicle center line to the reference path respectively, v x is the component of the speed at the center of the vehicle's rear axle on the x-axis of the vehicle body coordinate system, l fr is the wheelbase of the vehicle, δ f is the front wheel steering angle, κ ref is the radius of curvature at the reference path point; the design optimization performance index function is as follows: U min ≤HΔU k +U k ≤U max ; where ΔX(k) is the error between the state quantity at time k and the set value, U(k) is the control quantity at time k, H is the system matrix, and ΔU k is the control increment at time k, Q(k) represents the output error weighting at time k, P(k) represents the control quantity weighting at time k, and U max and U min are the maximum and minimum values of the control quantity at time k respectively. By solving this quadratic programming problem, for each time k, a sequence of control increments within the control time domain P starting from that time is obtained: ΔU k = [Δu k , Δu k+1 , …, Δu k+P-1 T ;​ After obtaining the future control increment sequence, use the first element of ΔU k as the actual control output u n (k - 1) increment, and use this increment as the output of the controlled vehicle. At this time, the control quantity becomes; u n u(k) = n u(k - 1)+Δu k .

Images