A lane detection method integrating spatiotemporal information
Through the lane line detection method that integrates space-time information, the feature extraction network and classification network are used, and combined with the optical flow method, the problems of poor real-time and low accuracy of lane line detection in the prior art are solved, and more efficient lane line detection is achieved.
Patent Information
- Application Number
- CN202210312729.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-28
AI Technical Summary
The existing lane line detection methods have shortcomings in real-time and detection accuracy. The traditional methods have poor detection effects and the deep learning methods have slow detection speed, making it difficult to meet the needs of real-time and accuracy.
The lane line detection method that integrates space-time information is adopted, and the detection speed and accuracy are improved by constructing a model of feature extraction network and classification network, and using data augmentation and optical flow method.
The speed and accuracy of lane line detection are significantly improved, and the problems of poor real-time detection and low accuracy are solved. The detection accuracy can be improved without reducing the speed, especially when lane line is blocked.
Smart Images

Figure CN114821520B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of perception technology of autonomous driving vehicles, and specifically relates to a lane line detection method that integrates spatiotemporal information. Background Art
[0002] Lane detection technology is a crucial function in advanced driver assistance systems (ADAS), and has enabled systems such as automatic lane departure warning and lane keeping. Currently, lane detection is mainly divided into two categories: traditional lane detection methods and deep learning-based lane detection methods;
[0003] Traditional lane line detection methods usually convert the road image collected by the camera from a color domain image to a grayscale value image, and filter the image to remove invalid areas in the image and retain the road information; then perform edge detection on the processed image to extract lane lines. The above traditional methods have small calculation amount and stable performance, but the detection effect is poor and it is difficult to meet the actual detection needs;
[0004] The lane line detection method based on deep learning uses the idea of image segmentation to classify each pixel in the image to achieve lane line detection. The detection accuracy is often high, but the detection speed is very slow and it is difficult to meet the real-time requirements.
[0005] In addition, lane line detection technology often requires processing real-time video data, but existing lane line detection methods generally extract frames from the video and focus on obtaining as many lane line features as possible from a picture. Even if deep learning methods are used, the accuracy of lane line detection is also limited. Summary of the invention
[0006] In view of the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide a lane line detection method that integrates spatiotemporal information, so as to solve the problems of poor real-time performance and low detection accuracy of lane line detection in the prior art; the method of the present invention defines lane line detection as a set of finding the positions of lane lines in certain rows in an image, and extracts spatiotemporal context information in the video, and integrates it with the detected lane information, thereby greatly improving the speed and accuracy of lane line detection.
[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:
[0008] A lane line detection method integrating spatiotemporal information of the present invention comprises the following steps:
[0009] Step 1: Obtain an image with annotated lane lines as a training data set for the lane line detection model, and perform data augmentation on the images in the training data set;
[0010] Step 2: Build a lane detection model, use the lane detection data set after data enhancement to train the lane detection model, and adjust the model hyperparameters to minimize the model loss value;
[0011] Step 3: The lane line video acquired in real time is used as the video to be detected, and the video to be detected is subjected to frame extraction processing to obtain the image to be detected;
[0012] Step 4: Determine whether the image to be detected is the first frame image, if so, proceed to step 5; if not, proceed to step 6;
[0013] Step 5: Input the image to be detected into the lane detection model trained in step 2, and post-process the model output to obtain the final lane detection result;
[0014] Step 6: Input the image to be detected into the lane line detection model trained in step 2, post-process the model output to obtain the first lane line detection result, use the optical flow method to obtain the second lane line detection result, and fuse the first lane line detection result and the second lane line detection result to obtain the final lane line detection result.
[0015] Furthermore, the lane line image with annotations in step 1 is captured by a vehicle-mounted camera, and the captured image is manually annotated.
[0016] Furthermore, the labeling tool adopts Labelme labeling tool.
[0017] Furthermore, the data enhancement in step 1 includes: normalization, random rotation, random cropping, and random horizontal and vertical flipping.
[0018] Furthermore, the lane line detection model in step 2 adopts a network architecture of a feature extraction network plus a classification network, the feature extraction network is a resnet18 backbone network, and the classification network is a two-layer fully connected layer plus a Softmax function;
[0019] The resnet18 backbone network includes: several residual modules, convolutional layers, batch normalization layers, and Relu layers; among them,
[0020] The residual module is divided into: a first residual module and a second residual module;
[0021] The first residual module includes two branches: the main branch is a 3×3 convolution layer, a batch normalization layer, a Relu layer, a 3×3 convolution layer, a batch normalization layer, and a Relu layer series branch; the bypass branch is a branch whose input is directly connected to the output;
[0022] The second residual module includes two branches: a main branch is a branch consisting of a 3×3 convolution layer, a batch normalization layer, a Relu layer, a 3×3 convolution layer, a batch normalization layer, and a Relu layer connected in series; a bypass branch is a branch consisting of a 1×1 convolution layer, a batch normalization layer, and a Relu layer connected in series;
[0023] The specific connection method of the resnet18 backbone network is:
[0024] The first layer of the network is a 7×7 convolutional layer, which is connected in series in the forward propagation direction of the network: a maximum pooling layer, a Relu layer, two first residual modules, a second residual module, a first residual module, a second residual module, a first residual module, a second residual module, a first residual module, and an average pooling layer;
[0025] The input of the classification network is a one-dimensional vector, which is obtained by flattening the output features of the feature extraction network, and the output is a three-dimensional tensor Pro∈R C×h×(w+1) , where R represents the real number domain, C is the maximum number of lane lines detected, h is the number of row anchor boxes in the height direction of the network input image, and w is the number of cells divided in the width direction of the network input image.
[0026] Furthermore, the row anchor frame refers to thin and long strips divided at equal intervals in the height direction of the image.
[0027] Furthermore, the cells refer to the same number of small grids of equal width divided from each row anchor box in the image width direction.
[0028] In particular, the row anchor box is only segmented in the lower 70% of the image area. Different images are segmented to obtain the same number of row anchor boxes, but different row anchor box heights.
[0029] Furthermore, the three-dimensional tensor output by the classification network represents the lane line cell position probability map, Pro i,j,k Corresponding to the probability that the i-th lane line exists in the k-th cell on the j-th row anchor box, when the probability value on the w+1-th cell is the largest, it means that the i-th lane line does not exist on the j-th row anchor box.
[0030] Furthermore, the lane detection model training process in step 2 is as follows:
[0031] Step 21: Use mini-batch stochastic gradient descent to train the lane detection model with a batch size of 32 and an initial learning rate of 4×10 -4 , the momentum value is 0.9, and the weight decay is 1×10 -4 ;
[0032] Step 22: Select the cross entropy loss function as the loss function of the lane detection model, input training samples to train the lane detection model, and adjust the hyperparameters according to the changes in the loss function so that the model reaches the optimal solution and saves the corresponding weights.
[0033] Furthermore, adjusting the model hyperparameters in step 2 specifically includes: adjusting the height of the image row anchor box and the number of divided cells, wherein the row anchor box height changes from small to large, and the number of cells changes from large to small, to obtain the maximum row anchor box height and the minimum number of cells without decreasing the loss function of the lane line detection model.
[0034] Furthermore, the size of the image to be detected in step 3 is consistent with the image size in the training data set, otherwise the height of the row anchor box in the height direction of the image and the number of cells in the width direction should be adjusted proportionally according to the image size.
[0035] Furthermore, the step 5 specifically includes:
[0036] The lane line cell position probability map is detected according to the trained lane line detection model to obtain the lane line cell position:
[0037]
[0038] In the formula, Loc i,j represents the cell position of the i-th lane line on the j-th row anchor box, Pro i,j,k represents the probability that the i-th lane line exists in the k-th cell on the j-th row anchor box;
[0039] The pixel coordinates of the lane line in the detection image are obtained based on the lane line cell position, the divided row anchor box, and the number of cells divided in the width direction, as the first lane line detection result:
[0040] (x1,y1)=(Loc i,j *grid_w,anchor j )
[0041] Where x1 and y1 are the pixel coordinates of the lane line pixel in the first lane line detection result in the x-axis and y-axis directions of the image, grid_w represents the pixel width of the cell in the width direction of the image, and anchor j Represents the pixel coordinate value corresponding to the j-th row anchor box in the y-axis direction of the image;
[0042] The lane line is fitted according to the coordinates of each pixel point obtained in the first lane line detection result to obtain the first lane line calculation result x re1 :
[0043] x re1 =k11 y re1 2 +k 12 y re1 +k 13
[0044] In the formula, k 11 , k 12 , k 13 are the polynomial coefficients respectively;
[0045] The first lane line calculation result is the final lane line detection result.
[0046] Furthermore, the step 6 specifically includes:
[0047] Use the method in step 5 to detect the lane line of the current frame image and obtain the first calculation result of the lane line of the current frame image:
[0048] x re1 ′=k 11 ′y re1 ' 2 +k 12 ′y re1 ′+k 13 '
[0049] Extract the optical flow v corresponding to the lane line pixel in the previous frame image ls1 :
[0050]
[0051] In the formula, are the horizontal and vertical velocity components of the lane line pixel points respectively;
[0052] According to the optical flow corresponding to the lane line pixel point and the lane line first detection result in the previous frame image, the lane line pixel point (x ls1 ,y ls1 ) in the current frame image at the corresponding position (x ls2 ,y ls2 ), as the second detection result of the lane line in the current frame image:
[0053]
[0054]
[0055] In the formula, Δt represents the time interval between two adjacent frames;
[0056] The lane line is fitted according to the pixel coordinates of each lane line obtained in the second detection result of the lane line of the current frame image, and the second calculation result of the lane line of the current frame image is obtained:
[0057] x re2 =k 21 y re2 2 +k 22 y re2 +k 23
[0058] In the formula, k 21 , k 22 , k 23 are the polynomial coefficients respectively;
[0059] The first lane line calculation result in the current frame image is merged with the second lane line calculation result to obtain the third lane line calculation result in the current frame image:
[0060]
[0061] The third calculation result of the lane line of the current frame image is the final detection result of the lane line.
[0062] Furthermore, the optical flow corresponding to the lane line pixel point is extracted by the Lucas-Kanade algorithm, and the specific steps are as follows:
[0063] Select the neighborhood S containing the lane line pixel points in the first detection result of the previous frame image m×m , compare the previous frame with the current frame image, and obtain the partial derivatives of the grayscale value of each pixel in the neighborhood with respect to the image x-axis, y-axis and time t;
[0064] According to the partial derivatives of the grayscale values of each pixel in the neighborhood with respect to the image x-axis, y-axis and time t, the optical flow corresponding to the lane line pixel in the previous frame is calculated as:
[0065]
[0066] In the formula, are the horizontal and vertical velocity components of the lane line pixel in the previous frame image, respectively. xi is the partial derivative of the gray value of the i-th pixel in the neighborhood with respect to the x-axis of the image, I yi is the partial derivative of the gray value of the i-th pixel in the neighborhood with respect to the y-axis of the image, I ti is the partial derivative of the grayscale value of the i-th pixel in the neighborhood with respect to time t, and n=m×m is the total number of pixels in the neighborhood.
[0067] Beneficial effects of the present invention:
[0068] The present invention defines lane line detection as finding the set of lane line positions in certain rows in an image. Compared with other lane line detection methods based on deep learning, which require pixel-by-pixel classification, the method of the present invention only needs to classify each cell. Since the number of cells is much smaller than the number of image pixels, the lane line detection speed can be significantly improved.
[0069] The lane detection model of the present invention is a classification based on a fully connected layer. The features used in the classification are global features, and the receptive field is the size of the entire image. By learning and utilizing visual clues and information at different positions in the image, the problem of difficult detection when the lane is severely obscured can be solved.
[0070] The present invention introduces spatiotemporal context information by extracting optical flow between adjacent frames in the video, corrects the lane line detection result, and improves the accuracy of lane line detection without reducing the lane line detection speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 Flow chart of the method of the present invention.
[0072] Figure 2 This is the schematic diagram of the lane detection model. DETAILED DESCRIPTION
[0073] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.
[0074] Reference Figure 1 , Figure 2 As shown, a lane line detection method integrating spatiotemporal information of the present invention comprises the following steps:
[0075] Step 1: Obtain an image with annotated lane lines as a training data set for the lane line detection model, and perform data augmentation on the images in the training data set;
[0076] The lane line image with annotations in step 1 is captured by a vehicle-mounted camera, and the captured image is manually annotated;
[0077] The labeling tool used is Labelme.
[0078] The data enhancement in step 1 includes: normalization, random rotation, random cropping, and random horizontal and vertical flipping.
[0079] Step 2: Build a lane detection model, use the lane detection data set after data enhancement to train the lane detection model, and adjust the model hyperparameters to minimize the model loss value;
[0080] The lane detection model in step 2 adopts a network architecture of feature extraction network plus classification network, the feature extraction network is a resnet18 backbone network, and the classification network is a two-layer fully connected layer plus a Softmax function;
[0081] The resnet18 backbone network includes: several residual modules, convolutional layers, batch normalization layers, and Relu layers; among them,
[0082] The residual module is divided into: a first residual module and a second residual module;
[0083] The first residual module includes two branches: the main branch is a 3×3 convolution layer, a batch normalization layer, a Relu layer, a 3×3 convolution layer, a batch normalization layer, and a Relu layer series branch; the bypass branch is a branch whose input is directly connected to the output;
[0084] The second residual module includes two branches: a main branch is a branch consisting of a 3×3 convolution layer, a batch normalization layer, a Relu layer, a 3×3 convolution layer, a batch normalization layer, and a Relu layer connected in series; a bypass branch is a branch consisting of a 1×1 convolution layer, a batch normalization layer, and a Relu layer connected in series;
[0085] The specific connection method of the resnet18 backbone network is:
[0086] The first layer of the network is a 7×7 convolutional layer, which is connected in series in the forward propagation direction of the network: a maximum pooling layer, a Relu layer, two first residual modules, a second residual module, a first residual module, a second residual module, a first residual module, a second residual module, a first residual module, and an average pooling layer;
[0087] The input of the classification network is a one-dimensional vector, which is obtained by flattening the output features of the feature extraction network, and the output is a three-dimensional tensor Pro∈R C×h×(w+1) , where R represents the real number domain, C is the maximum number of lane lines detected, h is the number of row anchor boxes in the height direction of the network input image, and w is the number of cells divided in the width direction of the network input image.
[0088] The row anchor frame refers to thin strips divided at equal intervals in the height direction of the image.
[0089] The cells refer to the same number of small grids of equal width divided from each row anchor box in the image width direction.
[0090] In particular, the row anchor box is only segmented in the lower 70% of the image area. Different images are segmented to obtain the same number of row anchor boxes, but different row anchor box heights.
[0091] The three-dimensional tensor output by the classification network represents the lane line cell location probability map, Pro i,j,k Corresponding to the probability that the i-th lane line exists in the k-th cell on the j-th row anchor box, when the probability value on the w+1-th cell is the largest, it means that the i-th lane line does not exist on the j-th row anchor box.
[0092] In this example, the lane detection model training process in step 2 is:
[0093] Step 21: Use mini-batch stochastic gradient descent to train the lane detection model with a batch size of 32 and an initial learning rate of 4×10 -4 , the momentum value is 0.9, and the weight decay is 1×10 -4 ;
[0094] Step 22: Select the cross entropy loss function as the loss function of the lane detection model, input training samples to train the lane detection model, and adjust the hyperparameters according to the changes in the loss function so that the model reaches the optimal solution and saves the corresponding weights.
[0095] Adjusting the model hyperparameters in step 2 specifically includes: adjusting the height of the image row anchor box and the number of divided cells, wherein the row anchor box height changes from small to large, and the number of cells changes from large to small, to obtain the maximum row anchor box height and the minimum number of cells without decreasing the loss function of the lane line detection model.
[0096] Step 3: The lane line video acquired in real time is used as the video to be detected, and the video to be detected is subjected to frame extraction processing to obtain the image to be detected;
[0097] The size of the image to be detected in step 3 is consistent with the image size in the training data set, otherwise the height of the row anchor box in the height direction of the image and the number of cells in the width direction should be adjusted proportionally according to the image size.
[0098] Step 4: Determine whether the image to be detected is the first frame image, if so, proceed to step 5; if not, proceed to step 6;
[0099] Step 5: Input the image to be detected into the lane detection model trained in step 2, and post-process the model output to obtain the final lane detection result;
[0100] Wherein, the step 5 specifically includes:
[0101] The lane line cell position probability map is detected according to the trained lane line detection model to obtain the lane line cell position:
[0102]
[0103] In the formula, Loci,j represents the cell position of the i-th lane line on the j-th row anchor box, Pro i,j,k represents the probability that the i-th lane line exists in the k-th cell on the j-th row anchor box;
[0104] The pixel coordinates of the lane line in the detection image are obtained based on the lane line cell position, the divided row anchor box, and the number of cells divided in the width direction, as the first lane line detection result:
[0105] (x1,y1)=(Loc i,j *grid_w,anchor j )
[0106] Where x1 and y1 are the pixel coordinates of the lane line pixel in the first lane line detection result in the x-axis and y-axis directions of the image, grid_w represents the pixel width of the cell in the width direction of the image, and anchor j Represents the pixel coordinate value corresponding to the j-th row anchor box in the y-axis direction of the image;
[0107] The lane line is fitted according to the coordinates of each pixel point obtained in the first lane line detection result to obtain the first lane line calculation result x re1 :
[0108] x re1 =k 11 y re1 2 +k 12 y re1 +k 13
[0109] In the formula, k 11 , k 12 , k 13 are the polynomial coefficients respectively;
[0110] The first lane line calculation result is the final lane line detection result.
[0111] Step 6: Input the image to be detected into the lane line detection model trained in step 2, post-process the model output to obtain the first lane line detection result, use the optical flow method to obtain the second lane line detection result, and fuse the first lane line detection result and the second lane line detection result to obtain the final lane line detection result; specifically including:
[0112] Use the method in step 5 to detect the lane line of the current frame image and obtain the first calculation result of the lane line of the current frame image:
[0113] x re1 ′=k 11 ′y re1 '2 +k 12 ′y re1 ′+k 13 '
[0114] Extract the optical flow v corresponding to the lane line pixel in the previous frame image ls1 :
[0115]
[0116] In the formula, are the horizontal and vertical velocity components of the lane line pixel points respectively;
[0117] According to the optical flow corresponding to the lane line pixel point and the lane line first detection result in the previous frame image, the lane line pixel point (x ls1 ,y ls1 ) in the current frame image at the corresponding position (x ls2 ,y ls2 ), as the second detection result of the lane line in the current frame image:
[0118]
[0119]
[0120] In the formula, Δt represents the time interval between two adjacent frames;
[0121] The lane line is fitted according to the pixel coordinates of each lane line obtained in the second detection result of the lane line of the current frame image, and the second calculation result of the lane line of the current frame image is obtained:
[0122] x re2 =k 21 y re2 2 +k 22 y re2 +k 23
[0123] In the formula, k 21 , k 22 , k 23 are the polynomial coefficients respectively;
[0124] The first lane line calculation result in the current frame image is merged with the second lane line calculation result to obtain the third lane line calculation result in the current frame image:
[0125]
[0126] The third calculation result of the lane line of the current frame image is the final detection result of the lane line.
[0127] Specifically, the optical flow corresponding to the lane line pixel point is extracted by the Lucas-Kanade algorithm, and the specific steps are as follows:
[0128] Select the neighborhood S containing the lane line pixel points in the first detection result of the previous frame image m×m , compare the previous frame with the current frame image, and obtain the partial derivatives of the grayscale value of each pixel in the neighborhood with respect to the image x-axis, y-axis and time t;
[0129] According to the partial derivatives of the grayscale values of each pixel in the neighborhood with respect to the image x-axis, y-axis and time t, the optical flow corresponding to the lane line pixel in the previous frame is calculated as:
[0130]
[0131] In the formula, are the horizontal and vertical velocity components of the lane line pixel in the previous frame image, respectively. xi is the partial derivative of the gray value of the i-th pixel in the neighborhood with respect to the x-axis of the image, I yi is the partial derivative of the gray value of the i-th pixel in the neighborhood with respect to the y-axis of the image, I ti is the partial derivative of the grayscale value of the i-th pixel in the neighborhood with respect to time t, and n=m×m is the total number of pixels in the neighborhood.
[0132] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.
Claims
1. A lane line detection method integrating spatiotemporal information, characterized in that: Here are the steps: Step 1: Obtain an image with annotated lane lines as a training data set for the lane line detection model, and perform data augmentation on the images in the training data set; Step 2: Build a lane detection model, use the lane detection data set after data enhancement to train the lane detection model, and adjust the model hyperparameters to minimize the model loss value; Step 3: The lane line video acquired in real time is used as the video to be detected, and the video to be detected is subjected to frame extraction processing to obtain the image to be detected; Step 4: Determine whether the image to be detected is the first frame image, if so, proceed to step 5; if not, proceed to step 6; Step 5: Input the image to be detected into the lane detection model trained in step 2, and post-process the model output to obtain the final lane detection result; Step 6: Input the image to be detected into the lane line detection model trained in step 2, post-process the model output to obtain the first lane line detection result, use the optical flow method to obtain the second lane line detection result, and fuse the first lane line detection result and the second lane line detection result to obtain the final lane line detection result; The lane line detection model in step 2 adopts a network architecture of feature extraction network plus classification network, the feature extraction network is a resnet18 backbone network, and the classification network is a two-layer fully connected layer plus a Softmax function; The resnet18 backbone network includes: several residual modules, convolutional layers, batch normalization layers, and Relu layers; among them, The residual module is divided into: a first residual module and a second residual module; The first residual module includes two branches: the main branch is a 3×3 convolution layer, a batch normalization layer, a Relu layer, a 3×3 convolution layer, a batch normalization layer, and a Relu layer series branch; the bypass branch is a branch whose input is directly connected to the output; The second residual module includes two branches: a main branch is a branch consisting of a 3×3 convolution layer, a batch normalization layer, a Relu layer, a 3×3 convolution layer, a batch normalization layer, and a Relu layer connected in series; a bypass branch is a branch consisting of a 1×1 convolution layer, a batch normalization layer, and a Relu layer connected in series; The specific connection method of the resnet18 backbone network is: The first layer of the network is a 7×7 convolutional layer, which is connected in series in the forward propagation direction of the network: a maximum pooling layer, a Relu layer, two first residual modules, a second residual module, a first residual module, a second residual module, a first residual module, a second residual module, a first residual module, and an average pooling layer; The input of the classification network is a one-dimensional vector, which is obtained by flattening the output features of the feature extraction network, and the output is a three-dimensional tensor Pro∈R C×h×(w+1) , where R represents the real number domain, C is the maximum number of lane lines detected, h is the number of row anchor boxes in the height direction of the network input image, and w is the number of cells divided in the width direction of the network input image; The step 5 specifically includes: The lane line cell position probability map is detected according to the trained lane line detection model to obtain the lane line cell position: In the formula, Loc i,j represents the cell position of the i-th lane line on the j-th row anchor box, Pro i,j,k represents the probability that the i-th lane line exists in the k-th cell on the j-th row anchor box; The pixel coordinates of the lane line in the detection image are obtained based on the lane line cell position, the divided row anchor box, and the number of cells divided in the width direction, as the first lane line detection result: (x1,y1)=(Loc i,j *grid_w,anchor j ) Where x1 and y1 are the pixel coordinates of the lane line pixel in the first lane line detection result in the x-axis and y-axis directions of the image, grid_w represents the pixel width of the cell in the width direction of the image, and anchor j Represents the pixel coordinate value corresponding to the j-th row anchor box in the y-axis direction of the image; The lane line is fitted according to the coordinates of each pixel point obtained in the first lane line detection result to obtain the first lane line calculation result x re1 : x re1 =k 11 and re1 2 +k 12 and re1 +k 13 In the formula, k 11 , k 12 , k 13 are the polynomial coefficients respectively; The first lane line calculation result is the final lane line detection result; The step 6 specifically includes: Use the method in step 5 to detect the lane line of the current frame image and obtain the first calculation result of the lane line of the current frame image: x re1 ′=k 11 'and re1 ′ 2 +k 12 'and re1 ′+k 13 ′ Extract the optical flow v corresponding to the lane line pixel in the previous frame image ls1 : In the formula, are the horizontal and vertical velocity components of the lane line pixel points respectively; According to the optical flow corresponding to the lane line pixel point and the lane line first detection result in the previous frame image, the lane line pixel point (x ls1 ,y ls1 ) in the current frame image at the corresponding position (x ls2 ,y ls2 ), as the second detection result of the lane line in the current frame image: In the formula, Δt represents the time interval between two adjacent frames; The lane line is fitted according to the pixel coordinates of each lane line obtained in the second detection result of the lane line of the current frame image, and the second calculation result of the lane line of the current frame image is obtained: x re2 =k 21 and re2 2 +k 22 and re2 +k 23 In the formula, k 21 , k 22 , k 23 are the polynomial coefficients respectively; The first lane line calculation result in the current frame image is merged with the second lane line calculation result to obtain the third lane line calculation result in the current frame image: The third calculation result of the lane line of the current frame image is the final detection result of the lane line.
2. The lane line detection method integrating spatiotemporal information according to claim 1, characterized in that: The data enhancement in step 1 includes: normalization, random rotation, random cropping, and random horizontal and vertical flipping.
3. The lane line detection method integrating spatiotemporal information according to claim 1, characterized in that: The row anchor frame refers to thin strips divided at equal intervals in the height direction of the image.
4. The lane line detection method integrating spatiotemporal information according to claim 1, characterized in that: The cells refer to the same number of small grids of equal width divided from each row anchor box in the image width direction.
5. The lane line detection method integrating spatiotemporal information according to claim 1, characterized in that: The lane detection model training process in step 2 is as follows: Step 21: Use mini-batch stochastic gradient descent to train the lane detection model with a batch size of 32 and an initial learning rate of 4×10 -4 , the momentum value is 0.9, and the weight decay is 1×10 -4 ; Step 22: Select the cross entropy loss function as the loss function of the lane detection model, input training samples to train the lane detection model, and adjust the hyperparameters according to the changes in the loss function so that the model reaches the optimal solution and saves the corresponding weights.
6. The lane line detection method integrating spatiotemporal information according to claim 1, characterized in that: The size of the image to be detected in step 3 is consistent with the image size in the training data set, otherwise the height of the row anchor box in the height direction of the image and the number of cells in the width direction should be adjusted proportionally according to the image size.
7. The lane line detection method integrating spatiotemporal information according to claim 1, characterized in that: The optical flow corresponding to the lane line pixel point is extracted by the Lucas-Kanade algorithm, and the specific steps are as follows: Select the neighborhood S containing the lane line pixel points in the first detection result of the previous frame image m×m , compare the previous frame with the current frame image, and obtain the partial derivatives of the grayscale value of each pixel in the neighborhood with respect to the image x-axis, y-axis and time t; According to the partial derivatives of the grayscale values of each pixel in the neighborhood with respect to the image x-axis, y-axis and time t, the optical flow corresponding to the lane line pixel in the previous frame is calculated as: In the formula, point are the horizontal and vertical velocity components of the lane line pixel in the previous frame image, I xi is the partial derivative of the gray value of the i-th pixel in the neighborhood with respect to the x-axis of the image, I yi is the partial derivative of the gray value of the i-th pixel in the neighborhood with respect to the y-axis of the image, I ti is the partial derivative of the grayscale value of the i-th pixel in the neighborhood with respect to time t, and n=m×m is the total number of pixels in the neighborhood.
