A lane detection method based on time series information and grid model

Abstract

The present invention discloses a lane line detection method based on a grid model of temporal information, comprising S1, obtaining sequential image frames of lane lines; S2, a data pre-storage stage; S3, a self-attention processing stage; S4, a memory extraction stage; S5, a decoding prediction stage; and S6, a prediction generation stage. The method achieves a network structure for memory generation and reading by combining self-attention, cross-attention, and multi-scale feature fusion. The present invention models the lane line detection problem as a classification problem by gridding the image, thereby reducing the amount of calculation and improving the calculation speed. The overall process of the model is divided into a current frame process and a historical frame process. The backbone of the model is the current frame process, and the historical process produces memory information for the current frame process to extract through a memory extraction module. The present invention reuses the features extracted from the image in the current frame process at previous moments in the historical frame process to avoid repeated calculations and further reduce the amount of calculation.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and relates to an image lane line detection method, specifically a lane line detection method based on a grid model of time series information. Background Art

[0002] Lane detection is widely used in today's autonomous driving technology. Within the deep learning landscape, modern lane detection methods primarily treat lane detection as a pixel segmentation problem. Traditional CNN models' disregard for context and global information leads to reduced detection efficiency under occlusion and strong light conditions. While the SCNN method emphasizes global information extraction, its cyclical computational process increases computational complexity. The UFSA method treats lane detection as a row-based selection problem using global features. It divides the image into a grid, where an image of (H, W) size is divided into grids of (h, w) size, where h and w are much smaller than H and W. UFSA transforms the pixel segmentation process into a grid classification process, focusing solely on the classification of each grid, significantly reducing the number of parameters and improving computational speed. While extracting global features, it significantly reduces computational cost.

[0003] However, given the temporal dynamics of human perception and the information received in autonomous driving applications, traditional lane detection models, including the UFSA model, only perform calculations on a single frame, ignoring the temporal relationships between frames. The MMA-Net model, based on an attention-based memory encoding and retrieval method in the field of image segmentation, implements a lane detection method based on temporal information extracted from historical frames. While incorporating temporal information significantly improves performance, its large input volume and complex memory encoding and retrieval modules reduce computational efficiency, making it unsuitable for real-time use. Summary of the Invention

[0004] The purpose of this invention is to address the problem of slow detection speed caused by the inefficient use of temporal information in current lane line detection technology models. By adjusting the MMA-Net network process to reduce repeated calculations and combining the "line anchor" multi-classification modeling method to reduce prediction parameters, a new lane line detection model with faster operation speed is proposed.

[0005] The present invention discloses a lane line detection method based on a grid model of time series information, comprising the following steps:

[0006] S1, obtain sequential image frames of lane lines and construct a dataset;

[0007] S2, data pre-storage stage

[0008] S2-1, intercept the first 20 frames of the sequential image frames and randomly select 5 frames as the initial historical frames;

[0009] S2-2, gridding the lane mask corresponding to the extracted image to obtain the probability matrix M of each grid class belonging to a certain lane;

[0010] S2-3, the probability matrix M is input into the feature extraction network E1 composed of the backbone network ResNet50 for feature extraction to obtain feature information and

[0011] S2-4, the extracted images are individually input into the feature extraction network E2 composed of the backbone network ResNet50 for feature extraction, and features representing the shallow and deep information of the image are obtained respectively;

[0012] S2-5, superimpose the feature copies representing the shallow and deep information of the image to obtain and The feature map with the same number of channels and size is denoted as and

[0013] S2-6, feature information and Respectively with feature maps and Corresponding addition to obtain the historical memory feature vector and

[0014] S2-7, repeat steps S2-1 to S2-6 for 5 times for the 5 frames of images extracted to obtain 5 sets of historical memory feature vectors and

[0015] S2-8, stack the 5 sets of feature vectors in order and in random order and

[0016] S3, self-attention processing stage

[0017] Received separately through the self-attention module and To extract shallow and deep self-attention features, shallow self-attention features are obtained by extracting shallow and deep self-attention features. and deep self-attention features That is, the output after processing the initial historical frame;

[0018] S4, memory retrieval stage

[0019] In the current frame process, the input is the current frame image f extracted from the dataset in chronological order, and the feature extraction network E1 is used to extract the shallow layer of the feature extraction network E1. and deep features The shallow and deep features Perform matrix multiplication to get and Respectively and Perform cross attention calculation to obtain a shallow layer T containing historical memory information f and deep features T d , the formula is as follows:

[0020]

[0021]

[0022] S5, decoding prediction stage

[0023] Input the current frame image f, shallow feature T f and deep features T d , feature fusion is performed through the feature pyramid network, firstly the deep feature T d Perform deconvolution to reduce the number of feature channels to shallow features T f Same as T, then upsample in length and width to the same size as T f The same, so that T' d With T f Same size, for T' d With T f Add, and repeat the above operation to get the feature vector and fuse it with the image f to get the final feature output T e ;

[0024] S6. Prediction generation stage

[0025] Through the full connection layer plus the relu activation layer as the prediction output decoding network, first T e Downsampling is performed in both the length and width dimensions, and the entire 3D feature map is then straightened into a one-dimensional feature vector, which is then input into a fully connected layer. The output undergoes Relu activation and is then input into another fully connected layer. The resulting predicted probability matrix M is obtained. The grid point with the highest probability in each row is the lane line location, and connecting the points to form a line identifies the lane line. Finally, the predicted probability matrix M is fed into the historical frame process as input for feature extraction to generate memory features. These features are then added to the original five sets of historical memory features, while the chronologically earliest historical memory features are deleted.

[0026] Preferably, the probability matrix M is such that the value of the grid (i, j) in the grid map corresponding to its channel k is the probability that the corresponding position in the original image is the lane line k. In addition, there is an extra column of grids in the grid map that has no corresponding position in the original image, and its value is the probability that lane k does not exist in this row of grids.

[0027] Preferably, the feature extraction network E1 is composed of ResNet50, which is composed of four modules stacked in sequence, each of which is composed of multiple convolutional networks stacked together. Each module takes the output of the previous module as input to further extract deep information. The output of the third and fourth modules in ResNet50 is used as the extracted feature information, which is recorded as and

[0028] Preferably, in step S3, the shallow self-attention feature Generation method: The two vector groups of extracted shallow self-attention features are stacked into a higher-dimensional vector, and generated by matrix multiplication The shallow self-attention feature is generated by the following formula

[0029]

[0030]

[0031]

[0032] in, The symbol indicates that the vectors are added in the first place. Calculation formula and same.

[0033] Preferably, the generating of deep self-attention features Method and shallow self-attention features The generation method is the same.

[0034] As a preference, it also includes

[0035] S7, memory storage stage

[0036] The generated probability matrix M and the shallow feature T extracted from the current frame f and deep features T d Store it as a historical frame for subsequent calculations.

[0037] The beneficial effects of the invention are as follows:

[0038] This paper remodels MMA-Net, modifying its lane detection model from an image segmentation task to a multi-classification task using gridded images. This reduces the task difficulty and the number of training parameters, thereby improving training and detection speed and accuracy. Furthermore, by adjusting the process, a smaller probability matrix is ​​used as input to E2, which is then fused with the image features from E1, rather than fusing the lane mask with the image before inputting it to E2. This avoids conflicting image data feature extraction and significantly reduces computational complexity. Experiments show that this fusion method does not degrade other model performance indicators.

[0039] By using self-attention and cross-attention to generate and read memories, combined with a grid-based classification modeling method, lane line detection tasks can be performed more quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the overall implementation scheme of the present invention;

[0041] Figure 2 is a schematic diagram of a memory retrieval module of the present invention;

[0042] Figure 3 Schematic diagram of the self-attention module of the present invention;

[0043] Figure 4 Schematic diagram of multi-scale feature fusion of the present invention;

[0044] Figure 5 It is a schematic diagram of image gridding of the present invention;

[0045] Figure 6 This is a schematic diagram of the effect of the embodiment of the present invention Figure 1 ;

[0046] Figure 7 This is a schematic diagram of the effect of the embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0047] The process of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] Reference Figure 1 It is a flow chart of the overall implementation scheme of the present invention, comprising the following steps:

[0049] S1. Acquire sequential image frames of lane lines. In this embodiment, the acquired sequential image frames of lane lines may use a data set VIL-100 with a maximum of 8 lane lines.

[0050] S2, data pre-storage stage: intercept the first 20 frames of the input sequential image frames, and extract 5 frames as the initial historical frames. Grid the lane mask corresponding to the image to obtain the probability matrix M of each grid classification belonging to a certain lane. Since the data set VIL-100 used in this embodiment can have a maximum of 8 lane lines, the number of channels in M ​​is 8. Figure 4 As shown, the value of grid cell (i, j) in the grid map corresponding to channel k represents the probability that the corresponding location in the original image is lane k. Furthermore, there is an extra column of grid cells at the end that has no corresponding location in the original image; its value represents the probability that lane k does not exist in that row. This probability matrix M is input into the feature extraction network E1 for feature extraction.

[0051] Specifically, the feature extraction network E1 is composed of ResNet50, which is stacked by four module steps. Each module takes the output of the previous module as input to further extract deep information. The present invention extracts the output of the third and fourth modules of ResNet50 as the feature information extracted in this step, which is recorded as and In addition, the extracted image is input into the feature extraction network E2, and the features representing the shallow and deep information of the image are obtained. and The feature map with the same number of channels and size is denoted as and Corresponding addition to obtain the historical memory feature vector and Repeat the above steps 5 times for the 5 frames of images extracted to obtain 5 sets of historical memory feature vectors and The five sets of feature vectors are stacked in order and in random order. and Used to focus on local temporal information and global intrinsic information.

[0052]

[0053]

[0054] S3, self-attention processing stage: This stage is well-established in the MMA-Net model, so it is not modified much in this embodiment and is only briefly described. and To extract shallow self-attention features and deep self-attention features.

[0055] Take the component with shallow combination as an example: the vector groups of the two shallow self-attention features of the input are stacked into a higher-dimensional vector, which is generated by matrix multiplication The shallow self-attention feature is generated by the following formula The symbol indicates that the vectors are added in the first place. The calculation formula is the same as

[0056]

[0057]

[0058]

[0059] Similarly, deep self-attention features are generated That is the output of history frame processing.

[0060] S4, memory retrieval stage: Figure 1 In the current frame process, the input is the current frame image f. For the feature extraction of the current frame, E1 is used to extract the shallow and deep features. and Similar to the historical frame processing method, matrix multiplication is performed to obtain and Respectively and Perform cross-attention calculations to obtain the shallow and deep features Tf and Td containing historical memory information. The formulas are as follows:

[0061]

[0062]

[0063] S5, decoding prediction stage: the input has three features, namely the image f of the current frame, the shallow feature T f and deep features T d Since the process of feature extraction using ResNet is to increase the number of channels and reduce the length and width, the sizes of the three inputs are different, and the fusion needs to adjust the size. The model of the present invention uses the feature pyramid structure principle commonly used in target detection to perform feature fusion. First, T d Perform deconvolution to reduce the number of feature channels to T f Same as T, then upsample in length and width to the same size as T f Similarly, after this treatment, T' d With T f Same size, for T' d With T fAdd, and repeat the above operation to get the feature vector and fuse it with the image f to get the final feature output T e The detailed formula is as follows:

[0064] T tmp =upSample(Conv2d(T d ))+T f (8)

[0065] T f =upSample(Conv2d(T tmp )+f) (9)

[0066] S6, prediction generation stage: Through the above steps, we can get a large size, small channel T e In multi-classification problems, a fully connected layer plus a relu activation layer is often used as a prediction output decoding method. e In order to reduce the number of parameters in the fully connected layer, the model of the present invention first e Downsampling is performed in the length and width dimensions, and the entire 3D feature map is then straightened into a one-dimensional feature vector, which is input into a fully connected layer. The output is activated by Relu and then input into a fully connected layer, resulting in the predicted probability matrix M. The grid point with the highest probability in each row is the location of the lane line, and connecting the points to form a line can be used to identify the lane line.

[0067] S7, memory storage stage: the generated probability matrix M, the shallow feature T extracted from the current frame f and deep features T d Store it and use it as the historical frame in subsequent calculations, that is, the input of historical frame processing.

[0068] The main innovation of this embodiment focuses on modifying the MMA-Net modeling method and reshaping the process to create a more computationally efficient model. This is achieved by converting MMA-Net's image segmentation modeling method to a "row anchor" classification modeling method to reduce decoder parameters; and by adjusting the data feature extraction process to reduce redundant computation.

[0069] First, in traditional image segmentation, the output image needs to be predicted in full size, that is, for an image with an input size of (w,h), the output also needs to be (w,h). When the image segmentation method is applied to lane line detection, a channel output is required for each lane line, and the output size is (n,w,h), where n is the number of lane lines. The number of parameters that need to be predicted is n×w×h. This invention adopts a multi-classification method based on "row anchors" to grid the image, such as Figure 4A lane line only appears once in a row of grid cells, so multi-classification calculations are performed only within that row of grid cells, calculating the probability that each grid cell belongs to each lane line. This transforms the lane line detection problem from a segmentation approach into a multi-classification problem for image grids. In this model, assuming a 20×20 pixel grid, the output size is (n, w / 20, h / 20), which is much smaller than the full-size prediction, reducing the computational effort during the prediction generation stage.

[0070] Secondly, the MMA-Net model uses ResNet50 to extract features from the original image size during both the memory generation and memory reading stages. Experiments have shown that the time for the two feature extractions accounts for about 70% of the model prediction time, making it an object worth optimizing. The present invention uses a new structural process for optimization: the present invention uses the smaller-sized gridded probability matrix output above as the input instead of the original-size image superimposed on the image mask as lane line position memory. And in order to prevent the loss of image features caused by this approach, the present invention also uses the features extracted from the previous moment of each historical frame in the current frame process as image feature memory, and superimposes them with the lane line position memory to generate a global historical memory. This approach reduces the repeated extraction of image features, thereby accelerating the model without reducing model performance. The process is as follows:

[0071] 1. Current frame process

[0072] As the name implies, the current frame process accepts the most recent input image frame in the time series input, first performs feature extraction through Encoder, E1 composed of ResNet50, and extracts the feature vector T cur Since this feature vector is extracted for the current frame, it contains the features of the current frame. The feature vector is fused with historical memory features through the memory extraction module, and the obtained features are input into the final decoder for prediction, generating the probability matrix M of each grid position corresponding to the current frame belonging to the lane line.

[0073] 2. Historical frame process

[0074] The historical frame process accepts the probability matrix M output by the model of the 5 frames before the current frame, and extracts the feature vector by Encoder E2, which is also composed of ResNet50. The generated memory information is extracted through the output probability matrix M. The memory only contains the position information of the lane line but not the image data information in the original frame. To solve this problem, the present invention extracts the feature vector T of the previous moment of each historical frame in the current frame process. cur As historical frame image data features and Add together to obtain the memory features containing lane line position information and image global data information The input of multiple historical frames generates multiple feature vectors, which are stacked in both sequential and random order. and It is used to focus on local temporal information and global inherent information, which together constitute memory information as the output of the historical frame process.

[0075] The current frame process reads the memory information output by the historical frame process through the memory reading module, obtains features containing temporal memory information and current image feature information, and inputs them into the decoder to output the multi-classification probability matrix of the grid.

[0076] The present invention achieves a network structure for memory generation and reading by combining self-attention, cross-attention and multi-scale feature fusion. The present invention models the lane line detection problem into a classification problem by gridding the image, thereby reducing the amount of calculation and improving the calculation speed. The overall process of the model is divided into the current frame process and the historical frame process. The backbone of the model is the current frame process, and the historical process produces memory information for the current frame process to extract through the memory extraction module. The present invention reuses the features extracted from the image in the current frame process at the previous moment in the historical frame process to avoid repeated calculations and further reduce the amount of calculation.

Claims

1. A lane detection method based on time series information and grid model, characterized in that: The following steps are involved: S1, obtain sequential image frames of lane lines and construct a dataset; S2, data pre-storage stage S2-1, intercept the first 20 frames of the sequential image frames and randomly select 5 frames as the initial historical frames; S2-2, gridding the lane mask corresponding to the extracted image to obtain the probability matrix M of each grid class belonging to a certain lane; S2-3, the probability matrix M is input into the feature extraction network E1 composed of the backbone network ResNet50 for feature extraction to obtain feature information and S2-4, the extracted images are individually input into the feature extraction network E2 composed of the backbone network ResNet50 for feature extraction, and features representing the shallow and deep information of the image are obtained respectively; S2-5, superimpose the feature copies representing the shallow and deep information of the image to obtain and The feature map with the same number of channels and size is denoted as and S2-6, feature information and Respectively with feature maps and Corresponding addition to obtain the historical memory feature vector and S2-7, repeat steps S2-1 to S2-6 for 5 times for the 5 frames of images extracted to obtain 5 sets of historical memory feature vectors and S2-8, stack the 5 sets of feature vectors in order and in random order and S3, self-attention processing stage Received separately through the self-attention module and To extract shallow and deep self-attention features, shallow self-attention features are obtained by extracting shallow and deep self-attention features. and deep self-attention features That is, the output after processing the initial historical frame; S4, memory retrieval stage In the current frame process, the input is the current frame image f extracted from the data set in time order, and the feature extraction network E1 is used to extract the shallow layer of the feature extraction of the current frame. and deep features The shallow and deep features Perform matrix multiplication to get and Respectively and Perform cross attention calculation to obtain a shallow layer T containing historical memory information f and deep features T d , the formula is as follows: S5, decoding prediction stage Input the current frame image f, shallow feature T f and deep features T d , feature fusion is performed through the feature pyramid network, firstly the deep feature T d Perform deconvolution to reduce the number of feature channels to shallow features T f Same as T, then upsample in length and width to the same size as T f The same, so that T' d With T f Same size, for T' d With T f Add, and repeat the above operation to get the feature vector and fuse it with the image f to get the final feature output T e ; S6. Prediction generation stage Through the full connection layer plus the relu activation layer as the prediction output decoding network, first T e Downsampling is performed in the length and width dimensions, and the entire 3D feature map is straightened into a one-dimensional feature vector and input into the fully connected layer. The output is activated by ReLU and then input into a fully connected layer. Finally, the predicted probability matrix M is obtained. The grid with the highest probability in each row is the point where the lane line is located. The points are connected to form a line to identify the lane line.

2. The lane line detection method based on time series information and grid model according to claim 1, characterized in that: The probability matrix M is that the value of the grid (i, j) in the grid map corresponding to its channel k is the probability that the corresponding position in the original image is lane line k. In addition, there is an extra column of grids in the grid map that has no corresponding position in the original image, and its value is the probability that lane k does not exist in this row of grids.

3. The lane line detection method based on time series information and grid model according to claim 2, characterized in that: The feature extraction network E1 is composed of ResNet50, which is composed of four modules stacked in sequence, each of which is composed of multiple convolutional networks stacked together. Each module uses the output of the previous module as input to further extract deep information. The output of the third and fourth modules in ResNet50 is used as the extracted feature information, which is recorded as and 4. The lane line detection method based on time series information and grid model according to claim 1, characterized in that: In step S3, the shallow self-attention feature Generation method: The two vector groups of extracted shallow self-attention features are stacked into a higher-dimensional vector, and generated by matrix multiplication The shallow self-attention feature is generated by the following formula in, The symbol indicates that the vectors are added in the first place. Calculation formula and same.

5. The lane line detection method based on time series information and grid model according to claim 4, characterized in that: The deep self-attention feature Generation method and shallow self-attention features The generation method is the same.

6. The lane line detection method based on time series information and grid model according to claim 1, characterized in that: Also includes S7, memory storage stage The generated probability matrix M and the shallow feature T extracted from the current frame f and deep features T d Store it as a historical frame for subsequent calculations.

Images