Lane line semantic segmentation method based on self-attention

Abstract

The invention provides a lane line semantic segmentation method based on self-attention, and the method comprises the steps: firstly carrying out the data collection and preprocessing, then achieving the binary semantic segmentation based on an ERFnet neural network model, and finally carrying out the final multi-target semantic segmentation based on an improved DeepLabV3Plus model according to a binary semantic segmentation map, the ERFnet is responsible for the preliminary coarse-grained segmentation, and the ERFnet is responsible for the initial coarse-grained segmentation. The lane line and the background are rapidly separated, the improved DeepLabV3Plus further refines the segmentation result, and the capture ability of the model to local and global features is enhanced by introducing a region self-attention unit and a channel self-attention unit. According to the scheme of the invention, by combining the ERFnet and the improved DeepLabV3Plus model, the accuracy and generalization ability of segmentation are improved, the real-time performance is maintained, the real-time and high-precision segmentation of the lane line is realized, and the method has strong adaptability to changeable complex road conditions.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lane line semantic segmentation method based on self-attention. Background Art

[0002] With the rapid development of computer artificial intelligence, autonomous driving has gradually become a research hotspot for major companies. Both automobile manufacturers and non-automotive manufacturers are vying to invest a lot of resources to strive to realize truly unmanned driving as soon as possible. In the current research direction, the autonomous driving system can be divided into three core parts according to its functions: perception layer, decision layer and execution layer. Among them, the perception layer refers to the vehicle's perception and identification of the vehicle itself and the driving environment through various sensors and GPS and other technologies; the decision layer refers to the vehicle software system that generates driving strategies and processing results through calculation and processing of various perception layer data to ensure normal communication and operation between various hardware and software subsystems; the execution layer refers to the vehicle hardware level executing specific decision instructions of the decision layer. It can be seen that the functions of different levels work together to form a complete autonomous driving system. The perception layer obtains environmental information, the decision-making layer analyzes and makes decisions on the information, and the execution layer converts the decision results into specific actions. In the entire autonomous driving system, the perception layer can obtain comprehensive driving environment information such as feasible roads, obstacles, traffic signals, location information, posture information, etc. The speed and accuracy of this layer's perception of the driving environment directly affects whether the driving strategy adopted by the decision-making layer is accurate and reliable, which in turn plays a vital role in the safe driving of the vehicle. Therefore, it is very necessary to study the driving environment of the perception layer.

[0003] In addition to relying on increasingly sophisticated hardware facilities for initial information collection, current autonomous driving systems also use deep learning algorithms to conduct a second targeted analysis of the extracted data to extract more intuitive and effective data information. The environmental data initially collected by the perception layer in autonomous driving has the following characteristics: complex and changeable environment, high real-time data, large data volume, etc. During driving, weather, traffic signs, pedestrians, and obstacles are all difficult to process secondary data. At the same time, deep learning models will encounter a large number of features when facing complex data, some of which contribute more to the task, while other features may not be important or even interfere.

[0004] The ERFnet model is a lightweight convolutional neural network architecture specifically designed for real-time semantic segmentation tasks. The DeepLabV3Plus model is a highly accurate and efficient deep learning architecture for semantic segmentation. The applications of both in the field of semantic segmentation are relatively mature, but each has some weaknesses. ERFnet is affected by the number of parameters and only focuses on the feature extraction of the local receptive field. When faced with complex scenes and multi-scale changes in monitoring targets, it may lead to unstable segmentation results. The DeepLabV3Plus model has a more complex structure and requires more computing resources and time. Therefore, in the current lane line semantic segmentation tasks, there are widespread problems of low robustness, low generalization, and low real-time level. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention proposes a lane line semantic segmentation method based on self-attention, which adopts the deep learning algorithm of the deep neural network model to learn and represent the high-level features of the data. Through the nonlinear transformation of multiple neural network layers, more complex and abstract features can be automatically extracted from the original data. These characteristics enable the deep learning network model to have better performance in processing driving environment data.

[0006] The present invention combines the ERFnet model and the DeepLabV3Plus model, and improves DeepLabV3Plus. ERFnet is used for coarse-grained lane feature extraction, and the input image is convolved multiple times and downsampled step by step by the encoder to obtain a lane feature map, and then the decoder is upsampled step by step to obtain a binary semantic segmentation map of the lane and background, thereby segmenting the original image (such as a dashcam screenshot) into lanes and background.

[0007] After the initial segmentation by ERFnet, the improved DeepLabV3Plus further refines the segmentation of lane lines and performs more detailed classification according to the characteristics of lane lines (such as single and double lines, yellow and white lines, and virtual and real lines). In order to further improve the generalization and semantic segmentation accuracy of DeepLabV3Plus, the regional self-attention unit and channel self-attention unit are introduced on the basis of the original model. The introduction of these two attention units improves the feature aggregation ability of the network model in different dimensions. In the low-dimensional feature extraction stage, the regional self-attention unit enables the network to focus on the spatial semantic information of the specified area in the image based on the segmentation result of ERFnet. The unit can weight the spatial features in the specified area, so that different types of lane lines obtain different attention weights, thereby improving the ability to capture and classify local feature information; in the high-dimensional feature extraction stage, the channel self-attention unit enables the network to focus on the channel semantic information of the image. By calculating the correlation between different channels in the feature map and adaptively assigning weights to the channel features, it helps to extract the correlation between different feature channels in the image, further improving the segmentation model's perception of global features. The introduction of regional self-attention units and channel self-attention units enables DeepLabV3Plus to have better segmentation capabilities in various application scenarios. These attention mechanisms allow the model to pay more attention to important areas and channels of the image during feature extraction and aggregation, thereby improving the accuracy and generalization of semantic segmentation.

[0008] The present invention proposes a lane line semantic segmentation method based on self-attention, which specifically includes the following steps:

[0009] Step S1: perform data collection and preprocessing, obtain the video stream collected by the vehicle camera, intercept the obtained video stream according to a fixed frame interval and mark the lane lines, and standardize the original image data set;

[0010] Step S2: training the ERFnet neural network model, and realizing binary semantic segmentation of image data based on the trained ERFnet neural network model, specifically: dividing the image data processed in step S1 into batches, and first sending the batch data into the convolution layer of the ERFnet model for feature extraction, and then the residual connection block splits the two-dimensional features after feature extraction into two one-dimensional convolutions, respectively extracting features in the horizontal and vertical directions, and then the residual connection operation fuses the processed feature map with the original input feature map to generate a set of feature maps, and finally these feature maps are further processed by the fully connected layer, and the probability distribution of each pixel is mapped to the corresponding category label through the Softmax activation function to generate a binary semantic segmentation result;

[0011] Step S3: According to the binary semantic segmentation map output by step S2, the final multi-target semantic segmentation is performed based on the improved DeepLabV3Plus model.

[0012] Furthermore, the step S3 further comprises:

[0013] Step S31: training the improved DeepLabV3Plus model;

[0014] Step S32: The regional self-attention unit of the trained model performs inner product on the original feature map and the lane-background binary classification weight mask generated by the ERFnet model to obtain the lane area semantic features, and performs linear transformation on these features through 1×1 convolution to generate query, key and value vectors, and generates the regional attention weight matrix by calculating the similarity between the query vector and the key vector;

[0015] Step S33: Perform weighted calculation on the regional attention weight matrix and the value vector to obtain more refined lane line regional features and input them into the Xception network of the model. The Xception network uses deep convolution and point-by-point convolution to perform independent convolution on the input feature map respectively.

[0016] Step S34: ASPP uses multiple hole convolution kernels of different sizes in parallel to extract features of different scales from the feature map output by Xception, and further captures global context information through global average pooling;

[0017] Step S35: The channel self-attention unit reduces the height and width dimensions of the features input by Xception through two one-dimensional convolution operations, and further optimizes them through nonlinear mapping. Then, the dot product of the compressed query vector and the key vector is calculated to generate a channel attention weight matrix, and the channel attention weight matrix is ​​applied to the value vector to complete the weighted operation. The weighted channel features are added element by element to the original features to form a new feature map.

[0018] Step S36: The feature fusion module fuses the high-dimensional features output by the channel self-attention module with the low-dimensional features output by the ASPP module, and then uses the multi-scale target inference module to perform multiple scaling and segmentation on the fused features at different scales to obtain the final semantic segmentation result.

[0019] Furthermore, the improved DeepLabV3Plus model includes an encoder module and a decoder module. The encoder module includes a regional self-attention module RSA, an Xception module, a channel self-attention module CSA and an ASPP module, and the decoder module includes a feature fusion module and a multi-scale target reasoning module.

[0020] Furthermore, the workflow of the RSA module is as follows: the input f of RSA is the original feature map and the weighted mask version with lane line semantics obtained by ERFnet model processing The inner product of the template and the original feature map is used to obtain the semantic features of the lane area. Then, the feature is linearly transformed through a convolution layer with a convolution kernel size of 1×1 and matched with the key vector f key Calculate the dimension and get the self-attention mechanism query vector The expression is:

[0021]

[0022] Among them, Conv 1×1 represents 1×1 convolution, ⊙ represents inner product;

[0023] Then calculate the key and value vectors in the regional self-attention mechanism. The specific expression is:

[0024] f key =Conv 1×1 (f)

[0025] f val =Conv 1×1 (f)

[0026] Through the above calculations, we get the Q, K, and V vectors required for self-attention, that is, f key 、f val ;

[0027] Then multiply the transpose of the Q matrix and the K matrix to get a similarity matrix, and divide the similarity matrix by the scaling factor Scale the weights, put the calculation results into the Softmax function for normalization, and finally add the attention weights and f val Multiply and calculate the weighted eigenvalue. The expression is as follows:

[0028]

[0029] Furthermore, the Xception model uses depthwise convolution and point-wise convolution to perform further convolution operations on the weighted feature map output by RSA.

[0030] Furthermore, the ASPP module uses multiple hole convolution kernels of different sizes in parallel to extract features of different scales from the feature maps output by Xception, and further captures global context information through global average pooling.

[0031] Furthermore, the workflow of the CSA module is as follows: the input of the CSA module is the high-dimensional feature map output by Xception, two one-dimensional convolutions are used for feature extraction, and a ReLU activation function is added between the two one-dimensional convolutions. The expression is as follows:

[0032] f x =Conv1D2(ReLU(Conv1D1(f′))),x∈{q,k,v}

[0033] Among them, f′ is the high-dimensional feature map output by Xception. After two one-dimensional convolution compressions and one nonlinear transformation, a 1×C-dimensional vector is obtained, where C is the number of channels;

[0034] Then use f q and f k Calculate the correlation on the channel, the result is a C×C weight matrix, sum the weight matrix in the first dimension and divide it by The weights are scaled, normalized by Softmax, and then the channel features f are multiplied by point multiplication. v After weighting, the channel self-attention representation is transformed from (1, 1, C) to (H, W, C) through the reshape function, and then weighted to the feature map f'. The expression is as follows:

[0035]

[0036] Furthermore, the feature fusion module in the decoder module fuses the high-dimensional features output by the CSA module with the low-dimensional features output by the ASPP module, and then the multi-scale object inference module scales and segments the fused features multiple times at different scales to obtain the final semantic segmentation result.

[0037] Compared with the prior art, the beneficial technical effects of the present invention are:

[0038] Real-time and high efficiency: The present invention combines the ERFnet and DeepLabV3Plus models in the lane line semantic segmentation task, and achieves a balance between real-time and high precision through two-level segmentation. With its low computational complexity and high real-time characteristics, ERFnet is responsible for the preliminary segmentation of the image, coarse-grained extraction of lane line features and generation of a preliminary binary segmentation map. Subsequently, the improved DeepLabV3Plus further refines the coarse segmentation results to improve segmentation accuracy without increasing the overall computational burden. As a lightweight network, ERFnet has high real-time efficiency and low computational complexity. It is suitable for performing preliminary coarse-grained segmentation of lane lines in vehicle-mounted systems with limited computing resources. It can quickly separate lane lines and backgrounds, providing a good foundation for the next step of fine segmentation, ensuring that the system maintains efficient response in real-time scenarios, and improving timeliness and computational efficiency;

[0039] Multi-scale scene adaptability: In view of the problem that ERFnet is unstable in extracting local features in complex and multi-scale scenes, the present invention improves the DeepLabV3Plus model and introduces regional self-attention units and channel self-attention units in the DeepLabV3Plus model. The regional self-attention unit focuses on the specified area in the image, so that the network can give differentiated attention weights to different types of lane lines (such as single and double lines, yellow and white lines, and virtual and real lines), thereby improving the segmentation accuracy in complex scenes and improving the model's ability to capture local features in complex scenes; the channel self-attention unit optimizes the feature weights of each channel and adaptively weights the features, so that the model can effectively capture the global semantic information of the lane lines when high-dimensional features are aggregated, thereby enhancing the adaptability to multi-scale changes;

[0040] Segmentation accuracy and generalization ability: Combining the regional and channel self-attention mechanisms, DeepLabV3Plus can effectively allocate attention resources and focus on key areas and important feature channels in the image. This improvement effectively improves the feature aggregation ability of the network model, making the model more generalizable in a variety of road scenes, and significantly improving the accuracy and stability of the segmentation results;

[0041] In summary, this paper effectively integrates the advantages of ERFnet and DeepLabV3Plus and rationally applies self-attention units, which significantly improves the accuracy and robustness of lane line semantic segmentation while ensuring high real-time performance and computational efficiency, especially maintaining good performance under changeable and complex road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 is a flow chart of a lane line semantic segmentation method based on self-attention provided by an embodiment of the present invention;

[0044] Figure 2 It is a network model diagram of lane line semantic segmentation provided by an embodiment of the present invention;

[0045] Figure 3 is an internal structure diagram of an RSA unit provided by an embodiment of the present invention;

[0046] Figure 4 It is a diagram of the internal structure of a CSA unit provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] A lane line semantic segmentation method based on self-attention proposed in the present invention is mainly divided into two parts. The first part is data preprocessing. This stage is based on the ERFnet neural network model to output a binary semantic segmentation map to prepare data for subsequent attention units. The second part is multi-target semantic segmentation. This stage is based on the DeepLabV3Plus neural network model combined with two attention units to achieve the final lane line semantic segmentation.

[0049] A lane semantic segmentation method based on self-attention, such as Figure 1 As shown, the method includes:

[0050] Step S1: data collection and preprocessing;

[0051] Step S2: Implement binary semantic segmentation based on the ERFnet neural network model;

[0052] Step S3: According to the binary semantic segmentation map output by step S2, the final multi-target semantic segmentation is performed based on the improved DeepLabV3Plus model.

[0053] Specifically, the step S1 further comprises:

[0054] Get the video stream captured by the vehicle camera, intercept the pictures at a fixed frame interval and mark the lane lines. The marked data is a grayscale image. According to the different types of lane lines, each pixel of the original image corresponds to a different value in the grayscale image.

[0055] When a neural network learns feature weights, if the input training data is not scaled proportionally, the distribution range of each feature value may be different, which will cause an imbalance when the learning rate is corrected in different dimensions, causing the model to over-adjust in some weight dimensions and almost no adjustment in other weight dimensions, which will make the training process difficult, slow down the convergence speed, and may cause the model performance to deteriorate. Therefore, the present invention scales the value range of each feature to a similar interval, so that each feature has the same influence on the training of the model, so that the model can consider each feature more balanced when updating the weight, avoiding excessive or insufficient adjustment, thereby improving the training effect.

[0056] In order to eliminate the offset in the data set and prevent data with too large numerical differences from causing negative interference to the training of the neural network model, the original image data set needs to be standardized. Since the original data is a three-channel image, the pixels of each channel are distributed between [0, 255]. Therefore, it is necessary to perform standardization on each channel separately. The present invention adopts mean variance standardization, also known as Z-score standardization. The standardization formula is as follows:

[0057] x * =(x-μ) / σ

[0058] Among them, x represents the sample value before standardization, that is, the pixel value of a single channel of the original image, μ and σ represent the mean and variance of all image data in each channel respectively, and x * Represents the normalized value. These image data are normalized to ensure that the pixel value range adapts to the input requirements of the ERFnet model, thereby avoiding training instability or convergence difficulties caused by inconsistent data distribution.

[0059] The step S2 further comprises:

[0060] The data after preprocessing in step S1 is used as the input of step S2. The input data is a set of standardized image data. Then the image data is divided into batches to improve the computational efficiency of the training and reasoning stages. The ERFnet neural network model is trained, and then the trained ERFnet neural network model is used to perform binary semantic segmentation on the input image data. The ERFnet model is a semantic segmentation network based on residual connection and separable convolution. It improves the efficiency of model processing frames without reducing the accuracy. It is particularly suitable for the real-time requirements in autonomous driving. With its lightweight architecture and efficient feature extraction capabilities, the model can quickly learn the feature representation of lane lines. During the training process, multi-level features of the image are extracted through convolutional layers and residual connections. These features are crucial for subsequent semantic segmentation.

[0061] The ERFnet neural network model is used to perform binary semantic segmentation on the input image data, including: the batch data is first sent to the convolution layer of the ERFnet model to start the feature extraction process. The convolution layer extracts the basic features of the image, such as edges and textures, through two-dimensional convolution operations. These features lay the foundation for subsequent advanced feature learning. In each convolution process, the convolution kernel covers the local area of ​​the image in a sliding window manner, and generates a feature map through weighted summation. This method extracts local pattern information while maintaining spatial relationships;

[0062] Then, the core module of the ERFnet model, the Non-bottleneck-1D residual connection block, further processes the input two-dimensional feature map and splits the two-dimensional convolutional features into two one-dimensional convolutions, which operate in the horizontal and vertical directions respectively. The first step is to extract features in the horizontal direction through one-dimensional convolution, and the second step is to extract features in the vertical direction through another set of one-dimensional convolutions. This decomposition method reduces the number of parameters while expanding the receptive field, thereby improving the context modeling ability of the model.

[0063] Then the residual connection operation fuses the processed feature map with the original input feature map, which not only enhances the ability to transmit information, but also ensures that the deep network retains important information in the shallow layer. After each two-dimensional convolution operation, the feature map will also pass through a nonlinear activation function to introduce nonlinear expression capabilities;

[0064] After the above multi-level feature extraction and processing, a set of feature maps are finally generated. These feature maps are further processed by the fully connected layer, and the probability distribution of each pixel is mapped to the corresponding category label through the Softmax activation function to generate a binary semantic segmentation result. The output binary semantic segmentation map is a two-dimensional matrix with the same size as the original image, in which the value of each pixel represents the classification result of the position. In this way, the ERFnet model achieves efficient and accurate semantic segmentation of targets such as lane lines.

[0065] The ERFnet model adopts the classic Encoder-Decoder architecture. In the encoding stage, the resolution of the feature map is gradually reduced through multi-layer convolution and pooling operations to extract more abstract semantic features. In the decoding stage, the feature map gradually restores the spatial resolution through upsampling. In order to avoid the loss of details of high-level semantic information, the decoding stage also combines the feature fusion strategy, that is, the shallow features of the encoding stage are spliced ​​with the features of the corresponding layer of the decoding stage, so that the model retains both local details and global context information.

[0066] The preliminary binary semantic segmentation map obtained in step S2 is mainly used to distinguish lane lines from background areas. In order to complete more refined multi-target semantic segmentation on this basis, step S3 uses an improved lane line semantic segmentation network, namely the improved DeepLabV3Plus model, for multi-target semantic segmentation. This model has a complex architecture and multi-scale feature fusion characteristics, and can provide more refined segmentation results. The core of the DeepLabV3Plus model lies in its encoder-decoder structure, in which the encoder extracts high-level features of the image through a deep convolutional network, while the decoder gradually restores the spatial resolution to achieve accurate pixel-level prediction.

[0067] The DeepLabV3Plus model is used to extract and segment the input data. The model extracts lane-related edge, texture, and geometric features through a deep convolutional network, and combines the contextual information of multi-scale features to gradually restore the spatial resolution of the feature map. In the decoder, by fusing shallow features, the model can better preserve lane details.

[0068] The step S3 further comprises:

[0069] The improved DeepLabV3Plus model is trained, and then the regional self-attention unit of the trained model performs inner product on the original feature map and the lane line-background binary classification weight mask generated by the ERFnet model to obtain the lane line area semantic features, and then these features are linearly transformed through 1×1 convolution to generate query, key and value vectors. The regional attention weight matrix is ​​generated by calculating the similarity between the query vector and the key vector. Finally, the weight matrix and the value vector are weighted to obtain a more refined lane line area feature input model's Xception network. The Xception network uses deep convolution and point-by-point convolution to perform independent convolutions on the input feature map respectively. ASPP uses multiple hole convolution kernels of different sizes in parallel to extract features of different scales from the feature map output by Xception, and further captures them through global average pooling. Global context information. At the same time, the features input to the channel self-attention unit by Xception are reduced in height and width dimensions through two one-dimensional convolution operations, and further optimized through nonlinear mapping. Then, the dot product of the compressed query vector and the key vector is calculated to generate the channel attention weight matrix, and it is applied to the value vector to complete the weighted operation. Then, the weighted channel features are added element by element to the original features to form a new feature map for enhancing the expression of the lane line area. Finally, the feature fusion module fuses the high-dimensional features output by the channel self-attention module with the low-dimensional features output by the ASPP module, and then the fused features are scaled and segmented multiple times at different scales through the multi-scale target reasoning module to generate a semantic segmentation map containing different lane line categories (such as solid lines, dotted lines, and double lines, etc.), and the final segmentation result is obtained, completing the model's semantic segmentation task for lane lines.

[0070] The attention mechanism can focus on important information with high weights and ignore irrelevant information with low weights, and can adaptively adjust weights so that important features can be focused in different environments. In specific downstream tasks, deep learning network models with attention mechanisms can fit specified features based on the annotations of the original data. The attention mechanism can improve the model's expressiveness, modeling capabilities, and robustness, enabling the model to handle complex data and tasks more flexibly, and provide interpretability and visualization capabilities. It is an important part of deep learning research and application.

[0071] Through the above steps, the improved DeepLabV3Plus model is expanded from binary segmentation maps to multi-target semantic segmentation, which not only enhances the segmentation capability of complex lane line scenes, but also significantly improves the accuracy and robustness of the segmentation results, providing a reliable foundation for subsequent lane line recognition and analysis.

[0072] The improved DeepLabV3Plus model of the present invention introduces regional self-attention units (RSA) and channel self-attention units (CSA) based on the existing DeepLabV3Plus model. The introduction of regional self-attention units enables the model to focus more on key areas in the image, such as the boundary area between lane lines and background, thereby enhancing the ability to distinguish features of different regions. The introduction of channel self-attention units further strengthens the model's weight allocation to different channel features, which helps to improve the accuracy of segmentation, especially when dealing with lane lines with complex textures and color changes. By combining regional self-attention and channel self-attention, the improved DeepLabV3Plus model refines the binary segmentation map to achieve the final multi-target semantic segmentation, including accurate recognition of different types of lane lines, such as solid lines, dotted lines, double lines, etc.; through the combination of regional self-attention and channel self-attention, the model can better utilize the spatial and channel correlations in the image, thereby enhancing the model's expressive ability and segmentation performance, and thus helping the model to more accurately locate and segment targets in the image. When faced with complex scenes and multi-category segmentation tasks, regional self-attention and channel self-attention can also improve the generalization ability of the model, making the model adaptable to images of different scales, shapes and perspectives.

[0073] The improved lane semantic segmentation network of the present invention, i.e., the improved DeepLabV3Plus model, adopts an encoder-decoder structure as a whole, wherein the encoder is composed of multiple convolutional layers and pooling layers, which are used to gradually reduce the spatial resolution of the input image and extract high-level semantic features, thereby effectively compressing the information of the input image and capturing its important feature information for more accurate segmentation in subsequent processing; the decoder gradually restores the spatial resolution using transposed convolution operations or upsampling operations, and at the same time fuses the low-level and high-level features of the encoder through jump connections, and finally maps the features extracted by the encoder back to the spatial size of the original image, and generates pixel-level segmentation results.

[0074] Specifically, the improved DeepLabV3Plus model includes an encoder module and a decoder module. The encoder module includes a regional self-attention module RSA, an Xception module, a channel self-attention module CSA and an ASPP (Atrous Spatial Pyramid Pooling) module. The decoder module includes a feature fusion module and a multi-scale target reasoning module. The specific structure is as follows: Figure 2 shown.

[0075] In the semantic segmentation of lane lines, considering that lane lines have rich contextual information in the image and have certain spatial consistency and continuity, adding self-attention to the traditional network structure can help the model better understand the semantic relationship between different regions in the image, thereby improving the accuracy and robustness of lane line segmentation. At the same time, taking advantage of the high real-time and low computational complexity of ERFnet, a weight mask is added to the self-attention unit, the attention weight matrix is ​​calculated on the specified area, and the relevant information between regions is extracted.

[0076] Specifically, the regional self-attention unit can automatically learn the degree of correlation between lane line features according to the needs of the lane line segmentation task. By calculating the similarity between different regions, it can assign a weight to each pixel to represent its correlation with other pixels. In this way, the model can weightedly aggregate the features of different regions according to the weights to extract important features related to the lane line. At the same time, the regional self-attention unit can make the model pay more attention to the lane line area and model the details of the lane line more accurately. It can help the model capture important features such as the shape, connectivity, and curvature of the lane line, as well as the interactive information between the lane line and other scene elements, which is critical to the accuracy and robustness of the lane line semantic segmentation, especially in complex road scenes.

[0077] The design of the regional self-attention unit is based on the self-attention mechanism, and its internal structure is as follows: Figure 3 shown.

[0078] The input of the regional self-attention unit is the original feature map f and the weighted mask version with lane line semantics obtained by ERFnet model processing. The inner product of the template and the original feature map is used to obtain the semantic features of the lane line area. Then, the feature is linearly transformed through a convolution layer with a convolution kernel size of 1×1 to improve the receptive field of spatial correlation and local structural information of different lane line areas and match them with the key vector f key Calculate the dimension and get the self-attention mechanism query vector The expression is as follows:

[0079]

[0080] Among them, Conv 1×1 represents 1×1 convolution, ⊙ represents inner product;

[0081] Then calculate the key and value vectors in the regional self-attention mechanism. The specific expressions are as follows:

[0082] f key =Conv 1×1 (f)

[0083] f val =Conv 1×1 (f)

[0084] The Q, K, and V vectors required for self-attention are obtained through the calculation of the above three formulas, that is, f key 、f val Then the similarity between Q and K is calculated. Specifically, the attention score is obtained by calculating the dot product of the Q vector and the K vector, which reflects the relationship between different elements in the input sequence. The calculation process involves multiplying the transpose of the Q matrix and the K matrix to obtain a similarity matrix, which represents the relevance of the lane line area features compared to the full image context information. Based on this similarity matrix, the attention score is obtained by dividing by the scaling factor The attention weights are scaled to avoid the problem of gradient vanishing or gradient exploding during the calculation process. The calculation results are normalized in the Softmax function to ensure the effectiveness and interpretability of the attention weights so that they meet the requirements of probability distribution, so as to better explain the degree of attention to different features. Finally, the attention weights and f val Multiply and calculate the weighted eigenvalue to highlight the lane feature information and suppress irrelevant noise information. The formula is as follows:

[0085]

[0086] The output of RSA is the image information weighted by the attention matrix. This module mainly focuses on the correlation between different spatial positions in the image. The input feature map is regarded as a spatial grid, and each position interacts with other positions. By calculating the similarity or correlation between each position and other positions, a weight matrix is ​​obtained for weighted aggregation features. This can capture the dependency between different positions, enabling the model to better understand the spatial structure and contextual information in the image, and improve the model's lane line classification accuracy in lane line scenes with multiple scales and occlusion relationships.

[0087] The backbone network uses the Xception model, which performs further convolution operations on the weighted feature maps output by RSA. The Xception model adopts a deep separable convolution architecture, splitting the traditional standard convolution into two steps: deep convolution and point-by-point convolution. This design enables the network to better learn the spatial and channel correlation of features and has better feature representation capabilities. At the same time, compared with traditional convolutional neural networks, deep separable convolution can reduce the number of model parameters and computational complexity, and greatly reduce the number of parameters of standard convolution, thereby improving the efficiency and training speed of the model, making it more suitable for lane line semantic segmentation applications with certain real-time requirements and complex segmentation scenarios.

[0088] The ASPP module is one of the key components in the DeepLabV3Plus model. This module can capture contextual information at different scales and improve the performance and accuracy of semantic segmentation. Its input is the high-dimensional features in Xception. The main purpose is to expand the receptive field so that the network can effectively perceive targets of different scales. This goal is achieved by applying expansion rates of different scales on multiple hole convolution layers. The expansion rate defines the sampling interval between pixels in the convolution kernel. A larger expansion rate can enrich the receptive field and capture a wider range of contextual information, enabling the network to better understand the targets and backgrounds in the image and generate more accurate segmentation results. ASPP uses multiple hole convolution kernels of different sizes (such as 1x1, 3x3, 5x5) in parallel to extract features from different scales, and further capture global contextual information through global average pooling.

[0089] The RSA unit mainly performs attention weighted operations on low-dimensional features. When extracting low-dimensional features, it has rich spatial semantic information. The original image still retains a smaller number of channels and a larger size, realizing the processing and reprocessing of spatial semantic information. However, the information contained in the image not only has a spatial dimension but also a channel dimension. After multiple convolutional layer feature extractions, when extracting high-dimensional features, the number of channels can be expanded from the original three channels to hundreds or thousands of channels. Processing channel information on this basis can enable the model to better recognize features.

[0090] Channel information focuses on the semantic differences and correlations of features represented by different channels. Features represented by different channels provide information on different aspects of the image, such as color, texture, shape, etc. By analyzing and understanding the correlation between different channels, different semantic features expressed by different channels can be identified, so as to better understand and distinguish different objects or scenes in the image. This paper designs a channel self-attention module CSA for extracting channel semantic information. Its internal structure is as follows: Figure 4 shown.

[0091] The input of the channel self-attention module is the high-dimensional feature map output by Xception, that is, a more abstract feature representation obtained after multiple convolutions of the neural network. The high-dimensional feature map usually contains multiple feature dimensions. Compared with calculating the channel self-attention weights in low dimensions, the results calculated on this basis have a larger receptive field and richer feature channels. Similar to the usual self-attention module, it is necessary to calculate the Q, K, and V vectors from the original feature map. Considering the need to compress the information in the two dimensions of height H and width W, the present invention chooses to use two one-dimensional convolutions for feature extraction. This dimensional compression method can learn more specific and local feature representations, reduce the semantic information lost during the dimensional compression process, and is suitable for more complex image feature extraction tasks. At the same time, a ReLU activation function is added between the two one-dimensional convolutions. By introducing nonlinear transformations in linear transformations, more complex features can be learned when adjusting weights. The formula is expressed as follows:

[0092] f x =Conv1D2(ReLU(Conv1D1(f′))),x∈{q,k,v}

[0093] In this formula, f′ is the high-dimensional feature map output by Xception. After the first layer of one-dimensional convolution Conv1D1 with a convolution kernel size of 1, the feature map is compressed from the dimension of H. Then the nonlinear activation function ReLU is introduced. Finally, the outer layer of one-dimensional convolution Conv1D2 with a convolution kernel size of 1 is used to compress the dimension of the feature map W. After two compressions and one nonlinear transformation, a 1×C (C is the number of channels) dimensional vector is obtained. This vector is the Q, K, and V input of the self-attention. The final channel self-attention formula is expressed as follows:

[0094]

[0095] i and j represent the rows and columns of the weight matrix respectively, using f q and f k Calculate the correlation on the channel, and the result is a C×C weight matrix, where each row represents f q Each element and f k The relevance of all elements in , and the subsequent weights need to be assigned to each channel to evaluate the importance of each channel, so the weight matrix is ​​summed in the first dimension, so that the weight matrix is ​​compressed into the sum f v The same shape, by dividing The weights are scaled to avoid the problem of gradient vanishing or gradient exploding during the calculation process. After the Softmax normalization operation, the probability distribution requirements of the weights are guaranteed. Then, the channel feature f is multiplied by point multiplication. vWeighting is performed to increase the attention to important channel information. Finally, the channel self-attention representation is repeatedly stacked and transformed from (1, 1, C) to (H, W, C) through the shaping function reshape, and then weighted to the feature map f' to realize the self-attention mechanism on the channel features.

[0096] After passing through the channel self-attention module, a weighted high-dimensional feature map will be obtained, while the feature map after ASPP processing is a low-dimensional feature map. The feature fusion module in the decoder fuses the high-dimensional features output by the channel self-attention module with the low-dimensional features output by the ASPP module to comprehensively utilize feature information at different levels, better retain the detailed information of the image, and improve the ability to segment small-sized objects and boundaries.

[0097] After integrating all the extracted feature information, semantic segmentation is performed through the multi-scale target inference module. The input image is scaled and segmented multiple times at different scales. After several upsamplings, the results are finally fused to obtain the final segmentation result, completing the model's semantic segmentation task for lane lines.

[0098] The essence of semantic segmentation is to make classification predictions for each pixel in the image. The output result of the network is essentially the classification probability distribution of each pixel. Therefore, when training the ERFnet model and the improved DeepLabV3Plus model, in order to minimize the error between the predicted classification and the true classification, the training uses the reverse iteration technique, which is based on the gradient descent algorithm. By propagating the error from the output layer to the input layer, the contribution of each parameter to the error is calculated, and the parameter value is updated according to the gradient information, thereby gradually optimizing the performance of the network model. In terms of the learning rate adjustment algorithm, the Adam algorithm is used. Compared with the traditional gradient descent algorithm, the Adam algorithm has better computational efficiency because it uses an adaptive learning rate for each parameter without the need for an additional learning rate adjustment process. In addition, the Adam algorithm also combines the estimation of momentum and second-order moments, which can better handle non-stationary objective functions and sparse gradients, thereby improving the convergence performance of the model. The specific formula is expressed as follows:

[0099]

[0100] In the above formula, reverse iteration requires initializing the learning rate α, decay exponents β1 and β2, and the smoothing term ε. t is the current number of iterations, and θ is the parameter that needs to be iteratively optimized. First, the forward propagation loss calculation function L and the gradient of the parameter θ are executed. Next, we need to calculate the first-order moment variable m and the second-order moment variable v of θ according to the above formula. In the first iteration, m and v are set to zero vectors. Then, according to the deviation correction of the first-order moment and the second-order moment, the corrected first-order moment estimate m is calculated according to the above formula. *and the second-order moment estimate v * , and finally update the parameter θ according to the above formula to complete the back propagation parameter optimization.

[0101] The selected loss calculation function is the cross entropy loss function, which is used to measure the mismatch between the predicted value of the classification model and the true label. In the lane line semantic segmentation task, the difference between the probability distribution predicted by the model and the true label value of each pixel is measured. The formula is expressed as follows:

[0102] L=-∑ i y i ·log(p i )

[0103] Among them, y i is a vector with the same length as the number of categories, representing the one-hot encoding form of the true label, p i It is also a vector of length equal to the number of categories, representing the predicted probability distribution of the model.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lane semantic segmentation method based on self-attention, characterized in that: The method comprises: Step S1: perform data collection and preprocessing, obtain the video stream collected by the vehicle camera, intercept the obtained video stream according to a fixed frame interval and mark the lane lines, and standardize the original image data set; Step S2: training the ERFnet neural network model, and realizing binary semantic segmentation of image data based on the trained ERFnet neural network model, specifically: dividing the image data processed in step S1 into batches, and first sending the batch data into the convolution layer of the ERFnet model for feature extraction, and then the residual connection block splits the two-dimensional features after feature extraction into two one-dimensional convolutions, respectively extracting features in the horizontal and vertical directions, and then the residual connection operation fuses the processed feature map with the original input feature map to generate a set of feature maps, and finally these feature maps are further processed by the fully connected layer, and the probability distribution of each pixel is mapped to the corresponding category label through the Softmax activation function to generate a binary semantic segmentation result; Step S3: According to the binary semantic segmentation map output by step S2, the final multi-target semantic segmentation is performed based on the improved DeepLabV3Plus model.

2. The method according to claim 1, characterized in that The step S3 further comprises: Step S31: training the improved DeepLabV3Plus model; Step S32: The regional self-attention unit of the trained model performs inner product on the original feature map and the lane-background binary classification weight mask generated by the ERFnet model to obtain the lane area semantic features, and performs linear transformation on these features through 1×1 convolution to generate query, key and value vectors, and generates the regional attention weight matrix by calculating the similarity between the query vector and the key vector; Step S33: Perform weighted calculation on the regional attention weight matrix and the value vector to obtain more refined lane line regional features and input them into the Xception network of the model. The Xception network uses deep convolution and point-by-point convolution to perform independent convolution on the input feature map respectively. Step S34: ASPP uses multiple hole convolution kernels of different sizes in parallel to extract features of different scales from the feature map output by Xception, and further captures global context information through global average pooling; Step S35: The channel self-attention unit reduces the height and width dimensions of the features input by Xception through two one-dimensional convolution operations, and further optimizes them through nonlinear mapping. Then, the dot product of the compressed query vector and the key vector is calculated to generate a channel attention weight matrix, and the channel attention weight matrix is ​​applied to the value vector to complete the weighted operation. The weighted channel features are added element by element to the original features to form a new feature map. Step S36: The feature fusion module fuses the high-dimensional features output by the channel self-attention module with the low-dimensional features output by the ASPP module, and then uses the multi-scale target inference module to perform multiple scaling and segmentation on the fused features at different scales to obtain the final semantic segmentation result.

3. The method according to claim 1, characterized in that The improved DeepLabV3Plus model includes an encoder module and a decoder module. The encoder module includes a regional self-attention module RSA, an Xception module, a channel self-attention module CSA and an ASPP module. The decoder module includes a feature fusion module and a multi-scale target reasoning module.

4. The method according to claim 3, characterized in that The workflow of the RSA module is as follows: The input f of RSA is the original feature map and the weighted mask version with lane line semantics obtained by processing the ERFnet model. The inner product of the template and the original feature map is used to obtain the semantic features of the lane area. Then, the feature is linearly transformed through a convolution layer with a convolution kernel size of 1×1 and matched with the key vector f key Calculate the dimension and get the self-attention mechanism query vector The expression is: Among them, Conv 1×1 represents 1×1 convolution, ⊙ represents inner product; Then calculate the key and value vectors in the regional self-attention mechanism: f key =Conv 1×1 (f) f val =Conv 1×1 (f) This gives us the Q, K, and V vectors needed for self-attention, namely f key 、f val ; Then multiply the transpose of the Q matrix and the K matrix to get a similarity matrix, and divide the similarity matrix by the scaling factor Scale the weights, put the calculation results into the Softmax function for normalization, and finally add the attention weights and f val Multiply and calculate the weighted eigenvalue. The expression is as follows:

5. The method according to claim 3, characterized in that: The Xception model uses depthwise convolution and point-by-point convolution to perform further convolution operations on the weighted feature map output by RSA.

6. The method according to claim 3, characterized in that The ASPP module uses multiple hole convolution kernels of different sizes in parallel to extract features of different scales from the feature maps output by Xception, and further captures global context information through global average pooling.

7. The method according to claim 3, characterized in that The workflow of the CSA module is as follows: the input of the CSA module is the high-dimensional feature map output by Xception, two one-dimensional convolutions are used for feature extraction, and a ReLU activation function is added between the two one-dimensional convolutions. The expression is as follows: f x =Conv1D2(ReLU(Conv1D1(f′))), x∈{q,k,v} Among them, f′ is the high-dimensional feature map output by Xception. After two one-dimensional convolution compressions and one nonlinear transformation, a 1×C-dimensional vector is obtained, where C is the number of channels; Then use f q and f k Calculate the correlation on the channel, the result is a C×C weight matrix, sum the weight matrix in the first dimension and divide it by The weights are scaled, normalized by Softmax, and then the channel features f are multiplied by point multiplication. v After weighting, the channel self-attention representation is transformed from (1, 1, C) to (H, W, C) through the reshape function, and then weighted to the feature map f'. The expression is as follows:

8. The method according to claim 3, characterized in that The feature fusion module in the decoder module fuses the high-dimensional features output by the CSA module with the low-dimensional features output by the ASPP module. Then the multi-scale object inference module scales and segments the fused features multiple times at different scales to obtain the final semantic segmentation result.

Images