A lane line semantic segmentation method based on self-attention

Abstract

The application provides a lane line semantic segmentation method based on self-attention, first data acquisition and preprocessing are carried out, then binary semantic segmentation is realized based on an ERFnet neural network model, and finally, according to the binary semantic segmentation graph, the improved DeepLabV3Plus model is used for final multi-target semantic segmentation, the ERFnet is responsible for preliminary coarse-grained segmentation, and the lane line and the background are quickly separated, and the improved DeepLabV3Plus further refines the segmentation result, and through the introduction of the regional self-attention unit and the channel self-attention unit, the model's capturing ability for local and global features is enhanced. The scheme combines the ERFnet and the improved DeepLabV3Plus model, not only improves the segmentation accuracy and the generalization ability, but also maintains the real-time performance, realizes real-time and high-precision segmentation of the lane line, and has strong adaptability to variable and complex road conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a lane line semantic segmentation method based on self-attention. BACKGROUND

[0002] With the rapid development of computer artificial intelligence field, automatic driving has gradually become the research hotspot of various enterprises, whether it is a car manufacturer or a non-car manufacturer, and they strive to realize the true sense of unmanned driving as soon as possible, in the current research direction, the automatic driving system can be divided into three core parts according to the function: the perception layer, the decision layer and the execution layer, wherein the perception layer refers to the vehicle through various sensors and GPS and other technologies to perceive and identify the vehicle itself and the driving environment; the decision layer refers to the vehicle software system to generate driving strategies and processing results by calculating and processing various perception layer data, to ensure the normal communication and operation between the software and hardware subsystems; the execution layer refers to the vehicle hardware layer to execute the specific decision instructions of the decision layer. As can be seen, the functions of different levels cooperate with each other to form a complete automatic driving system, the perception layer obtains environmental information, the decision layer analyzes and decides the information, and the execution layer converts the decision results into specific actions, the perception layer in the whole automatic driving system can obtain all-around driving environment information such as feasible road surface, obstacles, traffic signals, position information and attitude information, the speed and accuracy of the driving environment perception of this layer directly affect whether the driving strategy taken by the decision layer is accurate and reliable, and further plays a crucial role in the safe driving of the vehicle, therefore, it is necessary to study the driving environment of the perception layer.

[0003] The current automatic driving system perceives the environment by relying on more and more precise hardware facilities for initial information collection, and also uses deep learning algorithms to analyze the extracted data for the second time, so as to extract more intuitive and effective data information. The environmental data collected by the perception layer in automatic driving has the following characteristics: complex and changeable environment, high real-time data, large data volume, etc. In the driving process, weather, traffic signs, pedestrians and the shielding of obstacles are the difficulties of secondary data processing. At the same time, the deep learning model will encounter a large number of features when facing complex data, some of which contribute more to the task, while others may not be important or even interference.

[0004] The ERFnet model is a lightweight convolutional neural network architecture specially used for real-time semantic segmentation tasks, and the DeepLabV3Plus model is a high-accuracy and high-efficiency semantic segmentation deep learning architecture, both of which have relatively mature applications in the field of semantic segmentation, but each has some weaknesses, the ERFnet is affected by the parameter quantity and only focuses on local receptive field feature extraction, and when facing complex scenes and monitoring target multi-scale changes, the segmentation result may be unstable, the DeepLabV3Plus model structure is more complex, and more computing resources and time are required, therefore, in the current lane line semantic segmentation task, there are widespread problems of low robustness, low generalization and low real-time degree. SUMMARY

[0005] In view of the problems of the prior art, the application provides a lane line semantic segmentation method based on self-attention, which adopts a deep learning algorithm of a deep neural network model to learn and represent high-level features of data, and through nonlinear transformation of multiple neural network layers, more complex and abstract features can be automatically extracted from original data, so that the deep learning network model has better performance in processing driving environment data.

[0006] The application combines the ERFnet model and the DeepLabV3Plus model, and improves the DeepLabV3Plus. The ERFnet is used for coarse-grained lane line feature extraction, the input image is subjected to multiple convolution and step-by-step down-sampling of the encoder to obtain a lane line feature map, and then the decoder is subjected to step-by-step up-sampling to obtain a binary semantic segmentation map of the lane line and the background, so as to divide the original image (such as a screenshot of a driving recorder) into the lane line and the background.

[0007] After preliminary segmentation by ERFnet, the improved DeepLabV3Plus further refines the segmentation of lane lines and classifies them more finely according to their features (such as single and double lines, yellow and white lines, and dashed and solid lines). In order to further improve the generalization and semantic segmentation accuracy of DeepLabV3Plus, regional self-attention units and channel self-attention units are introduced based on the original model. The introduction of these two attention units improves the feature aggregation capability of the network model in different dimensions. In the low-dimensional feature extraction stage, the regional self-attention unit is based on the segmentation results of ERFnet, allowing the network to focus on the spatial semantic information of the specified region in the image. This unit can weight the spatial features within the specified region, allowing different types of lane lines to receive unequal attention weights, thereby improving the ability to capture and classify local feature information. In the high-dimensional feature extraction stage, the channel self-attention unit allows the network to focus on the channel semantic information of the image. By calculating the correlation between different channels in the feature map and adaptively assigning weights to the channel features, it helps to extract the relevance between different feature channels in the image, further improving the global feature perception ability of the segmentation model. The introduction of regional self-attention units and channel self-attention units enables DeepLabV3Plus to have better segmentation capability in various application scenarios. These attention mechanisms allow the model to pay more attention to important regions and channels during feature extraction and aggregation, thereby improving the accuracy and generalization ability of semantic segmentation.

[0008] The lane line semantic segmentation method based on self-attention proposed by the application specifically comprises the following steps:

[0009] Step S1: data acquisition and preprocessing, obtaining the video stream collected by the vehicle-mounted camera, cutting pictures at fixed frame intervals from the obtained video stream and labeling the lane lines, and standardizing the original picture dataset;

[0010] Step S2: training an ERFnet neural network model, and implementing binary semantic segmentation of image data based on the trained ERFnet neural network model, specifically: dividing the image data processed in step S1 into batches, the batch data is first sent to the convolutional layer of the ERFnet model for feature extraction, then the residual connection block splits the two-dimensional features after feature extraction into two one-dimensional convolutions for feature extraction in the horizontal and vertical directions, then the residual connection operation fuses the processed feature maps with the original input feature maps to generate a set of feature maps, finally these feature maps are further processed through the fully connected layer, and the probability distribution of each pixel point is mapped to the corresponding class label through the Softmax activation function to generate a binary semantic segmentation result;

[0011] Step S3: Based on the binary semantic segmentation map output in step S2, the final multi-target semantic segmentation is performed based on the improved DeepLabV3Plus model.

[0012] Further, the step S3 further comprises:

[0013] Step S31: training the improved DeepLabV3Plus model;

[0014] Step S32: the region self-attention unit of the trained model performs inner product on the original feature map and the lane-line-background binary classification weight mask generated by the ERFnet model to obtain lane-line region semantic features, and performs linear transformation on these features through 1x1 convolution to generate query, key and value vectors, and generates a region attention weight matrix by calculating the similarity between the query vector and the key vector;

[0015] Step S33: the region attention weight matrix and the value vector are weighted to obtain more refined lane-line region features and input into the Xception network of the model, and the Xception network uses deep convolution and point-wise convolution to independently convolve the input feature map;

[0016] Step S34: ASPP uses multiple different size hole convolution kernels in parallel to extract features of different scales on the feature map output by Xception, and further captures global context information through global average pooling;

[0017] Step S35: the channel self-attention unit reduces the height and width dimensions of the Xception input feature through two one-dimensional convolution operations respectively, and further optimizes through nonlinear mapping, then calculates the dot product of the compressed query vector and the key vector to generate a channel attention weight matrix, and the channel attention weight matrix is applied to the value vector to complete the weighting operation, the weighted channel features and the original features are added element by element to form a new feature map;

[0018] Step S36: the feature fusion module fuses the high-dimensional feature output by the channel self-attention module and the low-dimensional feature output by the ASPP module, and then performs multiple scaling and segmentation on the fused feature at different scales through the multi-scale target reasoning module to obtain the final semantic segmentation result.

[0019] Further, the improved DeepLabV3Plus model comprises an encoder module and a decoder module, the encoder module comprises a region self-attention module RSA, an Xception module, a channel self-attention module CSA and an ASPP module, and the decoder module comprises a feature fusion module and a multi-scale target reasoning module.

[0020] Further, the workflow of the RSA module is as follows: the input f of the RSA is the original feature map and the weight mask version with lane line semantics obtained by processing the ERFnet model The template is multiplied by the original feature map to obtain lane line region semantic features, and then a convolution layer with a kernel size of 1x1 is used to linearly transform the features and match the key vector f key The dimensions are calculated to obtain the query vector of the self-attention mechanism The expression is:

[0021]

[0022] where Conv 1×1 represents a 1x1 convolution, and represents an inner product.

[0023] Then the key and value vectors in the region self-attention mechanism are calculated, and the specific expression is:

[0024] f key = Conv 1×1 (f)

[0025] f val = Conv 1×1 (f)

[0026] Through the above calculation, the Q, K, and V vectors required by the self-attention are obtained, i.e. f key , f val .

[0027] Then, the transpose of the Q matrix and the K matrix are multiplied to obtain a similarity matrix, and the similarity matrix is divided by a scaling factor The weights are scaled, the calculation results are put into the Softmax function for normalization, and finally the attention weights are multiplied by f val to calculate the weighted feature values, and the expression is as follows:

[0028]

[0029] Further, the Xception model uses deep convolution and pointwise convolution to further convolve the weighted feature map output by the RSA.

[0030] Further, the ASPP module uses multiple different sizes of hole convolution kernels in parallel to extract features of different scales from the feature map output by the Xception, and further captures global context information through global average pooling.

[0031] Furthermore, the workflow of the CSA module is as follows: the input of the CSA module is the high-dimensional feature map output by Xception, two one-dimensional convolutions are used for feature extraction, and a ReLU activation function is added between the two one-dimensional convolutions. The expression is as follows:

[0032] f x =Conv1D2(ReLU(Conv1D1(f′))),x∈{q,k,v}

[0033] Among them, f′ is the high-dimensional feature map output by Xception. After two one-dimensional convolution compressions and one nonlinear transformation, a 1×C-dimensional vector is obtained, where C is the number of channels.

[0034] Then use f q and f k Calculate the correlation on the channel, the result is a C×C weight matrix, sum the weight matrix in the first dimension and divide it by The weights are scaled, normalized by Softmax, and then the channel features f are multiplied by point multiplication. v After weighting, the channel self-attention representation is transformed from (1, 1, C) to (H, W, C) through the reshape function, and then weighted to the feature map f'. The expression is as follows:

[0035]

[0036] Furthermore, the feature fusion module in the decoder module fuses the high-dimensional features output by the CSA module with the low-dimensional features output by the ASPP module. Then the multi-scale object inference module scales and segments the fused features multiple times at different scales to obtain the final semantic segmentation result.

[0037] Compared with the prior art, the beneficial technical effects of the present invention are:

[0038] Real-time and high efficiency: In the lane line semantic segmentation task, the application combines ERFnet and DeepLabV3Plus model, and realizes the balance of real-time and high precision through two-level segmentation. ERFnet is responsible for the preliminary segmentation of the image with its low computational complexity and high real-time characteristics, extracts the lane line features in a coarse-grained manner and generates a preliminary binary segmentation map. Then, the improved DeepLabV3Plus further refines the coarse segmentation result, improves the segmentation accuracy without increasing the overall computational burden. ERFnet, as a lightweight network, has efficient real-time performance and low computational complexity, is suitable for performing preliminary coarse-grained segmentation of lane lines in vehicle-mounted systems with limited computing resources, can quickly separate lane lines and backgrounds, and provides a good foundation for the next fine segmentation, ensures that the system maintains high efficiency in real-time scenarios, and improves timeliness and computational efficiency.

[0039] Multi-scale scene adaptability: In view of the problem that ERFnet is unstable in local feature extraction in complex and multi-scale scenes, the application improves the DeepLabV3Plus model, introduces a regional self-attention unit and a channel self-attention unit in the DeepLabV3Plus model, the regional self-attention unit focuses on the specified region in the image, so that the network can give different attention weights to different types of lane lines (such as single and double lines, yellow and white lines, and solid and dashed lines), thereby improving the segmentation accuracy in complex scenes and improving the local feature capture ability of the model in complex scenes; the channel self-attention unit optimizes the feature weights of each channel and adaptively weights the features, so that the model can effectively capture the global semantic information of the lane line when aggregating high-dimensional features, and enhance the adaptability to multi-scale changes.

[0040] Segmentation accuracy and generalization ability: The combination of regional and channel self-attention mechanisms enables DeepLabV3Plus to effectively allocate attention resources and focus on key regions and important feature channels in the image. This improvement effectively improves the feature aggregation ability of the network model, enabling the model to have stronger generalization ability in diversified road scenes, and significantly improving the accuracy and stability of the segmentation result.

[0041] In summary, the application effectively integrates the advantages of ERFnet and DeepLabV3Plus, reasonably applies the self-attention unit, and significantly improves the accuracy and robustness of lane line semantic segmentation under the premise of ensuring high real-time performance and computational efficiency, especially in complex and variable road conditions. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0043] Figure 1 is a flow chart of a lane line semantic segmentation method based on self-attention provided by an embodiment of the present application;

[0044] Figure 2 is a network model diagram of lane line semantic segmentation provided by an embodiment of the present application;

[0045] Figure 3 is an internal structure diagram of an RSA unit provided by an embodiment of the present application;

[0046] Figure 4 is an internal structure diagram of a CSA unit provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0048] The lane line semantic segmentation method based on self-attention proposed in the present application mainly includes two parts. The first part is data preprocessing. This stage is based on an ERFnet neural network model to output a binary semantic segmentation map, so as to prepare data for the subsequent attention unit. The second part is multi-target semantic segmentation. This stage is based on a DeepLabV3Plus neural network model combined with two attention units to realize the final lane line semantic segmentation.

[0049] A lane line semantic segmentation method based on self-attention, as shown in Figure 1 , the method comprises:

[0050] Step S1: data acquisition and preprocessing;

[0051] Step S2: realizing binary semantic segmentation based on an ERFnet neural network model;

[0052] Step S3: based on the improved DeepLabV3Plus model, performing final multi-target semantic segmentation according to the binary semantic segmentation map output in step S2.

[0053] Specifically, the step S1 further comprises:

[0054] The video stream collected by the vehicle-mounted camera is obtained, pictures are intercepted at fixed frame number intervals, and lane lines are labeled, the labeled data is a gray image, and according to different lane line types, each pixel of the original picture corresponds to different values in the gray image.

[0055] When the neural network learns the feature weight, if the input training data is not scaled, the distribution range of each feature value may be different, which may cause an imbalance in the correction of the learning rate in different dimensions, thereby causing the model to excessively adjust in some weight dimensions and hardly adjust in other weight dimensions, which may cause the training process to become difficult, the convergence speed to slow down, and the model performance to decrease, therefore, the value range of each feature is scaled to a similar interval, so that each feature has the same influence on the model, and the model can more balancedly consider each feature when updating the weight, thereby avoiding excessive or insufficient adjustment, and the training effect is improved.

[0056] In order to eliminate the bias in the data set and prevent data with too large numerical difference from negatively interfering with the training of the neural network model, the original picture data set needs to be standardized, since the original data is a three-channel picture, the pixels in each channel are distributed in [0, 255], therefore, the standardization needs to be performed on each channel, the present application adopts mean-variance standardization, also known as Z-score standardization, and the standardization formula is as follows:

[0057] x * =(x-μ) / σ

[0058] Wherein, x represents the sample value before standardization, i.e., the pixel value on a single channel of the original picture, μ and σ respectively represent the mean and variance of all picture data on each channel, and x * The image data is normalized to ensure that the pixel value range is adapted to the input requirements of the ERFnet model, thereby avoiding unstable training or convergence difficulty caused by inconsistent data distribution.

[0059] The step S2 further comprises:

[0060] The data after preprocessing by step S1 is input to step S2, and the input data is a set of standardized image data. Then, the image data is divided into batches to improve the computational efficiency of the training and inference stages. The ERFnet neural network model is trained, and then the trained ERFnet neural network model is used to perform binary semantic segmentation on the input image data. The ERFnet model is a semantic segmentation network based on residual connection and separable convolution. It improves the efficiency of processing frames without reducing accuracy, and is particularly suitable for real-time requirements in autonomous driving. The model has a lightweight architecture and efficient feature extraction capability, and can quickly learn the feature representation of lane lines. In the training process, multi-level features of the image are extracted through convolution layers and residual connections, which are crucial for subsequent semantic segmentation.

[0061] The ERFnet neural network model is used to perform binary semantic segmentation on the input image data, including: batch data is first sent to the convolution layer of the ERFnet model to start the feature extraction process. The convolution layer extracts basic features of the image such as edges and textures through two-dimensional convolution operations, which lays the foundation for subsequent advanced feature learning. In each convolution process, the convolution kernel covers the local area of the image in the form of a sliding window, and generates a feature map through weighted summation. This method preserves spatial relationships while extracting local pattern information.

[0062] Then, the core module of the ERFnet model, the Non-bottleneck-1D residual connection block, further processes the input two-dimensional feature map by splitting the two-dimensional convolution feature into two one-dimensional convolutions, which operate in the horizontal and vertical directions respectively. The first step is to extract features along the horizontal direction using one-dimensional convolution, and the second step is to extract features along the vertical direction using another set of one-dimensional convolution. This decomposition method reduces the number of parameters while expanding the receptive field, thereby improving the context modeling capability of the model.

[0063] Then, the residual connection operation fuses the processed feature map with the original input feature map, which enhances the information transmission capability and ensures that the deep network retains important information from the shallow layer. After each two-dimensional convolution operation, the feature map is also subjected to a nonlinear activation function to introduce nonlinear expression capability.

[0064] After the above multi-level feature extraction and processing, a set of feature maps is finally generated. These feature maps are further processed through a fully connected layer, and the probability distribution of each pixel point is mapped to the corresponding class label through a Softmax activation function to generate a binary semantic segmentation result. The output binary semantic segmentation map is a two-dimensional matrix consistent with the size of the original image, where the value of each pixel point represents the classification result of that position. In this way, the ERFnet model realizes efficient and accurate semantic segmentation of lane lines and other targets.

[0065] The ERFnet model adopts the classic Encoder-Decoder architecture. In the encoding stage, the resolution of the feature map is gradually reduced through multi-layer convolution and pooling operations, thereby extracting more abstract semantic features. In the decoding stage, the feature map gradually recovers the spatial resolution through upsampling. To avoid the loss of details of high-level semantic information, the decoding stage also combines a feature fusion strategy, i.e., concatenating the features of the shallow layers in the encoding stage with the features of the corresponding layers in the decoding stage, so that the model can retain both local details and global context information.

[0066] The preliminary binary semantic segmentation map obtained in step S2 is mainly used to distinguish lane lines from background areas. In order to complete more fine multi-target semantic segmentation on this basis, step S3 uses an improved lane line semantic segmentation network, i.e., an improved DeepLabV3Plus model, for multi-target semantic segmentation. This model has a complex architecture and multi-scale feature fusion characteristics, and can provide more fine segmentation results. The core of the DeepLabV3Plus model lies in its encoder-decoder structure, in which the encoder extracts high-level features of the image through a deep convolutional network, while the decoder gradually recovers the spatial resolution to achieve accurate pixel-level prediction.

[0067] The DeepLabV3Plus model is used for feature extraction and segmentation of input data. This model extracts edge, texture, and geometric features related to lane lines through a deep convolutional network, while combining the context information of multi-scale features to gradually recover the spatial resolution of the feature map. In the decoder, by fusing shallow features, the model can better retain lane line details.

[0068] The step S3 further comprises:

[0069] The improved DeepLabV3Plus model is trained, and then the region self-attention unit of the trained model performs inner product on the original feature map and the lane-line-background binary classification weight mask generated by the ERFnet model to obtain lane-line region semantic features. Then, the features are linearly transformed through 1x1 convolution to generate query, key and value vectors. The similarity between the query vector and the key vector is calculated to generate a region attention weight matrix. Finally, the weight matrix is weighted with the value vector to obtain more refined lane-line region features input into the Xception network of the model. The Xception network uses deep convolution and point-wise convolution to independently convolve the input feature map. The ASPP uses multiple different size hole convolution kernels in parallel to extract features of different scales from the Xception output feature map, and further captures global context information through global average pooling. At the same time, the features input into the channel self-attention unit are reduced in height and width dimensions through two one-dimensional convolution operations, and are further optimized through nonlinear mapping. Then, the dot product of the compressed query vector and the key vector is calculated to generate a channel attention weight matrix, which is applied to the value vector to complete the weighting operation. Then, the weighted channel features are added element by element with the original features to form a new feature map for enhancing the expression of the lane-line region. Finally, the feature fusion module fuses the high-dimensional features output by the channel self-attention module with the low-dimensional features output by the ASPP module, and then performs multiple scaling and segmentation on the fused features at different scales through the multi-scale target reasoning module to generate a semantic segmentation map containing different lane-line categories (such as solid lines, dashed lines and double lines, etc.), obtaining the final segmentation result and completing the semantic segmentation task of the model on the lane line.

[0070] The attention mechanism can focus on important information with high weight and ignore irrelevant information with low weight, and can adaptively adjust the weight so that it can focus on important features in different environments. In specific downstream tasks, deep learning network models with attention mechanisms can fit specified features according to the annotations of the original data. Through the attention mechanism, the model's expression ability, modeling ability and robustness can be improved, making the model more flexible in handling complex data and tasks, and providing explainability and visualization capabilities. It is an important part of deep learning research and application.

[0071] Through the above steps, the improved DeepLabV3Plus model is expanded from binary segmentation to multi-target semantic segmentation, which not only enhances the segmentation capability of complex lane line scenes, but also significantly improves the accuracy and robustness of the segmentation result, providing a reliable foundation for subsequent lane line recognition and analysis.

[0072] The improved DeepLabV3Plus model of the present application introduces a regional self-attention unit (RSA) and a channel self-attention unit (CSA) on the basis of the existing DeepLabV3Plus model. The introduction of the regional self-attention unit enables the model to focus more on the key regions in the image, such as the boundary regions of the lane lines and the background, thereby enhancing the ability to distinguish different regional features. The introduction of the channel self-attention unit further strengthens the weight distribution of different channel features of the model, which helps to improve the accuracy of segmentation, especially when dealing with lane lines with complex textures and color changes. By combining regional self-attention and channel self-attention, the improved DeepLabV3Plus model refines the binary segmentation map and realizes the final multi-object semantic segmentation, including accurate identification of different types of lane lines such as solid lines, dashed lines, double lines, etc. Through the combination of regional self-attention and channel self-attention, the model can better utilize the spatial and channel correlation in the image, thereby enhancing the expression ability and segmentation performance of the model, and thus helping the model to more accurately locate and segment the objects in the image. In the face of complex scenes and multi-class segmentation tasks, regional self-attention and channel self-attention can also improve the generalization ability of the model, making the model have good adaptability on images of different scales, shapes and perspectives.

[0073] The improved lane line semantic segmentation network, i.e. the improved DeepLabV3Plus model, adopts an encoder-decoder structure as a whole, wherein the encoder is composed of multiple convolutional layers and pooling layers, which are used to gradually reduce the spatial resolution of the input image and extract high-level semantic features, thereby effectively compressing the information of the input image and capturing its important feature information for more accurate segmentation in subsequent processing. The decoder uses transpose convolution operation or upsampling operation to gradually restore the spatial resolution, and at the same time, the low-level and high-level features of the encoder are fused through a skip connection, finally mapping the features extracted by the encoder back to the spatial size of the original image, and generating a pixel-level segmentation result.

[0074] Specifically, the improved DeepLabV3Plus model includes an encoder module and a decoder module, the encoder module includes a regional self-attention module RSA, an Xception module, a channel self-attention module CSA and an ASPP (Atrous Spatial Pyramid Pooling) module, and the decoder module includes a feature fusion module and a multi-scale object reasoning module, and the specific structure is as shown in Figure 2

[0075] ​In lane line semantic segmentation, considering that lane lines have rich contextual information in pictures, and have certain spatial consistency and continuity, adding self-attention in the traditional network structure can help the model better understand the semantic relationship between different regions in the image, thereby improving the accuracy and robustness of lane line segmentation. Meanwhile, by taking advantage of the high real-time and low computational complexity of ERFnet, a weight mask is added to the self-attention unit to calculate the attention weight matrix on the specified region and extract the relevant information between regions.

[0076] Specifically, the regional self-attention unit can automatically learn the correlation degree between lane line features according to the needs of the lane line segmentation task. By calculating the similarity between different regions, it can assign a weight to each pixel representing its correlation with other pixels. In this way, the model can aggregate the features of different regions according to the weights, thereby extracting important features related to lane lines. Meanwhile, the regional self-attention unit can make the model pay more attention to the lane line region and model the details of the lane line more accurately. It can help the model capture important features such as the shape, connectivity, and curvature of the lane line, as well as the interaction information between the lane line and other scene elements, which is crucial for the accuracy and robustness of lane line semantic segmentation, especially in complex road scenes.

[0077] The design of the regional self-attention unit is based on the self-attention mechanism, and its internal structure is as shown in Figure 3

[0078] The input of the regional self-attention unit is the original feature map f and the weight mask version of the lane line semantic obtained by processing the ERFnet model The template is multiplied by the original feature map to obtain the lane line region semantic feature. Then, a convolution layer with a kernel size of 1x1 is used to linearly transform the feature, improve the receptive field of the spatial correlation and local structure information of different lane line regions, and make it match the key vector f key The calculation dimension is obtained by calculating the self-attention mechanism query vector The expression is as follows:

[0079]

[0080] where Conv 1×1 represents a 1x1 convolution, and represents an inner product.

[0081] Then, the key and value vectors in the regional self-attention mechanism are calculated, and the specific expression is as follows:

[0082] f key = Conv 1×1 (f)

[0083] ​f val = Conv 1×1 (f)

[0084] The Q, K, and V vectors required by self-attention are obtained through the calculation of the above three formulas, that is, f f key , f val in the above formulas. Then the similarity between Q and K is calculated, and the attention score is obtained by calculating the dot product of the Q vector and the K vector, which reflects the relationship between different elements in the input sequence. The calculation process involves multiplying the transpose of the Q matrix and the K matrix to obtain a similarity matrix, which represents the relevance of the lane line region features compared to the global context information. On the basis of the similarity matrix, the attention weights are scaled by dividing by the scaling factor to avoid the problem of gradient vanishing or gradient explosion in the calculation process. The calculation result is put into the Softmax function for normalization to ensure the effectiveness and interpretability of the attention weights, so that they meet the requirements of probability distribution, so as to better explain the attention degree to different features. Finally, the attention weights are multiplied by f val to calculate the weighted feature values, highlighting the lane line feature information and suppressing irrelevant noise information. The formula is expressed as follows:

[0085]

[0086] The output of RSA is the picture information weighted by the attention matrix. Through this module, the correlation of different spatial positions in the image is mainly focused on. The input feature map is regarded as a spatial grid, and each position interacts with other positions. By calculating the similarity or correlation between each position and other positions, a weight matrix is obtained to weight and aggregate the features. In this way, the dependency relationship between different positions can be captured, so that the model can better understand the spatial structure and context information in the image, and improve the classification accuracy of the model in lane line scenes with multiple scales and occlusion relationships.

[0087] The backbone network selected is the Xception model. The Xception model performs further convolution operations on the weighted feature map output by RSA. The Xception model adopts the architecture of depth separable convolution, which splits the traditional standard convolution into two steps: depth convolution and point-wise convolution. This design enables the network to better learn the spatial and channel correlation of features, and has better feature representation capability. At the same time, compared with traditional convolutional neural networks, depth separable convolution can reduce the parameter quantity and computational complexity of the model, significantly reducing the parameter quantity of standard convolution, thereby improving the efficiency and training speed of the model, and being more suitable for lane line semantic segmentation applications with certain real-time requirements and complex segmentation scenarios.

[0088] The ASPP module is one of the key components in the DeepLabV3Plus model, which can capture context information at different scales and improve the performance and accuracy of semantic segmentation. The input of the module is the high-dimensional features in Xception, and the main purpose is to expand the receptive field so that the network can effectively perceive targets at different scales. This is achieved by applying different dilation rates to multiple atrous convolution layers. The dilation rate defines the sampling interval between pixels within the convolution kernel. A larger dilation rate can enrich the receptive field and capture more global context information, enabling the network to better understand the targets and backgrounds in the image and generate more accurate segmentation results. ASPP uses multiple atrous convolution kernels of different sizes (such as 1x1, 3x3, 5x5) in parallel to extract features from different scales, and further captures global context information through global average pooling.

[0089] The RSA unit mainly performs attention weighting operations on low-dimensional features. In low-dimensional feature extraction, there is rich spatial semantic information, and the original picture retains fewer channels and larger sizes, achieving spatial semantic information processing and reprocessing. However, in addition to spatial dimensions, there are also channel dimensions in the information contained in the picture. After multiple convolution layers of feature extraction, the number of channels can be expanded from the original three channels to hundreds or even thousands of channels. Processing channel information can enable the model to better identify features.

[0090] Channel information focuses on the semantic differences and correlations of features represented by different channels. Different channel representations provide different aspects of information in the image, such as color, texture, shape, etc. By analyzing and understanding the relevance between different channels, different semantic features expressed by different channels can be identified, thereby better understanding and distinguishing different objects or scenes in the image. The present application designs a channel self-attention module CSA to extract channel semantic information, and its internal structure is as shown in Figure 4

[0091] ​The input of the channel self-attention module is the high-dimensional feature map of the Xception output, that is, a more abstract feature expression obtained after multiple convolution of the neural network. The high-dimensional feature map usually contains multiple feature dimensions. Compared with calculating the channel self-attention weight in the low-dimensional, the result calculated on this basis has a larger receptive field and richer feature channels. Similar to the usual self-attention module, Q, K and V vectors need to be calculated from the original feature map. Considering the need to compress the information in the height H and width W dimensions, the application selects to use two one-dimensional convolutions for feature extraction. This dimension compression method can learn more specific and local feature representation, reduce the loss of semantic information in the dimension compression process, and is suitable for more complex image feature extraction tasks. At the same time, ReLU activation function is added between the two one-dimensional convolutions, which introduces non-linear transformation in linear transformation, so that the weight adjustment can learn more complex features. The formula is as follows:

[0092] f x =Conv1D2(ReLU(Conv1D1(f′))), x∈{q,k,v}

[0093] In the formula, f' is the high-dimensional feature map output by Xception. The feature map is compressed from the dimension of H by the first one-dimensional convolution Conv1D1 with a convolution kernel size of 1, then a nonlinear activation function ReLU is introduced, and finally the feature map is compressed in the W dimension by the outer one-dimensional convolution Conv1D2 with a convolution kernel size of 1. After two compressions and one nonlinear transformation, a 1xC(C is the number of channels) dimensional vector is obtained. The vector is the input of the self-attention Q, K and V. The final channel self-attention formula is as follows:

[0094]

[0095] i and j represent the rows and columns of the weight matrix respectively. The correlation on the channel is calculated by using f q and f k . The calculation result is a CxC weight matrix. Each row of the weight matrix represents the correlation between each element in f q and all elements in f k . The weight needs to be distributed to each channel to evaluate the importance of each channel. Therefore, the weight matrix is summed in the first dimension, so that the weight matrix is compressed to the same shape as f v . The weight is scaled by dividing by to avoid the problem of gradient disappearance or gradient explosion in the calculation process. After the normalization operation of Softmax, the probability distribution requirement of the weight is guaranteed. Then the channel feature f vThe channel self-attention module is used to weight the channel information, so as to improve the attention to important channel information, and finally the channel self-attention representation is reshaped from (1, 1, C) to (H, W, C) through a reshape function, and then is weighted to the feature map f' to realize the self-attention mechanism on the channel features.

[0096] After the channel self-attention module, a weighted high-dimensional feature map is obtained, and the feature map after the ASPP processing is a low-dimensional feature map. The feature fusion module in the decoder fuses the high-dimensional features output by the channel self-attention module and the low-dimensional features output by the ASPP module, so as to comprehensively utilize the feature information of different levels and better retain the detail information of the image, thereby improving the segmentation capability of small-size objects and boundaries.

[0097] After integrating all the extracted feature information, the multi-scale target reasoning module is used for semantic segmentation. The input image is scaled and segmented multiple times at different scales. After several upsampling, the results are fused to obtain the final segmentation result, and the semantic segmentation task of the lane line is completed.

[0098] The essence of semantic segmentation is to classify and predict each pixel in the picture. The output result of the network is essentially the classification probability distribution of each pixel. Therefore, in order to minimize the error between the predicted classification and the true classification, the backpropagation technique is used in the training of the ERFnet model and the improved DeepLabV3Plus model. This technique is based on the gradient descent algorithm. By propagating the error from the output layer to the input layer, the contribution of each parameter to the error is calculated, and the parameter value is updated according to the gradient information, so as to gradually optimize the performance of the network model. In the learning rate adjustment algorithm, the Adam algorithm is used. Compared with the traditional gradient descent algorithm, the Adam algorithm has better computational efficiency because it uses adaptive learning rate on each parameter without additional learning rate adjustment process. In addition, the Adam algorithm also combines the estimation of momentum and second moment, which can better handle non-stationary objective functions and sparse gradient cases, thereby improving the convergence performance of the model. The specific formula is as follows:

[0099]

[0100] In the above formula, the learning rate α, the decay exponents β1 and β2, and the smoothing term ε need to be initialized, t is the current iteration number, and θ is the parameter to be iteratively optimized. First, the forward propagation loss calculation function L and the gradient of the parameter θ are executed Then, the first-order moment variable m and the second-order moment variable v of θ are calculated according to the above formula. The first iteration m and v are set to zero vectors. Then, according to the deviation correction of the first-order moment and the second-order moment, the corrected first-order moment estimate m is calculated according to the above formula. *and second moment estimation v * Finally, the parameters θ are updated according to the above formula, and the back propagation parameter optimization is completed.

[0101] The selected loss calculation function is the cross-entropy loss function, which is used to measure the degree of mismatch between the predicted value of the classification model and the true label. In the lane line semantic segmentation task, the difference between the probability distribution predicted by the model and the true label value is measured, and the formula is expressed as follows:

[0102] L = -∑ i y i ·log(p i )

[0103] Where y i is a vector with a length of the number of categories, representing the one-hot encoding form of the true label, and p i is also a vector with a length of the number of categories, representing the predicted probability distribution of the model.

[0104] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A lane line semantic segmentation method based on self-attention, characterized in that, The method comprises: Step S1: data acquisition and preprocessing is performed, video stream collected by the vehicle-mounted camera is obtained, pictures are intercepted at fixed frame number intervals according to the obtained video stream, lane lines are labeled, and the original picture dataset is standardized; Step S2: an ERFnet neural network model is trained, and binary semantic segmentation of image data is realized based on the trained ERFnet neural network model, specifically: the image data processed in step S1 is divided into batches, the batch data is first sent to the convolutional layer of the ERFnet model for feature extraction, then the residual connection block splits the two-dimensional features extracted into two one-dimensional convolutions, which are respectively used for feature extraction in the horizontal direction and the vertical direction, then the residual connection operation fuses the processed feature map and the original input feature map to generate a group of feature maps, finally, the feature maps are further processed through the fully connected layer, and the probability distribution of each pixel point is mapped to the corresponding class label through the Softmax activation function to generate a binary semantic segmentation result; Step S3: based on the improved DeepLabV3Plus model, the final multi-target semantic segmentation is performed according to the binary semantic segmentation graph output in step S2, specifically including: Step S31: the improved DeepLabV3Plus model is trained; Step S32: the region self-attention unit of the trained model performs inner product on the original feature map and the lane line-background binary classification weight mask generated by the ERFnet model to obtain lane line region semantic features, and generates query, key and value vectors through 1x1 convolution for linear transformation of these features, and generates a region attention weight matrix by calculating the similarity of the query vector and the key vector; Step S33: the region attention weight matrix and the value vector are weighted and calculated to obtain more detailed lane line region features and input the Xception network of the model, and the Xception network uses deep convolution and point-by-point convolution to independently convolve the input feature map; Step S34: ASPP uses multiple different size hole convolution kernels in parallel to extract features of different scales from the feature map output by Xception, and further captures global context information through global average pooling; Step S35: the channel self-attention unit reduces the height and width dimensions of the features input by Xception through two one-dimensional convolution operations, and further optimizes through nonlinear mapping, then calculates the dot product of the compressed query vector and the key vector to generate a channel attention weight matrix, and the channel attention weight matrix is applied to the value vector to complete the weighting operation, and the weighted channel features and the original features are added element by element to form a new feature map; Step S36: the feature fusion module fuses the high-dimensional features output by the channel self-attention module and the low-dimensional features output by the ASPP module, and then performs multiple scaling and segmentation on the fused features at different scales through the multi-scale target reasoning module to obtain the final semantic segmentation result.

2. The method of claim 1, wherein, The improved DeepLabV3Plus model comprises an encoder module and a decoder module, the encoder module comprises a regional self-attention module RSA, an Xception module, a channel self-attention module CSA and an ASPP module, and the decoder module comprises a feature fusion module and a multi-scale target reasoning module.

3. The method of claim 1, wherein, The workflow of the RSA module is as follows: the input f of the RSA is the original feature map and the weight mask version with lane line semantics obtained through the ERFnet model The mask version is multiplied by the original feature map to obtain the lane line region semantic feature, and then the feature is linearly transformed through a convolution layer with a kernel size of 1x1, and is matched with the key vector f key The dimension is calculated to obtain the self-attention mechanism query vector The expression is: where Conv 1×1 denotes a 1 x 1 convolution, and denotes an inner product; Then, the key and value vectors in the regional self-attention mechanism are calculated: f key = Conv 1×1 (f) f val = Conv 1×1 (f) From which the Q, K, V vectors required for self-attention are obtained, i.e. f key , f val ; Then multiply the Q matrix and the transpose of the K matrix to get a similarity matrix, divide the similarity matrix by a scaling factor The weights are scaled, the results are put into the Softmax function for normalization, and finally the attention weights and f val are multiplied to calculate the weighted feature values, as shown in the following expression:

4. The method of claim 2, wherein, The Xception model further convolves the weighted feature map output by the RSA by using a deep convolution and a point-wise convolution.

5. The method of claim 2, wherein, The working process of the CSA module is as follows: the input of the CSA module is the high-dimensional feature map output by the Xception, two one-dimensional convolutions are used for feature extraction, and a ReLU activation function is added between the two one-dimensional convolutions, and the expression is as follows: Wherein, f' is the high-dimensional feature map output by the Xception, which is compressed by two one-dimensional convolutions and nonlinearly transformed once to obtain a 1xC vector, and C is the number of channels; Then the f q and f k correlation on the channel is calculated, and the result is a CxC weight matrix. The weight matrix is summed in the first dimension and divided by The weights are scaled and normalized by Softmax. Then the channel feature f v is weighted by dot product. Finally, the channel self-attention representation is reshaped from (1,1,C) to (H,W,C) by reshape, and then weighted to the feature map f’, as follows:

6. The method of claim 2, wherein, The feature fusion module in the decoder module fuses the high-dimensional feature output by the CSA module with the low-dimensional feature output by the ASPP module, and then the multi-scale target reasoning module scales and segments the fused feature at different scales to obtain the final semantic segmentation result.

Images