Road semantic segmentation method based on lightweight multi-scale fusion Transform network

Abstract

The invention relates to the technical field of artificial intelligence and automatic driving, in particular to a road semantic segmentation method based on a lightweight multi-scale fusion Transform network, which comprises a lightweight encoder network module, a lightweight multi-scale fusion Transform network module, a lightweight decoder network module and a space attention network module. The light-weight encoder network module mixes depth separable convolution and asymmetric dilated convolution, not only effectively expands a receptive field to capture rich context information in a road scene, but also significantly reduces calculation complexity, and the light-weight multi-scale fusion Transform network module reconstructs feature maps extracted from multiple scales, so that the reconstruction efficiency is improved. The module uses a separable cross-channel attention mechanism to effectively capture internal dependence and association between feature maps of different scales. The road scene semantic segmentation method designed by the invention is high in accuracy, low in complexity, good in real-time performance and suitable for vehicle-mounted equipment, realizes good balance between segmentation precision and calculation efficiency, and has important application value in the field of automatic driving.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and autonomous driving technology, and in particular to a road semantic segmentation method based on a lightweight multi-scale fusion Transformer network. Background Art

[0002] In the context of the rapid development of autonomous driving technology, road scene semantic segmentation, as one of the core technologies of the perception layer of the autonomous driving system, is gradually becoming a research hotspot. With the continuous breakthroughs and integration of advanced perception technologies, autonomous vehicles need to understand the complex and changing road environment more accurately to ensure safe and efficient driving. Road scene semantic segmentation technology assigns specific semantic category labels to each pixel in the image, thereby achieving fine recognition and distinction of key elements such as roads, vehicles, pedestrians, and traffic signs. This technology not only provides an accurate environmental model for the path planning of autonomous driving vehicles, but also significantly improves the vehicle's perception and response capabilities to potential dangers. Therefore, the road scene semantic segmentation algorithm has important research significance and application value for promoting the further development of autonomous driving technology and improving road traffic safety.

[0003] Compared with general scenes, the semantic segmentation task of road scenes in autonomous driving applications is particularly challenging. Specifically, it requires not only real-time image processing capabilities, but also the ability to represent low-level and high-level features that can adapt to the complex changes of different traffic elements.

[0004] The applicant found that the existing road scene semantic segmentation algorithm has the following main difficulties:

[0005] (1) The road scene semantic segmentation algorithm needs to meet the requirements of real-time or efficient processing while ensuring high segmentation accuracy. However, this pursuit of high accuracy is often accompanied by an increase in computational complexity, which in turn affects the processing speed. Therefore, how to find the best balance between accuracy and efficiency remains a bottleneck problem affecting the road scene semantic segmentation task.

[0006] (2) Road scenes are particularly challenging due to their complexity, where the size, shape, texture and other information of traffic elements show different characteristics at different scales. This requires that the applied model must be able to capture both low-level and high-level visual features. However, current algorithms still have significant deficiencies in deeply mining and effectively integrating these multi-scale feature information, which to some extent limits their application effect and performance in complex road scenes. Summary of the invention

[0007] In view of this, the purpose of the present invention is to propose a road semantic segmentation method based on a lightweight multi-scale fusion Transformer network, so as to solve the problem that the existing road scene semantic segmentation algorithms do not have the ability to capture both low-level and high-level visual features at the same time, resulting in deficiencies in in-depth mining and effective integration of these multi-scale feature information, which to some extent limits their application effects and performance in complex road scenes.

[0008] Based on the above purpose, the present invention provides a road semantic segmentation method based on a lightweight multi-scale fusion Transformer network, comprising the following steps:

[0009] Step S1, processing the road scene semantic segmentation dataset;

[0010] Step S2, designing a lightweight encoder network module;

[0011] Step S2 includes the following steps:

[0012] Step S2.1, using a lightweight hybrid convolution block in a lightweight encoder network module to extract road scene features;

[0013] Step S2.2: Given an input image I∈R W×H×3 , using 3×3 standard convolution Conv 3×3 (·|) to extract shallow features E0∈W×H×C0;

[0014] Step S2.3, lightweight hybrid convolutional block LMCB (·|) learns multi-scale feature maps The specific body can be expressed as;

[0015]

[0016] Where MaxPool(·|) is the maximum pooling operation, E i represents the output feature map of the i-th layer encoder, Indicates the relevant parameters of the encoder network module;

[0017] Step S3, designing a lightweight multi-scale fusion Transformer network module, wherein the lightweight multi-scale fusion Transformer network module is intended to capture the intrinsic dependencies and associations between feature maps of different scales;

[0018] Step S3 includes the following steps:

[0019] Step S3.1, the lightweight multi-scale fusion Transformer network module converts the multi-scale feature map output by the encoder into a visual sequence, and reconstructs the multi-scale feature map using the separable cross-channel attention mechanism, which can be specifically expressed as:

[0020] E1′,E2′,E3′,E4′=EMFFTrans(E1,E2,E3,E4θ tr )

[0021] Where EMFFTrans(·|) represents the lightweight multi-scale fusion Transformer network module, and the relevant parameters are θ tr , E j ′ represents the reconstructed feature map of the jth layer;

[0022] Step S4, designing a lightweight decoder network module;

[0023] Step S4 includes the following steps:

[0024] Step S4.1: The lightweight decoder network aggregates the decoded feature map through the lightweight hybrid convolution block LMCB(·|) Reconstructed feature map output by lightweight multi-scale fusion Transformer network module And perform step-by-step upsampling to restore pixel details, thereby achieving a comprehensive and detailed overall representation of the road scene, which can be specifically expressed as:

[0025]

[0026] Where D j represents the j-th layer decoding feature map, Represents the feature map concatenation operation;

[0027] Step S5: design a spatial attention network module;

[0028] Step S5 includes the following steps:

[0029] Step S5.1, the spatial attention network module SA(·|) assigns weights to different spatial pixels of the output feature map D0 of the lightweight decoder network module, and calculates the refined feature map Specifically, it can be expressed as:

[0030] F=SA(D0|θ sa )=σ(Conv 1×1 (D0|θ sa ))⊙D0

[0031] where ⊙ represents the element-wise product operation, θ sa are the parameters of the spatial attention network module;

[0032] Step S6: Model training and testing: Using NVIDIA TITAN X GPU and Ubuntu 18.04 operating system, we built a road semantic segmentation model based on a lightweight multi-scale fusion Transformer network through the Pytorch open source library, and trained and tested the model.

[0033] Preferably, step S2 further includes the following sub-steps:

[0034] Step S2.4, the lightweight hybrid convolution block adopts a residual structure, first reducing the number of channels through a 1×1 convolution layer, then using a lightweight hybrid convolution operation for semantic feature extraction, and finally re-adjusting the number of channels through another 1×1 convolution layer;

[0035] Step S2.5, the lightweight hybrid convolution block integrates the channel attention mechanism into the skip connection, thereby weighting the features of different channels. Finally, the channel rearrangement operation is applied to the end of the lightweight hybrid convolution block to enhance the generalization ability of the network.

[0036] Step S2.6, assuming that x is the input feature map, then after the lightweight mixed convolution operation, the output feature map y can be expressed as:

[0037] y=CS(CA(x)+Conv 1×1 (LMConv(Conv 1×1 (x)))

[0038] Where CS(·) and CA(·) represent channel reordering and channel attention operations respectively; Conv 1×1 (·) represents a 1×1 convolution; LMConv(·) represents a lightweight mixed convolution operation.

[0039] Preferably, in step S2.4, the lightweight hybrid convolution operation connects the 3×3 depthwise separable convolution with the 3×1 and 1×3 depthwise separable asymmetric dilated convolutions in parallel, and then uses 1×1 pointwise convolution for feature aggregation;

[0040] And the depth-wise separable convolution can reduce the computational cost by applying a K×K depth-wise convolution filter to each input channel, and then use a 1×1 point-by-point convolution kernel for cross-channel combination, thereby achieving lightweight feature extraction;

[0041] Asymmetric dilated convolution uses convolution kernels with a dilation rate of r that are decomposed into 1×K and K×1 shapes to expand the receptive field, thereby enhancing the global feature extraction capability without increasing complexity.

[0042] Preferably, step S3 also includes the following steps:

[0043] Step S3.2, image block embedding, using convolutional projection to embed image blocks, which not only reduces the computational cost but also can introduce spatial induction compilation;

[0044] Step S3.3, attention representation learning, the separable cross-channel attention mechanism can effectively learn the feature dependencies between embedding vectors in multiple sequences;

[0045] Step S3.4: feature map reconstruction.

[0046] Preferably, step S3.2 also includes the following sub-steps:

[0047] Step S3.21, firstly, the encoder feature map Divided into L×L two-dimensional pixel blocks, the block size is pw i ×ph i ;

[0048] Step S3.22, then use 1×1 convolution to reduce the number of image channels, and apply size pw i ×ph i , step length is pw i ×ph i The convolution kernel is used to filter the feature map and the number of channels is resized through another 1×1 convolution layer;

[0049] Step S3.23: Finally, flatten the feature map into L×L one-dimensional visual vectors as an embedding sequence It can be expressed by the following formula:

[0050]

[0051]

[0052] Among them, E i Represents the encoder multi-scale feature map, Conv 1×1 (·)and Represents the size of 1×1 and pw respectively i ×ph i Convolution kernel, Unfold(·) represents the feature map flattening operation, T i is the i-th embedding sequence, is a vector concatenation operation, and Represents the concatenated embedded feature sequence.

[0053] Preferably, step S3.3 also includes the following sub-steps:

[0054] Step S3.31: Generate query components separately through linear mapping Key Component With value component Specifically, it can be expressed as:

[0055] Q=σ(T ∑ W Q )

[0056]

[0057] Among them, σ(·) represents the sigmoid activation function, are the weights of the linear mapping respectively;

[0058] Step S3.32: Based on the query component Q and the key component K i With value component V i Apply a separable cross-channel attention mechanism in multiple embedding sequences to learn the feature dependencies of channels in different embedding sequences. The operation is specifically defined as:

[0059]

[0060] Among them, ⊙ and ∑ represent element-wise multiplication and element-wise summation operations respectively, ReLU(·) is the ReLU activation function, is the weight of the output linear mapping, is the output sequence.

[0061] Preferably, in step S3.4, the following sub-steps are also included:

[0062] Step S3.41: First, output the sequence Fold into a two-dimensional feature map;

[0063] Step S3.42: Next, upsample the folded two-dimensional feature map to match the encoder feature map E i The spatial dimension of

[0064] Step S3.43, finally, the upsampled feature map is concatenated with the encoder feature map by feature channel, and a 1×1 convolutional layer is applied for dimension transformation, which can be expressed as:

[0065]

[0066] Among them, fold(·), upsample(·), Conv 1×1 (·) represent feature folding, upsampling, feature map channel concatenation, and 1×1 convolution operations, respectively.

[0067] Preferably, in step S6, the following sub-steps are also included:

[0068] Step S6.1: Output the semantic segmentation result. The semantic segmentation result is obtained through a mapping function, that is, the feature map after the spatial attention network module is refined As input, it passes through a 1×1 convolutional layer and is followed by a softmax activation function to calculate the probability of each pixel in the image belonging to a certain category, which can be expressed as:

[0069] S=Seg(F|θ seg )=Softmax(Conv 1×1 (F|θ seg ))

[0070] Among them, Conv 1×1 (·) represents 1×1 convolution, Softmax(·|) is the softmax function, θ seg represents the relevant parameters involved in semantic segmentation, and S is the semantic segmentation result;

[0071] Step S6.2, loss function design. The road scene semantic segmentation task has the problem of unbalanced distribution of categories, in which the background category accounts for the majority and the foreground category accounts for the minority. In order to balance the loss weights of samples of different categories, the Focal loss and Dice loss are combined to optimize the model, which can be specifically expressed as:

[0072] L=λL focal +L dice

[0073]

[0074] Where N represents the total number of pixels in the image; M is the total number of label categories in the segmentation task; p n (m) represents the probability that the nth pixel is predicted to be of category m; g n (m) is an indicator function used to indicate whether the actual category is the real category m; TP(m), FN(m), and FP(m) represent the true positive rate, false negative rate, and false positive rate for category m, respectively, with weight factors α and β; λ is a hyperparameter used to balance the weights of focal loss and Dice loss; α, β, and λ are set to 1, 1, and 0.2, respectively;

[0075] Step S6.3: Use the trained model to test the performance of the road scene semantic segmentation model.

[0076] Beneficial effects of the present invention: The lightweight hybrid convolutional network module designed by the present invention adopts a deeply separable asymmetric dilated convolution, which not only effectively expands the receptive field to capture rich contextual information in road scenes, but also significantly reduces the computational complexity, and can achieve efficient and accurate semantic feature extraction of road scenes.

[0077] The lightweight multi-scale fusion Transformer network module reconstructs feature maps extracted at multiple scales. The module uses a separable cross-channel attention mechanism to effectively capture the intrinsic dependencies and associations between feature maps of different scales.

[0078] The lightweight decoder network module aggregates the decoded feature map and the reconstructed feature map output by the lightweight multi-scale fusion Transformer network module, and performs step-by-step upsampling to restore pixel details, thereby achieving a comprehensive and detailed overall representation of the road scene.

[0079] The spatial attention module refines key pixel features by weight allocation, which can more effectively focus on key information areas in road scenes.

[0080] In summary, the road scene semantic segmentation method based on the lightweight multi-scale fusion Transformer network designed in the present invention has high accuracy, low complexity, good real-time performance, is suitable for vehicle-mounted equipment, and achieves a good balance between segmentation accuracy and computational efficiency. The present invention has important application value in the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 is a flowchart of an embodiment of the present invention;

[0083] Figure 2 Schematic diagram of a lightweight hybrid convolution block according to an embodiment of the present invention;

[0084] Figure 3 Schematic diagram comparing standard convolution and different lightweight convolution operations in an embodiment of the present invention;

[0085] Figure 4 It is a lightweight multi-scale fusion Transformer network module of an embodiment of the present invention;

[0086] Figure 5 Schematic diagram of the overall framework of road semantic segmentation according to an embodiment of the present invention;

[0087] Figure 6 Schematic diagram of road semantic segmentation results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0089] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0090] like Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 As shown in FIG, the road semantic segmentation method based on the lightweight multi-scale fusion Transformer network includes the following steps:

[0091] Step S1, processing the road scene semantic segmentation dataset; using the CamVid dataset (https: / / mi.eng.cam.ac.uk / research / projects / VideoRec / CamVid / ) publicly released by the University of Cambridge, which focuses on the semantic segmentation task of urban road scenes and contains 701 image samples, including 367 training images, 101 verification images, and 233 test images. These images are all from real road scenes captured by autonomous vehicles while driving on the road. The resolution of each image is 720×960 pixels, ensuring the high definition of the image. At the same time, each pixel in each image is classified into one of 11 predefined categories, which cover key elements in road scenes, such as roads, cars, pedestrians, buildings, trees, etc., providing rich label information for semantic segmentation tasks. The present invention uses the following data augmentation techniques to increase the number of samples in the CamVid training set: scale scaling and random cropping, image scale scaling in the range of [0.5,2], and the cropping size of the image is 360×480.

[0092] Step S2, designing a lightweight encoder network module;

[0093] Step S2 includes the following steps:

[0094] Step S2.1: Figure 2 As shown, a lightweight hybrid convolution block is used in the lightweight encoder network module to extract road scene features;

[0095] Step S2.2: Given an input image I∈R W×H×3 , using 3×3 standard convolution Conv 3×3 (·|) to extract shallow features E0∈W×H×C0;

[0096] Step S2.3, lightweight hybrid convolutional block LMCB (·|) learns multi-scale feature maps The specific body can be expressed as;

[0097]

[0098] Where MaxPool(·|) is the maximum pooling operation, E i Represents the output feature map of the i-th layer encoder, that is, E i It refers to the multi-scale feature map of learning. The number of layers of the encoder network is i, so the scales of the feature maps output (learned) by different layers are different. Indicates the relevant parameters of the encoder network module;

[0099] Step S2.4, the lightweight hybrid convolution block adopts a residual structure, first reducing the number of channels through a 1×1 convolution layer, then using a lightweight hybrid convolution operation for semantic feature extraction, and finally re-adjusting the number of channels through another 1×1 convolution layer;

[0100] Step S2.5, the lightweight hybrid convolution block integrates the channel attention mechanism into the skip connection, thereby weighting the features of different channels. Finally, the channel rearrangement operation is applied to the end of the lightweight hybrid convolution block to enhance the generalization ability of the network.

[0101] Step S2.6, assuming that x is the input feature map, then after the lightweight mixed convolution operation, the output feature map y can be expressed as:

[0102] y=CS(CA(x)+Conv 1×1 (LMConv(Conv 1×1 (x)))

[0103] Where CS(·) and CA(·) represent channel reordering and channel attention operations respectively; Conv 1×1(·) represents 1×1 convolution; LMConv(·) represents a lightweight mixed convolution operation; this calculation formula is not for the encoder, but for the lightweight mixed convolution operation, which is a smaller concept.

[0104] In step S2.4, the lightweight hybrid convolution operation connects the 3×3 depthwise separable convolution with the 3×1 and 1×3 depthwise separable asymmetric dilated convolutions in parallel, and then uses 1×1 pointwise convolution for feature aggregation;

[0105] like Figure 3 As shown in the figure, the depth-wise separable convolution can reduce the computational cost by applying a K×K depth-wise convolution filter to each input channel, and then combining them across channels using a 1×1 point-wise convolution kernel, thereby achieving lightweight feature extraction.

[0106] Asymmetric dilated convolution uses convolution kernels with a dilation rate of r that are decomposed into 1×K and K×1 shapes to expand the receptive field, thereby enhancing the global feature extraction capability without increasing complexity. Depthwise separable dilated asymmetric convolution combines the high efficiency of depthwise separable convolution with the receptive field expansion advantage of asymmetric dilated convolution. First, asymmetric 1×K and K×1 convolution kernels with a dilation rate of r are applied to each input channel, and then channels are combined through 1×1 point-by-point convolution. This convolution method not only maintains the computational efficiency of depthwise separable convolution, but also expands the receptive field through asymmetric dilated convolution, which helps to capture richer global features.

[0107] Step S3: Design a lightweight multi-scale fusion Transformer network module, such as Figure 4 As shown in Figure 2, the lightweight multi-scale fusion Transformer network module is designed to capture the intrinsic dependencies and associations between feature maps of different scales;

[0108] Step S3 includes the following steps:

[0109] Step S3.1, the lightweight multi-scale fusion Transformer network module converts the multi-scale feature map output by the encoder into a visual sequence, and reconstructs the multi-scale feature map using the separable cross-channel attention mechanism, which can be specifically expressed as:

[0110] E1′,E2′,E3′,E4′=EMFFTrans(E1,E2,E3,E4θ tr )

[0111] Where EMFFTrans(·|) represents the lightweight multi-scale fusion Transformer network module, and the relevant parameters are θ tr , E j ′ represents the reconstructed feature map of the jth layer;

[0112] Step S3.2, image block embedding, using convolutional projection to embed image blocks, which not only reduces the computational cost but also can introduce spatial induction compilation;

[0113] Step S3.21, first, the encoder feature map Divided into L×L two-dimensional pixel blocks, the block size is pw i ×ph i ;

[0114] Step S3.22, then use 1×1 convolution to reduce the number of image channels, and apply size pw i ×ph i , step length is pw i ×ph i The convolution kernel is used to filter the feature map and the number of channels is resized through another 1×1 convolution layer;

[0115] Step S3.23: Finally, flatten the feature map into L×L one-dimensional visual vectors as an embedding sequence It can be expressed by the following formula:

[0116]

[0117]

[0118] Among them, E i Represents the encoder multi-scale feature map. The encoder multi-scale feature map refers to the different feature maps output by encoders at different layers, and these feature maps are multi-scale. Conv 1×1 (·)and Represents the size of 1×1 and pw respectively i ×ph i Convolution kernel, Unfold(·) represents the feature map flattening operation, T i is the i-th embedding sequence, is a vector concatenation operation, and Represents the concatenated embedded feature sequence.

[0119] Step S3.3, attention representation learning, the separable cross-channel attention mechanism can effectively learn the feature dependencies between embedding vectors in multiple sequences;

[0120] Step S3.31: Generate query components separately through linear mapping Key Component With value component Specifically, it can be expressed as:

[0121] Q=σ(T ∑ W Q )

[0122]

[0123] Among them, σ(·) represents the sigmoid activation function, are the weights of the linear mapping respectively;

[0124] Step S3.32: Based on the query component Q and the key component K i With value component V i Apply a separable cross-channel attention mechanism in multiple embedding sequences to learn the feature dependencies of channels in different embedding sequences. The operation is specifically defined as:

[0125]

[0126] Among them, ⊙ and ∑ represent element-wise multiplication and element-wise summation operations respectively, ReLU(·) is the ReLU activation function, is the weight of the output linear mapping, is the output sequence.

[0127] Step S3.4: feature map reconstruction.

[0128] Step S3.41: First, output the sequence Fold into a two-dimensional feature map;

[0129] Step S3.42: Next, upsample the folded two-dimensional feature map to match the encoder feature map E i The spatial dimension of

[0130] Step S3.43, finally, the upsampled feature map is concatenated with the encoder feature map by feature channel, and a 1×1 convolutional layer is applied for dimension transformation, which can be expressed as:

[0131]

[0132] Among them, fold(·), upsample(·), Conv 1×1 (·) represent feature folding, upsampling, feature map channel concatenation, and 1×1 convolution operations, respectively.

[0133] Step S4, designing a lightweight decoder network module;

[0134] Step S4 includes the following steps:

[0135] Step S4.1: The lightweight decoder network aggregates the decoded feature map through the lightweight hybrid convolution block LMCB(·|) Reconstructed feature map output by lightweight multi-scale fusion Transformer network module And perform step-by-step upsampling to restore pixel details, thereby achieving a comprehensive and detailed overall representation of the road scene, which can be specifically expressed as:

[0136]

[0137] Where D j represents the j-th layer decoding feature map, Represents the feature map concatenation operation;

[0138] Step S5: design a spatial attention network module;

[0139] Step S5 includes the following steps:

[0140] Step S5.1, the spatial attention network module SA(·|) assigns weights to different spatial pixels of the output feature map D0 of the lightweight decoder network module, and calculates the refined feature map Specifically, it can be expressed as:

[0141] F=SA(D0|θ sa )=σ(Conv 1×1 (D0|θ sa ))⊙D0

[0142] where ⊙ represents the element-wise product operation, θ sa are the parameters of the spatial attention network module;

[0143] Step S6: Model training and testing. Using NVIDIA TITANX GPU and Ubuntu 18.04 operating system, we built a road semantic segmentation model based on a lightweight multi-scale fusion Transformer network through the Pytorch open source library, and trained and tested the model. The overall architecture of the model is shown in the figure below. Figure 5 shown.

[0144] Step S6.1: Output the semantic segmentation result. The semantic segmentation result is obtained through a mapping function, that is, the feature map after the spatial attention network module is refined As input, it passes through a 1×1 convolutional layer and is followed by a softmax activation function to calculate the probability of each pixel in the image belonging to a certain category, which can be expressed as:

[0145] S=Seg(F|θ seg )=Softmax(Conv 1×1 (F|θ seg ))

[0146] Among them, Conv 1×1 (·) represents 1×1 convolution, Softmax(·|) is the softmax function, θ segrepresents the relevant parameters involved in semantic segmentation, and S is the semantic segmentation result;

[0147] Step S6.2, loss function design. The road scene semantic segmentation task has the problem of unbalanced distribution of categories, in which background categories (including roads or buildings) account for the majority, while foreground categories (such as pedestrians or bicycles) account for the minority. In order to balance the loss weights of samples of different categories, the model is optimized by combining Focal loss and Dice loss, which can be specifically expressed as:

[0148] L=λL focal +L dice

[0149]

[0150] Where N represents the total number of pixels in the image; M is the total number of label categories in the segmentation task; p n (m) represents the probability that the nth pixel is predicted to be of category m; g n (m) is an indicator function used to indicate whether the actual category is the real category m; TP(m), FN(m), and FP(m) represent the true positive rate, false negative rate, and false positive rate for category m, respectively, with weight factors α and β; λ is a hyperparameter used to balance the weights of focal loss and Dice loss; α, β, and λ are set to 1, 1, and 0.2, respectively;

[0151] Step S6.3: Use the trained model to test the performance of the road scene semantic segmentation model. Given a test image, resize the image to 720×960 and feed it into the trained network model. Calculate the road scene semantic segmentation result through the network forward propagation and evaluate the performance of the road scene semantic segmentation model. The image segmentation result diagram is shown in the figure. Figure 5 shown.

[0152] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0153] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A road semantic segmentation method based on a lightweight multi-scale fusion Transformer network, characterized by: The following steps are involved: Step S1, processing the road scene semantic segmentation dataset; Step S2, designing a lightweight encoder network module; Step S2 includes the following steps: Step S2.1, using a lightweight hybrid convolution block in a lightweight encoder network module to extract road scene features; Step S2.2: Given an input image I∈R W×H×3 , using 3×3 standard convolution Conv 3×3 (·|) to extract shallow features E0∈W×H×C0; Step S2.3, lightweight hybrid convolutional block LMCB (·|) learns multi-scale feature maps The specific body can be expressed as; Where MaxPool(·|) is the maximum pooling operation, E i represents the output feature map of the i-th layer encoder, Indicates the relevant parameters of the encoder network module; Step S3, designing a lightweight multi-scale fusion Transformer network module, wherein the lightweight multi-scale fusion Transformer network module is intended to capture the intrinsic dependencies and associations between feature maps of different scales; Step S3 includes the following steps: Step S3.1, the lightweight multi-scale fusion Transformer network module converts the multi-scale feature map output by the encoder into a visual sequence, and reconstructs the multi-scale feature map using the separable cross-channel attention mechanism, which can be specifically expressed as: E1′,E2′,E3′,E4′=EMFFTrans(E1,E2,E3,E4θ tr ) Where EMFFTrans(·|) represents the lightweight multi-scale fusion Transformer network module, and the relevant parameters are θ tr , E j ′ represents the reconstructed feature map of the jth layer; Step S4, designing a lightweight decoder network module; Step S4 includes the following steps: Step S4.1: The lightweight decoder network aggregates the decoded feature map through the lightweight hybrid convolution block LMCB(·|) Reconstructed feature map output by lightweight multi-scale fusion Transformer network module And perform step-by-step upsampling to restore pixel details, thereby achieving a comprehensive and detailed overall representation of the road scene, which can be specifically expressed as: Where D j represents the j-th layer decoding feature map, Represents the feature map concatenation operation; Step S5: design a spatial attention network module; Step S5 includes the following steps: Step S5.1, the spatial attention network module SA(·|) assigns weights to different spatial pixels of the output feature map D0 of the lightweight decoder network module, and calculates the refined feature map Specifically, it can be expressed as: F=SA(D0|θ sa )=σ(Conv 1×1 (D0|θ sa ))⊙D0 where ⊙ represents the element-wise multiplication operation, θ sa are the relevant parameters of the spatial attention network module; Step S6: Model training and testing: Using NVIDIA TITAN X GPU and Ubuntu 18.04 operating system, we built a road semantic segmentation model based on a lightweight multi-scale fusion Transformer network through the Pytorch open source library, and trained and tested the model.

2. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 1 is characterized in that: Step S2 also includes the following sub-steps: Step S2.4, the lightweight hybrid convolution block adopts a residual structure, first reducing the number of channels through a 1×1 convolution layer, then using a lightweight hybrid convolution operation for semantic feature extraction, and finally re-adjusting the number of channels through another 1×1 convolution layer; Step S2.5, the lightweight hybrid convolution block integrates the channel attention mechanism into the skip connection, thereby weighting the features of different channels. Finally, the channel rearrangement operation is applied to the end of the lightweight hybrid convolution block to enhance the generalization ability of the network. Step S2.6, assuming that x is the input feature map, then after the lightweight mixed convolution operation, the output feature map y can be expressed as: y=CS(CA(x)+Conv 1×1 (LMConv(Conv 1×1 (x))) Where CS(·) and CA(·) represent channel reordering and channel attention operations respectively; Conv 1×1 (·) represents a 1×1 convolution; LMConv(·) represents a lightweight mixed convolution operation.

3. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 2 is characterized in that: In step S2.4, the lightweight hybrid convolution operation connects the 3×3 depthwise separable convolution with the 3×1 and 1×3 depthwise separable asymmetric dilated convolutions in parallel, and then uses 1×1 pointwise convolution for feature aggregation; And the depth-wise separable convolution can reduce the computational cost by applying a K×K depth-wise convolution filter to each input channel, and then use a 1×1 point-by-point convolution kernel for cross-channel combination, thereby achieving lightweight feature extraction; Asymmetric dilated convolution uses convolution kernels with a dilation rate of r that are decomposed into 1×K and K×1 shapes to expand the receptive field, thereby enhancing the global feature extraction capability without increasing complexity.

4. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 3 is characterized in that: Step S3 also includes the following steps: Step S3.2, image block embedding, using convolutional projection to embed image blocks, which not only reduces the computational cost but also can introduce spatial induction compilation; Step S3.3, attention representation learning, the separable cross-channel attention mechanism can effectively learn the feature dependencies between embedding vectors in multiple sequences; Step S3.4: feature map reconstruction.

5. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 4 is characterized in that: Step S3.2 also includes the following sub-steps: Step S3.21, firstly, the encoder feature map Divided into L×L two-dimensional pixel blocks, the block size is pw i ×ph i ; Step S3.22, then use 1×1 convolution to reduce the number of image channels, and apply size pw i ×ph i , step length is pw i ×ph i The convolution kernel is used to filter the feature map and the number of channels is resized through another 1×1 convolution layer; Step S3.23: Finally, flatten the feature map into L×L one-dimensional visual vectors as an embedding sequence It can be expressed by the following formula: Among them, E i Represents the encoder multi-scale feature map, Conv 1×1 (·)and Represents the size of 1×1 and pw respectively i ×ph i Convolution kernel, Unfold(·) represents the feature map flattening operation, T i is the i-th embedding sequence, is a vector concatenation operation, and Represents the concatenated embedded feature sequence.

6. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 5 is characterized in that: Step S3.3 also includes the following sub-steps: Step S3.31: Generate query components separately through linear mapping Key Component With value component Specifically, it can be expressed as: Among them, σ(·) represents the sigmoid activation function, are the weights of the linear mapping respectively; Step S3.32: Based on the query component Q and the key component K i With value component V i Apply a separable cross-channel attention mechanism in multiple embedding sequences to learn the feature dependencies of channels in different embedding sequences. The operation is specifically defined as: Among them, ⊙ and ∑ represent element-wise multiplication and element-wise summation operations respectively, ReLU(·) is the ReLU activation function, is the weight of the output linear mapping, is the output sequence.

7. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 6 is characterized in that: In step S3.4, the following sub-steps are also included: Step S3.41: First, output the sequence Fold into a two-dimensional feature map; Step S3.42: Next, upsample the folded two-dimensional feature map to match the encoder feature map E i The spatial dimension of Step S3.43, finally, the upsampled feature map is concatenated with the encoder feature map by feature channel, and a 1×1 convolutional layer is applied for dimension transformation, which can be expressed as: Among them, fold(·), upsample(·), Conv 1×1 (·) represent feature folding, upsampling, feature map channel concatenation, and 1×1 convolution operations, respectively.

8. The road semantic segmentation method based on lightweight multi-scale fusion Transformer network according to claim 7 is characterized in that: In step S6, the following sub-steps are also included: Step S6.1: Output the semantic segmentation result. The semantic segmentation result is obtained through a mapping function, that is, the feature map after the spatial attention network module is refined As input, it passes through a 1×1 convolutional layer and is followed by a softmax activation function to calculate the probability of each pixel in the image belonging to a certain category, which can be expressed as: S=Seg(F|θ seg )=Softmax(Conv 1×1 (F|θ seg )) Among them, Conv 1×1 (·) represents 1×1 convolution, Softmax(·|) is the softmax function, θ seg represents the relevant parameters involved in semantic segmentation, and S is the semantic segmentation result; Step S6.2, loss function design. The road scene semantic segmentation task has the problem of unbalanced distribution of categories, in which the background category accounts for the majority and the foreground category accounts for the minority. In order to balance the loss weights of samples of different categories, the Focal loss and Dice loss are combined to optimize the model, which can be specifically expressed as: L=λL focal +L dice Where N represents the total number of pixels in the image; M is the total number of label categories in the segmentation task; p n (m) represents the probability that the nth pixel is predicted to be of category m; g n (m) is an indicator function used to indicate whether the actual category is the real category m; TP(m), FN(m), and FP(m) represent the true positive rate, false negative rate, and false positive rate for category m, respectively, with weight factors α and β; λ is a hyperparameter used to balance the weights of focal loss and Dice loss; α, β, and λ are set to 1, 1, and 0.2, respectively; Step S6.3: Use the trained model to test the performance of the road scene semantic segmentation model.

Images