Transform improvement-based real-time semantic segmentation method

Abstract

The invention relates to a real-time semantic segmentation method based on Transform improvement, and the method comprises the following steps: carrying out the preprocessing of an open-source automatic driving scene data set, distributing a class label for each pixel of each image, and dividing the image into a training set and a test set; constructing a real-time semantic segmentation model, wherein the real-time semantic segmentation model comprises a shallow convolutional layer, a semantic feature enhancement branch, a fusion branch, an edge enhancement branch, two semantic segmentation heads and an edge segmentation head; the real-time semantic segmentation model is trained; and performing semantic segmentation by using the trained real-time semantic segmentation model. According to the method, the problems of insufficient multi-branch feature interaction and limited salient region attention capability and multi-scale feature modeling in the existing real-time semantic segmentation method can be solved.

Description

Technical Field

[0001] This application relates to the field of real-time semantic segmentation technology, specifically to a real-time semantic segmentation method based on an improved Transformer. Background Technology

[0002] In autonomous driving scenarios, semantic segmentation enables accurate identification of lane lines, obstacles, traffic signs, and free space, supporting vehicle path planning and decision-making. In recent years, with the increasing availability of high-resolution semantic segmentation datasets, the computational complexity of network models has also increased. High-precision semantic segmentation networks, such as DeepLabv3+ and PSPNet, while exhibiting excellent segmentation accuracy, are difficult to deploy on low-power devices or in real-time inference scenarios. Therefore, real-time semantic segmentation networks have become a research hotspot, aiming to significantly improve inference speed while maintaining high accuracy.

[0003] Patent CN116612288B discloses a multi-scale lightweight real-time semantic segmentation method and system. It extracts shallow, mid-level, and deep features through a lightweight encoder and enhances them using an attention module. Simultaneously, it combines an object context feature fusion module to enrich semantic information, thereby achieving high segmentation accuracy on low-computing-power devices. Patent CN120823388A discloses a real-time semantic segmentation method based on hierarchical feature fusion and channel attention enhancement. By constructing a semantic segmentation model containing multi-level feature fusion and channel attention enhancement, it can alleviate the problems of local detail loss and inconsistency in intra-class semantic labels. Although existing technologies have made significant progress in structural optimization and speed improvement, the following key limitations still exist when facing complex traffic scenarios or multi-class fine-grained segmentation tasks: (1) Insufficient information coupling between feature branches: In the process of fusion of spatial features and semantic features, traditional bilateral or three-branch architectures still suffer from information loss and semantic drift, making it difficult to achieve the optimal balance between boundary details and global consistency; (2) Insufficient regional attention mechanism and lack of ability to focus on salient features: Traditional channel / spatial attention modules (such as SE and CBAM) mainly perform global weighting and lack an adaptive focusing mechanism for salient areas (such as buildings and roads) of different scales or forms; (3) Limited multi-scale feature modeling capability: Although the pyramid pooling module can expand the receptive field, it introduces a high computational overhead and cannot guarantee the consistency of the segmentation between the boundary and the main area, which is not conducive to deployment on resource-constrained terminal devices. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time semantic segmentation method based on an improved Transformer, which can solve the problems of insufficient multi-branch feature interaction, lack of salient region focus capability, and limited multi-scale feature modeling in existing real-time semantic segmentation methods.

[0005] The technical solution adopted in this invention is: a real-time semantic segmentation method based on an improved Transformer, comprising the following steps: S1: Preprocess the open-source autonomous driving scene dataset, assign a category label to each pixel of each image, and divide the autonomous driving scene dataset into a training set and a test set; S2: Construct a real-time semantic segmentation model, which includes a shallow convolutional layer, a semantic feature enhancement branch, a fusion branch, an edge enhancement branch, two semantic segmentation heads, and an edge segmentation head. The shallow convolutional layer is used to extract basic features and downsample the input image to generate a main feature map. The semantic feature enhancement branch enhances the region proportion features of the main feature map, and after outputting the semantic segmentation map through the first semantic segmentation head, it calculates the semantic cross-entropy loss L with the real segmentation map. s1 The fusion branch is used to output high-dimensional features to the semantic feature enhancement branch and the edge enhancement branch. The outputs of the semantic feature enhancement branch, the fusion branch, and the edge enhancement branch are downsampled, aligned, and then concatenated to form a joint feature. The joint feature is then used to output a probability map through the second semantic segmentation head, and then compared with the real segmentation map to calculate the fusion semantic cross-entropy loss L. s2 The edge enhancement branch is used to capture object edges and structural details in the main feature map, and after outputting an edge probability map through the edge segmentation head, it calculates the boundary cross-entropy loss L with the true boundary map. b ; S3: Train the real-time semantic segmentation model using semantic cross-entropy loss L. s1 , fusion semantic cross-entropy loss L s2 and boundary cross-entropy loss L b The weighted sum is used as the overall loss function; S4; Use a trained real-time semantic segmentation model for semantic segmentation.

[0006] Furthermore, the semantic feature enhancement branch includes two levels of residual convolutional layers and two multi-scale cross-attention modules; the main feature map is sequentially processed by convolution, normalization, ReLU activation, convolution, and normalization in the two levels of residual convolutional layers, and then added element-wise with the input features to form the first residual output; the first residual output and the output of the fusion branch are enhanced by the multi-scale cross-attention modules and then one path is input to the first semantic segmentation head, and the other path is enhanced again by the multi-scale cross-attention modules, and then fused with the outputs of the fusion branch and the edge enhancement branch.

[0007] Furthermore, in the multi-scale cross-attention module, the first residual output or the enhanced output is processed by a GPU-friendly attention mechanism; the output of the fusion branch is convolved and then residually connected, and then processed by the FFN module and residually connected again. One output is fused with the output of the GPU-friendly attention mechanism, and the fused result is used as the vector Q of the query feature. The other output is convolved and pooled to obtain the key vector K and the value vector V. The vector Q of the query feature, the key vector K and the value vector V are subjected to cross-scale semantic interaction through the cross-attention mechanism and then residually connected. Then, they are subjected to nonlinear enhancement by the FFN module and residually connected again. Finally, they are subjected to local refinement through the local window attention mechanism and then to nonlinear enhancement by the FFN module to obtain the final enhanced feature.

[0008] Furthermore, the edge enhancement branch includes a three-level residual convolutional layer and two feature fusion modules; the main feature map is sequentially processed by convolution, normalization, ReLU activation, convolution and normalization in the three-level residual convolutional layer, and then added element-wise with the input features to form a third residual output; the third residual output and the output of the fusion branch are sequentially fused twice to obtain one output path for the input edge segmentation head, and the other path is fused with the outputs of the fusion branch and the semantic feature enhancement branch.

[0009] Furthermore, the fusion branch includes two three-level residual convolutional layers, a region attention mechanism module, and a multi-scale dilated convolution and global pooling module. The main feature map, after sequentially undergoing convolution, normalization, ReLU activation, convolution, and normalization in the three-level residual convolutional layers, is element-wise added to the input features to form a second residual output, which is then input to the semantic feature enhancement branch and the edge enhancement branch for enhancement and fusion with the first and third residual outputs, respectively. Simultaneously, the second residual output is input to the region attention mechanism module, which dynamically weights salient regions to enhance the model's response to key geographic features. The output of the region attention mechanism module is input to the semantic feature enhancement branch and the edge enhancement branch for enhancement and fusion with the first and third residual outputs, respectively. It then undergoes another three-level residual convolutional layer processing and is input to the multi-scale dilated convolution and global pooling module to obtain a multi-scale fusion output, which is then fused with the outputs of the semantic feature enhancement branch and the edge enhancement branch.

[0010] Furthermore, the multi-scale dilated convolution and global pooling module includes a main path, a multi-level residual convolution feature enhancement branch, and a multi-scale dilated convolution branch. The main path includes a global average pooling layer, a convolutional layer, and an upsampling layer. The multi-level residual convolution feature enhancement branch includes four convolutional layers. The multi-scale dilated convolution branch includes four dilated convolutional layers. The output of the region attention mechanism module is processed by the main path through a global average pooling layer to extract global semantic information, and then through convolutional layers for channel compression and feature enhancement. After upsampling to align spatial dimensions, it is processed by the multi-level residual convolution feature enhancement branch for step-by-step feature extraction and residual connection, and by the multi-scale dilated convolution branch for multi-scale context information extraction and residual connection. Finally, the outputs of the main path, the multi-level residual convolution feature enhancement branch, and the multi-scale dilated convolution branch are concatenated, and channel fusion and feature smoothing are achieved through convolutional layers to obtain a multi-scale fused output.

[0011] Furthermore, the expression for the overall loss function is: ; in, Represents the overall loss function. L represents the semantic cross-entropy loss. s1 The weighting coefficients, L represents the boundary cross-entropy loss. b The weighting coefficients, L represents the semantic cross-entropy loss. s2 The weighting coefficients.

[0012] The beneficial effects of this invention are as follows: (1) The semantic enhancement branch of this invention introduces a multi-scale cross-attention module, which integrates the advantages of lightweight Transformer attention mechanism and convolutional feature extraction, and constructs an efficient feature aggregation framework that can capture local details and global contextual relationships at the same time. By dynamically modeling the dependency relationship between shallow and deep features, the global perception ability and detail preservation ability of the semantic branch are effectively enhanced, and adaptive interaction and fine-grained guidance between features are realized. This effectively bridges the expression gap between deep semantic features and shallow structural information, enhances the expression of semantic features, and improves the recognition accuracy and structural integrity of objects at various scales. (2) The present invention introduces a region attention module based on regional saliency in the fusion branch, which can adaptively adjust the attention distribution according to the target scale and structural complexity of the input image, enhance the network's response to key ground features, and improve the segmentation robustness in complex ground feature scenarios; (3) The present invention replaces the traditional pyramid pooling module with a multi-scale dilated convolution and a global average pooling module. By combining multi-scale dilated convolution with attention jump connection mechanism, the adaptive fusion of global context features and local spatial features is achieved. While maintaining high-resolution detail information, the consistency of boundary and main body region segmentation is significantly enhanced, and efficient multi-scale semantic fusion with low computational overhead is achieved. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the real-time semantic segmentation model in an embodiment of the present invention; Figure 3 This is a schematic diagram of the residual convolutional layer in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the multi-scale cross-attention module in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the multi-scale dilated convolution and global pooling module in an embodiment of the present invention; Figure 6 This is a comparison diagram of the semantic segmentation visualization structure of the embodiments of the present invention and the prior art. Detailed Implementation

[0015] To better understand the above-described objects, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention; however, the invention may be practiced in other ways different from those described herein, and therefore, the invention is not limited to the specific embodiments disclosed below.

[0016] This invention provides a real-time method based on Transformer enhancement and region attention fusion, comprising the following steps: S1: The open-source autonomous driving scene dataset is preprocessed, assigning a category label to each pixel of each image, and dividing the dataset into a training set and a test set. The category labels provide valuable data support for accurate object detection and semantic segmentation applications. In this embodiment, the Cityscapes real-time semantic segmentation dataset is used for testing and evaluation. It contains 5000 finely annotated images with a resolution of 2048×1024. The dataset covers 19 categories and is divided into a training set (2975 images), a validation set (500 images), and a test set (1525 images).

[0017] S2: Construct a real-time semantic segmentation model, the real-time semantic segmentation model as follows: Figure 2 As shown, the system includes a shallow convolutional layer, a semantic feature enhancement branch, a fusion branch, an edge enhancement branch, two semantic segmentation heads, and an edge segmentation head. The shallow convolutional layer is used to extract basic features and downsample the input image to generate a main feature map. In this embodiment, the size of the main feature map is 1 / 8 of the original image. The shallow convolutional layer consists of three convolutional units. The first two convolutional units include a 3×3 convolution with a stride of 1, batch normalization, and a ReLU activation function. The third convolutional unit includes a 3×3 convolution with a stride of 2, batch normalization, and a ReLU activation function.

[0018] The semantic feature enhancement branch enhances the region proportion features of the main feature map, and after outputting the semantic segmentation map through the first semantic segmentation head, it calculates the semantic cross-entropy loss L with the real segmentation map. s1 The fusion branch is used to output high-dimensional features to the semantic feature enhancement branch and the edge enhancement branch. The outputs of the semantic feature enhancement branch, the fusion branch, and the edge enhancement branch are downsampled, aligned, and then concatenated to form a joint feature. The joint feature is then used to output a probability map through the second semantic segmentation head, and then compared with the real segmentation map to calculate the fusion semantic cross-entropy loss L. s2 The edge enhancement branch is used to capture object edges and structural details in the main feature map, and after outputting an edge probability map through the edge segmentation head, it calculates the boundary cross-entropy loss L with the true boundary map. b .

[0019] In this embodiment of the invention, the semantic feature enhancement branch includes two levels of residual convolutional layers and two multi-scale cross-attention modules; such as Figure 3As shown, the main feature map is sequentially processed by convolution, normalization, ReLU activation, convolution, and normalization in two levels of residual convolutional layers, and then added element-wise with the input features to form the first residual output. The first residual output and the output of the fusion branch are enhanced by a multi-scale cross-attention module, and one path is input into the first semantic segmentation head. The other path is further enhanced by the multi-scale cross-attention module and then fused with the outputs of the fusion branch and the edge enhancement branch.

[0020] like Figure 4 As shown, in the multi-scale cross-attention module, the first residual output or the enhanced output is processed by a GPU-friendly attention mechanism; the output of the fusion branch is convolved and then residually connected, and then processed by the FFN module and residually connected again. One output is fused with the output of the GPU-friendly attention mechanism, and the fused result is used as the vector Q of the query feature. The other output is convolved and pooled to obtain the key vector K and the value vector V. The vector Q of the query feature, the key vector K and the value vector V are subjected to cross-scale semantic interaction by the cross-attention module and then residually connected, then nonlinearly enhanced by the FFN module and then residually connected again. Finally, the local window attention mechanism is used for local refinement, and then nonlinearly enhanced by the FFN module to obtain the final enhanced feature. Figure 4 In this context, 'a' represents the feature map obtained by concatenating the main feature map with two residual convolutional layers, and 'b' represents the feature map from the fusion branch. The local window attention mechanism divides the feature map into multiple local windows and calculates the self-attention weights only within each window. By scaling the dot product attention mechanism to model local spatial correlation, the local structural consistency and boundary representation ability are enhanced while reducing computational complexity.

[0021] The edge enhancement branch includes a three-level residual convolutional layer and two feature fusion modules. The main feature map is sequentially processed by convolution, normalization, ReLU activation, convolution, and normalization in the three-level residual convolutional layer, and then added element-wise with the input features to form a third residual output. The third residual output and the output of the fusion branch are sequentially fused twice to obtain one output path for the input edge segmentation head, and the other path is fused with the outputs of the fusion branch and the semantic feature enhancement branch.

[0022] The fusion branch includes two three-level residual convolutional layers, a region attention mechanism module, and a multi-scale dilated convolution and global pooling module. The main feature map, after sequentially undergoing convolution, normalization, ReLU activation, convolution, and normalization in the three-level residual convolutional layers, is element-wise added to the input features to form a second residual output. This second residual output is then input to the semantic feature enhancement branch and the edge enhancement branch, where it is enhanced and fused with the first and third residual outputs, respectively. Simultaneously, the second residual output is input to the region attention mechanism module, which dynamically weights salient regions to enhance the model's response to key feature areas. The output of the region attention mechanism module is input to the semantic feature enhancement branch and the edge enhancement branch, where it is enhanced and fused with the first and third residual outputs, respectively. This is then processed again by the three-level residual convolutional layer and input to the multi-scale dilated convolution and global pooling module to obtain a multi-scale fused output, which is then fused with the outputs of the semantic feature enhancement branch and the edge enhancement branch.

[0023] like Figure 5 As shown, the multi-scale dilated convolution and global pooling module includes a main path, a multi-level residual convolution feature enhancement branch, and a multi-scale dilated convolution branch. The main path includes a global average pooling layer, a convolutional layer, and an upsampling layer. The multi-level residual convolution feature enhancement branch includes four convolutional layers. The multi-scale dilated convolution branch includes four dilated convolutional layers. The output of the region attention mechanism module is processed by the main path through a global average pooling layer to extract global semantic information, and then through convolutional layers for channel compression and feature enhancement. After upsampling to align spatial dimensions, it is processed by the multi-level residual convolution feature enhancement branch for step-by-step feature extraction and residual connection, and by the multi-scale dilated convolution branch for multi-scale context information extraction and residual connection. Finally, the outputs of the main path, the multi-level residual convolution feature enhancement branch, and the multi-scale dilated convolution branch are concatenated, and channel fusion and feature smoothing are achieved through convolutional layers to obtain a multi-scale fused output. In this embodiment of the invention, the traditional pyramid pooling module is replaced with a multi-scale dilated convolution and global average pooling module. Compared with the traditional pyramid pooling module, the multi-scale dilated convolution and global pooling module achieves dynamic interaction and detail recovery of cross-scale features while maintaining global semantic consistency, effectively improving the segmentation accuracy of buildings and their edge regions.

[0024] S3: Train the real-time semantic segmentation model using semantic cross-entropy loss L. s1 , fusion semantic cross-entropy loss L s2 and boundary cross-entropy loss L b The weighted sum is used as the overall loss function. The expression for the overall loss function is: ; in, Represents the overall loss function. L represents the semantic cross-entropy loss. s1 The weighting coefficients, L represents the boundary cross-entropy loss. b The weighting coefficients, L represents the semantic cross-entropy loss. s2 The weighting coefficients.

[0025] Before model training, initial training was performed using the ImageNet dataset. The Cityscapes dataset was used with 640 training iterations, a batch size of 8, and an image resolution of 1024 × 1024. The weight decay rate was set to 0.0005, the initial learning rate to 0.01, and stochastic gradient descent (SGD) was employed for optimization, with a momentum parameter of 0.9. During training, semantic feature enhancement and edge enhancement branches were trained separately to stabilize feature extraction. A fusion branch was then introduced for overall optimization, achieving collaborative learning of semantics and structure.

[0026] S4; Use a trained real-time semantic segmentation model for semantic segmentation.

[0027] The technical effects of the embodiments of the present invention are verified through experiments below: By comparing the embodiments of the present invention with existing commonly used semantic segmentation methods, the following results can be obtained: Figure 6 The results shown are a visual comparison of semantic segmentation in an urban road scene. Figure 6 From top to bottom, the images show the original image, ground truth annotations, segmentation results from the BiSeNetV2 method, segmentation results from the PIDNet-S method, and segmentation results from this embodiment of the invention. White circles represent representative difficult regions (such as road edges, construction barriers, small pillars, vehicle boundaries, and shadow areas). Figure 6 It can be seen that the BiSeNetV2 method can achieve basic segmentation in the overall road area, but it has obvious omissions and boundary blurring problems in slender structures (such as road posts and pillars), complex boundaries, and occluded areas. The PIDNet-S method improves edge perception, but semantic confusion and local breaks still occur in salient areas adjacent to the background. In contrast, the embodiments of the present invention show clearer target contours, more complete structure preservation capabilities, and more stable category consistency in multiple scenarios: the boundaries at the intersection of roads and sidewalks are more accurate, the semantics are more explicit within target areas such as vehicles and fences, and the recognition ability of small-scale targets and local details is significantly enhanced. The visualization results show that the method of the present invention can effectively alleviate the problems of boundary blurring and semantic misunderstanding in complex urban scenes, and achieve higher quality and more refined semantic segmentation results while ensuring real-time performance.

[0028] Using the mean intersection-union ratio (mIoU) as the evaluation metric, the performance of the algorithm in the dataset segmentation task is measured. The comparison results are shown in Table 1. The specific expression for the mean intersection-union ratio (mIoU) is:

[0029] in, Indicates the total number of categories. +1 contains the background category, i represents the true category label, and j represents the predicted category label. Indicates the true category is But it was predicted to be The number of pixels, This represents the number of pixels whose true class is j but are predicted as i. This represents the number of pixels whose true class is i and which are correctly predicted as i, i.e., true instances.

[0030] Table 1. Experimental results of the method of the present invention and its comparative methods

[0031] method mIOU (%) FPS GPU BiSeNetV2 76.7 124.0 GTX 1080 Ti DDRNet-23 78.6 182.4 RTX 3090 PIDNet-S 79.3 154.9 RTX 3090 Embodiments of the present invention 81.2 158.6 RTX 3090 In the table, FPS stands for frames per second, representing the number of images the model can process per unit of time, used to measure the model's inference speed and real-time performance. As shown in Table 1, the BiSeNetV2 method achieves 76.7% mIoU and 124.0 FPS on the GTX 1080 Ti platform, exhibiting good real-time performance but relatively limited accuracy. The DDRNet-23 and PIDNet-S methods achieve 78.6% and 79.3% mIoU respectively on the RTX 3090 platform. The DDRNet-23 method has a higher inference speed, while the PIDNet-S method achieves a more balanced performance between accuracy and speed. In contrast, the embodiment of this invention achieves 81.2% mIoU and 158.6 FPS on the same GPU platform (RTX 3090), significantly improving segmentation accuracy while maintaining real-time inference capabilities, indicating that the proposed network structure has superior overall performance in terms of accuracy and efficiency.

[0032] In summary, this invention employs a multi-layered structure consisting of semantic feature enhancement branches, fusion branches, and edge enhancement branches, achieving parallel extraction and dynamic fusion of global semantic information and local boundary features. The semantic feature enhancement branch focuses on category recognition, the edge enhancement branch focuses on strengthening the target edge contour, and the fusion branch achieves global consistency optimization, thereby effectively improving the structural integrity and boundary accuracy of the segmentation results. In the multi-scale cross-attention module, a GPU-friendly attention mechanism and a local window attention mechanism are combined to significantly improve long-range dependency modeling capabilities while maintaining computational efficiency, achieving effective transmission of cross-layer semantic and detailed information. A region attention module is embedded in the fusion branch, enabling the network to adaptively focus on key feature regions through a region saliency weighting mechanism. The multi-scale dilated convolution and global average pooling modules achieve effective fusion of global and local scale information, and scale-adaptive feature enhancement is achieved through feature concatenation and residual correction, effectively improving the model's robustness to changes in target scale. This invention achieves multi-branch collaborative optimization through a weighted joint training mechanism of semantic, boundary, and fusion loss functions, significantly improving semantic consistency and structural fidelity.

[0033] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time semantic segmentation method based on an improved Transformer, characterized in that, Includes the following steps: S1: Preprocess the open-source autonomous driving scene dataset, assign a category label to each pixel of each image, and divide the autonomous driving scene dataset into a training set and a test set; S2: Construct a real-time semantic segmentation model, which includes a shallow convolutional layer, a semantic feature enhancement branch, a fusion branch, an edge enhancement branch, two semantic segmentation heads, and an edge segmentation head. The shallow convolutional layer is used to extract basic features and downsample the input image to generate a main feature map. The semantic feature enhancement branch enhances the region proportion features of the main feature map, and after outputting the semantic segmentation map through the first semantic segmentation head, it calculates the semantic cross-entropy loss L with the real segmentation map. s1 The fusion branch is used to output high-dimensional features to the semantic feature enhancement branch and the edge enhancement branch. The outputs of the semantic feature enhancement branch, the fusion branch, and the edge enhancement branch are downsampled, aligned, and then concatenated to form a joint feature. The joint feature is then used to output a probability map through the second semantic segmentation head, and then compared with the real segmentation map to calculate the fusion semantic cross-entropy loss L. s2 The edge enhancement branch is used to capture object edges and structural details in the main feature map, and after outputting an edge probability map through the edge segmentation head, it calculates the boundary cross-entropy loss L with the true boundary map. b ; S3: Train the real-time semantic segmentation model using semantic cross-entropy loss L. s1 , fusion semantic cross-entropy loss L s2 and boundary cross-entropy loss L b The weighted sum is used as the overall loss function; S4; Use a trained real-time semantic segmentation model for semantic segmentation.

2. The real-time semantic segmentation method based on an improved Transformer as described in claim 1, characterized in that, The semantic feature enhancement branch includes two levels of residual convolutional layers and two multi-scale cross-attention modules; The main feature map is sequentially processed through convolution, normalization, ReLU activation, convolution, and normalization in two residual convolutional layers, and then added element-wise with the input features to form the first residual output. The first residual output and the output of the fusion branch are enhanced by a multi-scale cross-attention module, and one path is input into the first semantic segmentation head. The other path is further enhanced by the multi-scale cross-attention module and then fused with the outputs of the fusion branch and the edge enhancement branch.

3. The real-time semantic segmentation method based on an improved Transformer as described in claim 2, characterized in that, In the multi-scale cross-attention module, the first residual output or the enhanced output is processed by a GPU-friendly attention mechanism; the output of the fusion branch is convolved and then residually connected, and then processed by the FFN module and residually connected again. One output is fused with the output of the GPU-friendly attention mechanism, and the fused result is used as the vector Q of the query feature. The other output is convolved and pooled to obtain the key vector K and the value vector V. The vector Q of the query feature, the key vector K and the value vector V are subjected to cross-scale semantic interaction through the cross-attention mechanism and then residually connected. After nonlinear enhancement by the FFN module, they are residually connected again. Finally, local refinement is performed through the local window attention mechanism, and then nonlinear enhancement by the FFN module to obtain the final enhanced feature.

4. The real-time semantic segmentation method based on an improved Transformer as described in claim 3, characterized in that, The edge enhancement branch includes a three-level residual convolutional layer and two feature fusion modules; The main feature map is sequentially processed through convolution, normalization, ReLU activation, convolution, and normalization in the three-level residual convolutional layer, and then added element-wise with the input features to form the third residual output. The third residual output and the output of the fusion branch are sequentially processed through two feature fusions to obtain one output path for the input edge segmentation head, and the other path is fused with the outputs of the fusion branch and the semantic feature enhancement branch.

5. The real-time semantic segmentation method based on an improved Transformer as described in claim 4, characterized in that, The fusion branch includes two three-level residual convolutional layers, a region attention mechanism module, and a multi-scale dilated convolution and global pooling module. The main feature map, after sequentially undergoing convolution, normalization, ReLU activation, convolution, and normalization in the three-level residual convolutional layers, is element-wise added to the input features to form a second residual output. This second residual output is then input to the semantic feature enhancement branch and the edge enhancement branch, where it is enhanced and fused with the first and third residual outputs, respectively. Simultaneously, the second residual output is input to the region attention mechanism module, which dynamically weights salient regions to enhance the model's response to key feature areas. The output of the region attention mechanism module is input to the semantic feature enhancement branch and the edge enhancement branch, where it is enhanced and fused with the first and third residual outputs, respectively. This is then processed again by the three-level residual convolutional layer and input to the multi-scale dilated convolution and global pooling module to obtain a multi-scale fused output, which is then fused with the outputs of the semantic feature enhancement branch and the edge enhancement branch.

6. The real-time semantic segmentation method based on an improved Transformer as described in claim 5, characterized in that, The multi-scale dilated convolution and global pooling module includes a main path, a multi-level residual convolution feature enhancement branch, and a multi-scale dilated convolution branch; the main path includes a global average pooling layer, a convolutional layer, and an upsampling layer; the multi-level residual convolution feature enhancement branch includes four convolutional layers; the multi-scale dilated convolution branch includes four dilated convolutional layers. The output of the region attention mechanism module is processed by a global average pooling layer through the main path to extract global semantic information. Then, it is processed by a convolutional layer for channel compression and feature enhancement. Next, it is processed by upsampling to align spatial dimensions. Then, it is processed by a multi-level residual convolutional feature enhancement branch for step-by-step feature extraction and residual connection. Finally, it is processed by a multi-scale dilated convolutional branch for multi-scale context information extraction and residual connection. Finally, the outputs of the main path, the multi-level residual convolutional feature enhancement branch, and the multi-scale dilated convolutional branch are concatenated and channel fusion and feature smoothing are achieved through a convolutional layer to obtain a multi-scale fused output.

7. A real-time semantic segmentation method based on an improved Transformer as described in claim 6, characterized in that, The expression for the overall loss function is: ； in, Represents the overall loss function. L represents the semantic cross-entropy loss. s1 The weighting coefficients, L represents the boundary cross-entropy loss. b The weighting coefficients, L represents the fusion semantic cross-entropy loss. s2 The weighting coefficients.

Images