Real-time semantic segmentation method based on double-branch coding and decoding network

Abstract

The invention relates to a real-time semantic segmentation method based on a double-branch coding and decoding network. The real-time semantic segmentation method comprises the following steps: step 1, extracting feature information of an initial image through a CNN branch and a Transform branch; a CNN-Transform fusion module is used to fuse the output feature map of the CNN branch and the output feature map of the Transform branch by using the attention mask; and inputting the feature map fused by the CNN-Transformer into a segmentation head for prediction so as to obtain a segmentation map. According to the real-time semantic segmentation method, the CNN can be used for extracting local level information, and the Transform can be used for extracting global context information, so that the attention refining module can enhance the feature representation of significant features while enhancing the global feature representation; according to the method, a pyramid pooling module can extract multi-scale stripe information while extracting multi-scale context information, and a feature fusion module can dynamically adjust feature weights, so that the performance of real-time semantic segmentation is improved; therefore, a very good result is achieved between the segmentation precision and the reasoning speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a real-time semantic segmentation method based on a dual-branch encoding-decoding network. Background Art

[0002] Semantic segmentation is a basic task in computer vision. Its purpose is to classify each pixel in an image and determine its belonging semantic category. Semantic segmentation has a wide range of applications in many aspects, such as medical image processing, autonomous driving, human-computer interaction, remote sensing images, etc. In recent years, the number of real-time application scenarios of semantic segmentation has increased, and people have begun to pay attention to its real-time performance. Real-time semantic segmentation has certain requirements for the accuracy of the task and the inference speed. Therefore, how to improve the real-time performance of the algorithm while ensuring the accuracy of the segmentation result is a major challenge in the current semantic segmentation field.

[0003] In recent years, the rapid development of deep learning has made convolutional neural networks shine in the field of semantic segmentation. The CNN model performs well in extracting local hierarchical information, but its receptive field and the ability to learn long-range dependencies are limited, so that it cannot extract global semantic information well.

[0004] Transformer can fully capture the long-distance dependencies of an image by using the self-attention mechanism to establish the relationship between each element in the input sequence. Although it can model the global information of the image, its ability to capture fine spatial details is limited, and the computational cost is large and the complexity is high. In contrast, CNN has the ability to extract local information and location information well, and the computational cost is more friendly.

[0005] The Deep Aggregation Pyramid Pooling Module (DAPPM) collects global semantic information from feature maps at different levels through average pooling operations of different sizes. In addition, it increases the receptive field size without introducing additional parameters, thereby improving the network's perception range of the entire image. However, it does not consider the acquisition of strip information.

[0006] The attention refinement module can enhance the network's understanding of image semantics by introducing more detailed information. This enables the network to more accurately segment different objects and regions. BiSeNet proposes that the attention refinement module (ARM) only uses global average pooling to integrate context information into the feature representation at each position, improving the ability to understand global semantics. However, it does not consider strengthening important significant semantic information.

[0007] Element-wise addition of feature maps and concatenation along the channel dimension are two traditional feature fusion methods, which can strengthen important features and weaken noise. The network learns features at different spatial positions, improving the network's stability and generalization ability. However, their fusion ability is limited and cannot handle scenarios that require dynamic adjustment of feature weights and enhancement of network discrimination ability.

[0008] In summary, how to enable the segmentation model to simultaneously capture local-level information and global context information while balancing accuracy and speed, consider multi-level global information and strip information when using the pyramid pooling module, consider context information and salient information when using attention to refine the image, and dynamically adjust feature weights during the feature integration process are urgent problems to be solved in the field of real-time semantic segmentation. Summary of the Invention

[0009] The purpose of the present invention is to provide a real-time semantic segmentation method based on a dual-branch encoding-decoding network to improve the performance of real-time semantic segmentation.

[0010] To achieve the above purpose, the technical solution adopted by the present invention is: a real-time semantic segmentation method based on a dual-branch encoding-decoding network, including the following steps:

[0011] Step 1, extract the feature information of the initial image through the CNN branch and the Transformer branch;

[0012] (1) Input the initial image into the backbone network STDC of the CNN branch. STDC extracts local-level information from the initial image and outputs feature maps with different resolutions in the S1-S5 stages respectively; then input the feature map output in the S5 stage into the depth strip pyramid pooling module to first obtain a feature map containing multi-scale strip information and context information, and then reduce the number of channels of the output feature map through 1×1 convolution, and then perform an attention operation with the output feature map of the initial image after reducing the number of channels; at the same time, input the feature maps output in the S3 and S4 stages into the feature pooling refinement module to strengthen the global semantic and salient semantic information; finally, perform a decoding operation on the feature map processed by the depth strip pyramid pooling module in the S5 stage and the feature maps processed by the feature pooling refinement module in the S3 and S4 stages to obtain the output feature map of the CNN branch;

[0013] (2) Use the sum of the upsampled feature maps in the S3 and S4 stages of the CNN branch as the input of the Transformer branch. After extracting global information through two PoolFormer blocks, obtain the preliminary output P1, and then sum P1 with the upsampled feature map in the S5 stage. The sum result is further processed through two PoolFormer blocks to extract global information and obtain the final output P2 as the output feature map of the Transformer branch;

[0014] Step 2: Use the CNN-Transformer fusion module to fuse the output feature maps of the CNN branch and the output feature maps of the Transformer branch using an attention mask.

[0015] Step 3: The feature maps after CNN-Transformer fusion are input into the segmentation head for prediction to obtain the segmentation map.

[0016] Specifically, in Step 1, the operation method of the depth strip pyramid pooling module is as follows: First, perform large kernel pooling on the input feature maps with different kernels and different strides to generate feature maps with different resolutions, and use global average pooling to generate the global information of the entire feature map. Then, first use 1×1 convolution on the feature maps with different resolutions to reduce the number of channels and computational complexity, and then perform convolution operations on them with strips of different kernel sizes to extract strip information of different scales. Next, the feature maps with different resolutions after strip convolution are upsampled and concatenated to obtain a feature map containing multi-scale strip information and context information. Finally, the concatenated feature map is passed through 1×1 convolution to reduce the number of channels, and an attention operation is performed with the output feature map of the initial image after reducing the number of channels.

[0017] Specifically, in Step 1, the operation method of the feature pooling refinement module is as follows: First, the feature maps output in the S3 stage and the S4 stage are respectively subjected to global average pooling and global maximum pooling operations. Then, they respectively pass through the Conv-BN-Sigmoid layer to obtain the maximum pooling attention map and the average pooling attention map. The maximum pooling attention map is multiplied by the original feature map, the average pooling attention map is multiplied by the original feature map, and the results of the two multiplications are summed. Finally, a residual connection is made with the original input feature map to obtain the output.

[0018] Specifically, the decoding operation in Step 1 is as follows: After the feature map output in the S5 stage is processed by the depth strip pyramid pooling module, it is first upsampled to the same resolution as the feature map output in the S4 stage, and then added to the feature map of the feature map output in the S4 stage after being processed by the feature pooling refinement module. The result of the addition is then upsampled to the same resolution as the feature map output in the S3 stage, and finally added to the feature map of the feature map output in the S3 stage after being processed by the feature pooling refinement module to obtain the output of the CNN branch.

[0019] Specifically, in step 2, the operation of the CNN-Transformer fusion module is as follows: The output feature maps of the CNN branch and the Transformer branch are first concatenated in the channel dimension, and then the number of channels is reduced through a Conv-BN-ReLU layer to match the number of channels of the original feature map. The feature map with the reduced number of channels then goes through a Conv-BN-Sigmoid layer to obtain the attention mask α. Next, α is directly multiplied with the output feature map of the Transformer branch and residual connection is performed. At the same time, the relative attention mask 1-α is multiplied with the output feature map of the CNN branch and residual connection is also performed. Finally, the feature images after attention and residual processing of these two branches are summed pixel-wise to obtain the output result.

[0020] Specifically, in step 3, the segmentation head consists of a 3×3 Conv-BN layer and a 1×1 convolutional layer.

[0021] The beneficial effects of the present invention are as follows: The present invention proposes an effective dual-branch encoder-decoder network (DBEDNet) for real-time semantic segmentation. It consists of a long CNN branch to extract local hierarchical information and a short Transformer branch to extract global context information. The depth strip aggregation pyramid pooling module (DSAPPM) can extract multi-scale strip information while extracting global information. The feature pooling refinement module (FPRM) can optimize the feature representations at different stages in the network. The CNN-Transformer fusion module (CTFM) can adaptively fuse the information of the CNN branch and the Transformer branch. The present invention has conducted a large number of experiments to verify its effectiveness. On an RTX3090 GPU, it achieved 80.3% mIoU and 78.6 FPS on the Cityscapes dataset, 80.0% mIoU and 137.5 FPS on the CamVid dataset, and 40.9% mIoU and 253.7 FPS on the ADE20K dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0023] Figure 1 It is a schematic structural diagram of the dual-branch encoder-decoder network (DBEDNet) in the present invention;

[0024] Figure 2 This is a schematic diagram of the Deep Strip Aggregation Pyramid Pooling Module (DSAPPM) in the present invention;

[0025] Figure 3 This is a schematic diagram of the Feature Pooling Refinement Module (FPRM) in the present invention;

[0026] Figure 4 This is a schematic diagram of the CNN-Transformer Fusion Module (CTFM) in the present invention;

[0027] Figure 5 This is the segmentation comparison result between the Dual-Branch Encoder-Decoder Network (DBEDNet) of the present invention and other methods on the Cityscapes dataset;

[0028] Figure 6 This is the segmentation comparison result between the Dual-Branch Encoder-Decoder Network (DBEDNet) of the present invention and other methods on the CamVid dataset;

[0029] Figure 7 This is the segmentation comparison result between the Dual-Branch Encoder-Decoder Network (DBEDNet) and other methods on the ADE20K dataset. Detailed implementation manner

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but it shall not be used as a basis for any limitation to the invention.

[0031] Embodiment 1: A real-time semantic segmentation method based on a dual-branch encoder-decoder network. Taking the images in the Cityscapes dataset as an example, first, the images in the Cityscapes dataset are subjected to data augmentation techniques such as random cropping and padding, random resizing, random horizontal flipping, and color jittering, and the image resolution of 2048×1024 is cropped to 1024×512 and 1536×768 as the initial image size; then, as Figure 1 shown, the following steps are carried out:

[0032] Step 1, input the images with a resolution of 1024×512 or 1536×768 into the backbone network (STDC) in the CNN branch, and STDC extracts local hierarchical information; STDC has five stages S1-S5, and the resolutions of the feature maps output by these five stages are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image resolution respectively, and the number of channels are 64, 128, 256, 512, and 1024 respectively;

[0033] Step 2: The feature map output in the S5 stage of Step 1 is input into the Depth Strip Pyramid Pooling Module (DSAPPM) to obtain context and strip information, so as to solve the problem that the Depth Aggregation Pyramid Pooling Module (DAPPM) does not consider obtaining strip information.

[0034] As Figure 2 shown, the detailed operations of the DSAPPM module are as follows: First, large kernel pooling with different kernels and different strides is performed on the input feature map (i.e., the feature map output in the S5 stage) to generate feature maps with resolutions of 1 / 128, 1 / 256, and 1 / 512. In addition, global average pooling is also used to generate the global information of the entire feature map. After that, 1×1 convolutions are first used on the feature maps with different resolutions to reduce the number of channels and computational complexity, and then convolution operations are performed on the strips with different kernel sizes to extract strip information at different scales. For the input feature map x, the feature map y at each stage can be written as:

[0035] where C m×n represents a convolution of m×n. For example, C 1×1 and C 3×3 represent 1×1 convolution and 3×3 convolution respectively; U represents bilinear interpolation upsampling operation; P j,k represents average pooling with a stride of k and a kernel size of j, where j = 2 i +1 and k = 2 i -1; P global represents global average pooling.

[0036] Furthermore, after strip convolution and upsampling of the feature maps with different resolutions, they are concatenated. This can make the feature map contain multi-scale strip information and context information. These feature maps then pass through 1×1 convolution to reduce the number of channels, and then a simple attention operation is performed with the original feature map after reducing the number of channels. The specific steps can be represented by the following formulas: F c = Concat(y1, y2, y3, y4, y5), F cd = C 1×1 (F c ), F d = C 1×1 (x), F output = BiAttention(F cd , F d ),

[0037] where Concat represents the channel concatenation operation; C 1×1represents a 1×1 convolution; BiAttention can perform attention processing on the processed feature map and the feature map with reduced number of channels, so as to enhance the feature representation of important regions and balance the influence of new and old features; F c is the output after concatenating y1, y2, y3, y4, y5; F cd is for F c is the output after reducing the number of channels through a 1×1 convolution; F d is the output after reducing the number of channels of the initial input feature map x through a 1×1 convolution; F output the final output of the entire DSAPPM module, which is obtained from F d and F cd after passing through BiAttention.

[0038] Step 3, perform a decoding operation on the feature maps of the last three stages of the CNN branch. Before performing the decoding operation, the feature maps output by the S3 and S4 stages are first input into the Feature Pooling Refinement Module (FPRM) to strengthen the global semantic and significant semantic information, so as to solve the problem that the Attention Refinement Module (ARM) does not consider strengthening the significant semantic information.

[0039] As Figure 3 shown, the detailed operations of FPRM are as follows: First, the input feature maps are respectively subjected to global average pooling and global max pooling operations. Global average pooling can summarize the information of the entire feature map, thereby capturing more extensive semantic information. Global max pooling can extract the most significant feature information from the entire feature map, emphasizing the most important semantic information in the image;

[0040] Then, they respectively pass through the Conv-BN-Sigmoid layer to obtain the max pooling attention map and the average pooling attention map. The max pooling attention map is multiplied by the original feature map to strengthen the feature representation, thereby improving the network's perception ability of important semantic regions. The average pooling attention map is multiplied by the original feature map to enhance and improve the network's understanding ability of global semantics; then the two perform pixel-level summation, and this operation can increase both of these abilities at the same time; where the "original feature map" is the feature map output by the S3 and S4 stages;

[0041] Finally, a residual connection is made with the original input feature map to obtain the output, which can balance the influence of new and old features. Among them, the original input feature map is the feature map output by the S3 and S4 stages. The specific steps can be represented by the following formula: F a = CCBS(Avg(F s )) F m = CCBS(Max(F s )) F output = F s + F m · F s + F a · F s ,

[0042] where F s is the output of the S3 and S4 stages in the backbone network; Avg represents the global average pooling operation; Max represents the global max pooling operation; CBS is a Conv-BN-Sigmoid layer;

[0043] F a is the average pooling attention map obtained by performing global average pooling on F s as the input feature map and then being guided by a Conv-BN-Sigmoid layer;

[0044] F m is the average pooling attention map obtained by performing global max pooling on F s as the input feature map and then being guided by a Conv-BN-Sigmoid layer;

[0045] F output is the output of the entire FPRM module, which is obtained by adding the result of multiplying F m and F s to the result of multiplying F a and F s , and then adding F s .

[0046] The decoding operation of the feature maps in the last three stages of the CNN branch is as follows: The feature map processed by DSAPPM in the S5 stage is first upsampled to the same resolution as the feature map in the S4 stage, and then added to the feature map in the S4 stage that has passed through FPRM. The result of their addition is then upsampled to the same resolution as the feature map in the S3 stage, and finally added to the feature map in the S3 stage that has passed through FPRM to obtain the output of the CNN branch.

[0047] Step 4: Larger-resolution feature maps contain more semantic information, but the PoolFormer block also processes large-resolution feature maps very slowly. After weighing performance and speed, it is chosen to use the sum of the upsampled and channel-adjusted feature maps of the S3 stage and the S4 stage of the backbone network as the input of the Transformer branch. After being processed by two PoolFormer blocks, the preliminary output P1 is obtained; then P1 is added to the result of the upsampled and channel-adjusted feature map of the S5 stage, and then processed by two PoolFormer blocks to obtain the output P2 of the Transformer branch. This can enable the feature maps of the Transformer branch to be guided by the CNN branch when gradually extracting information.

[0048] Step 5: In semantic segmentation of the CNN branch feature maps, emphasis is placed on multi-scale information, local hierarchical information, and high-level abstract features; the Transformer branch focuses more on global dependencies, non-local feature associations, and high interpretability. Considering the different characteristics of the feature maps of the above two branches, a CNN-Transformer fusion module (CTFM) is used to fuse them using the idea of an attention mask to solve the problem that traditional feature fusion methods cannot dynamically adjust feature weights.

[0049] As Figure 4 shown, the detailed operations of CTFM are as follows: The output feature maps of the CNN branch and the output feature maps of the Transformer branch are first concatenated in the channel dimension. At this time, the feature maps have both hierarchical position information and global context information. Then, the number of channels is reduced through a Conv-BN-ReLU layer to match the number of channels of the original feature maps. The feature maps with reduced channels are then passed through a Conv-BN-Sigmoid layer to obtain the attention mask α. After obtaining the attention mask α, α is directly multiplied by the output feature maps of the Transformer branch and residual connections are made. At the same time, the relative attention mask 1-α is multiplied by the output feature maps of the CNN branch and residual connections are made. Multiplying the attention map by the original feature map can strengthen the feature representation of important positions. Residual connections introduce additional information focusing on key positions while retaining the original information, which helps improve the expressive power and generalization ability of the model. Finally, the pixel-level sum of the feature images of the two branches after attention and residual processing is obtained as the output. The specific steps can be expressed by the following formula: F cat =Concate(F c ,F t ), α=CBS(CBU(F cat )), F at =α·Ft +F t , F ac = (1 - α)·F c +F c , F output = F at +F ac ,

[0050] Among them, F c is the output feature map of the CNN branch; Concat represents the channel concatenation operation; F t is the output feature map of the Transformer branch; CBU represents a Conv - BN - ReLU layer; CBS represents a Conv - BN - Sigmoid layer;

[0051] F cat is the result of concatenating F c and F t in the channel dimension;

[0052] F at is the result of multiplying the attention mask α with F t and then adding them together;

[0053] F output is the feature map output by the entire CTFM module.

[0054] Step six, the feature map fused by CTFM is input into the segmentation head for pixel - level prediction. The segmentation head consists of a 3×3 Conv - BN layer and a 1×1 convolution. The 3×3 Conv - BN layer is used to further aggregate feature information, and the 1×1 convolution is used to adjust the number of channels of the feature map to the number of feature classes to be predicted.

[0055] In addition, two auxiliary losses are used to supervise the training of the output feature maps of the CNN branch and the Transformer branch. They are only used in training and do not affect the inference speed of the model.

[0056] To verify the effect of the present invention, the following experiments are carried out:

[0057] The following are three commonly used semantic segmentation datasets: (1) The Cityscapes dataset is a dataset widely used for semantic segmentation tasks, containing 5000 high - resolution images collected from the perspective of cars in different cities. Among them, 2975 images are used for training, 500 for validation, and 1525 for testing. The resolution of each image is 2048×1024, which poses a great challenge to real - time semantic segmentation. The annotations include 30 categories such as roads, buildings, pedestrians, vehicles, etc. Among them, 19 are used for semantic segmentation tasks.

[0058] (2) The CamVid dataset is a small-scale dataset for road scenes. It contains 701 densely annotated frames extracted from video sequences. Among them, 367 frames are used for training, 101 frames are used for validation, and 233 frames are used for testing. The image resolution of this dataset is 720×960. And each pixel is labeled as one of 32 different classes, and a subset of 11 classes is used for semantic segmentation.

[0059] (3) ADE20K is a large-scale scene parsing dataset for semantic segmentation tasks, containing high-resolution images from real scenes. Each pixel is labeled as one of 150 different classes, including various objects, scenes, and background elements. This dataset has 25K images with a resolution of 512×512, of which 20K are used for training, 2K are used for validation, and 3K are used for testing.

[0060] The present invention conducts experiments on the above three datasets, and respectively compares and evaluates the performance with other advanced methods on these three datasets. The main evaluation metrics are the mean intersection over union (mIoU) and the frames per second (FPS), and the parameter quantity of the present invention is also measured. The experimental environment is based on PaddlePaddle 2.4.0, CUDA 11.6, CUDNN 8.2.1, and TensorRT 8.2.1.8, and the experiments are conducted on RTX2080Ti and RTX3090 GPUs.

[0061] The experimental settings are as follows: All datasets use the Stochastic Gradient Descent (SGD) optimizer to update the model parameters. The momentum of SGD is set to 0.9, and the weight decay is 5e-4. A polynomial decay learning rate scheduler and a learning rate warm-up strategy are also used during training. For the Cityscapes dataset, we crop it to resolutions of 512×1024 and 768×1536 as inputs. The batch size corresponding to these two inputs is 8, the maximum number of iterations is 320000, the initial learning rate is 0.005, and the learning rate warm-up strategy is used in the first 1000 iterations. For the CamVid dataset and the ADE20K dataset, we do not perform cropping. The input resolutions are 960×720 and 512×512 respectively. All other settings are the same as the training settings of Cityscapes.

[0062] In addition, some data augmentation methods are also used, including random cropping and padding, random resizing, random horizontal flipping, and color jitter. The scale range of random resizing is [0.5, 2.0]. In the inference stage, we first export DBEDNet to the ONNX format, and then measure the model speed through TensorRT.

[0063] To verify the effectiveness of the modules and structures of the present invention, ablation experiments were conducted on the dual-branch encoding structure, the depth strip pyramid pooling module (DSAPPM), the feature pooling refinement module (FPRM), the CNN-Transformer fusion module (CTFM), and the loss function. The experimental results are shown in Tables 1, 2, 3, and 4. The present invention uses mIoU and FPS as evaluation criteria, and the ablation experiments were carried out using the basic version of DBEDNet on the Cityscapes dataset with an input resolution of 1024×512 and an RTX 2080Ti.

[0064] To verify the effectiveness of the dual-branch encoding encoder structure designed by the present invention, different structures were compared through experiments, where CNN-ED represents the CNN branch with a lightweight encoder-decoder structure. When feature fusion is required in the experiment, the low-resolution feature map after adjusting the number of channels is directly upsampled and then added to the high-resolution feature map. The ablation experiment results are shown in Table 1. The inference speed using only Transformer as the backbone is much slower than that of CNN, so CNN was selected as the backbone. Compared with other structures, the dual-branch encoding and decoding achieved the best performance. Although the inference speed of the dual-branch encoder-decoder structure decreased compared with other structures, it significantly improved the segmentation performance. Therefore, the dual-branch encoder-decoder structure is effective. Table 1 Comparison results of different structures on the Cityscapes validation set CNN CNN-ED Transformer Params ↓ mIoU(%)↑ FPS↑ √ 9.6M 73.2 322.9 √ 9.8M 77.4 257.7 √ 13.2M 77.8 66.3 √ √ 9.9M 77.7 176.6 √ √ 10.1M 78.0 130.7

[0065] To verify the effectiveness of the depth strip aggregation pyramid pooling module (DSAPPM) proposed by the present invention, DSAPPM was compared with several other pyramid pooling modules. The ablation experiment results are shown in Table 2. It can be seen from Table 2 that compared with the PPM, DAPPM, and PAPPM modules, the mIoU of the DSAPPM module of the present invention increased by 1.8%, 0.6%, and 0.7% respectively. After using the DSAPPM module, the mIoU of the model increased from 78.4% to 79.6%, and the number of parameters increased by 2.5M. These experimental results show that the DSAPPM module significantly improves the network's ability to parse scenes with a slightly reduced inference speed. Table 2 Comparison results of DSAPPM and other pyramid pooling modules on the Cityscapes validation set

[0066] The present invention also verifies the effectiveness of the proposed Feature Pool Refinement Module (FPRM) and CNN-Tranformer Fusion Module (CTFM) through experiments. Since the proposed FPRM and TAFM are both plug-and-play modules, the ablation of FPRM is to directly delete and replace it with ARM and CAM, while for CTFM, the fusion method is directly changed to the addition of the feature maps of the CNN and Transformer branches. Table 3 shows the results of the ablation experiments. After using FPRM and TAFM, the model accuracy has increased by 0.5% mIoU and 0.3% mIoU respectively. After using both methods simultaneously, the accuracy reaches 79.6% mIoU. Moreover, with a slight increase in the number of parameters, FPRM is superior to ARM and CAM in terms of segmentation inference and inference speed. These experimental results indicate that FPRM and CTFM have good effects in improving the model performance. Table 3 Comparison results of different refinement modules and fusion modules on the Cityscapes validation set

[0067] The dual-branch encoding-decoding network (DBEDNet) of the present invention introduces two additional auxiliary losses at the outputs of the CNN branch and the Transformer branch to promote the optimization of the entire network. They are only used during training and do not affect the inference speed of the model. The ablation experiments also verify their effectiveness. CrossEntropyLoss (CE) and OhemCrossEntropyLoss (OHEM) are two common losses in semantic segmentation, and their effects on the model performance are compared through experiments. The ablation results of the auxiliary losses and the loss function types are shown in Table 4. It can be seen from the table that the auxiliary losses help to increase the feature perception ability of the model, and OhemCrossEntropyLoss has a better effect. Table 4 Comparison results of different types of loss functions and auxiliary losses on the Cityscapes validation set Type Main Loss Auxiliary Loss mIoU(%)↑ CE √ 78.1 CE √ √ 78.3 OHEM √ 79.1 OHEM √ √ 79.6

[0068] The proposed DBEDNet of the present invention is evaluated on the Cityscapes dataset using input resolutions of 1024×512 and 1536×768 respectively. The backbone network is STDC1, and the models with the input image size being 50% and 75% of the original image are called DBEDNet-S1 and DBEDNet-S2. The backbone network is STDC2, and the models with the input image size being 50% and 75% of the original image are called DBEDNet-B1 and DBEDNet-B2.

[0069] The experimental results of DBEDNet on the Cityscapes dataset are shown in Table 5. The table shows the name of the network model, input resolution, number of parameters, segmentation accuracy, and inference speed. As can be seen from Table 5, on RTX3090, DBEDNet-B2 achieved 80.3% mIoU at 78.6 FPS, and DBEDNet-S1 achieved 78.0% mIoU at 233.2 FPS. Compared with some previous state-of-the-art methods, DBEDNet achieved a very good balance between inference speed and accuracy. Figure 5 The visualization results of DBEDNet and several other methods on the Cityscapes dataset are shown. As can be seen from the figure, the segmentation effect of DBEDNet-B in some detail parts is better than other networks. In general, DBEDNet achieved satisfactory results in terms of accuracy and speed on the Cityscapes dataset. Table 5 Comparison results of DBEDNet and other methods on the Cityscapes dataset

[0070] In Table 5, it indicates that the inference speed of this model was measured after being accelerated by TensorRT. ★ indicates that the inference speed of this model was measured after being accelerated by TensorRT on the PaddlePaddle platform. - indicates that the corresponding paper of this model did not report relevant parameters.

[0071] To further demonstrate the performance of DBEDNet proposed in the present invention, it was comprehensively compared with other network models on the CamVid dataset. Similar to other works, the input resolution for training and inference was set to 960×720. Table 6 shows the experimental comparison results between the present invention and other network models. On RTX3090, DBEDNet-S achieved 79.1% mIoU at 175.0 FPS, and DBEDNet-B achieved 80.0% mIoU at 137.5 FPS. Compared with STDCNet with the same backbone network, DBEDNet had a significant improvement in accuracy while the inference speed decreased. Figure 6 The visualization segmentation results of DBEDNet and RTFormer on the CamVid dataset are shown, from which it can be seen that DBEDNet is better at segmenting the contours of some objects. Generally speaking, DBEDNet achieved good results on the CamVid dataset, achieving a very good balance between accuracy and speed. Table 6 Comparison results of DBEDNet and other methods on the CamVid dataset

[0072] In Table 6, It indicates that the inference speed of the model is measured after being accelerated by TensorRT. ★ indicates that the inference speed of the model is measured after being accelerated by TensorRT on the PaddlePaddle platform. * indicates that the model is pre-trained on Cityscapes. $ indicates that the segmentation result of the model is obtained by training on the PaddlePaddle platform.

[0073] To further demonstrate the performance of DBEDNet, experiments were also conducted on the ADE20K dataset for verification. Similar to other works, the input resolutions for training and inference were set to 512×512. Table 7 shows the experimental comparison results with other methods. On the REX3090, DBEDNet-S achieved 38.5% mIoU with 320.1 FPS, and DBEDNet-B achieved 40.9% mIoU with 253.7 FPS. In addition, DBEDNet achieved the fastest inference speed. Figure 7 It shows the visual segmentation results of DBEDNet and RTFormer on the ADE20K dataset, from which it can be seen that DBEDNet performs better in segmenting some object structures. Compared with some previous methods, our DBEDNet is competitive in terms of both speed and accuracy. Table 7 Comparison results of DBEDNet with other methods on the ADE20K dataset

[0074] In Table 7, It indicates that the inference speed of the model is measured after being accelerated by TensorRT. $ indicates that the segmentation result of the model is obtained by training on the PaddlePaddle platform.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those of ordinary skill in the art should understand that the specific implementation manners of the present invention can be modified or equivalently replaced with reference to the above embodiments. Any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention are within the protection scope of the claims pending for approval.

Claims

1. A real-time semantic segmentation method based on a dual-branch encoding-decoding network, characterized in that, It includes the following steps: Step 1: Extract the feature information of the initial image through the CNN branch and the Transformer branch; (1) Input the initial image into the backbone network STDC of the CNN branch. STDC extracts the local hierarchical information of the initial image and outputs feature maps with different resolutions in the S1 - S5 stages respectively. Then, input the feature map output in the S5 stage into the depth strip pyramid pooling module. First, obtain the feature map containing multi-scale strip information and context information, then reduce the number of channels of the output feature map through 1×1 convolution, and then perform an attention operation with the output feature map of the initial image after reducing the number of channels. At the same time, input the feature maps output in the S3 and S4 stages into the feature pooling refinement module to enhance the global semantic and significant semantic information. Finally, perform a decoding operation on the feature map processed by the depth strip pyramid pooling module in the S5 stage and the feature maps processed by the feature pooling refinement module in the S3 and S4 stages to obtain the output feature map of the CNN branch; (2) Use the sum of the upsampled feature maps in the S3 and S4 stages of the CNN branch as the input of the Transformer branch. After extracting the global information through two PoolFormer blocks, obtain the preliminary output P1. Then, sum P1 with the upsampled feature map in the S5 stage. The sum result is further processed through two PoolFormer blocks to extract the global information and obtain the final output P2, which is used as the output feature map of the Transformer branch; Step 2: Use the CNN-Transformer fusion module to fuse the output feature map of the CNN branch and the output feature map of the Transformer branch using an attention mask; Step 3: Input the feature map after CNN-Transformer fusion into the segmentation head for prediction to obtain the segmentation map.

2. The real-time semantic segmentation method based on a dual-branch encoding and decoding network according to claim 1, wherein In Step 1, the operation method of the depth strip pyramid pooling module is as follows: First, perform large kernel pooling on the input feature map with different kernels and different strides to generate feature maps with different resolutions, and use global average pooling to generate the global information of the entire feature map. Then, first use 1×1 convolution on the feature maps with different resolutions to reduce the number of channels and computational complexity, and then perform convolution operations on strips with different kernel sizes to extract strip information at different scales. Next, splice the feature maps with different resolutions after strip convolution and upsampling to obtain the feature map containing multi-scale strip information and context information. Finally, reduce the number of channels of the spliced feature map through 1×1 convolution and perform an attention operation with the output feature map of the initial image after reducing the number of channels.

3. The real-time semantic segmentation method based on a dual-branch encoding and decoding network according to claim 1, wherein In step 1, the operation method of the feature pooling refinement module is as follows: First, the feature maps output in the S3 stage and the S4 stage are respectively subjected to global average pooling and global max pooling operations; then, they respectively pass through a Conv-BN-Sigmoid layer to obtain the max pooling attention map and the average pooling attention map. The max pooling attention map is multiplied by the original feature map, the average pooling attention map is multiplied by the original feature map, and the results of the two multiplications are summed. Finally, a residual connection is made with the feature map of the original input to obtain the output.

4. The real-time semantic segmentation method based on a dual-branch encoding and decoding network according to claim 1, wherein The decoding operation in step 1 is as follows: After the feature map output in the S5 stage is processed by the deep strip pyramid pooling module, it is first upsampled to the same resolution as the feature map output in the S4 stage, and then added to the feature map of the feature map output in the S4 stage processed by the feature pooling refinement module; the result of the addition is upsampled to the same resolution as the feature map output in the S3 stage, and finally added to the feature map of the feature map output in the S3 stage processed by the feature pooling refinement module to obtain the output of the CNN branch.

5. The real-time semantic segmentation method based on a dual-branch encoding and decoding network according to claim 1, characterized in that In step 2, the operation of the CNN-Transformer fusion module is as follows: The output feature maps of the CNN branch and the output feature maps of the Transformer branch are first concatenated in the channel dimension, and then the number of channels is reduced through a Conv-BN-ReLU layer to match the number of channels of the original feature map. The feature map with the reduced number of channels then passes through a Conv-BN-Sigmoid layer to obtain the attention mask α; then, α is directly multiplied by the output feature map of the Transformer branch and a residual connection is made. At the same time, the relative attention mask 1-α is multiplied by the output feature map of the CNN branch and a residual connection is made; finally, the feature images of these two branches after attention and residual processing are pixel-wise summed to obtain the output result.

6. The real-time semantic segmentation method based on a dual-branch encoding and decoding network according to claim 1, characterized in that In step 3, the segmentation head consists of a 3×3 Conv-BN layer and a 1×1 convolutional layer.

Images