A lightweight image semantic segmentation method based on deep learning

Abstract

This invention discloses a lightweight image semantic segmentation method based on deep learning. First, a lightweight image semantic segmentation model, Mobile-SegFormer, is constructed. Then, the constructed lightweight image semantic segmentation model is trained using a pre-segmented set of sample images to obtain a trained lightweight image semantic segmentation model. Finally, the image to be segmented is fed into the trained lightweight image semantic segmentation model, which outputs the segmented image. This invention improves both the SegFormer and MobileNetv3 models, achieving high segmentation accuracy without pre-training on the ImageNet dataset, and exhibits good portability. Experimental comparisons show that our proposed Mobile-SegFormer network has more accurate segmentation results compared to common lightweight networks, and compared to large segmentation networks, it has a smaller parameter count and lower floating-point operations per second, achieving a better balance between Params, FLOPS, and MIoU metrics.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and specifically to a lightweight image semantic segmentation method based on deep learning. Background Technology

[0002] Image semantic segmentation is an important topic in computer vision. Current deep neural network semantic segmentation models are widely studied due to their high segmentation accuracy; however, their large number of parameters and slow inference speed hinder their widespread application in real-world scenarios. For example, applications such as autonomous driving place high demands on the accuracy and timeliness of road scene segmentation. Furthermore, the development of mobile vision applications in recent years has spurred the demand for lightweight semantic segmentation techniques. Although significant progress has been made, current lightweight semantic segmentation networks still suffer from low accuracy and excessive parameter counts. Summary of the Invention

[0003] The present invention addresses the problems of low accuracy and excessive parameters in existing image semantic segmentation methods by providing a lightweight image semantic segmentation method based on deep learning.

[0004] To solve the above problems, the present invention is achieved through the following technical solution:

[0005] A lightweight image semantic segmentation method based on deep learning includes the following steps:

[0006] Step 1: Construct a lightweight image semantic segmentation model;

[0007] The lightweight image semantic segmentation model consists of one convolutional batch normalized activation module, 15 bottleneck residual modules, four multilayer perception modules with gated attention, one spatial pyramid cascade module, one gated attention mechanism module, one pruning module, two 1×1 convolutional modules, and one bilinear interpolation module. The input of the convolutional batch normalized activation module forms the input of the lightweight image semantic segmentation model, and the output of the convolutional batch normalized activation module is connected to the input of the first bottleneck residual module. The 15 bottleneck residual modules are cascaded sequentially. The output of the third bottleneck residual module is connected to the first input of the spatial pyramid cascade module through one multilayer perception module with gated attention, and the output of the sixth bottleneck residual module is connected to the spatial pyramid cascade module through one multilayer perception module with gated attention. The second input of the cascaded module and the output of the twelfth bottleneck residual module are connected to the third input of the spatial pyramid cascaded module through a multilayer perception module with fused gated attention. The output of the fifteenth bottleneck residual module is connected to the fourth input of the spatial pyramid cascaded module through a multilayer perception module with fused gated attention. The output of the spatial pyramid cascaded module is connected to the input of the gated attention mechanism module, the output of the gated attention mechanism module is connected to the input of the pruning module, the output of the pruning module is connected to the input of the first 1×1 convolution module, the output of the first 1×1 convolution module is connected to the input of the second 1×1 convolution module, the output of the second 1×1 convolution module is connected to the input of the bilinear interpolation module, and the output of the bilinear interpolation module forms the output of the lightweight image semantic segmentation model.

[0008] Step 2: Train the lightweight image semantic segmentation model constructed in Step 1 using the pre-segmented sample image set to obtain the trained lightweight image semantic segmentation model.

[0009] Step 3: Feed the image to be segmented into the lightweight image semantic segmentation model trained in Step 2. The trained lightweight image semantic segmentation model outputs the segmented image.

[0010] The bottleneck residual modules of the aforementioned lightweight image semantic segmentation model are divided into basic bottleneck residual modules and CA attention bottleneck residual modules; the first bottleneck residual modules to the third bottleneck residual modules and the seventh bottleneck residual modules to the tenth bottleneck residual modules are basic bottleneck residual modules; the fourth bottleneck residual modules to the sixth bottleneck residual modules and the eleventh bottleneck residual modules to the fifteenth bottleneck residual modules are CA attention bottleneck residual modules.

[0011] The aforementioned basic bottleneck residual module consists of two convolutional batch normalized activation modules, one spatial pyramid fusion module, and one fusion module. The input of the first convolutional batch normalized activation module forms the input of the basic bottleneck residual module. The output of the first convolutional batch normalized activation module is connected to the input of the spatial pyramid fusion module. The output of the spatial pyramid fusion module is connected to the input of the second convolutional batch normalized activation module. The output of the second convolutional batch normalized activation module and the input of the first convolutional batch normalized activation module are simultaneously connected to the input of the fusion module. The output of the fusion module forms the output of the basic bottleneck residual module.

[0012] The aforementioned CA attention bottleneck residual module consists of two convolutional batch normalized activation modules, one spatial pyramid fusion module, one CA attention mechanism module, and one fusion module. The input of the first convolutional batch normalized activation module forms the input of the CA attention bottleneck residual module. The output of the first convolutional batch normalized activation module is connected to the input of the spatial pyramid fusion module. The output of the spatial pyramid fusion module is connected to the input of the CA attention mechanism module. The output of the CA attention mechanism module is connected to the input of the second convolutional batch normalized activation module. The output of the second convolutional batch normalized activation module and the input of the first convolutional batch normalized activation module are simultaneously connected to the input of the fusion module. The output of the fusion module forms the output of the CA attention bottleneck residual module.

[0013] In the bottleneck residual module, the spatial pyramid fusion module consists of five depthwise separable dilated convolutional modules, one max pooling module, one upsampling module, one fusion module, one batch normalization module, and one activation module. The inputs of the first, second, third, and fourth depthwise separable dilated convolutional modules, along with the input of the max pooling module, collectively form the input of the spatial pyramid fusion module. The output of the max pooling module is connected to the input of the fifth depthwise separable dilated convolutional module, and the output of the fifth depthwise separable dilated convolutional module is connected to the input of the upsampling module. The inputs and outputs of the first, second, third, and fourth depthwise separable dilated convolutional modules, as well as the output of the upsampling module, are simultaneously connected to the input of the fusion module. The output of the fusion module is connected to the input of the batch normalization module, and the output of the batch normalization module is connected to the input of the activation module. The output of the activation module forms the output of the spatial pyramid fusion module.

[0014] The lightweight image semantic segmentation model described above, with its fused gated attention multilayer perception module, consists of a gated attention mechanism module, an image size flattening module, a layer normalization module, a fully connected module, an activation module, and an upsampling module. The input of the gated attention mechanism module forms the input of the fused gated attention multilayer perception module. The output of the gated attention mechanism module is connected to the input of the image size flattening module. The output of the image size flattening module is connected to the input of the layer normalization module. The output of the layer normalization module is connected to the input of the fully connected module. The output of the fully connected module is connected to the input of the activation module. The output of the activation module is connected to the input of the upsampling module. The output of the upsampling module forms the output of the fused gated attention multilayer perception module.

[0015] The spatial pyramid concatenated module of the aforementioned lightweight image semantic segmentation model consists of four depthwise separable dilated convolution modules, one stitching module, one batch normalization module, and one activation module.

[0016] The input of the first depthwise separable dilated convolution module forms the first input of the spatial pyramid cascade module; the input of the second depthwise separable dilated convolution module forms the second input; the input of the third depthwise separable dilated convolution module forms the third input; and the input of the fourth depthwise separable dilated convolution module forms the fourth input. The outputs of the first, second, third, and fourth depthwise separable dilated convolution modules are simultaneously connected to the input of the splicing module. The output of the splicing module is connected to the input of the batch normalization module. The output of the batch normalization module is connected to the input of the activation module. The output of the activation module forms the output of the spatial pyramid cascade module.

[0017] Compared with existing technologies, this invention proposes a lightweight image semantic segmentation model, Mobile-SegFormer, based on an improved SegFormer network and a MobileNetv3 model with fused CA attention mechanism, to achieve lightweight semantic segmentation. Mobile-SegFormer uses the improved MobileNetv3 as the encoder for feature extraction, effectively reducing the number of model parameters. To further improve model accuracy, a spatial pyramid fusion module is constructed in the bottleneck residual module as a basic module for multi-scale learning, improving the model's segmentation accuracy. Simultaneously, Mobile-SegFormer improves the SegFormer decoder by proposing a multi-layer perception module with fused gated attention combined with a spatial pyramid cascade module, further enhancing the model's ability to extract semantic information features. Experiments were conducted on the Cityscape, Camvid, and ADE20K datasets. The results show that the proposed Mobile-SegFormer model, without using ImageNet dataset pre-trained weights, achieves higher MIoU values ​​on the three experimental datasets compared to the original model, with fewer parameters and lower FLOPS. Attached Figure Description

[0018] Figure 1 is a schematic diagram of the lightweight image semantic segmentation model (Mobile-SegFormer).

[0019] Figure 2 The diagram shows the structural principle of the bottleneck residual module (BR); (a) basic bottleneck residual module, (b) CA attention bottleneck residual module.

[0020] Figure 3 This is a structural schematic diagram of the Space Pyramid Fusion Module (SPM).

[0021] Figure 4 This is a schematic diagram of the structure of a multilayer perceptron module (MLP).

[0022] Figure 5 This is a schematic diagram of the structure of a Space Pyramid Cascaded Module (SPC).

[0023] Figure 6 Visualizations of different algorithms on the Cityscape dataset: (a) Input; (b) Ground Truth; (c) SegFormer_B0; (d) SegFormer_MobileNetv3; (e) Ours.

[0024] Figure 7 Params and MIoU plots for different models on the Cityscape dataset.

[0025] Figure 8 FLOPS and MIoU plots of different models on the Cityscape dataset. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific examples.

[0027] A lightweight image semantic segmentation method based on deep learning includes the following steps:

[0028] Step 1: Construct a lightweight image semantic segmentation model.

[0029] Referring to Figure 1, the lightweight image semantic segmentation model consists of one convolutional batch normalized activation module (ConvBNACT), 15 bottleneck residual modules (BR), four multilayer perceptron (MLP) modules with fused gated attention, one spatial pyramid cascade module (SPC), one gated attention mechanism module (GCTB0), one pruning module (Dropout), two 1×1 convolutional modules (1X1Conv), and one bilinear interpolation module (Interpolate). The input of the convolutional batch normalized activation module forms the input of the lightweight image semantic segmentation model, and the output of the convolutional batch normalized activation module is connected to the input of the first bottleneck residual module. Fifteen bottleneck residual modules are connected in series. The output of the third bottleneck residual module is connected to the first input of the spatial pyramid cascade module through a multilayer perception module with gated attention. The output of the sixth bottleneck residual module is connected to the second input of the spatial pyramid cascade module through a multilayer perception module with gated attention. The output of the twelfth bottleneck residual module is connected to the third input of the spatial pyramid cascade module through a multilayer perception module with gated attention. The output of the fifteenth bottleneck residual module is connected to the fourth input of the spatial pyramid cascade module through a multilayer perception module with gated attention. The output of the spatial pyramid cascade module is connected to the input of the gated attention mechanism module. The output of the gated attention mechanism module is connected to the input of the pruning module. The output of the pruning module is connected to the input of the first 1×1 convolutional module. The output of the first 1×1 convolutional module is connected to the input of the second 1×1 convolutional module. The output of the second 1×1 convolutional module is connected to the input of the bilinear interpolation module. The output of the bilinear interpolation module forms the output of the lightweight image semantic segmentation model.

[0030] The lightweight image semantic segmentation model of this invention is an improvement upon SegFormer, which is based on an encoder and decoder structure. The encoder consists of one convolutional batch normalized activation module and 15 bottleneck residual modules. The decoder consists of four multilayer perceptron modules with fused gated attention, one spatial pyramid cascade module, one gated attention mechanism module, one pruning module, two 1×1 convolutional modules, and one bilinear interpolation module.

[0031] In the encoder of the lightweight image semantic segmentation model, the convolutional batch normalized activation module preprocesses the input image. Then, the preprocessed image is fed into four feature extraction networks with different channels and sizes, which are composed of 15 bottleneck residual modules. Specifically, the first feature extraction network is formed by the first bottleneck residual module to the third bottleneck residual module; the second feature extraction network is formed by the fourth bottleneck residual module to the sixth bottleneck residual module; the third feature extraction network is formed by the seventh bottleneck residual module to the twelfth bottleneck residual module; and the fourth feature extraction network is formed by the thirteenth bottleneck residual module to the fifteenth bottleneck residual module. This results in four feature maps with different channels and sizes.

[0032] The bottleneck residual module is an improvement on the ResNet structure. It extracts features from the input feature map through a convolutional batch normalization activation module. After feature extraction by the convolutional batch normalization activation module, it is sent to the spatial pyramid fusion module to extract features again. The CA parameters in Table 1 determine whether to perform CA attention for feature extraction. Then, it is sent to another convolutional batch normalization activation module to perform residual connection with the original feature map before output.

[0033] In this invention, the 15 bottleneck residual modules are divided into two categories: one category is basic bottleneck residual modules, of which the first to third bottleneck residual modules and the seventh to tenth bottleneck residual modules are basic bottleneck residual modules; the other category is CA attention bottleneck residual modules, of which the fourth to sixth bottleneck residual modules and the eleventh to fifteenth bottleneck residual modules are CA attention bottleneck residual modules.

[0034] Basic bottleneck residual module such as Figure 2As shown in (a), it consists of two convolutional batch normalized activation modules, one spatial pyramid fusion module, and one fusion module. The input of the first convolutional batch normalized activation module forms the input of the basic bottleneck residual module. The output of the first convolutional batch normalized activation module is connected to the input of the spatial pyramid fusion module. The output of the spatial pyramid fusion module is connected to the input of the second convolutional batch normalized activation module. The output of the second convolutional batch normalized activation module and the input of the first convolutional batch normalized activation module are simultaneously connected to the input of the fusion module. The output of the fusion module forms the output of the basic bottleneck residual module.

[0035] CA attention bottleneck residual module, such as Figure 2 As shown in (b), it consists of two convolutional batch normalized activation modules, one spatial pyramid fusion module, one CA attention mechanism module, and one fusion module. The input of the first convolutional batch normalized activation module forms the input of the CA attention bottleneck residual module. The output of the first convolutional batch normalized activation module is connected to the input of the spatial pyramid fusion module. The output of the spatial pyramid fusion module is connected to the input of the CA attention mechanism module. The output of the CA attention mechanism module is connected to the input of the second convolutional batch normalized activation module. The output of the second convolutional batch normalized activation module and the input of the first convolutional batch normalized activation module are simultaneously connected to the input of the fusion module. The output of the fusion module forms the output of the CA attention bottleneck residual module.

[0036] To enable the bottleneck residual module to have a larger receptive field and a stronger ability to capture global features, a spatial pyramid fusion module consisting of depthwise separable dilated convolutions and max pooling with dilation rates of 1, 2, 4, and 8 is used. Upsampling in the spatial pyramid fusion module uses bilinear interpolation, ensuring that multiple feature maps can be summed at the end, thus obtaining more semantic feature information with only a small amount of memory and parameters. The spatial pyramid fusion module effectively increases the model's receptive field, as shown in Equation 1, and efficiently extracts image features.

[0037] N = k + (k-1)(d-1)(1)

[0038] In the formula, N represents the receptive field, k represents the original convolutional kernel size, and d represents the dilation rate. For deep convolutional networks, a larger receptive field means more global information is contained, effectively reducing the loss of spatial information.

[0039] In this invention, the spatial pyramid fusion module in the bottleneck residual module is as follows: Figure 3As shown, the spatial pyramid fusion module consists of five depthwise separable dilated convolutional modules (DSDConv), one max pooling module, one upsampling module, one fusion module, one batch normalization (BN) module, and one activation module (ReLU). The dilation rates of the first and fifth DSDConv modules are 1, the second DSDConv module has a dilation rate of 2, the third DSDConv module has a dilation rate of 4, and the fourth DSDConv module has a dilation rate of 8. The inputs of the first, second, third, and fourth DSDConv modules, along with the input of the max pooling module, together form the input of the spatial pyramid fusion module. The output of the max pooling module is connected to the input of the fifth DSDConv module, and the output of the fifth DSDConv module is connected to the input of the upsampling module. The inputs and outputs of the first, second, and third depthwise separable dilated convolutional modules, the fourth depthwise separable dilated convolutional module, and the output of the upsampling module are all connected to the input of the fusion module. The output of the fusion module is connected to the input of the batch normalization module, the output of the batch normalization module is connected to the input of the activation module, and the output of the activation module forms the output of the spatial pyramid fusion module.

[0040] The network parameters of the encoder are shown in Table 1.

[0041] Table 1 Network parameters of the encoder

[0042]

[0043] Table 1 shows the network parameters only for the Cityscape dataset; the only difference between the Camvid and ADE20K datasets is the input size. For the Cityscape dataset, the 2048×1024 resolution image was cropped to 1024×512 resolution before being input into the model network. This invention improves upon the original MobileNetv3 network structure by replacing the SE attention mechanism with the CA attention mechanism and replacing the ResidualBlock module with our proposed bottleneck residual module BottleneckResidualBlock. Other structural features remain the same as the original MobileNetv3-Large structure.

[0044] In the decoder of the lightweight image semantic segmentation model, four multilayer perception modules with fused gated attention apply a gated attention mechanism to four feature maps to further focus on the effective semantic information in the feature maps. The length and width of the feature maps passed through the gated attention mechanism are flattened, turning the two-dimensional image into a one-dimensional feature queue. Then, a fully connected operation is performed on each of the four feature queues to adjust the number of channels in the feature maps. The adjusted feature queues are then normalized to avoid gradient explosion. The normalized feature maps are then activated by the ReLU function. Finally, the feature maps of different layers are reduced to 1 / 4 of the original image size through bilinear interpolation. The feature maps output by the four multilayer perception modules with fused gated attention are further extracted through a spatial pyramid cascade module. Since low semantic information feature maps contain less effective information and high semantic information feature maps contain more semantic information, DSDConv with a lower dilation rate is used to process low semantic information feature maps, and DSDConv with a higher dilation rate is used to process high semantic information feature maps. This allows the output of the final spatial pyramid cascade module to have higher-dimensional channel information. Then, the gated attention mechanism module focuses on the semantic information of the feature map that should be of more attention. The pruning module then prunes the feature map to avoid overfitting and reduces the number of channels with a 1×1 convolution. Another 1×1 convolution adjusts the number of channels to the number of segmentation categories. Finally, the bilinear interpolation module outputs a feature map of the same size as the original image as the prediction result.

[0045] To achieve better feature fusion, we pass the feature map of the input multilayer perception module with fusion gated attention through the GCT_B0 gated attention mechanism, and perform image size flattening operation on the feature map to synthesize the height and width of the image. We then normalize the synthesized feature map using layer normalization, and extract features through a fully connected module to reduce the ICS (Internal Convariate Shift) phenomenon. We use the ReLU activation module function for activation, which can effectively avoid the gradient vanishing problem. Finally, we upsample the feature map to 1 / 4 of the original image size using bilinear interpolation before outputting it.

[0046] In this invention, a multi-layer perception module that integrates gated attention is as follows: Figure 4As shown, the system consists of a gated attention mechanism module (GCTB0), an image size flattening module (Flatten), a layer normalization module (Layer Normalization), a fully connected module (Linear), an activation module (ReLU), and an upsampling module (Upsample). The input of the gated attention mechanism module forms the input of the multilayer perception module with fused gated attention. The output of the gated attention mechanism module is connected to the input of the image size flattening module, the output of the image size flattening module is connected to the input of the layer normalization module, the output of the layer normalization module is connected to the input of the fully connected module, the output of the fully connected module is connected to the input of the activation module, the output of the activation module is connected to the input of the upsampling module, and the output of the upsampling module forms the output of the multilayer perception module with fused gated attention.

[0047] The four-layer feature maps output by the multilayer perception module with fused gating attention are processed by a spatial pyramid concatenation module, which consists of 3×3 depthwise separable dilated convolutions with dilation rates of 1, 2, 4 and 8. The convolution results are then concatenated, normalized by batch normalization, and activated by the ReLU activation function before being output.

[0048] In this invention, the spatial pyramid cascade module is as follows: Figure 5 As shown, it consists of four depthwise separable dilated convolutional modules (DSDConv), one concatenation module, one batch normalization (BN) module, and one activation module (ReLU). The dilation rate of the first DSDConv module is 1, the second DSDConv module's dilation rate is 2, the third DSDConv module's dilation rate is 4, and the fourth DSDConv module's dilation rate is 8. The input of the first DSDConv module forms the first input of the spatial pyramid cascade module, the input of the second DSDConv module forms the second input, the input of the third DSDConv module forms the third input, and the input of the fourth DSDConv module forms the fourth input. The outputs of the first, second, third, and fourth depthwise separable dilated convolutional modules are simultaneously connected to the input of the stitching module. The output of the stitching module is connected to the input of the batch normalization module. The output of the batch normalization module is connected to the input of the activation module. The output of the activation module forms the output of the spatial pyramid cascade module.

[0049] Step 2: Train the lightweight image semantic segmentation model constructed in Step 1 using the pre-segmented sample image set to obtain the trained lightweight image semantic segmentation model.

[0050] Step 3: Feed the image to be segmented into the lightweight image semantic segmentation model trained in Step 2. The trained lightweight image semantic segmentation model outputs the segmented image.

[0051] The effectiveness and robustness of the algorithm proposed in this invention will be demonstrated through experiments below.

[0052] The experimental hardware platform consisted of a Tesla V100 GPU with 32GB of video memory and a Gold Intel processor with 24 cores. The experimental software environment used Ubuntu 16.04, Python 3.7.4, the deep learning framework PaddlePaddle 2.4.0, and gcc 7.3.0.

[0053] The experiments used the Cityscapes, Camvid, and ADE20K datasets. For the Cityscapes, Camvid, and ADE20K training sets, random cropping was employed, cropping the training set images to resolutions of 1024×512, 960×720, and 480×480, respectively. Random scaling, random horizontal flipping, random padding, and random distortion were used within the range of 0.5 to 2.0 to increase the diversity of the training set images and the complexity of the input images, thereby ensuring the model can adapt to more complex environments and improving its robustness.

[0054] Cityscapes is a dataset collected from 50 cities in Germany and neighboring countries, encompassing street scenes in spring, summer, and autumn. It contains 5000 images with a resolution of 2048×1024, with 2975 images in the training set, 500 in the validation set, and 1525 in the test set. It features dense pixel annotations across 19 categories. For the Cityscapes dataset, the AdamW optimizer was used to update the network parameters, with a learning rate set to 4×10⁻⁶. -4 The final learning rate is 1×10. -6 The weight decay value is 0.0125, the momentum is 1, the batch size is set to 4, and training is started using the Warmup method with 1500 Warmup iterations and an initial learning rate of 1×10⁻⁶. -6 The number of iterations, Epoch, is set to 175.

[0055] Camvid is a road and driving scene image segmentation dataset from Cambridge, consisting of 701 images with a resolution of 960x720. The training set contains 367 images, the validation set 101 images, and the test set 233 images, with dense pixel annotations for 32 categories. For the Camvid dataset, the AdamW optimizer was used to update the network parameters, with a learning rate set to 8×10⁻⁶. -4 The weight decay value is 0.0125, the momentum is 1, the batch size is set to 4, and the number of iterations (Epoch) is set to 175.

[0056] ADE20K is a dataset released by MIT for semantic segmentation, object recognition, and other applications. It contains 25,562 images of varying resolutions: 20,210 for training, 2,000 for validation, and 3,352 for testing. It features dense pixel annotations across 150 categories. For the ADE20K dataset, the AdamW optimizer is used to update the network parameters, with a learning rate set to 8×10⁻⁶. -4 The final learning rate is 1×10. -6 The weight decay value is 0.0125, the momentum is 1, the batch size is set to 16, and training is started using the Warmup method with 1500 Warmup iterations and an initial learning rate of 1×10⁻⁶. -6 The iterations are set to 150K.

[0057] The experiment uses commonly used evaluation metrics in semantic segmentation: total number of parameters (params), floating-point operations per second (FLOPS), and mean intersection over union (MIoU). FLOPS and params describe the complexity of the model; higher values ​​indicate a more complex model and higher computational cost. A higher MIoU value indicates better segmentation performance.

[0058] I. Ablation Experiment

[0059] Ablation experiments were designed for each module to verify the effectiveness of each module. The ablation experiments were evaluated on the Cityscape dataset and the Camvid dataset. For ease of representation, the Cityscape dataset is labeled as (1) and the Camvid dataset is labeled as (2). The corresponding input sizes are 1024×512 and 960×720, respectively, and the number of iterations (Epoch) is 175.

[0060] 1) Improved attention mechanisms

[0061] Table 2 Comparison of evaluation indicators for different attention mechanisms

[0062]

[0063] As shown in Table 2, for both the Cityscape and Camvid datasets, replacing the SE attention mechanism with the CA attention mechanism effectively reduced the number of model parameters and resulted in higher MIoU values. Therefore, this invention selected the lighter CA attention mechanism as the attention module for the improved MobileNetv3.

[0064] 2) Backbone network improvement

[0065] Table 3 Comparison of evaluation indicators for different backbone networks

[0066]

[0067]

[0068] As shown in Table 3, for the Cityscape dataset, MobileNetv3 with the addition of the SPM Block, combined with the original SE attention mechanism, achieves a MIoU of 65.76%. Our proposed SPM Block improves the MIoU of the model by 2.76% with only 0.41M Params and 0.84G FLOPS, demonstrating the good performance of the SPM module. Meanwhile, our improved Our_MobileNetv3 model achieves an MIoU of 66.29%. Compared to the unimproved MobileNetv3 backbone, our improved MobileNetv3 backbone improves the MIoU by 3.29% while reducing the number of Params by 0.81M. For the Camvid dataset, MobileNetv3 with the addition of the SPM Block combined with the original SE attention mechanism achieves an MIoU of 54.59%. Our proposed SPM Block improves the MIoU by 2.32% with only an increase of 0.4M Params and 1.08G FLOPS. However, the MIoU of our improved Our_MobileNetv3 model using the CA attention mechanism is lower than that using the SE attention mechanism. Weighing the Params values ​​of the CA and SE attention mechanisms, we ultimately choose the CA attention mechanism as the attention mechanism for improving the MobileNetv3 network.

[0069] 3) MLP Layer Improvement

[0070] Table 4 Comparison of Evaluation Indicators for the Improved MLP

[0071]

[0072] As shown in Table 4, for the Cityscape dataset, adding the improved MLP to SegFormer_B0 improved MIoU by 0.6% without increasing the model's Params and FLOPS, while using the improved MLP in SegFormer_MobileNetv3 improved MIoU by 0.62% without increasing the model's Params and FLOPS. For the Camvid dataset, adding the improved MLP to SegFormer_B0 improved MIoU by 2.98%, while using the improved MLP in SegFormer_MobileNetv3 improved MIoU by 1.43% without increasing the model's Params and FLOPS.

[0073] 4) Add SPC Block

[0074] Table 5 Comparison of Evaluation Metrics for Adding SPC Blocks

[0075]

[0076]

[0077] As shown in Table 5, for the Cityscape dataset, SegFormer_B0 improved MIoU by 1.23% with only a 0.01M increase in model params and a 0.44G increase in FLOPS using the SPC Block, while SegFormer_MobileNetv3 improved MIoU by 1.21% with only a 0.02M increase in model params and a 0.44G increase in FLOPS using the SPC Block. For the Camvid dataset, SegFormer_B0 improved MIoU by 3.53% with only a 0.01M increase in model params and a 0.57G increase in FLOPS using the SPC Block, while SegFormer_MobileNetv3 improved MIoU by 1.15% with only a 0.02M increase in model params and a 0.57G increase in FLOPS using the SPC Block.

[0078] II. Comparison Experiment of Different Models

[0079] The model proposed in this invention was compared with 10 lightweight image semantic segmentation models, including SegFormer, FCN, Deeplabv3, ESPNetV2, BiSeNetV2, O-CRNet, Topformer, LRASPP, and PSPNet, on the Cityscape, Camvid, and ADE20K datasets. The comparison results are shown in Table 6.

[0080] Table 6 Comparison of different models on the experimental dataset.

[0081]

[0082] As shown in Table 6, for the Cityscape dataset, compared to other classic lightweight networks such as ESPNetV2 and BiSeNetV2, the SegFormer model of this invention has a superior MIoU value. It not only improves accuracy by 7.77% and 5.74% respectively compared to the original models SegFormer_B0 and SegFormer_MobileNetv3, but also reduces the number of parameters by 1.32M and 0.79M respectively. The floating-point operations per second (FLOPS) are increased by 1.27G compared to SegFormer_MobileNetv3, and compared to the original S... The SegFormer_B0 model reduced the MIoU by 0.95G. Compared to other models with large parameter counts and high FLOPS, such as Deeplabv3_Resnet50, the MIoU of the SegFormer model of this invention lagged behind by 3.77%, but its Params were only 8.95% and FLOPS were only 5.55% of Deeplabv3_Resnet50. Compared to medium-sized models such as FCN_HRNetW18, the SegFormer model of this invention has fewer Params, fewer FLOPS, and higher MIoU. For the Camvid and ADE20K datasets, the SegFormer model of this invention also balances the three evaluation metrics of Params, FLOPS, and MIoU. Compared with the SegFormer_B0 model, it improves the MIoU value by 3.54% and 3.35% on the Camvid and ADE20K datasets, respectively, and improves the MIoU value by 2.12% and 1.85% compared with SegFormer_MobileNetv3, respectively. In the end, the model of this invention achieves a good balance among the three evaluation metrics and can better meet the needs of real-world scenarios.

[0083] To more intuitively compare the segmentation performance of SegFormer before and after the improvement, five sets of segmentation images were set up on the Cityscape dataset for visual comparison and analysis, such as... Figure 6As shown in the first row, it can be observed that neither the SegFormer_B0 nor the SegFormer_MobileNetv3 networks effectively distinguished the two pedestrians on the right, while the model of this invention effectively segmented both pedestrians. The SegFormer_B0 and SegFormer_MobileNetv3 networks exhibited omissions in segmenting the buildings on the right and their handling of vehicles on the left was also relatively coarse. The model of this invention shows improvements in these areas. In the third row, when segmenting the people and fences on both sides, the SegFormer_B0 and SegFormer_MobileNetv3 networks suffered from omissions, discontinuous segmentation, and blurred details. The model of this invention achieved significantly more continuous and refined segmentation results.

[0084] Figure 7 and Figure 8 The visualization results of various metrics of different models on the Cityscape dataset are presented. By observing the points in the image, it can be found that the model proposed in this invention still maintains a good MIoU index with a low number of parameters (Params and FLOPS).

[0085] In summary, to achieve a balance between lightweight image segmentation and model accuracy, this invention improves both the SegFormer and MobileNetv3 models. It achieves high segmentation accuracy without pre-training on the ImageNet dataset and offers good portability. Experimental comparisons show that our proposed Mobile-SegFormer network, compared to common lightweight networks, delivers more accurate segmentation results. Compared to large segmentation networks, it has a smaller parameter count and lower floating-point operations per second, effectively balancing Params, FLOPS, and MIoU metrics.

[0086] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A lightweight image semantic segmentation method based on deep learning, characterized in that, The steps include the following: Step 1: Construct a lightweight image semantic segmentation model; The lightweight image semantic segmentation model consists of one convolutional batch normalized activation module, 15 bottleneck residual modules, four multilayer perception modules with gated attention, one spatial pyramid cascade module, one gated attention mechanism module, one pruning module, two 1×1 convolutional modules, and one bilinear interpolation module. The spatial pyramid cascade module consists of four depthwise separable dilated convolutional modules, one stitching module, one batch normalization module, and one activation module. The input of the first depthwise separable dilated convolutional module forms the first input of the spatial pyramid cascade module, the input of the second depthwise separable dilated convolutional module forms the second input, the input of the third depthwise separable dilated convolutional module forms the third input, and the input of the fourth depthwise separable dilated convolutional module forms the fourth input. The outputs of the first, second, third, and fourth depthwise separable dilated convolutional modules are simultaneously connected to the input of the stitching module. The output of the stitching module is connected to the input of the batch normalization module. The output of the batch normalization module is connected to the input of the activation module. The output of the activation module forms the output of the spatial pyramid cascade module. The input to the convolutional batch normalized activation module forms the input to the lightweight image semantic segmentation model. The output of the convolutional batch normalized activation module is connected to the input of the first bottleneck residual module. Fifteen bottleneck residual modules are cascaded sequentially. The output of the third bottleneck residual module is connected to the first input of the spatial pyramid cascade module through a multilayer perception module with fused gated attention. The output of the sixth bottleneck residual module is connected to the second input of the spatial pyramid cascade module through a multilayer perception module with fused gated attention. The output of the twelfth bottleneck residual module is connected to the spatial pyramid cascade module through a multilayer perception module with fused gated attention. The third input, the output of the fifteenth bottleneck residual module, is connected to the fourth input of the spatial pyramid cascade module through a multilayer perception module with fused gated attention. The output of the spatial pyramid cascade module is connected to the input of the gated attention mechanism module, the output of the gated attention mechanism module is connected to the input of the pruning module, the output of the pruning module is connected to the input of the first 1×1 convolution module, the output of the first 1×1 convolution module is connected to the input of the second 1×1 convolution module, the output of the second 1×1 convolution module is connected to the input of the bilinear interpolation module, and the output of the bilinear interpolation module forms the output of the lightweight image semantic segmentation model. Step 2: Train the lightweight image semantic segmentation model constructed in Step 1 using the pre-segmented sample image set to obtain the trained lightweight image semantic segmentation model. Step 3: Feed the image to be segmented into the lightweight image semantic segmentation model trained in Step 2. The trained lightweight image semantic segmentation model outputs the segmented image.

2. The lightweight image semantic segmentation method based on deep learning according to claim 1, characterized in that, The bottleneck residual modules are the basic bottleneck residual modules and the CA attention bottleneck residual modules; the first bottleneck residual modules to the third bottleneck residual modules and the seventh bottleneck residual modules to the tenth bottleneck residual modules are the basic bottleneck residual modules; the fourth bottleneck residual modules to the sixth bottleneck residual modules and the eleventh bottleneck residual modules to the fifteenth bottleneck residual modules are the CA attention bottleneck residual modules.

3. The lightweight image semantic segmentation method based on deep learning according to claim 2, characterized in that, The basic bottleneck residual module consists of two convolutional batch normalized activation modules, one spatial pyramid fusion module, and one fusion module; The input of the first convolutional batch normalized activation module forms the input of the basic bottleneck residual module. The output of the first convolutional batch normalized activation module is connected to the input of the spatial pyramid fusion module. The output of the spatial pyramid fusion module is connected to the input of the second convolutional batch normalized activation module. The output of the second convolutional batch normalized activation module and the input of the first convolutional batch normalized activation module are simultaneously connected to the input of the fusion module. The output of the fusion module forms the output of the basic bottleneck residual module.

4. A lightweight image semantic segmentation method based on deep learning according to claim 2, characterized in that, CA The attention bottleneck residual module consists of two convolutional batch normalized activation modules, one spatial pyramid fusion module, one CA attention mechanism module, and one fusion module. The input of the first convolutional batch normalized activation module forms the input of the CA attention bottleneck residual module. The output of the first convolutional batch normalized activation module is connected to the input of the spatial pyramid fusion module. The output of the spatial pyramid fusion module is connected to the input of the CA attention mechanism module. The output of the CA attention mechanism module is connected to the input of the second convolutional batch normalized activation module. The output of the second convolutional batch normalized activation module and the input of the first convolutional batch normalized activation module are simultaneously connected to the input of the fusion module. The output of the fusion module forms the output of the CA attention bottleneck residual module.

5. A lightweight image semantic segmentation method based on deep learning according to claim 3 or 4, characterized in that, The spatial pyramid fusion module consists of 5 depthwise separable dilated convolution modules, 1 max pooling module, 1 upsampling module, 1 fusion module, 1 batch normalization module, and 1 activation module. The inputs of the first, second, third, and fourth depthwise separable dilated convolutional modules, along with the input of the max pooling module, collectively form the input of the spatial pyramid fusion module. The output of the max pooling module is connected to the input of the fifth depthwise separable dilated convolutional module, and the output of the fifth depthwise separable dilated convolutional module is connected to the input of the upsampling module. The inputs and outputs of the first, second, third, and fourth depthwise separable dilated convolutional modules, as well as the output of the upsampling module, are simultaneously connected to the input of the fusion module. The output of the fusion module is connected to the input of the batch normalization module, and the output of the batch normalization module is connected to the input of the activation module. The output of the activation module forms the output of the spatial pyramid fusion module.

6. A lightweight image semantic segmentation method based on deep learning according to claim 1, characterized in that, The multilayer perception module with gated attention consists of a gated attention mechanism module, an image size flattening module, a layer normalization module, a fully connected module, an activation module, and an upsampling module. The input of the gated attention mechanism module forms the input of the multilayer perception module that integrates gated attention. The output of the gated attention mechanism module is connected to the input of the image size flattening module. The output of the image size flattening module is connected to the input of the layer normalization module. The output of the layer normalization module is connected to the input of the fully connected module. The output of the fully connected module is connected to the input of the activation module. The output of the activation module is connected to the input of the upsampling module. The output of the upsampling module forms the output of the multilayer perception module that integrates gated attention.

Images