A semantic segmentation method based on improved ASPP and fusion module in a complex scene

Abstract

The application aims to provide a semantic segmentation method based on improved ASPP and fusion module in a complex scene, comprising the following steps: building a Deeplabv3+ model under a Pytorch framework; designing a RA-ASPP module based on a traditional ASPP structure; designing a CBB module; replacing the ASPP module in the Deeplabv3+ model with the RA-ASPP module, and replacing the 3*3 standard convolution in the decoding fusion part with the CBB module; training the model by using a frozen training method, and respectively using Xception and MobileNetV2 as the backbone part to perform an ablation experiment on a PASCAL VOC07+12 dataset, and comparing the performance of different models. The improved module provided in the application improves the segmentation effect of the Deeplabv3+, and different backbone parts also provide more choices for the semantic segmentation task in a complex scene.

Description

Technical Field

[0001] The present invention relates to an image processing method. Background Technology

[0002] The main task of image semantic segmentation is to understand the semantics of an image and segment objects with different semantic meanings. Semantics refers to the meaning represented by objects in an image. For example, in complex scenes, pedestrians, vehicles, roads, and obstacles all have different meanings. It is a process of dividing a digital image into multiple sets (pixel sets), where pixels with the same label in the segmented image have similar characteristics.

[0003] Semantic segmentation accuracy directly impacts the precision of scene understanding in unmanned systems. It plays a crucial role in autonomous driving, robotic navigation, drone landing systems, and smart security monitoring. For example, in autonomous driving, cars require image recognition and segmentation capabilities to fully understand environmental changes during operation; in medical image diagnosis, it helps doctors analyze images of patients' lesions, improving diagnostic efficiency. Higher segmentation accuracy leads to better subsequent image processing tasks; image segmentation is a prerequisite for image processing and a fundamental task with significant importance in computer vision. However, complex scenes often involve unstructured objects, diverse targets, irregular shapes, varying lighting, and object occlusion, all posing significant challenges to semantic segmentation accuracy. For instance, segmenting small objects and small, strip-shaped regions of objects is difficult, such as the legs of tables and chairs in indoor scenes, or thin strips like utility poles and streetlights in road scenes. Distinguishing between different targets with similar appearances and the same target with different appearances is also challenging. For example, ground surfaces with similar textures and appearance to trees are mistakenly classified as trees. They are not very adaptable to changes in lighting and seasons in complex environments, exhibiting poor robustness.

[0004] With the development of deep convolutional neural networks, many computer vision tasks have seen significant improvements in performance. Semantic segmentation, as one of the most important tasks in computer vision, has also made considerable progress thanks to deep learning technologies. Semantic segmentation is a challenging task in computer vision because foreground and background objects in reality are highly complex, often exhibiting different shapes, sizes, and colors. Although deep convolutional neural networks have significantly improved the performance of traditional semantic segmentation tasks, there is still a gap in handling complex semantic segmentation scenarios applied to real-world situations. The increasing complexity of segmentation scenes places increasingly stringent demands on segmentation algorithms. In scene perception technology, efficiently and accurately extracting target classification information from environmental information is extremely difficult, mainly for two reasons: firstly, accurate segmentation is impossible for objects with similar attributes; secondly, semantic segmentation relies on object shape, which demonstrates high segmentation accuracy for static objects with stable or relatively unchanged shapes, but becomes inaccurate when the object is moving or undergoes frequent and significant shape changes. Therefore, further in-depth research is needed to improve the accuracy of image semantic segmentation. Improving semantic segmentation by optimizing network structure is crucial for achieving accurate perception in complex scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic segmentation method based on an improved ASPP and a fusion module for complex scenarios, which can offer more options for semantic segmentation tasks in complex scenarios.

[0006] The objective of this invention is achieved as follows:

[0007] This invention provides a semantic segmentation method for complex scenarios based on an improved ASPP and a fusion module, characterized by:

[0008] (1) Build the Deeplabv3+ model under the PyTorch framework;

[0009] (2) Design the RA-ASPP module based on the traditional ASPP structure;

[0010] (3) Design the CBB module;

[0011] (4) The ASPP module in the Deeplabv3+ model is replaced by the RA-ASPP module, and the 3×3 standard convolution in the decoding fusion part is replaced by the CBB module;

[0012] (5) The model was trained using the freeze training method, and ablation experiments were conducted on the PASCAL VOC07+12 dataset using Xception and MobileNetV2 as backbones respectively to compare the performance of different models.

[0013] The present invention may also include:

[0014] 1. Step (1) includes the following steps:

[0015] (1.1) The Deeplabv3+ network structure is built using the Xception network model as the backbone, and the backbone switches between Xception and MobileNetV2 to meet different application requirements.

[0016] (1.2) Based on the ASPP structure, a RA-ASPP module is proposed. First, the residual network structure is used to achieve denser multi-scale feature extraction. Then, the asymmetric convolution module and the dilated convolution module are combined to form a new AACB module, which is used to replace the 3×3 dilated convolution module in ASPP.

[0017] (1.3) After decoding and fusion, a parallel fusion structure CBB combining 1×1 standard convolution and bottleneck module is proposed.

[0018] 2. The AACB module in step (2) is used to replace the 3×3 dilated convolution module in ASPP. The AACB module continues the dilation rate in dilated convolution, that is, the sampling rate of the AACB module is rate={6,12,18}.

[0019] 3. The CBB module structure in step (3) is based on the bottleneck module in ResNet. After the 3×3 convolution operation, an SE attention mechanism module is added. The reduction factor of the SE module is 16.

[0020] 4. In step (5), the PASCAL VOC07+12 dataset was used for network training. 10,582 additionally labeled images were used for training, and 1,449 images were used for verification and testing. The initial learning rate was 0.007. The stochastic gradient descent network model optimizer was used with a momentum of 0.9 and a weight decay rate of 0.0001. The learning rate descent method was cosine. The input image size was 512×512. The Freeze batch size was 8 and the Freeze epoch was 100. The Unfreeze batch size was 8 and the Unfreeze epoch was 200. A total of 300 epochs were trained.

[0021] The advantages of this invention are as follows: This invention proposes a RA-ASPP module combining residual networks and asymmetric dilated convolution modules, further enriching the scale of feature extraction and achieving denser multi-scale feature extraction, significantly enhancing the network's representational ability. It also proposes a parallel fusion structure CBB combining 1×1 standard convolution and a bottleneck module, reducing information loss during the entire network transmission process. To meet the requirements of high accuracy and real-time performance in complex scenarios, this invention uses Xception and MobileNetV2 as backbones for experimental verification on the PASCAL VOC07+12 dataset. Experimental results show that when the backbone is Xception, the proposed method achieves an average intersection-union (MIoU) of 79.78%, improving speed by 2.81% compared to the previous method with only a 1.72 FPS sacrifice. The proposed module significantly improves semantic segmentation accuracy, achieving segmentation results comparable to advanced semantic segmentation algorithms and meeting the high accuracy requirements. When the backbone is divided into MobileNetV2 components, the proposed method achieves a speed of 37.54 FPS, an improvement of 17.34 FPS compared to the previous method, while maintaining a MIoU of 73.32%, ensuring a balance between real-time segmentation speed and accuracy. The improved module proposed in this invention enhances the segmentation performance of Deeplabv3+, and the different backbone components also provide more options for semantic segmentation tasks in complex scenarios. Attached Figure Description

[0022] Figure 1 Here is a diagram of the Deeplabv3+ architecture;

[0023] Figure 2 A diagram of the improved Deeplabv3+ architecture;

[0024] Figure 3 Here is a diagram of the RA-ASPP structure;

[0025] Figure 4a This is a schematic diagram of the residual network elements (residual network structure) of RA-ASPP. Figure 4b A schematic diagram of the residual network element of RA-ASPP (improved element);

[0026] Figure 5 A schematic diagram of dilated convolution with different dilation rates;

[0027] Figure 6 This is a schematic diagram of asymmetric convolution;

[0028] Figure 7a This is a CBB module structure diagram (CBB structure diagram). Figure 7b This is a structural diagram of the CBB module (SE module);

[0029] Figure 8The loss training curve for Xception as the backbone;

[0030] Figure 9 The MIoU variation curve for Xception as the backbone;

[0031] Figure 10 A comparison chart of IoU performance of Deeplabv3+, Ours1, and Ours2 on the PASCAL VOC07+12 dataset;

[0032] Figure 11 Comparison of segmentation results for different methods (Input image, labeled image, Deeplabv3+, Ours1 (Xception), Ours2 (MobileNetV2)). Detailed Implementation

[0033] The invention will now be described in more detail with reference to the accompanying drawings:

[0034] Combination Figure 1-11 The traditional Deeplabv3+ model structure is as follows: Figure 1 As shown, this invention improves the structure of Deeplabv3+. Deeplabv3+ is a typical semantic segmentation network architecture, consisting of an encoder and a decoder, capable of pixel-level image segmentation, which makes it effective in image classification. Compared to Deeplabv3, Deeplabv2, and Deeplabv1, Deeplabv3+ adds a simple yet effective decoder module, forming an encoder-decoder structure. This allows for the collection of more pixel information, resulting in higher accuracy of the segmented images. Deeplabv3+ uses Xception instead of ResNet to deepen the network. Image feature information is extracted through depth-separable convolutional layers with different channels in the backbone Xception model. High-level semantic information is obtained using parallel dilated convolutions at different rates in the spatial pyramid pooling module, and channel compression is performed through 1×1 convolutions. The decoder fuses the low-level features extracted from the backbone network with high-level features upsampled by 4x bilinear interpolation, and then uses 3×3 convolutions to recover spatial information and 4x bilinear interpolation upsamples to refine target boundaries. The decoding structure improves the recovery of edge information, thereby improving accuracy.

[0035] This invention proposes a semantic segmentation method for complex scenarios based on an improved ASPP and a fusion module, the network structure of which is as follows: Figure 2 As shown in the figure. The method improves the recognition accuracy and segmentation precision of the network by designing an RA-ASPP module and adopting a parallel fusion structure CBB at the decoding end.

[0036] Includes the following steps:

[0037] Step 1: Input an RGB image with a size of 512×512; (e.g., Figure 2 )

[0038] Step 2: Input the image into the skeleton section ( Figure 2 Backbone (in the middle) is used to extract features from the image.

[0039] This invention employs two switchable backbone components: Xception and MobileNetV2. Xception is the traditional backbone of Deeplabv3+, while MobileNetV2 is a lightweight deep neural network proposed by Google for embedded devices such as mobile phones. MobileNetV2 is an upgraded version of MobileNet, significantly reducing model parameters and computational cost compared to traditional convolutional neural networks with only a slight decrease in accuracy. Compared to Xception, MobileNetV2 offers advantages such as lower computational cost and real-time performance, meeting the real-time requirements of semantic segmentation tasks. MobileNetV2 introduces depthwise separable convolutions to replace ordinary convolutions, and introduces linear bottlenecks and inverse residual structures to avoid information loss and improve accuracy, greatly reducing the number of model parameters and computational cost, thereby enhancing the network's representational capabilities. To meet the real-time requirements of semantic segmentation in complex scenarios, this invention replaces Xception with MobileNetV2, thus providing a semantic segmentation method that better meets the requirements of real-time applications.

[0040] Step 3: To address the limitations of ASPP and considering the diverse characteristics of targets in complex scenarios, this invention proposes an RA-ASPP module to supplement the information lost during the ASPP feature extraction process, thereby achieving better target segmentation results. Figure 3 This is a schematic diagram of the RA-ASPP structure proposed in this invention.

[0041] The traditional ASPP module utilizes high-level semantic feature maps obtained from the backbone network for multi-scale sampling to generate multi-scale feature maps. It combines dilated convolutions to expand the receptive field of the convolutional kernels without losing resolution. ASPP consists of two parallel parts: the first part contains a 1×1 convolutional layer and three 3×3 dilated convolutional layers with sampling rates of {6, 12, 18}, a total of 256 kernels, and includes batch normalization layers; the second part is the image-level feature representation, specifically achieved by applying global average pooling followed by a 1×1 convolutional layer with 256 kernels, and finally performing bilinear upsampling to obtain the desired spatial dimension. The ASPP structure incorporates parallel sampling of dilated convolutions at different sampling rates. Dilated convolutions can expand the receptive field of the feature map without losing image resolution. High resolution allows for precise target localization, while different receptive fields can sense different scale information. By combining different sampling rates in parallel, multiple receptive fields of different sizes can be obtained, thus achieving the goal of classifying targets of arbitrary sizes. However, ASPP's limitations also lie in dilated convolution. High-sampling-rate dilated convolution performs well for recognizing large objects, but loses valuable information about small objects; low-sampling-rate dilated convolution can obtain semantic location information of small objects, but loses more contour edge information of large objects. Parallel combination of dilated convolutions with different sampling rates compensates to some extent for the information lost by dilated convolution, but the valuable content within that lost information is still not fully utilized.

[0042] Step 3 includes:

[0043] Step 3.1: Construct the residual network unit of RA-ASPP. Figure 4 is a schematic diagram of the residual network unit of RA-ASPP.

[0044] To address the overfitting and vanishing gradient problems common in convolutional neural networks, residual networks were proposed. The basic idea is to assume that an optimal number of layers exists in a deep network, meaning that some redundant layers are defined as redundant layers. These redundant layers are designated as identity layers, enabling an identity mapping between input and output, and these identity layers can be adaptively learned during network training.

[0045] Figure 4(a) is a schematic diagram of the residual network structure. The residual network proposes the concept of a shortcut, that is, skipping one or more layers and directly adding the input result to the bottom layer. The calculation formula of the residual network is shown in Equation (1).

[0046] H(x)=x+F(x) (1)

[0047] In equation (1): H(x) is the mapping at the bottom layer; x is the input result; F(x) is the output result of the hidden layer in the network. Feature extraction of the image is performed by adding the output of multiple cascaded convolutional layers to the input.

[0048] In convolutional neural network (CNN) structures, the deeper the network, the more errors are generated during training. The emergence of residual networks has, to some extent, solved the problem of performance degradation in deep CNNs under extremely deep conditions. As shown in Figure 4(b), this invention adds a residual network to the traditional ASPP structure to avoid network performance degradation and further enriches the scale of feature extraction by utilizing residual connections, achieving denser multi-scale feature extraction, thereby increasing the accuracy of network model segmentation.

[0049] Step 3.2: Setup Figure 3 The AACB module in RA-ASPP combines asymmetric convolution with dilated convolution to form a new asymmetric atrous convolution (AACB) module. Addressing the shortcomings of dilated convolution, this module combines asymmetric convolution with dilated convolution to replace the 3×3 dilated convolution module in ASPP. The AACB module also retains the dilation rate of dilated convolution, with a sampling rate of {6, 12, 18}. The proposed AACB module, on the one hand, utilizes dilated convolution to expand the receptive field and capture multi-scale contextual information; on the other hand, it effectively supplements the spatially missing information in dilated convolution, resulting in better continuity of the entire network.

[0050] The Deeplabv3+ network model uses the ASPP module to enrich the semantic information of the context during feature extraction. However, dilated convolutions with multiple dilation rates are prone to causing a checkerboard effect, resulting in the loss of small-scale targets and discontinuous segmentation. Figure 5 This is a schematic diagram of dilated convolutions with different dilation rates. Compared with ordinary convolutional layers, dilated convolutions introduce a new parameter called "dilation rate," which defines the spacing between values ​​when the convolutional kernel processes data. Dilated convolutions have the advantages of preserving the integrity of the data structure and not using downsampling. However, multi-layered dilated convolutions also have the disadvantage of disrupting data continuity. The convolutional kernel of a dilated convolution is implemented by padding zeros on an ordinary convolutional kernel, which increases the dilation rate of the network. The relationship between the dilation rate and the size of the dilated convolutional kernel is shown in equation (2):

[0051] kd size =(γ-1)(k) size -1)+k size (2)

[0052] In the formula: γ represents the dilation coefficient of dilated convolution; k size Indicates the size of a regular convolution kernel; kd size This indicates the size of the dilated convolution kernel; when γ = 1, it is a normal convolution kernel.

[0053] The work of ACNet shows that the skeleton portion of a standard square convolutional kernel is more important than its corner portions. Discarding the weights of the kernel skeleton portion during training reduces model accuracy, while discarding the weights of the kernel corner portions increases model accuracy. Enhancing the kernel skeleton portion allows for the capture of more image features, improving model accuracy. Therefore, this invention introduces asymmetric convolution to improve the dilated convolution portion. Figure 6 This is a schematic diagram of asymmetric convolution. The asymmetric convolution module adds horizontal and vertical asymmetric convolution kernels to the standard square convolution. The input image is processed by three convolution kernels of different shapes: 3×3, 1×3 and 3×1, to extract different branch features, as shown in Equation (3):

[0054]

[0055] Where: M :,:,k ∈R U×V×C This represents the feature map of the k-th channel with an input size of U×V; O represents the j-th convolutional kernel of the k-th channel; O:,:, represents the output feature map corresponding to the j-th convolutional kernel.

[0056] To enhance the skeleton of the 3×3 convolution kernel, ACB uses the additivity of convolution to fuse three parallel 3×3, 1×3 and 3×1 convolution kernels to obtain the fused feature output. The feature dimension of the fused output is consistent with the dimension of the input feature, as shown in Equation (4):

[0057]

[0058] Where I is the input feature map matrix, K (1) and K (2) These represent two 2D convolutional kernels whose sizes are compatible with each other. ACNet works by summing the corresponding positions of two 2D convolutional kernels to form an asymmetric convolutional module, replacing the original 3×3 convolutional kernel to enhance the kernel's skeleton. The core of ACNet is replacing the conventional square convolutional kernel with an asymmetric form. By enhancing the kernel's skeleton and increasing its weights, ACNet can capture more image features and improve the accuracy of the baseline model.

[0059] Step 4: Construct a parallel fusion structure CBB combining 1×1 convolutions and bottleneck modules. The CBB module proposed in this invention can reduce the problem of partial information loss due to upsampling and improve the accuracy of the network.

[0060] The decoding and fusion part of Deeplabv3+ uses 3×3 convolution to perform simple feature fusion on the merged total features, and finally uses bilinear interpolation upsampling of 4 times to obtain the segmentation result. In the Deeplabv3+ network, bilinear interpolation upsampling is used twice. The specific matrix of bilinear interpolation is shown in Equation (5). Taking (0,0), (1,0), (0,1), (1,1) as an example, interpolation is performed in the x-axis and y-axis directions in this square.

[0061]

[0062] Assuming x and y are the x-axis and y-axis coordinates of the target point, respectively, when interpolating to (x,y) between (0,0), (1,0), (0,1), and (1,1), the relationship between this point and (0,0) is f(1,0)×(1-x)×(1-y), the relationship with (1,0) is f(1,0)×x×(1-y), the relationship with (0,1) is f(0,1)×(1-x)×y, and the relationship with (1,1) is f(1,1)×x×y. From formula (5), it can be concluded that the grayscale value of the target point is obtained by averaging the grayscale values ​​of the surrounding four pixels. This method considers the size of the surrounding pixel values ​​but does not consider the influence of the rate of change of adjacent points, which will cause some detail information to be lost after magnification.

[0063] To address the information loss issue caused by bilinear interpolation sampling, the proposed CBB module structure, as shown in Figure 7(a), combines a 1×1 convolution and a bottleneck module in parallel to form the CBB module, replacing the traditional 3×3 convolution module. In the CBB module, 1×1 convolutions are added to adjust the channels and resolution. The CBB module is based on the bottleneck module of traditional ResNet, with an SE module added after the 3×3 convolution operation, and a reduction factor of 16 (r = 16). The SE module automatically acquires the importance of each feature channel through learning, assigning different weights to feature channels, emphasizing features useful for the current detection task and suppressing invalid features, thereby improving the efficiency of feature processing. It can be flexibly embedded into other network models.

[0064] Figure 7(b) shows the structural diagram of the SE module. The SE module first processes the input feature map using global average pooling, then passes it through two fully connected layers to reduce the dimension of the feature map and then increase the dimension. Finally, after processing by the sigmoid activation function, the corresponding weights are obtained. The weights are multiplied by the original input feature map at the corresponding positions to obtain the output result, which means that feature maps of different importance can be processed accordingly.

[0065] Step 5: Experiment setup.

[0066] The initial learning rate was set to 0.007 in the experiment. Xception was used as the backbone network, and the Stochastic Gradient Descent (SGD) network model optimizer was employed with a momentum of 0.9. To prevent overfitting, the weight decay rate was set to 0.0001. The learning rate descent method used was cosine. The input image size was 512×512. Due to limitations in hardware processing capabilities, the experiment employed a frozen training method to accelerate training efficiency. The Freeze batch size was 8, and the Freeze epoch was 100; the Unfreeze batch size was 8, and the Unfreeze epoch was 200; a total of 300 epochs were trained.

[0067] The experiments used the PASCAL VOC07+12 dataset, which is a combination of the standard semantic segmentation datasets PASCAL VOC2007 and VOC2012. It contains 21 semantic segmentation categories, including 20 foreground categories and 1 background category. The PASCAL VOC07+12 dataset was used for training with 10,582 additionally labeled images and for validation and testing with 1,449 images; no separate test set was used. The PASCAL VOC07+12 dataset is currently the most commonly used dataset in the field of semantic segmentation. Its large data volume allows the trained semantic segmentation model to exhibit stronger generalization ability. The experimental program was mainly implemented using the mainstream deep learning framework PyTorch, and the hardware and software configurations are shown in Table 1.

[0068] Table 1 Experimental Hardware Configuration Table

[0069]

[0070] Step 6: Experimental results and analysis.

[0071] Step 6.1: This invention uses the PASCAL VOC07+12 dataset to train a semantic segmentation method based on an improved ASPP and fusion module for complex scenarios, thereby verifying the effectiveness of the algorithm. The training of the improved network with Xception as the backbone is shown here as an example.

[0072] Figure 8 The chart shows the trend of loss during training for the proposed model. Due to the use of frozen training to improve training efficiency, the loss curve decreases rapidly overall. Both train loss and value loss continuously decrease until they become constant, demonstrating the good training performance of the proposed model. When the epoch reaches 100, the loss curve gradually flattens out. Figure 9 This is a schematic diagram of the MIOU change curve during the training process. When the Epoch is 195, the MIOU of 79.78% is the optimal value.

[0073] Step 6.2: Comparison of extensive ablation experiments.

[0074] Table 2 shows extensive ablation experiments on the PASCAL VOC07+12 dataset.

[0075]

[0076] To verify the effectiveness of the improved modules, based on the experimental conditions in Table 1, ablation experiments were conducted on the Deeplabv3+ model on the PASCAL VOC07+12 dataset. Table 2 shows the results of the ablation experiments. The proposed RA-ASPP and CBB modules significantly improved the segmentation accuracy of the Deeplabv3+ model with only a small increase in parameters. Specifically, Ours1 improved MioU, MPA, and PA by 2.81%, 1.86%, and 0.76% respectively compared to the traditional Deeplabv3+, with a parameter value of 64.157MB, an increase of 9.443MB compared to the original model, and achieved a real-time semantic segmentation speed of 18.48 FPS. To further improve the model's segmentation speed, the backbone was switched to the more lightweight MobileNetV2 in Ours2. Ours2 achieved MioU, MPA, and PA of 73.32%, 82.01%, and 94.05% respectively. Compared to Deeplabv3+ and Ours1, Ours2 sacrifices some accuracy, but its parameter size is only 7.299MB, while its segmentation speed reaches 37.54 FPS, which is 17.34 FPS faster than Deeplabv3+. Ours2 achieves a good balance between speed and accuracy. Ablation experiments demonstrate that the two improved modules proposed in this invention significantly improve the segmentation accuracy of Deeplabv3+, and both proposed improved Deeplabv3+ models exhibit good accuracy and real-time performance.

[0077] Table 3 Speed ​​Comparison on Different GPUs

[0078]

[0079] Table 3 compares the semantic segmentation speed of the models on different GPUs. Ours1 uses Xception as its backbone, while Ours2 uses MobileNetV2. The segmentation speed of Ours1 is close to that of Deeplabv3+, with a difference of only about 1 FPS. Ours2 has a significant speed advantage over the traditional Deeplabv3+ and Ours1.

[0080] Table 4 shows the MIOU of different networks on the PASCAL VOC07+12 dataset.

[0081]

[0082] As shown in Table 4, Ours1 achieves an MIOU of 79.78% on the PASCAL VOC07+12 dataset, which is higher than SegNet, FCN-8s, Deeplabv1, Deeplabv2, Deeplabv3, and Deeplabv3+ in the same series. It achieves an accuracy comparable to advanced semantic segmentation algorithms and has good semantic segmentation performance. Ours2 achieves an MIOU of 73.32%, which is higher than SegNet, FCN-8s, and Deeplabv1, but lower than the other algorithms in Table 4. This is mainly because Ours2 uses a lightweight backbone, sacrificing some accuracy while gaining speed improvement.

[0083] Figure 10 The paper presents the IoU comparison results of Deeplabv3+, Ours1 (Xception), and Ours2 (MobileNetV2) on the PASCALVOC07+12 dataset for each category. The improved Deeplabv3+ proposed in this invention achieves accuracy comparable to current mainstream semantic segmentation algorithms, demonstrating excellent semantic segmentation performance.

[0084] Figure 11 The comparison images show the segmentation results of different methods. Ours1 achieves better segmentation results than Deeplabv3+, providing more comprehensive shape information and smoother edge contours. Ours2 performs worse than Deeplabv3+ and Ours1, but still predicts the corresponding segmentation results well. When the backbone is segmented into Xception, the improved Deeplabv3+ performs better than the traditional Deeplabv3+; when the backbone is segmented into the lighter MobileNetV2, the improved Deeplabv3+ performs worse than Deeplabv3+.

Claims

1. A semantic segmentation method based on improved ASPP and fusion module in complex scenes, characterized by: (1) building a Deeplabv3+ model under the Pytorch framework; (1.1) using the Xception network model as the backbone to build the Deeplabv3+ network structure, wherein the backbone part is switched between Xception and MobileNetV2 to meet different application requirements; (1.2) proposing a RA-ASPP module based on the ASPP structure, first using a residual network structure to achieve more dense multi-scale feature extraction; then combining an asymmetric convolution module with a dilated convolution module to form a new AACB module, which is used to replace the 3x3 dilated convolution module in ASPP; (1.3) proposing a parallel fusion structure CBB combining 1x1 standard convolution and bottleneck module after decoding fusion; (2) designing the RA-ASPP module based on the traditional ASPP structure; (3) designing the CBB module; (4) replacing the ASPP module in the Deeplabv3+ model with the RA-ASPP module, and replacing the 3x3 standard convolution in the decoding fusion part with the CBB module; (5) training the model using the freeze training method, and using Xception and MobileNetV2 as the backbone to perform ablation experiments on the PASCAL VOC07+12 dataset and compare the performance of different models.

2. The semantic segmentation method based on improved ASPP and fusion module in a complex scene according to claim 1, characterized in that: The AACB module in step (2) is used to replace the 3x3 dilated convolution module in ASPP, and the AACB module continues the dilation rate in dilated convolution, i.e. the sampling rate rate = {6, 12, 18} of the AACB module.

3. The method according to claim 1, characterized in that: The CBB module structure in step (3) is based on the bottleneck module in ResNet, and adds an SE attention mechanism module after the 3x3 convolution operation, and the reduction factor of the SE module is 16.

4. The method according to claim 1, characterized in that: In step (5), the PASCAL VOC07+12 dataset is used for network training, 10582 additional labeled images are used for training, 1449 images are used for verification and testing, the initial learning rate is 0.007, the stochastic gradient descent network model optimizer is used, the momentum is 0.9, the weight decay rate is set to 0.0001, the learning rate reduction method is cos, the input image size is 512x512, the Freeze batch size is 8, the Freeze epoch is 100; the Unfreeze batch size is 8, the UnFreeze epoch is 200; a total of 300 epochs are trained.

Images