Multi-scale information fused depth asymmetric bottleneck real-time semantic segmentation method, system and device and medium
By introducing self-corrected DAB module and cascading cross-stage ASPP module in deep asymmetric bottleneck network, integrating multi-scale information, the problem of the existing network lacking multi-scale information extraction and global context guidance in semantic segmentation is achieved. Higher semantic segmentation accuracy and real-time.
Patent Information
- Application Number
- CN202411071225.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-12
AI Technical Summary
The existing deep asymmetric bottleneck network lacks adaptive multi-scale information extraction and global context guidance in semantic segmentation, resulting in inaccurate segmentation results, discontinuous large-size object analysis, and mis-analysis of small target objects.
A real-time semantic segmentation method for the depth asymmetric bottleneck that integrates multi-scale information is proposed. By introducing a self-corrected DAB module and a cascading cross-stage ASPP module, multi-scale information is captured and fused to enhance the effectiveness and accuracy of the network.
It realizes more accurate and complete target object positioning, improves semantic segmentation accuracy, especially in road scenarios, improves the segmentation ability of large and small targets, while maintaining the lightweight structure and real-time nature of the network.
Smart Images

Figure CN118941796A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a deep asymmetric bottleneck real-time semantic segmentation method, system, device and medium that integrates multi-scale information. Background Art
[0002] In recent years, semantic segmentation has been applied in all aspects of life and has been widely used in fields such as medical care, satellite remote sensing and autonomous driving. For example, in the field of autonomous driving, real-time images are transmitted through vehicle-mounted radar or cameras to automatically identify various targets on the road and analyze different targets at the semantic level, which helps improve driving safety.
[0003] Deep convolutional neural networks (CNNs) have made significant progress in semantic segmentation. The effectiveness of these networks depends largely on the depth and width of the network. The network parameters are large, very complex, and usually require a lot of computing resources. In the field of autonomous driving, not only are computing resources limited, but the algorithm is also required to have high real-time performance. Therefore, it is particularly important to study real-time and efficient lightweight semantic segmentation methods.
[0004] Currently, many network models achieve effective and efficient segmentation by designing efficient feature extraction modules. For example, FDDWNet proposes an extremely efficient residual module EERM, and uses a deeply separable dilated convolution to learn semantic features from receptive fields of different scales with fewer model parameters; DABNet proposes a deep asymmetric bottleneck DAB structure, which extracts local information and contextual information at the same time, greatly reducing parameters and providing comparable accuracy and inference speed. This type of network seems to be effective for smaller input resolutions, but it easily ignores the appearance of details around boundaries and small objects, and lacks the fusion of multi-scale information. Summary of the invention
[0005] Purpose of the invention: In view of the problems that DABNet cannot adaptively extract multi-scale information and lacks the guidance of global context information, resulting in inaccurate final segmentation results, discontinuous analysis of large-size objects, and incorrect analysis of small target objects, the present invention proposes a deep asymmetric bottleneck real-time semantic segmentation method that integrates multi-scale information for road scenes. While ensuring the lightweight of the network, the effectiveness of the network is improved as much as possible, achieving a good balance between real-time performance and accuracy.
[0006] Technical solution: A deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information, including the following steps:
[0007] Get the input image;
[0008] Input the input image into the pre-trained semantic segmentation network to obtain the semantic segmentation result of the input image;
[0009] The semantic segmentation network consists of a downsampling part, a self-correction part and an upsampling part; the self-correction part is constructed based on a self-correction DAB module, and the upsampling part is constructed by introducing a CCSP-ASPP module.
[0010] Furthermore, the downsampling part consists of three 3×3 convolutional layers and a first downsampling block; three 3×3 convolutional layers are used to extract initial features, and the same first downsampling block as the ENet initial block is used, and the first downsampling block is a cascade of 3×3 convolution with a step size of 2 and 2×2 maximum pooling.
[0011] Furthermore, a long-range skip connection is established between the input image and the first downsampling block.
[0012] Further, the self-correction part includes a self-correction part 1 and a self-correction part 2;
[0013] The self-correction part 1 is composed of three self-correction DAB modules in cascade, and when the self-correction part 1 outputs, the input of the first self-correction DAB module is stacked with the output of the last self-correction DAB module;
[0014] The self-correction part 2 is composed of 6 self-correction DAB modules in cascade, and when the self-correction part 2 outputs, the input of the first self-correction DAB module and the output of the last self-correction DAB module are stacked;
[0015] The output of the self-correction part 1 reaches the self-correction part 2 through the second down-sampling block; and a long-range skip connection is established between the input image and the second down-sampling block;
[0016] The output of the second self-correction part passes through a 1×1 convolution to reach the upsampling part, and a long-range skip connection is established between the input image and the 1×1 convolution.
[0017] Furthermore, each self-correction DAB module includes: a first convolution layer, a rear convolution layer and a terminal convolution layer; the first convolution layer is a self-calibration convolution layer, and the rear convolution layer is a bottleneck structure consisting of a deep asymmetric dilated convolution and two branches;
[0018] The bottleneck structure composed of the deep asymmetric dilated convolution and two branches includes a branch for extracting local feature information and another branch for using dilated convolution to expand the receptive field. The output results of the two branches are added to reach the terminal convolution layer, which is a 1×1 point-by-point convolution layer; the one branch is composed of 3×1 Dconv and 1×3 Dconv, and the other branch is composed of 3×1 Ddconv and 1×3 Ddconv.
[0019] Furthermore, in the self-calibration convolutional layer, a segmentation step, a self-correction step and a splicing step are performed;
[0020] The segmentation step includes: dividing the input X into two parts {X1, X2} on average;
[0021] The self-correction step includes: assuming that there is a set of shapes (C, C, k h , k w ) of the filter bank K, where k h and k w are the height and width of the filter respectively. The filter bank K is evenly divided into four parts to obtain four filters, which are expressed as The shape of each filter is (C / 2, C / 2, k h , k w ), use the three filters {K2, K3, K4} to perform a self-correction operation on X1 to obtain the result Y1; use K1 to perform a convolution operation to obtain the result Y2;
[0022] The stitching step includes: stitching the result Y1 and the result Y2 to obtain a self-calibration result.
[0023] Furthermore, the self-correction operation of X1 using the three filters {K2, K3, K4} to obtain the result Y1 specifically includes:
[0024] Use average pooling of size r×r to perform pooling operation on the input X1:
[0025] T1=AvgPool(X1)
[0026] The obtained T1 is sent to the K2 filter to perform a convolution operation, and the feature map is upsampled to the same size as X1 to obtain X1′:
[0027] X1′=Up(f2(T1))=Up(T1*K2)
[0028] Among them, Up(·) represents the process of upsampling by bilinear interpolation;
[0029] Perform self-calibration operation according to the following formula:
[0030] Y1′=f3(X1)·σ(X1+X1′)=(X1*K3)·σ(X1+X1′)
[0031] Among them, σ represents the Sigmoid activation function, and “·” represents the product of the corresponding elements;
[0032] The output of the self-correction operation is:
[0033] Y1=f4(Y1′)=Y1′*K4.
[0034] Further, the upsampling part includes a CCSP-ASPP module and an upsampling block; the upsampling block is used to restore the image output by the CCSP-ASPP module to the input image size;
[0035] The CCSP-ASPP module is composed of a left branch with a residual edge function and a right branch using a cascaded ASPP module; the left branch includes a first 1×1 convolutional layer; the right branch includes a second 1×1 convolutional layer, a third 1×1 convolutional layer, a first 3×3 convolutional layer, a second 3×3 convolutional layer, a third 3×3 convolutional layer, a global average pooling layer, a fourth 1×1 convolutional layer, an upsampling layer, a first concat layer, a fifth 1×1 convolutional layer, a second concat layer and a sixth 1×1 convolutional layer;
[0036] For the right branch, the input of the CCSP-ASPP module passes through the second 1×1 convolution layer, and is sent to the third 1×1 convolution layer, the first 3×3 convolution layer, and the global average pooling layer respectively. The output of the first 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the second 3×3 convolution layer. The output of the second 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the third 3×3 convolution layer. The output of the global average pooling layer passes through the fourth 3×3 convolution layer and the upsampling layer in turn and is sent to the first concat layer. At the same time, the outputs of the third 1×1 convolution layer, the first 3×3 convolution layer, the second 3×3 convolution layer, and the third 3×3 convolution layer are all sent to the first concat layer. The output of the first concat layer passes through the fifth 1×1 convolution layer, the second concat layer, and the sixth 1×1 convolution layer in turn to obtain the output of the CCSP-ASPP module.
[0037] For the left branch, the input of the CCSP-ASPP module is fed into the first 1×1 convolutional layer, and the output of the first 1×1 convolutional layer is fed into the second concat layer.
[0038] The present invention discloses a deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information, comprising:
[0039] An image acquisition module, used for acquiring an input image;
[0040] The semantic segmentation module is used to input the input image into the pre-trained semantic segmentation network to obtain the semantic segmentation result of the input image;
[0041] The semantic segmentation network consists of a downsampling part, a self-correction part and an upsampling part; the self-correction part is constructed based on a self-correction DAB module, and the upsampling part is constructed by introducing a CCSP-ASPP module.
[0042] Further, the downsampling part is composed of three 3×3 convolutional layers and a first downsampling block; the three 3×3 convolutional layers are used to extract the initial features, and the first downsampling block is the same as the ENet initial block, and the first downsampling block is a cascade of a 3×3 convolution with a stride of 2 and a 2×2 maximum pooling;
[0043] A long-range skip connection is established between the input image and the first downsampling block.
[0044] Further, the self-correction part includes a self-correction part 1 and a self-correction part 2;
[0045] The self-correction part 1 is composed of three self-correction DAB modules in cascade, and when the self-correction part 1 outputs, the input of the first self-correction DAB module is stacked with the output of the last self-correction DAB module;
[0046] The self-correction part 2 is composed of 6 self-correction DAB modules in cascade, and when the self-correction part 2 outputs, the input of the first self-correction DAB module and the output of the last self-correction DAB module are stacked;
[0047] The output of the self-correction part 1 reaches the self-correction part 2 through the second down-sampling block; and a long-range skip connection is established between the input image and the second down-sampling block;
[0048] The output of the second self-correction part passes through a 1×1 convolution to reach the upsampling part, and a long-range skip connection is established between the input image and the 1×1 convolution.
[0049] Furthermore, each self-correction DAB module includes: a first convolution layer, a rear convolution layer and a terminal convolution layer; the first convolution layer is a self-calibration convolution layer, and the rear convolution layer is a bottleneck structure consisting of a deep asymmetric dilated convolution and two branches;
[0050] The bottleneck structure composed of the deep asymmetric dilated convolution and two branches includes a branch for extracting local feature information and another branch for using dilated convolution to expand the receptive field. The output results of the two branches are added to reach the terminal convolution layer, which is a 1×1 point-by-point convolution layer. The one branch is composed of 3×1 Dconv and 1×3 Dconv, and the other branch is composed of 3×1 Ddconv and 1×3 Ddconv.
[0051] Furthermore, in the self-calibration convolutional layer, a segmentation step, a self-correction step and a splicing step are performed;
[0052] The segmentation step includes: dividing the input X into two parts {X1, X2} on average;
[0053] The self-correction step includes: assuming that there is a set of shapes (C, C, k h , k w ) of the filter bank K, where k h and k w are the height and width of the filter respectively. The filter bank K is evenly divided into four parts to obtain four filters, which are expressed as The shape of each filter is (C / 2, C / 2, k h , k w ), use the three filters {K2, K3, K4} to perform a self-correction operation on X1 to obtain the result Y1; use K1 to perform a convolution operation to obtain the result Y2;
[0054] The stitching step includes: stitching the result Y1 and the result Y2 to obtain a self-calibration result.
[0055] Furthermore, the self-correction operation of X1 using the three filters {K2, K3, K4} to obtain the result Y1 specifically includes:
[0056] Use average pooling of size r×r to perform pooling operation on the input X1:
[0057] T1=AvgPool(X1)
[0058] The obtained T1 is sent to the K2 filter to perform a convolution operation, and the feature map is upsampled to the same size as X1 to obtain X1′:
[0059] X1′=Up(f2(T1))=Up(T1*K2)
[0060] Among them, Up(·) represents the process of upsampling by bilinear interpolation;
[0061] Perform self-calibration operation according to the following formula:
[0062] Y1′=f3(X1)·σ(X1+X1′)=(X1*K3)·σ(X1+X1′)
[0063] Among them, σ represents the Sigmoid activation function, and “·” represents the product of the corresponding elements;
[0064] The output of the self-correction operation is:
[0065] Y1=f4(Y1′)=Y1′*K4.
[0066] Further, the upsampling part includes a CCSP-ASPP module and an upsampling block; the upsampling block is used to restore the image output by the CCSP-ASPP module to the input image size;
[0067] The CCSP-ASPP module is composed of a left branch with a residual edge function and a right branch using a cascaded ASPP module; the left branch includes a first 1×1 convolutional layer; the right branch includes a second 1×1 convolutional layer, a third 1×1 convolutional layer, a first 3×3 convolutional layer, a second 3×3 convolutional layer, a third 3×3 convolutional layer, a global average pooling layer, a fourth 1×1 convolutional layer, an upsampling layer, a first concat layer, a fifth 1×1 convolutional layer, a second concat layer and a sixth 1×1 convolutional layer;
[0068] For the right branch, the input of the CCSP-ASPP module passes through the second 1×1 convolution layer, and is sent to the third 1×1 convolution layer, the first 3×3 convolution layer, and the global average pooling layer respectively. The output of the first 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the second 3×3 convolution layer. The output of the second 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the third 3×3 convolution layer. The output of the global average pooling layer passes through the fourth 3×3 convolution layer and the upsampling layer in turn and is sent to the first concat layer. At the same time, the outputs of the third 1×1 convolution layer, the first 3×3 convolution layer, the second 3×3 convolution layer, and the third 3×3 convolution layer are all sent to the first concat layer. The output of the first concat layer passes through the fifth 1×1 convolution layer, the second concat layer, and the sixth 1×1 convolution layer in turn to obtain the output of the CCSP-ASPP module.
[0069] For the left branch, the input of the CCSP-ASPP module is fed into the first 1×1 convolutional layer, and the output of the first 1×1 convolutional layer is fed into the second concat layer.
[0070] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information are implemented.
[0071] The present invention discloses a storage medium storing a semantic segmentation program. When the semantic segmentation program is executed by at least one processor, the steps of a deep asymmetric bottleneck real-time semantic segmentation method fusing multi-scale information are implemented.
[0072] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0073] (1) The present invention proposes a depth-wise asymmetric bottleneck network with multi-scales for real-time semantic segmentation (MS-DABNet), which is a network model including a self-calibration DAB (Self-Calibration DAB) structure and a cascaded and cross stage partial ASPP (CCSP-ASPP) module. The network effectiveness is enhanced by capturing and fusing multi-scale information, and it has the advantages of multi-scale feature extraction and fusion.
[0074] (2) The self-correcting DAB module proposed in the present invention has the advantage of adaptively acquiring multi-scale information, and is used to extract and aggregate multi-scale information to solve the multi-scale problem of semantic segmentation tasks; the self-correcting DAB module proposed in the present invention is obtained by improving the original DAB module. The self-correcting DAB module has the advantages of the original DAB module and can expand the receptive field as much as possible, capture multi-scale information, generate richer and more discriminative feature representations, and locate large and small target objects more accurately and completely;
[0075] (3) The cascaded cross-stage partial ASPP module proposed in the present invention is improved on the basis of the ASPP module. It uses the cross-stage feature fusion strategy and bottleneck structure to greatly reduce the amount of calculation while broadening the network structure, so that each network layer can learn richer features; the present invention increases the information interaction of feature graphs, effectively obtains more comprehensive multi-scale information, and is conducive to the reuse of semantic information, thereby enhancing the detailed features in the road scene;
[0076] (4) The present invention designs the MS-DABNet network based on the self-correcting DAB module and the CCSP-ASPP module, which can not only adaptively extract multi-scale features, but also fuse multi-scale information to enhance global feature information and obtain more comprehensive and effective feature information, so that the network can accurately segment large and small targets on the road, and the network still has a lightweight structure;
[0077] (5) The MS-DABNet network proposed in this invention improves mIoU from 70.10% to 72.84%, achieves a running speed of 38.6FPS on a single RTX3090 GPU to process high-resolution images (1024×2048), and the model size is only 1.33M. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] Figure 1 is a flow chart of the method of the present invention;
[0079] Figure 2 A network structure diagram of a lightweight real-time semantic segmentation network that integrates multi-scale information proposed by the present invention;
[0080] Figure 3 This is the structure diagram of the self-correction DAB module;
[0081] Figure 4 This is the structural diagram of the self-correction convolution SCConv;
[0082] Figure 5 It is a structural diagram of the cascaded cross-stage part of the ASPP module. DETAILED DESCRIPTION
[0083] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.
[0084] Embodiment 1:
[0085] like Figure 1 As shown, this embodiment discloses a deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information, including the following steps:
[0086] Step 1: Construct a depth-wise asymmetric bottleneck network with multi-scales for real-time semantic segmentation (MS-DABNet) that integrates multi-scale information; Figure 2 As shown, it includes a downsampling part, a self-correction part 1, a self-correction part 2 and an upsampling image restoration part which are connected together.
[0087] The construction process of the deep asymmetric bottleneck real-time semantic segmentation network integrating multi-scale information is now described in detail. For the convenience of description, the deep asymmetric bottleneck real-time semantic segmentation network integrating multi-scale information is referred to as MS-DABNet network.
[0088] Design of the downsampling part of the MS-DABNet network: The downsampling part consists of three 3×3 convolutions and downsampling blocks. First, three 3×3 convolutions are used to extract the initial features, and the stride of the first convolution is set to 2 to downsample the image. Then, the same downsampling block as the initial block of ENet is used, which is a cascade of 3×3 convolutions with a stride of 2 and 2×2 maximum pooling. Only three downsamplings are used to obtain a feature map with a resolution of 1 / 8 of the input image. Through the downsampling operation, the resolution of the feature map can be reduced and the receptive field can be expanded, thereby improving the ability to extract contextual information. Finally, a long-range jump connection is established between the input image and the downsampling block, which facilitates feature reuse and compensates for information loss.
[0089] Design of the self-correction part 1 of the MS-DABNet network: This self-correction part 1 consists of a cascade of three self-correction DAB modules for dense feature extraction. In order to enhance the spatial connection between features, at the output of the self-correction part 1, the input of the first self-correction DAB module is stacked with the output of the last self-correction DAB module to combine high-level features with low-level features. Then, after passing through the downsampling block, the input image is connected to the downsampling block to build a long-range jump connection to reach the self-correction part 2. The expansion rate of the three self-correction DAB modules in the self-correction part 1 is 2.
[0090] Design of the self-correction part 2 of the MS-DABNet network: The self-correction part 2 consists of 6 cascaded self-correction DAB modules, and the expansion rates of the 6 self-correction DAB modules are 4, 4, 8, 8, 16, and 16. At the output of the self-correction part 2, the input of the first self-correction DAB module and the output of the last self-correction DAB module are stacked to combine high-level features with low-level features.
[0091] Figure 3 The structure of each self-correction DAB module is shown. For each self-correction DAB module, the first convolution layer is the self-calibration convolution SCConv (Self-Calibration Convolution, SCConv), and the subsequent convolution layer is a bottleneck structure consisting of a deep asymmetric dilated convolution and two branches. The self-correction DAB module effectively fuses multi-scale features through a dual-branch structural design. One branch is used to extract local feature information, and the other branch uses dilated convolution to expand the receptive field. Finally, the two branches are added together, and then a 1×1 point-by-point convolution is performed at the end of each self-correction DAB module to restore the number of channels and fuse all channel information. Specifically, as Figure 3 As shown, one branch is composed of 3×1 Dconv and 1×3 Dconv, and the other branch is composed of 3×1 Ddconv and 1×3 Ddconv.
[0092] The structure of self-calibrated convolution SCConv is as follows Figure 4 As shown. In the self-calibration convolution SCConv, there are three steps, namely splitting (Split), self-calibration (Self-Calibration), and concatenation (Concat). The original filter bank is divided into four parts, each responsible for a different function. Suppose there is a set of shapes (C, C, k h , k w ) of the filter bank K, where k h and k w are the height and width of the filter respectively, and the filter bank K is evenly divided into four parts, each of which is responsible for different functions. After even division, four filters are obtained, which are expressed as The shape of each filter is (C / 2, C / 2, k h , k w ).
[0093] The calculations performed in the self-calibration convolution SCConv are described as follows.
[0094] Assuming the input X of self-correction part one, the input X is evenly divided into two parts {X1, X2}, and then each part is sent to a different operation for extracting different types of context information.
[0095] The first operation is to use the three filters {K2, K3, K4} to perform self-correction operation on X1 to obtain the result Y1. Specifically, it includes:
[0096] The input X1 is pooled using an average pooling operation of size r×r, that is:
[0097] T1=AvgPool(X1)
[0098] The obtained T1 is sent to the K2 filter to perform the convolution operation, and the feature map is upsampled to the same size as X1 to obtain X1′, that is:
[0099] X1′=Up(f2(T1))=Up(T1*K2)
[0100] Wherein, Up(·) represents the process of upsampling by bilinear interpolation.
[0101] Perform self-calibration operation according to the following formula:
[0102] Y1′=f3(X1)·σ(X1+X1′)=(X1*K3)·σ(X1+X1′)
[0103] Among them, σ represents the Sigmoid activation function, and “.” represents the product of the corresponding elements.
[0104] The output of the self-correction operation is:
[0105] Y1=f4(Y1′)=Y1′*K4
[0106] The second operation is to use K1 to perform a simple convolution operation to obtain Y2, the purpose of which is to retain the original spatial context features, namely:
[0107] Y2=f1(X2)=X2*K1
[0108] The final output Y of the self-correction convolution is the concatenation of the result of the self-correction operation and the result of the ordinary convolution operation, that is:
[0109] Y=Concat(Y1, Y2)={Y1, Y2}
[0110] Design of upsampling image restoration part: In this upsampling image restoration part, the CCSP-ASPP module is first introduced to enhance the global feature information by fusing multi-scale information to obtain more comprehensive and effective feature information, so that the network can accurately segment large and small objects on the road. After the CCSP-ASPP module, 8 times upsampling is directly performed to restore the image to the input image size.
[0111] The CCSP-ASPP module is now further described.
[0112] The structure of CCSP-ASPP module is as follows Figure 5 As shown in the figure. The overall structure of the CCSP-ASPP module is similar to the bottleneck structure. First, the feature map from the upper layer is sent to two 1×1 convolutions for dimensionality reduction, which halve the number of feature channels and send them to the left branch and the right branch respectively. The left branch has the function of residual edge, and the right branch uses the cascaded ASPP module to better capture multi-scale features. Finally, the features of the two branches are spliced to obtain the final multi-scale features. By using the split and merge strategy across stages, the number of feature channels is reduced by half, the computational complexity is reduced by half, and the probability of duplication can be effectively reduced during the information integration process.
[0113] The right branch is the cascaded ASPP module, which includes 1×1 convolution, global average pooling and 3×3 convolution with dilation rates of 6, 12 and 18 respectively, for extracting image features. In addition, the three middle 3×3 convolution branches of the module adopt a layer-by-layer stacking method to achieve dense pixel sampling, enabling the network to extract richer high-level features. The feature maps output by the upper 1×1 convolution layer are used as the input of the cascaded ASPP, and then these feature maps are convolved in 4 different ways to obtain feature maps of the same size. Since the last branch is a pooling layer, the number of channels will not be changed, and 1×1 convolution and upsampling are introduced to obtain feature maps of the same size as the other 4 branches. Finally, the feature maps obtained by the 5 branches are spliced to increase the information interaction of the feature maps, thereby solving the problem of imbalance between feature maps and receptive fields. At the same time, the cascaded ASPP structure can effectively obtain more comprehensive multi-scale information, which is conducive to the reuse of semantic information, thereby enhancing the detailed features in road scenes.
[0114] Specifically, Figure 5 As shown in the figure, the CCSP-ASPP module consists of a left branch with a residual edge function and a right branch using a cascaded ASPP module; the left branch includes the first 1×1 convolution layer; the right branch includes the second 1×1 convolution layer, the third 1×1 convolution layer, the first 3×3 convolution layer, the second 3×3 convolution layer, the third 3×3 convolution layer, the global average pooling layer, the fourth 1×1 convolution layer, the upsampling layer, the first concat layer, the fifth 1×1 convolution layer, the second concat layer and the sixth 1×1 convolution layer;
[0115] For the right branch, the input of the CCSP-ASPP module passes through the second 1×1 convolution layer, and is sent to the third 1×1 convolution layer, the first 3×3 convolution layer, and the global average pooling layer respectively. The output of the first 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the second 3×3 convolution layer. The output of the second 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the third 3×3 convolution layer. The output of the global average pooling layer passes through the fourth 3×3 convolution layer and the upsampling layer in turn and is sent to the first concat layer. At the same time, the outputs of the third 1×1 convolution layer, the first 3×3 convolution layer, the second 3×3 convolution layer, and the third 3×3 convolution layer are all sent to the first concat layer. The output of the first concat layer passes through the fifth 1×1 convolution layer, the second concat layer, and the sixth 1×1 convolution layer in turn to obtain the output of the CCSP-ASPP module.
[0116] For the left branch, the input of the CCSP-ASPP module is fed into the first 1×1 convolutional layer, and the output of the first 1×1 convolutional layer is fed into the second concat layer.
[0117] This embodiment improves the DAB module, proposes a self-correction DAB module, introduces a self-calibration convolution SCConv to encode multi-scale information, and adaptively extracts surrounding context features. Compared with the standard convolution, the self-calibration convolution SCConv does not increase the amount of additional parameters and calculations.
[0118] In this embodiment, the CCSP-ASPP module is introduced in the final stage of encoding, that is, before 8-fold upsampling, to enhance the global feature information by fusing multi-scale information to obtain more comprehensive and effective feature information, so that the network can accurately segment large and small objects on the road.
[0119] The MS-DABNet network based on the self-correcting DAB module and the CCSP-ASPP module has a lightweight structure and can be embedded in mobile devices to meet application scenarios such as autonomous driving. The MS-DABNet network proposed in this embodiment can segment both large and small target objects well while maintaining a lightweight network, and achieves a good balance between real-time segmentation speed and accuracy.
[0120] Step 2: The deep asymmetric bottleneck real-time semantic segmentation network integrating multi-scale information is trained using the training set of finely annotated images in the Cityscapes dataset. During the training phase, the training epoch size is set to 1000, the batch size is set to 8, and the stochastic gradient descent (SGD) method is used with a momentum of 0.9 and a weight decay of 4e. -5 , the learning rate strategy adopts the "poly" method, and the initial learning rate is set to 0.045. On the Windows system, the experimental environment is built with Python and PyTorch deep learning framework, and the training is completed on the NVIDIA GeForce RTX 3090 GPU device to obtain a pre-trained deep asymmetric bottleneck real-time semantic segmentation network that integrates multi-scale information;
[0121] Step 3: The semantic segmentation network of this embodiment is for high-resolution images, so the input image resolution is 1024×2048. Since the dataset used has a very high resolution, the real-time semantic segmentation task is very challenging. We now use MIoU and FPS as two evaluation indicators to evaluate the segmentation accuracy and speed, and use the model's Params to evaluate the model's parameters. The formulas for MIoU, FPS, and Params are as follows:
[0122]
[0123] Where i represents the i-th category, p ii represents the number of pixels predicted as the i-th category, i.e., the true positive TP, p ijrepresents the number of pixels predicted as the jth category in the i-th category, i.e., the false negative example FN, p ji It represents the number of pixels predicted as the i-th category in the j-th category, i.e., false positive examples FP.
[0124]
[0125] Where N is the total number of processed images, T i is the time required to process the i-th image. The larger the FPS value, the faster the image processing speed, which proves that the performance of the network model is better.
[0126] Params = K h ×K w ×C in ×C out
[0127] Among them, k h ×k w is the size of the convolution kernel, C in is the number of channels of the input feature, C out is the number of channels of the output feature, H and W are the height and width of the input feature.
[0128] Step 4: Implement semantic segmentation based on the pre-trained deep asymmetric bottleneck real-time semantic segmentation network that integrates multi-scale information.
[0129] The real-time semantic segmentation network proposed in this embodiment uses only 1.33M parameters to achieve relatively superior results. As shown in Table 1, the real-time semantic segmentation network proposed in this embodiment achieves 72.84% MIoU on the Cityscapes test set without pre-training and other assistance, which is significantly better than the existing real-time segmentation network model. Compared with the original DABNet network, the accuracy is improved by 2.74%. Compared with other lightweight real-time semantic segmentation networks in the table, the accuracy MIoU is also improved to a certain extent, ranging from 1.34% to 14.54%.
[0130] Table 1 Performance comparison of different semantic segmentation models
[0131] Method Inputsize MIoU / % FPS (frames / s) GPU Params / M ERFNet 512×1024 68.00 41.7 TitianX 2.1 ENet 512×1024 58.30 76.9 TitianX 0.36 ICNet 1024×2048 69.50 30.3 TitianX 6.68 CGNet 1024×2048 64.80 91.4 RTX 3090 0.52 FDDWNet 512×1024 71.50 72.7 RTX 3090 0.80 GFEN 1024×2048 60.50 - A5000 1.18 FANet 1024×2048 71.20 - - - Fast-SCNN 1024×2048 68.00 193.1 RTX 3090 1.11 DABNet 1024×2048 70.10 67.6 RTX 3090 0.76 Ours 1024×2048 72.84 38.60 RTX 3090 1.33
[0132] In order to compare the speed with other network models fairly, all speed experiments were performed on a single RTX 3090 GPU on a Windows system. MS-DABNet can process 1024×2048 images at a speed of 38.60FPS. Although it is lower than the segmentation speed of Fast-SCNN, DABNet, FDDWNet and CGNet, it is the most superior in terms of accuracy. Obviously, the real-time semantic segmentation network proposed in this embodiment achieves a better balance between MIoU and inference speed.
[0133] Embodiment 2:
[0134] Based on Example 1, this example proposes a deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information, including:
[0135] An image acquisition module, used for acquiring an input image;
[0136] The semantic segmentation module is used to input the input image into the pre-trained semantic segmentation network to obtain the semantic segmentation result of the input image;
[0137] Among them, the semantic segmentation network consists of a downsampling part, a self-correction part and an upsampling part; the self-correction part is constructed based on the self-correction DAB module, and the upsampling part is constructed by introducing the CCSP-ASPP module.
[0138] Embodiment 3:
[0139] This embodiment discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps disclosed in any of the above embodiments are implemented.
[0140] Embodiment 4:
[0141] This embodiment discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps disclosed in any of the above embodiments are implemented.
[0142] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0143] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0144] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information, characterized by: The following steps are involved: Get the input image; Input the input image into the pre-trained semantic segmentation network to obtain the semantic segmentation result of the input image; The semantic segmentation network consists of a downsampling part, a self-correction part and an upsampling part; the self-correction part is constructed based on a self-correction DAB module, and the upsampling part is constructed by introducing a CCSP-ASPP module.
2. According to claim 1, a deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information is characterized by: The downsampling part consists of three 3×3 convolutional layers and a first downsampling block; three 3×3 convolutional layers are used to extract initial features, and the same first downsampling block as the ENet initial block is used. The first downsampling block is a cascade of 3×3 convolution with a step size of 2 and 2×2 maximum pooling.
3. The deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information according to claim 2 is characterized by: A long-range skip connection is established between the input image and the first downsampling block.
4. The deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information according to claim 1, characterized in that: The self-correction part includes a self-correction part 1 and a self-correction part 2; The self-correction part 1 is composed of three self-correction DAB modules in cascade, and when the self-correction part 1 outputs, the input of the first self-correction DAB module is stacked with the output of the last self-correction DAB module; The self-correction part 2 is composed of 6 self-correction DAB modules in cascade, and when the self-correction part 2 outputs, the input of the first self-correction DAB module and the output of the last self-correction DAB module are stacked; The output of the self-correction part 1 reaches the self-correction part 2 through the second down-sampling block; and establishing a long-range skip connection between the input image and the second downsampling block; The output of the second self-correction part passes through a 1×1 convolution to reach the upsampling part, and a long-range skip connection is established between the input image and the 1×1 convolution.
5. The deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information according to claim 1, characterized in that: The self-correction DAB module includes: a first convolution layer, a rear convolution layer and a terminal convolution layer; the first convolution layer is a self-calibration convolution layer, and the rear convolution layer is a bottleneck structure consisting of a deep asymmetric dilated convolution and two branches; The bottleneck structure composed of the deep asymmetric dilated convolution and two branches includes a branch for extracting local feature information and another branch for using dilated convolution to expand the receptive field. The output results of the two branches are added to reach the terminal convolution layer, which is a 1×1 point-by-point convolution layer; the one branch is composed of 3×1 Dconv and 1×3 Dconv, and the other branch is composed of 3×1 Ddconv and 1×3 Ddconv.
6. The deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information according to claim 5, characterized in that: In the self-calibration convolutional layer, a segmentation step, a self-correction step and a splicing step are performed; The segmentation step includes: dividing the input X into two parts {X1, X2} on average; The self-correction step includes: assuming that there is a set of shapes (C, C, k h ,k w ) of the filter bank K, where k h and k w are the height and width of the filter respectively. The filter bank K is evenly divided into four parts to obtain four filters, which are expressed as The shape of each filter is (C / 2, C / 2, k h ,k w ), use the three filters {K2, K3, K4} to perform a self-correction operation on X1 to obtain the result Y1; use K1 to perform a convolution operation to obtain the result Y2; The stitching step includes: stitching the result Y1 and the result Y2 to obtain a self-calibration result.
7. The deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information according to claim 6, characterized in that: The self-correction operation of X1 using the three filters {K2, K3, K4} to obtain the result Y1 specifically includes: Use average pooling of size r×r to perform pooling operation on the input X1: T1=AvgPool(X1) The obtained T1 is sent to the K2 filter to perform a convolution operation, and the feature map is upsampled to the same size as X1 to obtain X1′: X1′=Up(f2(T1))=Up(T1*K2) Among them, Up(·) represents the process of upsampling by bilinear interpolation; Perform self-calibration operation according to the following formula: Y1′=f3(X1)·σ(X1+X1′)=(X1*K3)·σ(X1+X1′) Among them, σ represents the Sigmoid activation function, "·" represents the product of the corresponding elements; The output of the self-correction operation is: Y1=f4(Y1′)=Y1′*K4.
8. The deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information according to claim 1, characterized in that: The upsampling part includes a CCSP-ASPP module and an upsampling block; the upsampling block is used to restore the image output by the CCSP-ASPP module to the input image size; The CCSP-ASPP module is composed of a left branch with a residual edge function and a right branch using a cascaded ASPP module; the left branch includes a first 1Δ1 convolution layer; the right branch includes a second 1×1 convolution layer, a third 1×1 convolution layer, a first 3×3 convolution layer, a second 3×3 convolution layer, a third 3×3 convolution layer, a global average pooling layer, a fourth 1×1 convolution layer, an upsampling layer, a first concat layer, a fifth 1×1 convolution layer, a second concat layer and a sixth 1×1 convolution layer; For the right branch, the input of the CCSP-ASPP module passes through the second 1×1 convolution layer, and is sent to the third 1×1 convolution layer, the first 3×3 convolution layer, and the global average pooling layer respectively. The output of the first 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the second 3×3 convolution layer. The output of the second 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the third 3×3 convolution layer. The output of the global average pooling layer passes through the fourth 3×3 convolution layer and the upsampling layer in turn and is sent to the first concat layer. At the same time, the outputs of the third 1×1 convolution layer, the first 3×3 convolution layer, the second 3×3 convolution layer, and the third 3×3 convolution layer are all sent to the first concat layer. The output of the first concat layer passes through the fifth 1×1 convolution layer, the second concat layer, and the sixth 1×1 convolution layer in turn to obtain the output of the CCSP-ASPP module. For the left branch, the input of the CCSP-ASPP module is fed into the first 1×1 convolutional layer, and the output of the first 1×1 convolutional layer is fed into the second concat layer.
9. A deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information, characterized by: include: An image acquisition module, used for acquiring an input image; The semantic segmentation module is used to input the input image into the pre-trained semantic segmentation network to obtain the semantic segmentation result of the input image; The semantic segmentation network consists of a downsampling part, a self-correction part and an upsampling part; the self-correction part is constructed based on a self-correction DAB module, and the upsampling part is constructed by introducing a CCSP-ASPP module.
10. The deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information according to claim 9, characterized in that: The downsampling part consists of three 3×3 convolutional layers and a first downsampling block; three 3×3 convolutional layers are used to extract initial features, and the same first downsampling block as the ENet initial block is used, and the first downsampling block is a cascade of 3×3 convolution with a step size of 2 and 2×2 maximum pooling; A long-range skip connection is established between the input image and the first downsampling block.
11. The deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information according to claim 9, characterized in that: The self-correction part includes a self-correction part 1 and a self-correction part 2; The self-correction part 1 is composed of three self-correction DAB modules in cascade, and when the self-correction part 1 outputs, the input of the first self-correction DAB module is stacked with the output of the last self-correction DAB module; The self-correction part 2 is composed of 6 self-correction DAB modules in cascade, and when the self-correction part 2 outputs, the input of its first self-correction DAB module is stacked with the output of its last self-correction DAB module; The output of the self-correction part 1 reaches the self-correction part 2 through the second down-sampling block; and establishing a long-range skip connection between the input image and the second downsampling block; The output of the second self-correction part passes through a 1×1 convolution to reach the upsampling part, and a long-range skip connection is established between the input image and the 1×1 convolution.
12. The deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information according to claim 9, characterized in that: Each self-correction DAB module includes: a first convolution layer, a rear convolution layer and a terminal convolution layer; the first convolution layer is a self-calibration convolution layer, and the rear convolution layer is a bottleneck structure consisting of a deep asymmetric dilated convolution and two branches; The bottleneck structure composed of the deep asymmetric dilated convolution and two branches includes a branch for extracting local feature information and another branch for using dilated convolution to expand the receptive field. The output results of the two branches are added to reach the terminal convolution layer, which is a 1×1 point-by-point convolution layer; the one branch is composed of 3×1 Dconv and 1×3 Dconv, and the other branch is composed of 3×1 Ddconv and 1×3 Ddconv.
13. The deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information according to claim 12, characterized in that: In the self-calibration convolutional layer, a segmentation step, a self-correction step and a splicing step are performed; The segmentation step includes: dividing the input X into two parts {X1, X2} on average; The self-correction step includes: assuming that there is a set of shapes (C, C, k h ,k w ) of the filter bank K, where k h and k w are the height and width of the filter respectively. The filter bank K is evenly divided into four parts to obtain four filters, which are expressed as The shape of each filter is (C / 2,C / 2,k h ,k w ), use the three filters {K2, K3, K4} to perform a self-correction operation on X1 to obtain the result Y1; use K1 to perform a convolution operation to obtain the result Y2; The stitching step includes: stitching the result Y1 and the result Y2 to obtain a self-calibration result.
14. The deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information according to claim 13, characterized in that: The self-correction operation of X1 using the three filters {K2, K3, K4} to obtain the result Y1 specifically includes: Use average pooling of size r×r to perform pooling operation on the input X1: T1=AvgPool(X1) The obtained T1 is sent to the K2 filter to perform a convolution operation, and the feature map is upsampled to the same size as X1 to obtain X1′: X1′=Up(f2(T1))=Up(T1*K2) Among them, Up(·) represents the process of upsampling by bilinear interpolation; Perform self-calibration operation according to the following formula: Y1′=f3(X1)·σ(X1+X1′)=(X1*K3)·σ(X1+X1′) Among them, σ represents the Sigmoid activation function, "·" represents the product of the corresponding elements; The output of the self-correction operation is: Y1=f4(Y1′)=Y1′*K4.
15. The deep asymmetric bottleneck real-time semantic segmentation system integrating multi-scale information according to claim 9, characterized in that: The upsampling part includes a CCSP-ASPP module and an upsampling block; the upsampling block is used to restore the image output by the CCSP-ASPP module to the input image size; The CCSP-ASPP module is composed of a left branch with a residual edge function and a right branch using a cascaded ASPP module; the left branch includes a first 1×1 convolutional layer; the right branch includes a second 1×1 convolutional layer, a third 1×1 convolutional layer, a first 3×3 convolutional layer, a second 3×3 convolutional layer, a third 3×3 convolutional layer, a global average pooling layer, a fourth 1×1 convolutional layer, an upsampling layer, a first concat layer, a fifth 1×1 convolutional layer, a second concat layer and a sixth 1×1 convolutional layer; For the right branch, the input of the CCSP-ASPP module passes through the second 1×1 convolution layer, and is sent to the third 1×1 convolution layer, the first 3×3 convolution layer, and the global average pooling layer respectively. The output of the first 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the second 3×3 convolution layer. The output of the second 3×3 convolution layer is added to the output of the second 1×1 convolution layer and sent to the third 3×3 convolution layer. The output of the global average pooling layer passes through the fourth 3×3 convolution layer and the upsampling layer in turn and is sent to the first concat layer. At the same time, the outputs of the third 1×1 convolution layer, the first 3×3 convolution layer, the second 3×3 convolution layer, and the third 3×3 convolution layer are all sent to the first concat layer. The output of the first concat layer passes through the fifth 1×1 convolution layer, the second concat layer, and the sixth 1×1 convolution layer in turn to obtain the output of the CCSP-ASPP module. For the left branch, the input of the CCSP-ASPP module is fed into the first 1×1 convolutional layer, and the output of the first 1×1 convolutional layer is fed into the second concat layer.
16. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a deep asymmetric bottleneck real-time semantic segmentation method integrating multi-scale information as described in any one of claims 1 to 8 are implemented.
17. A storage medium, characterized in that: The storage medium stores a semantic segmentation program, which, when executed by at least one processor, implements the steps of a deep asymmetric bottleneck real-time semantic segmentation method that integrates multi-scale information as described in any one of claims 1 to 8.
Citation Information
Cited By
Remote sensing image semantic segmentation method based on feature interaction enhancement
CN119992099A
