A lightweight semantic segmentation method based on deep neural network

Abstract

A lightweight semantic segmentation method based on a deep neural network, the method comprising: S100: constructing a lightweight semantic segmentation network, the lightweight semantic segmentation network being a global-local context network GLCNet, the GLCNet network structure comprising a convolution block module, a down-sampling module, a global-local context GLC module and a multi-resolution fusion MRF module; S200: performing semantic segmentation on an input image by using the lightweight semantic segmentation network. The method achieves a good balance among segmentation precision, efficiency and parameter quantity.

Description

Technical Field

[0001] This disclosure belongs to the field of neural network and image processing technology, and specifically relates to a lightweight semantic segmentation method based on deep neural networks. Background Technology

[0002] Semantic segmentation divides an image into regions with different semantic categories. It can be viewed as a pixel classification task and has been successfully applied in many vision-related scenarios, such as autonomous driving, medical image segmentation, and 3D scene recognition. In recent years, the superior performance of semantic segmentation has benefited from the development of convolutional neural networks, especially the introduction of fully convolutional networks (FCNs), which utilizes convolutional neural networks (CNNs) to predict pixel categories.

[0003] Currently, more and more visual application scenarios (such as autonomous driving, robotics, etc.) have high requirements for the speed of segmentation algorithms. A series of segmentation methods based on FCN mostly use deep and wide backbone networks to extract features, which requires a large number of parameters and a long inference time. They ignore factors such as computational efficiency, number of parameters and memory usage, making it difficult to apply them to scenarios with limited system resources. People have conducted a lot of research on building lightweight segmentation models. Existing lightweight semantic segmentation networks can be roughly divided into two categories: (1) methods based on continuous downsampling and channel compression to reduce redundant information and thus improve inference speed. (2) methods based on dual-branch modules to reduce the number of parameters. The above two types of methods have achieved good results in balancing accuracy and speed, but continuous downsampling will gradually lose the spatial information of the input features, and dual-branch modules ignore the role of global information in segmentation.

[0004] In summary, most current semantic segmentation algorithms based on convolutional neural networks involve a large number of parameters and high computational complexity, limiting their application in real-time processing scenarios. Therefore, it is necessary to construct a lightweight semantic segmentation network to address its application in resource-constrained environments. Summary of the Invention

[0005] To address the aforementioned problems, this disclosure provides a lightweight semantic segmentation method based on deep neural networks, which includes:

[0006] S100: Construct a lightweight semantic segmentation network, which is a Global-Local Context Network (GLCNet). The GLCNet network structure includes a convolutional block module, a downsampling module, a Global-Local Context (GLC) module, and a Multi-Resolution Fusion (MRF) module.

[0007] S200: Use this lightweight semantic segmentation network to perform semantic segmentation on the input image.

[0008] Through the above technical solution, the lightweight semantic segmentation network GLCNet achieves a good balance between segmentation accuracy and speed while reducing the number of parameters and saving memory. The lightweight semantic segmentation network GLCNet mainly includes a global-local context module and a multi-resolution fusion module. The global-local context module consists of a global feature extractor and a local context feature extractor, which extract different feature information respectively, then concatenate the two branches of features, and model the correlation between features through residual connections, which is beneficial for learning feature information in complex scenes. Multi-spectral channel attention and convolution operations are used to learn the global and local context information of the image, while residual connections are used to model the correlation between information; this module has low computational cost. The multi-resolution fusion module first uses multi-spectral channel attention to focus on important information in low-level features and upsamples high-level features; then, it propagates the spatial information of high-resolution features to low-resolution features to enhance the spatial information of low-resolution features; this module has low parameter cost. The effectiveness of this method is verified on the Cityscapes and Camvid datasets, achieving mIoU of 69.89% and 68.86% respectively, demonstrating good segmentation performance. When the input image size is 512×1024, the speed can reach 87 frames per second; when the input image size is 360×480, the speed can reach 122 frames per second, which can meet the requirements of real-time segmentation. Attached Figure Description

[0009] Figure 1 This is a flowchart of a lightweight semantic segmentation method based on a deep neural network provided in one embodiment of this disclosure;

[0010] Figure 2 This is a structural diagram of GLCNet in one embodiment of this disclosure;

[0011] Figure 3 This is a structural diagram of a GLC module in one embodiment of this disclosure;

[0012] Figure 4 This is a visualization comparison of the Cityscapes dataset in one embodiment of this disclosure;

[0013] Figure 5 This is a visualization comparison of the Camvid dataset in one embodiment of this disclosure. Detailed Implementation

[0014] The following is in conjunction with the appendix Figures 1 to 5 The present invention will now be described in further detail.

[0015] In one embodiment, see Figure 1 It discloses a lightweight semantic segmentation method based on deep neural networks, which includes:

[0016] S100: Construct a lightweight semantic segmentation network, which is a Global-Local Context Network (GLCNet). The GLCNet network structure includes a convolutional block module, a downsampling module, a Global-Local Context (GLC) module, and a Multi-Resolution Fusion (MRF) module.

[0017] S200: Use this lightweight semantic segmentation network to perform semantic segmentation on the input image.

[0018] In this embodiment, the method mainly consists of a Global-Local Context (GLC) module and a Multi-Resolution Fusion (MRF) module. The GLC module learns global and local context information of the image, and uses residual connections to enhance the dependencies between features. Based on this, a multi-resolution fusion module is proposed to aggregate features from different stages. First, low-resolution features are upsampled, and then fused with high-resolution features to enhance the spatial information of high-level features.

[0019] Compared to methods that only consider segmentation performance, this approach aims to design a lightweight network architecture to achieve a balance between performance, parameter count, and speed. This method designs a lightweight global-local contextual semantic segmentation network, GLCNet, which mitigates the spatial information loss caused by successive downsampling. This approach achieves a better balance between parameter count, speed, and performance, making GLCNet more suitable for resource-constrained systems.

[0020] This method uses the proposed GLC module as the main component to design a lightweight GLCNet network, the detailed structure of which is shown below. Figure 2 As shown. The design principle of this network is to reduce the number of parameters, save memory, and improve speed. Unlike deep convolutional networks (VGG, ResNet, etc.), this structure is mainly formed by stacking several small-channel convolutions. Secondly, this method reduces the resolution of the original image to 1 / 8, which is beneficial for preserving spatial information. This is because high-precision semantic segmentation models usually use 5 downsampling operations to obtain a feature map with a resolution of 1 / 32 of the original image to obtain abstract feature information of objects; however, multiple downsampling operations lose spatial information. Figure 2 The two downsampling modules in the model have the same structure.

[0021] In another embodiment, the convolutional block module in step S100 includes a 3×3 convolution for dimensionality transformation, a 1×1 convolution for inter-channel interaction, and a 3×3 convolution with a stride of 2.

[0022] In this embodiment, the method first uses three convolutions to extract initial features C1, where the 3×3 convolution is used for dimensional transformation, mapping low-dimensional features to high-dimensional features. Then, a 1×1 convolution is used to achieve interaction between channels. Finally, a 3×3 convolution with a stride of 2 is used to reduce the resolution of the original image to 1 / 2.

[0023] In another embodiment, the downsampling module in step S100 includes a 3×3 convolution with a stride of 2 and a 2×2 pooling operation with a stride of 2.

[0024] In this embodiment, two methods are used to reduce the image resolution: 3×3 convolution with a stride of 2 and 2×2 pooling with a stride of 2. The outputs of the two branches are then concatenated as the input for the next stage. This structure has strong feature representation capabilities. Using this downsampling method, the resolution of the original image is reduced to 1 / 4 and 1 / 8.

[0025] By using the GLC module after downsampling, GLCNet can aggregate global and local contextual information at both spatial and semantic levels. Compared to structures that capture contextual information only at the semantic stage, GLCNet can capture both global and local contextual information of objects at every stage.

[0026] Then, the MRF module is used to propagate low-level spatial information to high-level features, enabling features from different stages to learn from each other. The aggregated feature information is then input into a 3×3 depthwise convolution to enhance the interaction between information flows, and finally, a 1×1 convolution is used to output the pixel category.

[0027] In another embodiment, the Global-Local Context (GLC) module in step S100 consists of a global information extractor branch and a local context information extractor branch.

[0028] In this embodiment, the GLC module consists of two branches: a global information extractor and a local context information extractor, such as... Figure 3 As shown.

[0029] In another embodiment, at the beginning of each Global-Local Context (GLC) module, a 3×3 convolution is used to process the input features F. in ∈R C×H×W The number of channels is halved, and its output is represented as F. out This is used as the input to both the global information extractor branch and the local context information extractor branch, where R is the real number field and C is the F... in The number of channels, H is Fin The height, W is F in The width.

[0030] In this embodiment, a 3×3 convolution is used because it has a larger receptive field, using the features after the channels are halved as inputs to both branches.

[0031] In another embodiment, the global information extractor branch uses multi-spectral channel attention to capture feature information. The specific process of capturing feature information is as follows: first, F... out ∈R C′×H×W The features are divided into i groups, with c = C′ / i after grouping. Each group's features are preprocessed using a 2D Discrete Cosine Transform (DCT). Then, a heuristic two-step criterion is used to select the corresponding frequency components for each group's features; where F... out This means using 3×3 convolution to transform the input features F in The output feature obtained after halving the number of channels is C′=C / 2, where C is F in The number of channels, C′ is F out The number of channels, H is F out The height, W is F out The width.

[0032] In this embodiment, the global information extractor branch primarily uses multi-spectral channel attention to capture feature information, first by transferring F... out They are divided into i groups, denoted as i. In this method, i = 16, and the number of channels after grouping is c = C′ / i. Each group of features is preprocessed using 2D DCT, and the output for each group is as follows:

[0033] F out =Conv 3×3 (F in )

[0034] Wherein, this formula will input feature F in The number of channels is halved, Conv 3x3 This is a convolution operation with a kernel size of 3x3.

[0035]

[0036] This formula is the 2D discrete cosine transform formula. 2d Let H be the height of the input feature and W be the width of the input feature. h belongs to the set {0, 1, ..., H-1}, and w belongs to the set {0, 1, ..., W-1}.

[0037]

[0038] This formula calculates the frequency components of each set of input features using a 2D discrete cosine transform. u and v are the 2D indices of the frequency composition, and Freq... i This is the output result.

[0039] In another embodiment, the use of a heuristic two-step criterion to select the corresponding frequency component for each set of features specifically involves: first, evaluating the importance of each channel frequency component; then, selecting the top i high-performance frequency components and concatenating them; inputting the concatenated frequency components into a fully connected layer to learn and obtain an attention map; and then multiplying them with the input features for weighting.

[0040] In this embodiment, the importance of each channel frequency component is first evaluated. Then, the top 16 high-performance frequency components are selected and concatenated. The concatenated frequency components are then input into a fully connected layer to learn and obtain an attention map, which is then multiplied by the input features for weighting. The above process is represented as follows:

[0041] Freq = cat([Freq 0 Freq 1 , ..., Freq i-1 ])

[0042] Wherein, the formula represents the frequency components of all groups, and Freq is the output result.

[0043] F glo = sigmoid(fc(Freq))

[0044] Wherein, this formula calculates the weights for the frequency components, fc represents a fully connected network, sigmoid is the activation function, and F... glo This is the output of the process.

[0045] In another embodiment, the local context information extractor branch consists of convolution decomposition and deep dilated convolution.

[0046] In this embodiment, the standard 3×3 convolution is first decomposed into asymmetric 3×1 and 1×3 convolutions to capture local information. Then, dilated convolutions are used to capture contextual information at different scales, while depthwise convolutions are used to reduce model parameters and memory usage. Finally, the outputs of the two branches are concatenated to enhance mutual learning between the features while maintaining the number of feature channels. Residual connections are used to connect the input and output, enhancing the correlation between spatial and semantic information. This method promotes the propagation of information flow between features with fewer parameters and lower computational cost. The above operations are represented as follows:

[0047] F1 = Conv 3×1 (Conv l×3 (F out ))

[0048] F locc =DW-D-Conv 3×3 (F1)

[0049] output = F in +cat(F locc F glo )

[0050] In the above representation, F out This represents the output after dimension adjustment and the inputs Conv to the two branches. m×n This represents a convolution operation with a kernel size of m×n, DW-D-Conv. 3×3 This represents a depthwise dilated convolution. The outputs of the two branches are F... glo and F locc，output This indicates the output of the GLC module.

[0051] In another embodiment, the multi-resolution fusion MRF module first uses multi-spectral channel attention to process feature map C1 to focus on important information, and then applies a 1×1 convolution to adjust the number of channels; next, feature maps C2 and C3 are upsampled by 2x and 4x respectively to match the spatial resolution of the low-level feature maps; then, the processed C1 feature map is added to the 2x upsampled feature map, and a 1×1 convolution is applied to adjust the dimension of the added feature; finally, the dimension-adjusted feature map is fused with the 4x upsampled feature map C3, and the output is input into a 3×3 convolution to further fuse the features.

[0052] In this embodiment, to fully utilize the feature information at each stage, a multi-resolution fusion module is proposed. First, an attention mechanism is used to process feature map C1 to focus on important information, and then a 1×1 convolution is applied to adjust the number of channels to 64. Next, feature maps C2 and C3 are upsampled by 2x and 4x respectively to match the spatial resolution of the lower-level feature maps. The processed feature map C1 is added to the 2x upsampled feature map, and a 1×1 convolution is applied to adjust the dimension of the added feature map to 128. Then, the dimension-adjusted feature map is fused with the 4x upsampled feature map C3. The above process is represented as follows:

[0053] C′1=MSCA(C1)

[0054]

[0055] Here, MSCA represents multi-spectral channel attention, and its output is represented as C′1. p This represents the upsampling operation, where α represents the upsampling factor, and C... i Represents the i-th feature. This indicates that for feature map C′ i-1Perform a 1×1 convolution.

[0056] In another embodiment, feature map C1 is the initial feature map, feature map C2 is the output feature map of four stacked global-local context (GLC) modules after the first downsampling module, and feature map C3 is the output feature map of five stacked global-local context (GLC) modules after the second downsampling module.

[0057] In this embodiment, different numbers of GLC modules are stacked after downsampling to obtain global and local context information. First, four GLC modules are stacked after the feature map size is 1 / 4, with each module having a corresponding dilatancy of 1, resulting in feature map C2. Second, five GLC modules are stacked after the feature map size is 1 / 8, with corresponding dilatancy of (4, 4, 8, 8, 12), resulting in feature map C3.

[0058] In another embodiment, the datasets used to validate the effectiveness of this method are the Cityscapes and CamVid datasets. The Cityscapes dataset consists of 5,000 finely annotated images and 20,000 coarsely annotated images. This method uses only the finely annotated images, which are 1024×2048 pixels in size, with each pixel contained in 19 predefined classes. These images are divided into training, validation, and test sets, containing 2,975, 500, and 1,525 images respectively.

[0059] CamVid is another challenging street view dataset. It consists of 701 images extracted from different video sequences, each image being 720×960 pixels in size, with each pixel contained in one of 11 predefined classes. These images were divided into training, validation, and test sets, containing 367, 101, and 233 images respectively.

[0060] Evaluation metrics: Segmentation accuracy is measured by mean intersection-to-union ratio (mIoU), segmentation speed is measured by frames per second (FPS), and memory consumption is evaluated by the number of parameters (Params).

[0061] Experimental environment: The system environment is Ubuntu 18.04, Python 3.6, PyTorch 1.0, CPU is Intel(R) Xeon(R) Gold 6136, GPU is NVIDIA TITAN V, and CUDA version is 10.2.

[0062] Parameter settings: During the training phase, the Adam optimizer is used to update the network parameters, with the exponential decay rate set to the default value and the weight decay rate set to 2 × 10⁻⁴. A Poly learning strategy is used to adaptively adjust the learning rate, ensuring that the learning rate after each iteration is base_lr × (1 - iter / iter). max ), where base_lr is 1×10⁻³, iter is the current iteration number, and iter max The maximum number of iterations was set to 16. The proposed method was trained for 1000 epochs on both datasets with a batch size of 16. Random horizontal flipping and random scaling were used to augment the training data, with the scaling range being [0.5, 2]. The input image sizes for the Cityscapes and Camvid datasets were 1024×512 and 480×360, respectively. During the testing phase, only a single scale was used for testing.

[0063] Figure 4 and Figure 5 Visualizations of our proposed method versus other segmentation algorithms on the Cityscapes and Camvid datasets are presented respectively. The results demonstrate that our method accurately identifies and segments objects. Figure 4 In the first and third rows shown, our method more accurately identifies vehicle pixels in the image, while other methods produce incomplete vehicle contour segmentation. In the fourth row, our method better identifies road pixels, while EDANet, FastSCNN, and CGNet are affected by interference from other pixels. In the fifth row, EDANet and FastSCNN fail to better identify traffic light signs, but our method accurately determines the object's location and presents clear segmentation results. For pixels such as buildings, railings, and small objects, our method achieves better segmentation results.

[0064] like Figure 5 In the first row, the segmentation methods CGNet and DABNet perform relatively coarse segmentation of streetlights. In the second row, other segmentation methods classify edge pixels of trees as part of other pixels, while our method identifies edge pixels better. In the third row, the segmentation methods CGNet, ERFNet, and DABNet fail to identify wall pixels well, but our method provides clearer segmentation results for walls. In the fourth row, our method identifies road pixels better. Furthermore, our method also has advantages in identifying pixels of traffic signs, vehicles, and buildings.

[0065] analyze Figure 4 and Figure 5 It can be concluded that this method can better identify pixel categories and present better visualization results.

[0066] The quantitative analysis of algorithm performance is as follows. For the Cityscapes dataset, Table 1 presents the comparison results of different segmentation methods on the Cityscapes test set, where None indicates no backbone network is used. The results show that this method maintains a better balance between parameter count, speed, and accuracy. Specifically, the analysis focuses on the following three aspects: Parameter count: Compared to ENet and FSSNet, which have only 1 / 3 the number of parameters of this method, their corresponding accuracies are approximately 10% and 5% lower, respectively. Speed: Compared to algorithms with similar parameter counts such as EDANet, CGNet, NDNet, and CEPNet, this method is not only faster but also more accurate. Accuracy: Compared to algorithms using lightweight backbone networks such as SGCPNet, DFANet, and BiseNet, this method achieves higher accuracy in all aspects, but is slightly slower. This is because lightweight backbone networks filter out a large amount of irrelevant information in the early stages, resulting in relatively faster speeds in later operations. Compared to ICNet, SwiftNet, and BSDNet algorithms, this method achieves higher accuracy with fewer parameters and faster speed. While some algorithms, such as BiseNetV2, do outperform this method in performance and speed, BiseNetV2 has 3.4M parameters, while this method has only 0.68M. The reason for the slower speed but fewer parameters in this method lies in the convolution operations and significant upsampling operations in the multi-resolution fusion module. In summary, this method achieves higher accuracy and faster speed with fewer parameters.

[0067]

[0068] Table 1

[0069] To further verify the effectiveness of this method, Table 2 presents the comparison results of this method with other segmentation methods on the Camvid test set. The results show that the performance of this method is second only to FBSNet. Compared with models with fewer parameters, the accuracy of this method is improved by 17.3% and 3.2% compared to ENet and CGNet models, respectively. Compared with models such as DABNet, FDDWNet, and LDPNet with similar parameter counts, this method achieves higher segmentation performance. Furthermore, compared with models such as DFANet, RGPNet, and BSDNet with larger parameter counts, this method achieves higher segmentation performance with fewer parameters.

[0070]

[0071] Table 2

[0072] This method was tested on the Camvid dataset to verify the effectiveness of each module. The model was trained using the training and validation sets of the Camvid dataset to verify the impact of each module on accuracy and parameter count. Evaluation was performed on the Camvid test set, and the results are shown in Table 3. RC represents residual connections, and MRF represents multi-resolution fusion modules.

[0073]

[0074] Table 3

[0075] Fusion Method: First, the aggregation method of two branches in the GLC module was studied. The results show that the splicing operation is not only more effective than the addition operation, but also reduces the number of parameters by 0.13M. Furthermore, residual connections are used to establish relationships between information, enhancing the propagation of information flow. The results show that this operation improves the segmentation accuracy by 0.17%, and none of the above operations affect the number of parameters while maintaining high computational efficiency.

[0076] Attrition Rate: Due to the varying sizes of objects in the dataset, a higher attrition rate for small objects can introduce interference and affect discrimination, while a lower attrition rate for large objects is detrimental to obtaining contextual information. Therefore, designing a reasonable attrition rate plays a crucial role in acquiring contextual information. This method designed four combinations of attrition rates to study their impact on the model. Table 3 shows that adjusting the number of GLC modules and the attrition rate combination at different stages can improve performance. Higher accuracy was achieved when the number of GLC modules was 4 and 5, respectively. Changing the attrition rate while adjusting the number of GLC modules showed that the attrition rate combination d = (1, 1, 1, 1, 4, 4, 8, 8, 12) achieved better results, with a segmentation accuracy of 68.15%, and this operation did not increase the number of parameters.

[0077] Multi-resolution fusion module: The multi-resolution fusion module can aggregate feature information from different stages. Unlike the skip structure, this module first aggregates the low-level features from the first two stages, and then propagates the low-level information flow to the high-level features. Experimental results show that compared with not using the multi-resolution fusion module, the performance is improved by 0.7%, while the number of parameters added by this module is only 0.1M.

[0078] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A lightweight semantic segmentation method based on deep neural networks, comprising the following steps: S100: Construct a lightweight semantic segmentation network, which is a Global-Local Context Network (GLCNet). The GLCNet network structure includes a convolutional block module, a downsampling module, a Global-Local Context (GLC) module, and a multi-resolution fusion (MRF) module. S200: Use this lightweight semantic segmentation network to perform semantic segmentation on the input image; The Global-Local Context (GLC) module in step S100 consists of a global information extractor branch and a local context information extractor branch. The global information extractor branch uses multi-spectral channel attention to capture feature information. The specific process for capturing feature information is as follows: First, [the following is a list of steps, likely related to a global information extractor branch]... Divide into i groups, and the number of channels after grouping is For each set of features, a 2D Discrete Cosine Transform (DCT) is used for preprocessing; then, a heuristic two-step criterion is used to select the corresponding frequency components for each set of features; where, This indicates that the input features are processed using a 3×3 convolution. The output features obtained after halving the number of channels C is The number of channels, for The number of channels, H is The height, W is The width; wherein, the use of the heuristic two-step criterion to select the corresponding frequency component for each group of features is specifically as follows: first, evaluate the importance of each channel frequency component, then select the top i high-performance frequency components and concatenate them, input the concatenated frequency components into the fully connected layer to learn and obtain the attention map, and then multiply them with the input features for weighting; The local context information extractor branch consists of convolution decomposition and deep dilated convolution.

2. The method according to claim 1, wherein, The convolutional block module in step S100 includes a 3×3 convolution for dimensionality transformation, a 1×1 convolution for inter-channel interaction, and a 3×3 convolution with a stride of 2.

3. The method according to claim 1, wherein, The downsampling module in step S100 includes a 3×3 convolution with a stride of 2 and a 2×2 pooling with a stride of 2.

4. The method according to claim 1, wherein, At the beginning of each Global-Local Context (GLC) module, a 3×3 convolution is used to process the input features. The number of channels is halved, and its output is represented as... This is used as the input to both the global information extractor branch and the local context information extractor branch, where R is the real number field and C is... The number of channels, H is The height, W is The width.

5. The method according to claim 1, wherein, The multi-resolution fusion MRF module first uses multi-spectral channel attention to process the feature map. To focus on important information, a 1×1 convolution is applied to adjust the number of channels; then, the feature maps are processed separately. and Upsampling is performed at 2x and 4x to match the spatial resolution of the low-level feature maps; then the processed... The feature map is added to the feature map after a 2x upsampling, and a 1×1 convolution is applied to adjust the dimension of the added feature map; finally, the dimension-adjusted feature map is fused with the feature map after a 4x upsampling. The output is then fed into a 3×3 convolution to further fuse features.

6. The method according to claim 5, wherein, The feature map This is the initial feature map, the feature map The output feature maps are the result of stacking four Global-Local Context (GLC) modules after the first downsampling module; the feature maps The output feature map is the result of stacking five global-local context (GLC) modules after the second downsampling module.

Images