Real-time semantic segmentation method based on efficient feature extraction and context aggregation

Abstract

The present invention relates to a real-time semantic segmentation method based on efficient feature extraction and context aggregation, belonging to the technical field of image semantic segmentation. The method constructs an EFCANet model and mainly includes: an encoding stage: a bilateral structure is adopted, in which the spatial branch uses pooling operations and standard convolution to retain shallow spatial information and enhance feature propagation; a context branch is used to provide a large receptive field and capture deep context information; these two branches are merged at different stages of the network to enhance information propagation between different layers; and a decoding stage: a context fusion module (CFM) is constructed by combining depthwise separable convolution and channel rearrangement operations to fuse feature information at different layers to obtain a feature fusion map. The present invention improves the segmentation effect of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation and relates to a real-time semantic segmentation method based on efficient feature extraction and context aggregation. Background Art

[0002] Autonomous driving technology primarily utilizes onboard sensors to perceive road conditions and accurately and rapidly identify and process the captured image information to ensure driving safety. Semantic segmentation is a challenging research topic in computer vision, aiming to assign a category to each pixel in a given image. Currently, autonomous driving applications often demonstrate a strong demand for real-time inference speed and high accuracy. Therefore, designing segmentation algorithms that balance segmentation accuracy and processing speed is a hot topic in autonomous driving research.

[0003] Early image segmentation methods primarily separated objects from backgrounds based on low-level image features (such as color and shape), but this approach had significant limitations when applied to complex scenes. With the development of deep learning, multiple research branches have emerged in this field, with the fully convolutional network (FCN) proposed by Long et al. being a major research direction.

[0004] Based on the FCN architecture, deep learning semantic segmentation models are categorized into high-quality and high-efficiency models. Research on high-quality models focuses on achieving high accuracy without paying much attention to inference speed, such as the DeepLab series, PSPNet, and AML. These high-precision algorithms are often computationally intensive and require many parameters, failing to meet application requirements. High-efficiency models strike a balance between segmentation accuracy and computational resources. ENet, the first real-time segmentation network, reduces image resolution to improve inference speed, but at the expense of some spatial information. ERFNet decomposes the original n×n convolution kernel into an n×1 convolution and a 1×n convolution, significantly reducing network parameters. The paper "Si H, Zhang Z, Lv F, et al. Real-time semantic segmentation via multiply spatial fusion network[J].arXiv preprint arXiv:1911.07217,2019" uses a multi-feature fusion module to enhance inter-layer information flow, expanding the receptive range and restoring spatial information while minimizing computational overhead. FRSSNet introduces the Ghost module (based on a set of intrinsic feature maps, using linear operations to reveal the information behind the intrinsic features) to generate redundant information of the network's intermediate feature maps at the cost of a small number of parameters, reducing computational consumption; BiseNet V2 introduces spatial paths and semantic paths to reduce computation.

[0005] Most previous networks have neglected segmentation efficiency, meaning their computational and storage requirements are extremely high, making them difficult to meet the needs of real-world applications where information must interact rapidly with the environment. Furthermore, electronic devices, such as robots, mobile phones, and telemedicine, have limited storage capacity and computational cost, making them unable to support large and complex algorithms.

[0006] Therefore, it is urgent to design a lightweight and efficient network to balance the segmentation accuracy and running speed in autonomous driving scenarios. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a real-time semantic segmentation method based on efficient feature extraction and context aggregation to solve the problem of finding a balance between segmentation accuracy and running speed in existing unmanned driving scenarios and improve the segmentation effect.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A real-time semantic segmentation method based on efficient feature extraction and context aggregation is proposed, specifically by constructing an EFCANet network structure model, including encoding and decoding stages;

[0010] Encoding stage: A bilateral structure is adopted, in which the spatial branch uses pooling operations and standard convolution to preserve shallow spatial information and enhance feature propagation; the context branch is used to provide a large receptive field and capture deep contextual information. These two branches are merged at different stages of the network to enhance information propagation between different layers.

[0011] Decoding stage: Combining depth-wise separable convolution and channel rearrangement operations to construct a context fusion module (CFM) to fuse feature information at different levels and further improve the segmentation effect of the model.

[0012] Furthermore, the context branch structure is:

[0013] Layers 1 to 3: 3×3 convolution;

[0014] Layer 4: downsampling;

[0015] Layers 5 to 7: Efficient Feature Extraction Unit A, or EFEU-A. The Efficient Feature Extraction Unit EFEU is constructed using a residual structure.

[0016] Layer 8: 3×3 convolution;

[0017] Layers 9 to 14: Efficient Feature Extraction Unit B, or EFEU-B.

[0018] Furthermore, the efficient feature extraction unit EFEU-A extracts shallow features through 3×3 convolution, 1×1 convolution, 3×3 dilated convolution and channel rearrangement operations in sequence.

[0019] Furthermore, the efficient feature extraction unit EFEU-B captures deep contextual information through 3×1 convolution, 1×3 convolution, 1×1 convolution, 5×5 depth-expanded convolution and channel rearrangement operations in sequence.

[0020] Furthermore, the PReLU activation function and batch normalization operation are introduced in EFEU.

[0021] Furthermore, the CFM module first inputs the feature information of different layers into the depth-wise separable convolutional layer to extract features; then average pools the feature map with 1 / 2 resolution and downsamples it to 1 / 4 to obtain X1, and at the same time bilinearly upsamples the feature map with 1 / 8 resolution to 1 / 4 to obtain X3, adds the obtained feature maps, and increases the information exchange between different channels through channel rearrangement operation, thereby further improving the model performance.

[0022] The beneficial effects of the present invention are that, compared with existing image segmentation models, the EFCANet model constructed by the present invention can achieve a better balance between segmentation accuracy and inference speed, and improve the accuracy and efficiency of the segmentation model. This is specifically reflected in the following aspects:

[0023] First, a residual structure is used to construct an efficient feature extraction unit (EFEU) to better perceive spatial and semantic information. Second, a bilateral structure is adopted, in which the spatial branch is combined with a pooling operation to retain shallow spatial information, and the context branch is used to provide a large receptive field and capture deep contextual information. These two branches are merged at different stages of the network to strengthen information propagation between different layers. Finally, a context fusion module is constructed by combining depthwise separable convolution and channel reordering operations to fuse information between different layers and further improve the segmentation effect of the model. Specifically, it achieves 73.2% mIOU on the Cityscapes test dataset at a speed of 102.3FPS for 512×1024 input.

[0024] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0026] Figure 1 The structural diagram of the EFCANet model constructed for the present invention;

[0027] Figure 2 It is the structural diagram of the efficient feature extraction unit (EFEU), where Figure 2 (a) is the structural diagram of EFEU-A. Figure 2 (b) is the structural diagram of EFEU-B;

[0028] Figure 3 This is the structural diagram of the context fusion module (CFM). DETAILED DESCRIPTION

[0029] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0030] See also Figures 1 to 3 , Figure 1 This is the model structure diagram of the EFCANet proposed in this paper, which mainly consists of two parts: the encoding stage and the decoding stage. The encoding stage consists of four parts: the initial module, the context branch, the spatial branch, and the long connection. The context branch consists of two stages (the detailed architecture of the context branch is shown in Table 1, where r represents the hole rate). This branch mainly collects deep context information, while the spatial branch uses pooling operations and standard convolution to preserve shallow spatial information and enhance feature propagation. The decoding stage begins with the context fusion module (CFM).

[0031] Table 1 Detailed structure of context branches

[0032] layer operate Output size 1-3 3×3 convolution 256×512×16 4 Downsampling 128×256×64 5-7 EFEU-A (r=2,2,2) 128×256×64 8 3×3 convolution 64×128×128 9-11 EFEU-B (r = 4, 4, 8) 64×128×128 12-14 EFEU-B (r = 8, 16, 16) 64×128×128

[0033] Each module consists of:

[0034] 1) Efficient Feature Extraction Unit (EFEU)

[0035] The Efficient Feature Extraction Unit (EFEU) designs the corresponding modules according to the image resolution, thereby improving the feature extraction performance while maintaining a good inference speed. When the image size is 128×256, the designed feature unit is as follows: Figure 2As shown in (a), 3×3 convolution, 1×1 convolution, 3×3 dilated convolution and channel rearrangement operations are performed in sequence. Since this module is used to extract shallow features, 3×3 dilated convolution is mainly used to increase the receptive field of the model without reducing the accuracy of feature extraction; 1×1 convolution is used before 3×3 dilated convolution mainly to fuse the information between channels, and channel rearrangement operation is used to strengthen the information exchange between channels. When the image size is 64×128, the input feature unit is designed as follows Figure 2 As shown in (b), through 3×1 convolution, 1×3 convolution, 1×1 convolution, 5×5 depth-expanded convolution, and channel rearrangement operations, because the image resolution is further reduced at this time, accuracy needs to be further improved. At this time, stacking one-dimensional convolution is slower than two-dimensional convolution. Therefore, the 3×3 convolution is decomposed into 3×1 convolution and 1×3 convolution. At the same time, the 3×3 depth-expanded convolution is replaced with a 5×5 depth-expanded convolution. This does not increase the computational burden too much while improving accuracy. In addition, the PReLU activation function and batch normalization operation are introduced in EFEU to speed up network training and increase the network's expressive power.

[0036] 2) Context Fusion Module (CFM)

[0037] The purpose of feature fusion is to combine relevant features in the image into a feature vector with more discriminative information than the input features. Based on this idea, the context fusion module (CFM) is designed to integrate multi-scale context information, such as Figure 3 shown.

[0038] The CFM module first inputs the feature information of different layers into the depthwise separable convolutional layer to extract features, effectively reduce parameters, and expand the receptive field. It then performs average pooling on the 1 / 2 resolution feature map and downsamples it to 1 / 4 to obtain X1. At the same time, the 1 / 8 resolution feature map is bilinearly upsampled to 1 / 4 to obtain X3. The resulting feature maps are added together, and the channel rearrangement operation is used to increase information exchange between different channels, further improving model performance.

[0039] 1. Verification experiment:

[0040] 1) Comparison of speed and accuracy on the Cityscapes dataset

[0041] This experiment verifies the algorithm model on the Cityscapes test set and compares it with the current excellent algorithm models. The experimental results are shown in Tables 2 and 3.

[0042] Table 2 Evaluation results of different algorithms on the Cityscapes test set

[0043] Algorithm Model Para(M) mIOU (%) FPS ENet 0.36 58.3 76.9 LEDNet 0.94 69.2 71 DABNet 0.76 70.1 27.7 EDANet 0.68 67.3 108.7 DFANet 7.8 71.3 100 BiseNet 5.8 68.4 105 MSCFNet 1.15 71.9 50 BSDNet 1.2 68.3 84.6 EFCANet (the present invention) 1.23 73.2 102.3

[0044] Table 3 IOU (%) performance of each category on the Cityscapes test set

[0045]

[0046]

[0047] The network proposed in this paper has 1.23M parameters. Despite having fewer parameters compared to ENet, its accuracy is 14.9 percentage points lower than EFCANet proposed in this paper, a significant gap in the segmentation field. LEDNet and DABNet use residual connections and factorized convolutions to improve efficiency and accuracy, but both perform worse than EFCANet. EDANet combines dilated convolutions and dense connections to achieve high efficiency, running 6.4 FPS faster than EFCANet, but its accuracy is 5.9 percentage points lower. DFANet uses multi-scale feature extraction and class-based relationships to improve segmentation accuracy. It also uses a lightweight network as a backbone to enhance real-time performance, but its large parameter count makes it difficult to apply in practice. BiseNet achieves higher accuracy by combining multiple branches, but still lags behind EFCANet. MSCFNet and BSDNet achieve good accuracy by introducing an attention mechanism, but this introduces some computational overhead, resulting in a decrease in inference speed. Compared to these models, EFCANet proposed in this paper strikes a better balance between accuracy and inference speed.

[0048] Table 3 further gives the segmentation accuracy value (IOU) of each category on the Cityscapes dataset. The results show that the segmentation accuracy of most categories exceeds that of other algorithms, which illustrates the effectiveness of the algorithm proposed in this paper.

[0049] 2. Ablation experiment:

[0050] 1) Ablation study of different voiding rates

[0051] Dilated convolutions are used in EFEU to effectively aggregate multi-scale contextual information. To explore the impact of different dilation rates on network performance, while the dilation rate in EFEU-A remains unchanged, EFEU-B was tested with different dilation rates: {4,4,4,4,4,4}, {4,4,8,8,16,16}, {3,3,7,7,13,13}, and {5,5,9,9,19,19}. The results are shown in Table 4. When the dilation rate is set too low, the receptive field is insufficient, resulting in insufficient contextual information extraction. When the dilation rate is set too high, the continuity of information is lost and long-range information becomes irrelevant, affecting segmentation performance.

[0052] Table 4 Ablation studies with different voiding rates

[0053]

[0054]

[0055] 2) Ablation study of different decoders

[0056] To explore the impact of different decoders on algorithm performance, this experiment used the decoder structures of ASPP, PPM, and ERFNet to verify the effectiveness of the CFM structure. As shown in Table 5, CFM achieved an accuracy improvement of 0.7 percentage points compared to the baseline network (the encoding portion of the network), 0.8 percentage points compared to the ASPP module, and 0.3 percentage points compared to the PPM module. Although it decreased by 0.9 percentage points compared to the ERFNet decoder, the significant computational cost incurred by the decoder led to a significant reduction in running speed. Experimental results show that using the CFM module at the end of the network achieves the most significant performance improvement at the lowest computational cost.

[0057] Table 5 Evaluation results of different decoders

[0058] Baseline Network CFM ASPP PPM ERF-Decoder mIOU (%) FPS √ 72.5 115.6 √ √ 73.2 102.3 √ √ 72.4 90.6 √ √ 72.9 102.7 √ √ 74.1 56.1

[0059] 3) Dual-branch ablation study

[0060] As the network depth increases, semantic information continues to increase, while shallow spatial information continues to be lost. Therefore, spatial branches are used to preserve spatial information. To explore whether dual branches can effectively improve network performance, this experiment designed three sets of comparative experiments: one without the spatial branch, one only aggregated once in the first stage, and the last group aggregated once in both the first and second stages to evaluate the bilateral structure. As shown in Table 6, when using the context branch alone, the network's inference speed reached 109.7 FPS, but the segmentation accuracy dropped significantly. When the two branches were aggregated in the first stage, the accuracy increased from 69.8% to 71.7%, and when aggregated again in the second stage, the accuracy increased from 71.7% to 73.2%. This shows that although the introduction of the spatial branch incurs additional computational effort, the bilateral structure significantly improves network accuracy without significantly affecting inference speed.

[0061] Table 6 Evaluation results of dual-branch structure

[0062]

[0063]

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A real-time semantic segmentation method based on efficient feature extraction and context aggregation, characterized in that: The method is to input the image dataset into the EFCANet model to realize image semantic segmentation, including encoding and decoding stages; Encoding stage: A bilateral structure is adopted, in which the spatial branch uses pooling operations and standard convolution to retain shallow spatial information; the context branch is used to provide a large receptive field and capture deep context information; these two branches are merged at different stages of the network; The context branch structure is: Layers 1 to 3: 3×3 convolution; Layer 4: downsampling; Layers 5-7: Efficient Feature Extraction Unit A, or EFEU-A, extracts shallow features through 3×3 convolution, 1×1 convolution, 3×3 dilated convolution, and channel rearrangement operations. Among them, the efficient feature extraction unit EFEU is constructed using the residual structure; Layer 8: 3×3 convolution; Layers 9-14: Efficient Feature Extraction Unit B, or EFEU-B, captures deep contextual information through 3×1 convolution, 1×3 convolution, 1×1 convolution, 5×5 depthwise dilated convolution, and channel reordering operations. Decoding stage: Combine depth-wise separable convolution and channel rearrangement operations to construct a context fusion module (CFM) to fuse feature information at different levels and obtain a feature fusion graph.

2. The real-time semantic segmentation method according to claim 1, characterized in that The PReLU activation function and batch normalization operation are introduced in EFEU.

3. The real-time semantic segmentation method according to claim 1, characterized in that The CFM module first inputs the feature information of different layers into the depth-separable convolutional layer to extract features; Then the 1 / 2 resolution feature map is average pooled and downsampled to 1 / 4 to obtain X1. At the same time, the 1 / 8 resolution feature map is bilinearly upsampled to 1 / 4 to obtain X3. The obtained feature maps are added, and the information exchange between different channels is increased through channel rearrangement operation.

Images