A Real-Time Semantic Segmentation Method for Remote Sensing Images Based on Dual-Path Structure
By employing a dual-path structure for real-time semantic segmentation of remote sensing images, combining dilated convolution and decomposed convolution, MobileNetV2, and an asymmetric multi-scale fusion module, the problem of high computational resources in high-resolution remote sensing image segmentation is solved, achieving high-precision and high-efficiency semantic segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUXI UNIV
- Filing Date
- 2023-07-24
- Publication Date
- 2026-05-26
AI Technical Summary
Existing networks have high computational resource requirements for high-resolution remote sensing image segmentation, resulting in limited real-time performance. Lightweight networks have not yet achieved an ideal balance between accuracy and speed.
A real-time semantic segmentation method for remote sensing images based on a dual-path structure is adopted, which combines dilated convolution and decomposed convolution to obtain deep spatial information. MobileNetV2 is used as the backbone network and an asymmetric multi-scale fusion module is introduced. The efficiency of feature extraction and fusion is improved through a dual-path feature aggregation module and a spatial attention mechanism.
While reducing the number of parameters and computational load, it significantly improves the segmentation accuracy and speed of remote sensing images, achieving efficient semantic segmentation results.
Smart Images
Figure CN117058166B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a real-time semantic segmentation method for remote sensing images based on a dual-path structure. Background Technology
[0002] To achieve better segmentation accuracy for high-resolution remote sensing images, existing networks introduce a large number of parameters and computational loads into their structures, significantly increasing computation time and requiring substantial computer hardware resources. This severely impacts real-time performance and hinders the practical application of semantic segmentation networks in the remote sensing field. To better balance accuracy and speed in semantic segmentation, many lightweight networks have been designed, and research has increasingly focused on lightweight networks with shallow structures. Among these, the dual-path structure most effectively addresses the trade-off between accuracy and inference speed, achieving a balance between the two. The existing dual-path real-time semantic segmentation network, Aerial-BiseNet, achieves a balance between high accuracy and real-time efficiency. To reduce the difference between the spatial and semantic dual paths, a channel attention based feature fusion module (CAFFM) is proposed to ensure effective fusion across different levels. Although some lightweight network structures have been proposed, a high degree of balance between accuracy and speed in real-time semantic segmentation of high-resolution remote sensing images has not yet been achieved. Summary of the Invention
[0003] Purpose of the invention: The purpose of this invention is to provide a real-time semantic segmentation method for remote sensing images based on a dual-path structure that enables fast and accurate segmentation of high-resolution remote sensing images.
[0004] Technical solution: The real-time semantic segmentation algorithm for remote sensing images of the present invention includes the following steps:
[0005] S1. Obtain the Potsdam remote sensing image dataset and the GID-15 remote sensing image dataset. After preprocessing the two remote sensing image datasets, divide them into training set, validation set and test set according to the set ratio.
[0006] S2, a real-time semantic segmentation network for remote sensing images based on a dual-path structure is constructed, including a spatial path, a contextual path, and a dual-path feature aggregation part; wherein, the spatial path is composed of multiple sets of decomposed convolutions with different dilation rates; the contextual path is composed of a backbone network and an asymmetric multi-scale fusion module; the dual-path feature aggregation part is composed of a dual-path feature aggregation module and a spatial attention mechanism;
[0007] S3. Input the remote sensing image datasets of the preprocessed training set and validation set from step S1 into the real-time semantic segmentation network for remote sensing images for training, calculate the cross-entropy loss function and perform backpropagation, update the network parameters, obtain a model that meets the requirements and save the model.
[0008] S4. Input the preprocessed test set from step S1 into the model obtained in step S3, and output the accurate segmentation map of the remote sensing image.
[0009] Further, in step S1, the dataset preprocessing specifically involves: cropping the orthophotos and label images in the Potsdam remote sensing image dataset into corresponding 224×224 sub-images in sequence and performing data augmentation to obtain a new dataset A; cropping the orthophotos and label images in the GID-15 remote sensing image dataset into corresponding 224×224 sub-images in sequence and performing data augmentation to obtain a new dataset B;
[0010] The data enhancement includes rotation and contrast enhancement;
[0011] Datasets A and B are randomly divided into three parts: a training set (60%), a validation set (20%), and a test set (20%).
[0012] Furthermore, in step S2, the steps for building a real-time semantic segmentation network for remote sensing images based on a dual-path structure are as follows:
[0013] S21, input the orthophoto image into the spatial path to obtain multi-scale spatial features;
[0014] Simultaneously, the orthophoto image is input into the context path to obtain multi-scale semantic features;
[0015] S22, input spatial features and semantic features into the dual-path feature aggregation part to obtain the feature map after the aggregation of the two features;
[0016] S23, take the feature map output from step 22 as input, and obtain the final accurate segmentation result after 8 times upsampling.
[0017] Furthermore, the network structure of the spatial path is as follows:
[0018] It mainly consists of four sets of decomposed convolutions, batch normalization, and ReLU activation functions with different porosity;
[0019] The deconvolution is performed using 3×1 and 1×3 convolution operations;
[0020] Perform decomposition convolution operations with dilation rates of 1, 2, 3, and 5;
[0021] The output resolution of the spatial path is 1 / 16 of the original input image.
[0022] Furthermore, the network structure of the context path is as follows:
[0023] The context path mainly consists of the backbone network and asymmetric multi-scale fusion modules;
[0024] The backbone network is mainly composed of MobileNetV2 with the last two pointwise convolutions removed. It has 17 Bottleneck layers and one standard convolutional layer. Each Bottleneck contains two pointwise convolutional layers and one depthwise convolutional layer.
[0025] The asymmetric multi-scale fusion module consists of upsampling, downsampling, ECA module, cascaded and one-dimensional convolution;
[0026] The upsampling involves performing bilinear interpolation on small-scale features by a factor of 2 to obtain a feature map of twice the size.
[0027] The downsampling involves performing pooling operations of 2x and 4x on large-scale features to obtain feature maps of 1 / 2 and 1 / 4 size.
[0028] The ECA module uses non-dimensionality-reduced GAP to aggregate convolutional features, first adaptively determines the kernel size k, then performs one-dimensional convolution, and finally uses the Sigmoid function to learn the weights of important channel features.
[0029] Furthermore, the network structure of the dual-path feature aggregation part is as follows:
[0030] The dual-path feature aggregation part mainly consists of a dual-path feature aggregation module and a spatial attention mechanism;
[0031] The dual-path feature aggregation module mainly consists of cascaded layers, convolutional layers, and pooling layers.
[0032] The cascaded layer performs a cascade operation on two inputs and a cascade operation on one-dimensional feature maps;
[0033] The convolutional layer includes: a 1×1 convolution containing a BN layer and a ReLU activation function for dimensionality reduction; 3×1 and 1×3 decomposed convolutions containing a BN layer and a ReLU activation function with dilatancy ratios of 1, 2, and 3 for convolution operations with different receptive fields; and a 1×1 convolution containing a BN layer and a Sigmoid activation function for dimensionality reduction and obtaining spatially dependent weights.
[0034] The pooling layer includes: spatial global average pooling and spatial global max pooling, to obtain spatial dependencies;
[0035] The spatial attention mechanism takes the aggregated feature map output by the dual-path aggregation module and the feature map output by the three convolutional blocks in the spatial path as input.
[0036] Furthermore, in step S3, the steps for training the real-time semantic segmentation network for remote sensing images are as follows:
[0037] S31, Randomly initialize the parameters of the remote sensing image semantic segmentation network, input the preprocessed training set and validation set data in step S1 into the remote sensing image real-time semantic segmentation network based on dual-path structure, generate the semantic segmentation probability map of the remote sensing image, and calculate the cross-entropy loss.
[0038] The formula for calculating the cross-entropy loss function is:
[0039]
[0040] Where y is the actual label image, y ′ The predicted label map is N, where N is the number of pixels in the image.
[0041] S32, backpropagation of loss, updating network parameters, minimizing the loss function as the optimization objective, obtaining and saving models that meet the conditions.
[0042] Compared with the prior art, the significant advantages of this invention are as follows:
[0043] 1. For spatial paths, this invention combines dilated convolution and decomposed convolution, using four sets of decomposed convolution structures with different dilation rates as spatial paths to obtain deep and rich spatial information; for contextual paths, this invention uses MobileNetV2 as the backbone network for extracting semantic features, and proposes an efficient asymmetric multi-scale fusion module to fuse features at different scales, and adds an efficient channel attention module, effectively improving the network's extraction of important features with a small number of parameters, thereby improving the richness and effectiveness of semantic feature extraction.
[0044] 2. For dual-path features, this invention proposes a dual-path feature aggregation module, which efficiently aggregates different features extracted from the two paths, avoiding redundant information and noise from feature aggregation from affecting the segmentation effect. Furthermore, a spatial attention mechanism is added to further refine the extracted spatial features and aggregated spatial-semantic features, thereby improving segmentation accuracy without reducing the segmentation rate. Attached Figure Description
[0045] Figure 1 This is a schematic diagram of the process of the present invention;
[0046] Figure 2 This is a structural diagram of the real-time semantic segmentation model for remote sensing images in this invention;
[0047] Figure 3 (a) is a structural diagram of MobileNetV2 in this invention.
[0048] (b) is a bottleneck layer with a reverse residual structure.
[0049] (c) is a bottleneck layer without a reverse residual structure;
[0050] Figure 4 This is a structural diagram of the asymmetric multi-scale fusion module in this invention;
[0051] Figure 5 This is a network structure diagram of the dual-path feature aggregation part in this invention;
[0052] Figure 6 Column (a) in the image shows the orthophotos input from the Potsdam dataset.
[0053] (b) Column shows the label plot of the Potsdam dataset input.
[0054] (c) shows the segmentation results of U-Net on the Potsdam dataset.
[0055] (d) shows the segmentation results of BiSeNet on the Potsdam dataset.
[0056] (e) shows the segmentation results of Aerial-BiSeNet on the Potsdam dataset.
[0057] (f) shows the segmentation results of MFANet on the Potsdam dataset.
[0058] (g) shows the segmentation results of this invention on the Potsdam dataset;
[0059] Figure 7 Column (a) in the image shows the orthophotos input from the GID dataset.
[0060] (b) Column shows the label graph input from the GID dataset.
[0061] (c) shows the segmentation results of U-Net on the GID dataset.
[0062] (d) shows the segmentation results of BiSeNet on the GID dataset.
[0063] (e) shows the segmentation results of Aerial-BiSeNet on the GID dataset.
[0064] (f) shows the segmentation results of MFANet on the GID dataset.
[0065] (g) shows the segmentation results of this invention on the GID dataset. Detailed Implementation
[0066] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0067] The real-time semantic segmentation algorithm for remote sensing images based on a dual-path structure provided in this embodiment, such as... Figure 1 As shown, it includes the following steps:
[0068] Step 1: Obtain the Potsdam and GID-15 remote sensing image datasets, which include orthophotos and corresponding label maps. After preprocessing the datasets, divide them into training, validation, and test sets according to a set ratio.
[0069] The steps for obtaining the dataset, preprocessing the dataset, and splitting the dataset are as follows:
[0070] The datasets were obtained as follows: the Potsdam remote sensing image dataset and the GID-15 remote sensing image dataset were obtained separately. The Potsdam remote sensing images contain IRRG images and corresponding label images, with a size of 6000×6000; the GID-15 remote sensing images contain RGB images and corresponding label images, with a size of 6800×7200.
[0071] The dataset preprocessing specifically involves cropping the orthophotos and label images from the Potsdam and GID-15 remote sensing image datasets into 224×224 sub-images in sequence and performing data augmentation to obtain two new datasets.
[0072] Data augmentation includes operations such as rotation and contrast enhancement;
[0073] The specific method for dividing the dataset is as follows: two new datasets of size 224×224 are randomly divided into three parts, with the training set accounting for 60%, the validation set accounting for 20%, and the test set accounting for 20%, resulting in two sets of training sets, two sets of validation sets, and two sets of test sets.
[0074] Step two involves constructing a real-time semantic segmentation network for remote sensing images based on a dual-path structure, including spatial path, contextual path, and dual-path feature aggregation components, such as... Figure 2 As shown, the spatial path consists of four sets of decomposed convolutions with different dilation rates; the context path consists of a backbone network and an asymmetric multi-scale fusion module; and the dual-path feature aggregation part consists of a dual-path feature aggregation module and a spatial attention mechanism.
[0075] The steps to build a real-time semantic segmentation network for remote sensing images based on a dual-path structure are as follows:
[0076] Step 21: Input the orthophoto image into the spatial path to obtain multi-scale spatial information. The spatial path mainly consists of four sets of decomposition convolutions with different dilation rates. Spatial information is obtained through convolution, batch normalization, and ReLU activation function operations. Specifically, the input image is first subjected to four sets of decomposition convolutions with different dilation rates, batch normalization, and ReLU activation function operations in sequence to achieve spatial feature extraction. The resolution of the spatial path output becomes 1 / 16 of the input image.
[0077] Simultaneously, the orthophoto image is input into the context path to obtain multi-scale semantic information. The context path mainly consists of a backbone network and an asymmetric multi-scale fusion module. The backbone network is mainly composed of MobileNetV2 with the last two pointwise convolutions removed. The asymmetric multi-scale fusion module utilizes an asymmetric structure to achieve efficient fusion of multi-scale features extracted by the backbone network. Specifically, the orthophoto image is first input into the backbone network to extract semantic features at different scales. Then, the asymmetric multi-scale fusion module performs different upsampling, downsampling, and channel attention mechanisms on the last five feature maps extracted by the backbone network to obtain feature maps of the same size, and performs feature concatenation and one-dimensional convolution operations. The final resolution of the extracted multi-scale features is 1 / 16 of the original input image.
[0078] Step 22: Input the spatial features obtained through the spatial path in Step 21 and the semantic features obtained through the context path into the dual-path feature aggregation to obtain two feature maps after feature aggregation. The dual-path feature aggregation part mainly consists of a dual-path feature aggregation module and a spatial attention mechanism. The dual-path feature aggregation module recalibrates the feature relationship between the two paths from the perspective of space and channels through convolution operations, and efficiently realizes dual-path feature aggregation. The spatial attention mechanism further refines the aggregated spatial-semantic features and spatial features. Specifically, the feature maps extracted by the two paths are first concatenated, and then the concatenated feature maps are dimensionality reduced. The dimensionality-reduced feature maps are then subjected to spatial global average pooling and global max pooling operations. The dimensionality-reduced feature maps are then further dimensionality-reduced to generate a one-dimensional feature map, and then dilated convolution operations are performed to output three feature maps with different receptive fields. Six feature maps in a one-dimensional spatial direction are concatenated to capture dependencies in that direction. The concatenated feature maps are then compressed one-dimensionally and passed through a sigmoid function to obtain a weighted feature map with global dependencies. This weighted feature map is then summed with the first dimensionality-reduced feature map to output the aggregated feature map from the dual-path feature aggregation module. Figure 5As shown. Then, the feature maps output by the three convolutional blocks in the spatial path of step 21 are used to perform feature calibration on the fused spatial-semantic features through a spatial attention mechanism. Spatial information is used as guiding information for the fused features, so that the network pays more attention to the pixel regions that play a decisive role in image segmentation and ignores irrelevant regions. At the same time, the resolution becomes 1 / 8 of the original input.
[0079] Step 23: Take the feature map output from step 22, which has a resolution of 1 / 8 of the original input features, as input, and upsample it by 8 times to obtain the final accurate segmentation result.
[0080] The network structure of the spatial path is as follows:
[0081] The spatial path mainly consists of four sets of decomposed convolutions, batch normalization, and ReLU activation functions with different hole rates;
[0082] Decompose the convolution and perform 3×1 and 1×3 convolution operations;
[0083] Perform decomposition convolution operations with dilation rates of 1, 2, 3, and 5;
[0084] The spatial path output resolution is 1 / 16 of the original input image.
[0085] The network structure of the context path is as follows:
[0086] The context path mainly consists of the backbone network and asymmetric multi-scale fusion modules;
[0087] The backbone network mainly consists of MobileNetV2 with the last two pointwise convolutions removed, and has a total of 17 Bottleneck layers and one standard convolutional layer. Each Bottleneck layer contains two pointwise convolutional layers and one depthwise convolutional layer. The network structure of MobileNetV2 is as follows: Figure 3 As shown in (a)-3(c).
[0088] like Figure 4 As shown, the asymmetric multi-scale fusion module consists of operations such as upsampling, downsampling, ECA module, cascading, and one-dimensional convolution;
[0089] Upsampling involves performing bilinear interpolation on small-scale features by a factor of 2 to obtain a feature map of twice the size.
[0090] Downsampling involves performing pooling operations of 2x and 4x on large-scale features to obtain feature maps of 1 / 2 and 1 / 4 size;
[0091] The ECA module uses non-dimensionality-reduced GAP to aggregate convolutional features, first adaptively determines the kernel size k, then performs one-dimensional convolution, and finally uses the Sigmoid function to learn the weights of important channel features.
[0092] The asymmetric multi-scale fusion module takes the last five feature maps of different sizes output from the backbone network in step 21 as input. The five feature sizes are 56×56×24, 28×28×32, 14×14×96, 7×7×320, and 1×1×320. First, the smallest one-dimensional feature is weighted by a sigmoid function and multiplied with the 7×7×320 feature map to obtain a recalibrated 7×7×320 feature map. Then, it is upsampled by 2 to obtain a 14×14×320 feature map. The 14×14×96 feature map remains unchanged. The 28×28×32 and 56×56×24 feature maps are downsampled by 2 and 4 times, respectively, and then processed by the ECA module to obtain enhanced 14×14×32 and 14×14×24 feature maps. Finally, the four feature maps of the same size are concatenated and subjected to one-dimensional convolution to obtain multi-scale semantic features with global dependencies.
[0093] like Figure 5 As shown, the network structure of the dual-path feature aggregation part is as follows:
[0094] The dual-path feature aggregation part mainly consists of a dual-path feature aggregation module and a spatial attention mechanism;
[0095] The dual-path feature aggregation module mainly consists of cascaded layers, convolutional layers, and pooling layers;
[0096] Cascaded layers include: cascading two inputs and cascading one-dimensional feature maps;
[0097] The convolutional layers include: 1×1 convolutions containing BN layers and ReLU activation functions for dimensionality reduction; 3×1 and 1×3 decomposed convolutions containing BN layers and ReLU activation functions with dilatancy ratios of 1, 2, and 3 for convolution operations with different receptive fields; and 1×1 convolutions containing BN layers and Sigmoid activation functions for dimensionality reduction and obtaining spatially dependent weights.
[0098] The pooling layer includes: spatial global average pooling and spatial global max pooling, to obtain spatial dependencies;
[0099] The dual-path feature aggregation module takes the spatial and semantic features output from step 21 as input. First, it concatenates the feature maps of the two paths, then performs a dimensionality reduction on the concatenated feature map. Next, it performs spatial global average pooling, spatial global max pooling, and a second dimensionality reduction operation on the first dimensionality-reduced feature map. Then, it performs three decomposition convolution operations with different dilation rates on the one-dimensional feature map of the second dimensionality reduction, outputting three one-dimensional feature maps with different receptive fields and spatial dependencies. Then, it concatenates the six one-dimensional feature maps with spatial dependencies, performs a dimensionality reduction operation on the concatenated feature map, and outputs spatial weights through the Sigmoid activation function. These weights are multiplied by the first dimensionality-reduced feature map for feature weighting and summation, outputting the aggregated feature map of this module.
[0100] The spatial attention mechanism takes the aggregated feature map output by the dual-path aggregation module in step 22 and the feature map output by the three convolutional blocks in the spatial path of step 21 as input. First, average pooling and max pooling operations are applied to the spatial features in step 21. Then, the generated one-dimensional feature maps are concatenated to form a feature descriptor. Next, a convolution operation is performed to generate the required spatial weight map. This spatial weight map is multiplied with the spatial-semantic features in step 22 to obtain the final generated feature, while the resolution is reduced to 1 / 8 of the original input.
[0101] Step 3: Input the two preprocessed training sets and two validation sets of remote sensing image datasets from Step 1 into the real-time semantic segmentation network for remote sensing images from Step 2 for training. Calculate the cross-entropy loss function and perform backpropagation to update the network parameters and obtain the optimal parameter model. Specifically, this includes:
[0102] Step 31: Randomly initialize the parameters of the remote sensing image semantic segmentation network, input the two sets of training sets and two sets of validation sets that have been preprocessed in Step 1 into the remote sensing image real-time semantic segmentation network based on the dual-path structure in Step 2, generate the semantic segmentation probability map of the remote sensing image, and calculate the cross-entropy loss.
[0103] Step 32: Backpropagate the loss, update the network parameters, minimize the loss function as the optimization objective, obtain the optimal parameter model and save it.
[0104] In step three, the loss function used during the training of the remote sensing image semantic segmentation network is the cross-entropy loss function, and the formula for calculating the cross-entropy loss function L is as follows:
[0105]
[0106] Where y is the actual label image, y ′ The image is the predicted label map, where N is the number of pixels in the image.
[0107] Step four: Input the two preprocessed test sets from step one into the optimal parameter model trained in step three, and output the accurate segmentation map of the remote sensing image.
[0108] The following description, in conjunction with the structural diagrams of each module provided in the embodiments, further illustrates the following:
[0109] The MobileNetV2 structure provided in this embodiment is as follows: Figure 3 As shown, the MobileNetV2 model has 17 Bottleneck layers, one standard convolutional layer, and two pointwise convolutional layers. Each Bottleneck contains two pointwise convolutional layers and one depthwise convolutional layer, for a total of 54 trainable parameter layers. MobileNetV2 optimizes the network using linear bottlenecks and inverse residual structures, resulting in a deeper network but a smaller model size and faster speed. The input orthophoto image is passed through a regular convolution to reduce its size and increase the number of channels. Then it passes through 17 linear bottlenecks, where a residual structure is used when the stride is 1, and not used when the stride is 2. Furthermore, the last pointwise convolutional layer of the bottleneck structure uses a Linear activation function, which can extract features better without adding too many additional parameters and computational cost. The residual structure is based on the ResNet residual structure and is an optimized version of it. The proposed inverse residual effectively solves the gradient vanishing problem that occurs as the network depth increases during training, allowing shallower layers of the deep network to obtain gradients during backpropagation, and enabling the parameters of shallower layers to be trained, thereby increasing the feature representation capability. Finally, in this embodiment, pwConv9-1 and pwConv11-1 in MobileNetV2 are removed to further improve the speed of semantic feature extraction.
[0110] The structure of the asymmetric multi-scale fusion module provided in this embodiment is as follows: Figure 4As shown, the asymmetric multi-scale fusion module mainly concatenates feature maps of different sizes after upsampling and downsampling operations. The last five feature maps of different sizes output from the linear bottleneck structure with a stride of 2 in MobileNetV2 are used as inputs: feature maps of sizes 1×1×320, 7×7×320, 14×14×96, 28×28×32, and 56×56×24 are represented by x1, x2, x3, x4, and x5, respectively. x1 is multiplied by x2 using a sigmoid function and then upsampled by a factor of 2 to obtain a 14×14×320 feature map. x4 is downsampled by a factor of 2 and subjected to an ECA attention mechanism to obtain a 14×14×32 feature map. x5 is downsampled by a factor of 4 and subjected to an ECA attention mechanism to obtain a 14×14×24 feature map. Finally, the four feature maps of the same size are concatenated and subjected to a 1×1 convolution operation to create dependencies between channels. Meanwhile, an ECA attention mechanism was added during the two downsampling processes, which effectively avoided the impact of dimensionality reduction on the channel attention learning effect, and only a small number of parameters were involved to effectively capture cross-channel interactions.
[0111] The structure of the dual-path feature aggregation module provided in this embodiment is as follows: Figure 5 As shown, the dual-path feature aggregation module captures the mapping relationship between the two path features from a global perspective, based on the extraction of two types of features, to achieve efficient fusion of spatial and semantic features. The feature maps of the two paths are concatenated, and then the concatenated feature map undergoes dimensionality reduction to obtain the dependency relationship between the two feature maps. Then, the first dimensionality-reduced feature map is subjected to spatial global average pooling, spatial global max pooling, and a second dimensionality reduction operation. Next, the second dimensionality-reduced one-dimensional feature map is subjected to three decomposition convolution operations with different dilation rates, outputting three one-dimensional feature maps with different receptive fields and spatial dependencies. These six one-dimensional feature maps with different spatial dependencies are then concatenated. The concatenated feature map undergoes dimensionality reduction and is passed through a Sigmoid activation function to output spatial weights. These weights are multiplied by the first dimensionality-reduced feature map for feature weighting and summation, outputting the module's aggregated feature map.
[0112] The proposed dual-path feature aggregation network was deployed on the PyTorch platform. The experiment used Python 3.8.13 + PyTorch 1.12.1 framework on Windows 10. The hardware environment consisted of a 12th Gen Intel(R) Core(TM) i9-12900KF@3.19GHz, an NVIDIA GeForce RTX 3090 GPU, and 32GB + 24GB of RAM. For the learning rate, a poly learning rate strategy was adopted—the learning rate was multiplied by a certain factor for each iteration during training. i represents the iteration number, max_i represents the maximum number of iterations (set to 200), and p represents power (set to 0.9). The optimizer uses the adaptive momentum estimation Adam, with an initial learning rate of 0.01, momentum and weight decay coefficients of 0.9 and 0.0005 respectively, and a batch size of 12. To comprehensively evaluate the model's segmentation accuracy and speed, we used four metrics: OA, F1, params, and FLOPs. OA and F1 were used as accuracy metrics, while params and FLOPs were used as real-time performance metrics. Table 1 shows the comparison results of the real-time semantic segmentation algorithm for remote sensing images based on a dual-path structure provided in the example with other segmentation networks on the Potsdam(P) and GID(G) datasets.
[0113] Table 1 Comparison of segmentation accuracy and real-time performance of different segmentation networks
[0114]
[0115] To demonstrate the effectiveness of the real-time semantic segmentation algorithm for remote sensing images based on a dual-path structure provided in this embodiment, the model was trained, validated, and tested using Potsdam and GID remote sensing data. (Table 1...) Figure 6 Columns (a)-(g) in the text and Figure 7 As can be seen from columns (a) to (g), the segmentation accuracy of this embodiment is higher than that of existing segmentation networks, while the number of parameters and computational load are greatly reduced, and the segmentation effect is closest to that of remote sensing labels.
Claims
1. A real-time semantic segmentation method for remote sensing images based on a dual-path structure, characterized in that, Includes the following steps: S1. Obtain the Potsdam remote sensing image dataset and the GID-15 remote sensing image dataset. After preprocessing the two remote sensing image datasets, divide them into training set, validation set and test set according to the set ratio. S2, a real-time semantic segmentation network for remote sensing images based on a dual-path structure is constructed, including a spatial path, a contextual path, and a dual-path feature aggregation part; wherein, the spatial path is composed of multiple sets of decomposed convolutions with different dilation rates; the contextual path is composed of a backbone network and an asymmetric multi-scale fusion module; the dual-path feature aggregation part is composed of a dual-path feature aggregation module and a spatial attention mechanism; S3. Input the remote sensing image datasets of the preprocessed training set and validation set from step S1 into the real-time semantic segmentation network for remote sensing images for training, calculate the cross-entropy loss function and perform backpropagation, update the network parameters, obtain a model that meets the requirements and save the model. S4. Input the preprocessed test set from step S1 into the model obtained in step S3, and output the accurate segmentation map of the remote sensing image. In step S2, the steps for building a real-time semantic segmentation network for remote sensing images based on a dual-path structure are as follows: S21, input the orthophoto image into the spatial path to obtain multi-scale spatial features; Simultaneously, the orthophoto image is input into the context path to obtain multi-scale semantic features; S22, input spatial features and semantic features into the dual-path feature aggregation part to obtain the feature map after the aggregation of the two features; S23, take the feature map output from step S22 as input, and obtain the final accurate segmentation result after 8 times upsampling; The network structure of the spatial path is as follows: It mainly consists of four sets of decomposed convolutions, batch normalization, and ReLU activation functions with different porosity; The deconvolution is performed using 3×1 and 1×3 convolution operations; Perform decomposition convolution operations with dilation rates of 1, 2, 3, and 5; The output resolution of the spatial path is 1 / 16 of the original input image; The network structure of the context path is as follows: The context path mainly consists of the backbone network and asymmetric multi-scale fusion modules; The backbone network is mainly composed of MobileNetV2 with the last two pointwise convolutions removed. It has 17 Bottleneck layers and one standard convolutional layer. Each Bottleneck contains two pointwise convolutional layers and one depthwise convolutional layer. The asymmetric multi-scale fusion module consists of upsampling, downsampling, ECA module, cascaded and one-dimensional convolution; The upsampling involves performing bilinear interpolation on small-scale features by a factor of 2 to obtain a feature map of twice the size. The downsampling involves performing pooling operations of 2x and 4x on large-scale features to obtain feature maps of 1 / 2 and 1 / 4 size. The ECA module uses non-dimensionality-reduced GAP to aggregate convolutional features, first adaptively determines the kernel size k, then performs one-dimensional convolution, and finally uses the Sigmoid function to learn the weights of important channel features. The network structure of the dual-path feature aggregation part is as follows: The dual-path feature aggregation part mainly consists of a dual-path feature aggregation module and a spatial attention mechanism; The dual-path feature aggregation module mainly consists of cascaded layers, convolutional layers, and pooling layers. The cascaded layer performs a cascade operation on two inputs and a cascade operation on one-dimensional feature maps; The convolutional layer includes: a 1×1 convolution containing a BN layer and a ReLU activation function for dimensionality reduction; 3×1 and 1×3 decomposed convolutions containing a BN layer and a ReLU activation function with dilatancy ratios of 1, 2, and 3 for convolution operations with different receptive fields; and a 1×1 convolution containing a BN layer and a Sigmoid activation function for dimensionality reduction and obtaining spatially dependent weights. The pooling layer includes: spatial global average pooling and spatial global max pooling, to obtain spatial dependencies; The spatial attention mechanism takes the aggregated feature map output by the dual-path aggregation module and the feature map output by the three convolutional blocks in the spatial path as input.
2. The real-time semantic segmentation method for remote sensing images based on a dual-path structure according to claim 1, characterized in that, In step S1, the dataset preprocessing specifically involves: cropping the orthophotos and label images from the Potsdam remote sensing image dataset into corresponding 224×224 sub-images in sequence and performing data augmentation to obtain a new dataset A; cropping the orthophotos and label images from the GID-15 remote sensing image dataset into corresponding 224×224 sub-images in sequence and performing data augmentation to obtain a new dataset B; The data enhancement includes rotation and contrast enhancement; Datasets A and B are randomly divided into three parts: the training set accounts for 60%, the validation set accounts for 20%, and the test set accounts for 20%.
3. The real-time semantic segmentation method for remote sensing images based on a dual-path structure according to claim 1, characterized in that, In step S3, the steps for training the real-time semantic segmentation network for remote sensing images are as follows: S31, Randomly initialize the parameters of the remote sensing image semantic segmentation network, input the preprocessed training set and validation set data in step S1 into the remote sensing image real-time semantic segmentation network based on dual-path structure, generate the semantic segmentation probability map of the remote sensing image, and calculate the cross-entropy loss. The formula for calculating the cross-entropy loss function is: ， Where y is the actual label image. The predicted label map is N, where N is the number of pixels in the image. S32, backpropagation of loss, updating network parameters, minimizing the loss function as the optimization objective, obtaining and saving models that meet the conditions.