Road segmentation method based on U-shaped network model in spatial and frequency domains for remote sensing images

By combining spatial and frequency domain U-shaped network models, the problems of insufficient local feature learning and high computational cost in remote sensing images are solved, achieving efficient segmentation of narrow and long roads in remote sensing images and improving segmentation accuracy and computational efficiency.

CN117689888BActive Publication Date: 2026-11-17NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311727812.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-15
Publication Date
2026-11-17
Estimated Expiration
2043-12-15

AI Technical Summary

Technical Problem

Existing remote sensing image road segmentation methods suffer from insufficient local feature learning ability, difficulty in capturing long-distance spatial dependencies and global semantic interpretation when dealing with remote sensing images with complex topological characteristics, low color contrast and cluttered backgrounds. In addition, frequency domain convolution has a large computational cost and high hardware requirements.

Method used

A U-shaped network model based on spatial and frequency domains is adopted, which combines spatial and frequency domain convolutional layers to form a hybrid global-local feature representation scheme. Furthermore, by improving the frequency domain separable convolutional structure, the computational load and parameters are reduced, thereby achieving fast road segmentation.

Benefits of technology

It improves the accuracy and efficiency of segmenting narrow and long roads in remote sensing images, reduces the amount of computation and parameters, and is suitable for mobile computing and edge computing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117689888B_ABST
    Figure CN117689888B_ABST
Patent Text Reader

Abstract

This invention proposes a road segmentation method for remote sensing images based on a U-shaped network model in the spatial and frequency domains. The method includes: preparing a publicly available remote sensing image dataset; manually labeling remote sensing images containing roads to separate the background and roads; performing data augmentation on the training and validation sets through rotation, scaling, flipping, and mirroring operations; inputting the augmented training and validation sets into the proposed network model for training; using the trained model as a prediction model, and inputting the test set into the prediction model to segment the roads, thus obtaining the segmentation result. This invention solves the problem of capturing long and thin road objects in remote sensing images by combining spatial and frequency domain convolutions to form a hybrid global-local feature representation scheme.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image recognition technology, and in particular relates to a road segmentation method for remote sensing images based on a U-shaped network model in the spatial and frequency domains. Background Technology

[0002] Road segmentation in remote sensing satellite imagery is crucial for fields such as Geographic Information Systems (GIS), urban road planning, and autonomous driving, and is a fundamental research topic in geographic mapping, traffic control, and autonomous driving. With the development of sensor and aerospace technologies, high-resolution remote sensing images rich in ground feature information are becoming increasingly easier to acquire. However, due to the complex topological characteristics, low color contrast, and cluttered backgrounds inherent in remote sensing images, road segmentation from them remains a significant challenge.

[0003] Traditional remote sensing image processing methods typically segment images based on thresholds, edges, or regions. However, these methods are limited to specific scenarios, lacking versatility and requiring manual adjustment of many parameters. In recent years, however, deep learning methods have increasingly been used to solve remote sensing segmentation tasks. These methods possess feature representation learning and parameter sharing capabilities, enabling faster and more accurate identification and segmentation of narrow roads in remote sensing images.

[0004] The mainstream techniques for road segmentation in deep learning-based remote sensing images employ spatial domain convolution due to its excellent ability to learn and represent local features. However, for road objects with long spans in remote sensing images, the limited receptive field also faces the challenge of capturing long-distance spatial dependencies and global semantic interpretation within the image. While the receptive field can be expanded by stacking spatial domain convolutional layers and downsampling, this leads to a more complex model and a greater susceptibility to overfitting. Compared to spatial domain convolution, frequency domain convolution can more efficiently consider global information. By combining spatial and frequency domain convolution, local and non-local receptive fields can be effectively integrated, further improving model performance. However, complex frequency domain convolution suffers from excessive computational cost, often requiring powerful computing hardware. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a road segmentation method for remote sensing images based on a U-shaped network model in the spatial and frequency domains. This method utilizes a combination of spatial and frequency domain convolutional layers to extract features, forming a hybrid global-local feature representation scheme that more accurately extracts narrow and long roads from remote sensing images. Simultaneously, an improved frequency-domain separable convolutional structure is employed, which, while retaining the high receptive field advantage of frequency-domain convolution, significantly reduces the computational load and parameters of the network model, achieving faster road segmentation.

[0006] To achieve the above objectives, the technical solution adopted in this invention is a remote sensing image road segmentation method based on a U-shaped network model in the spatial and frequency domains, specifically including the following steps:

[0007] A remote sensing image road segmentation method based on a spatial and frequency domain U-shaped network model includes the following steps:

[0008] Step 1: Prepare a publicly available remote sensing image dataset. Manually label the remote sensing images containing roads to separate the background and roads, unify their size, and divide them into training, validation, and test sets according to the proportions.

[0009] Step 2: Perform data augmentation on the training and validation sets;

[0010] Step 3: Input the training set and validation set into the U-shaped network model for training; the U-shaped network model consists of 9 layers connected in series, specifically including:

[0011] The first layer consists of a spatial domain convolutional layer, a frequency domain convolutional layer, and a downsampling layer connected in series.

[0012] The second and third layers are both composed of two frequency domain convolutional layers and one downsampling layer connected in series.

[0013] The fourth layer consists of two spatial domain convolutional layers and a downsampling layer connected in series.

[0014] The fifth layer consists of two frequency domain convolutional layers directly connected in series;

[0015] The sixth layer consists of an upsampling layer and two spatial domain convolutional layers connected in series. After upsampling, the feature maps with the same length and width dimensions as those in the fourth layer are concatenated and then passed together through two spatial domain convolutional layers.

[0016] The seventh and eighth layers are both composed of an upsampling layer and two frequency domain convolutional layers connected in series. After upsampling, the seventh layer is concatenated with the feature maps of the same length and width in the third layer, and then they pass through two frequency domain convolutional layers together. After upsampling, the eighth layer is concatenated with the feature maps of the same length and width in the second layer, and then they pass through two frequency domain convolutional layers together.

[0017] The ninth layer consists of an upsampling layer, two frequency domain convolutional layers, and a sigmoid layer connected in series. After upsampling, the feature maps with the same dimensions as those in the first layer are concatenated, and then the layers enter the frequency domain convolutional layer.

[0018] Step 4: Use the trained U-shaped network model as the prediction model, input the test set into the prediction model to segment the roads, and the segmentation results will be obtained.

[0019] Preferably, the frequency domain convolutional layer comprises:

[0020] Frequency domain convolution (FC), layer normalization layer (LN), and fully connected layer (FCL) are connected sequentially.

[0021] The local linear transformation (LLT) is connected to the output of the frequency domain convolution (FC).

[0022] Preferably, the frequency domain convolutional FC includes: horizontal convolution ConvH, vertical convolution ConvV, global average pooling GAP, and an attention mechanism layer;

[0023] Complex feature maps of size C×H×W are subjected to horizontal pooling and vertical pooling respectively, and then asymmetric convolution is performed in their respective directions.

[0024] Meanwhile, complex feature maps of size C×H×W are processed by global average pooling, and the smoothed feature maps then enter the attention mechanism layer.

[0025] The intermediate results of the above calculations are combined to complete the separable convolution operation.

[0026] Preferably, the spatial domain convolutional layers are composed of a single-connected spatial domain 3×3 convolutional module, a batch normalization module, and a GeLu activation function module.

[0027] Preferably, the dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] 1. The frequency domain convolutional layer designed in this invention has the feature of being plug-and-play. This module can be directly replaced by the spatial domain convolutional layer of other deep learning network models. By combining frequency convolution and local linear transformation, it effectively integrates local and non-local receptive fields, thereby further improving the performance of the network.

[0030] 2. This invention designs a U-shaped network model that combines spatial domain convolution and frequency domain convolution, forming a hybrid global-local feature representation scheme. Compared with the current mainstream spatial domain convolutional network model, this method has richer feature representation. It can not only use the frequency domain convolutional layer to represent global features and complete the overall capture of tortuous and continuous road objects, but also use the small-scale fine-grained representation capability of the spatial domain convolutional layer to enable the network to complete the detailed capture of small and narrow road objects in remote sensing images.

[0031] 3. This invention designs a separable convolution method in the frequency domain.

[0032] Because spatially separable convolutions are often prone to underfitting, and for spatially separable convolutions, not every kernel can be divided into two kernels due to the possibility of non-positive definite kernel matrices.

[0033] However, these problems do not exist in the frequency domain. By projecting the spectrum from the orthogonal direction and performing asymmetric convolution operations in the corresponding direction, a balance can be struck between reducing the number of trainable parameters and maintaining effective feature extraction.

[0034] To prevent the asymmetric structure from introducing cluttered background channel features, this method also employs an attention mechanism to recalibrate channel features, enhancing semantic features while suppressing non-semantic features.

[0035] Compared to performing two-dimensional convolutions directly in the frequency domain, this structure can retain the performance advantages of performing convolutions in the frequency domain while significantly reducing computational cost and parameters. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the U-shaped network model structure based on the spatial and frequency domains in this invention;

[0037] Figure 2 for Figure 1 A schematic diagram of the structure of a mid-space domain convolutional layer;

[0038] Figure 3 for Figure 1 A schematic diagram of the structure of a mid-frequency domain convolutional layer;

[0039] Figure 4 for Figure 3 A schematic diagram of the structure of mid-frequency domain convolution;

[0040] Figure 5 for Figure 4 A schematic diagram of the structure of the separable convolution method;

[0041] Figure 6 This is a schematic diagram comparing the road segmentation effects in remote sensing images. Detailed Implementation

[0042] The following will be combined with the appendix Figure 1-5 This paper provides a more detailed description of the remote sensing image road segmentation method based on a U-shaped network model in the spatial and frequency domains of the present invention, illustrating only a portion, not all, of the embodiments. It should be understood that those skilled in the art can modify the invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and not as a limitation thereof.

[0043] The remote sensing image road segmentation method based on a spatial and frequency domain U-shaped network model of the present invention includes the following steps:

[0044] Step 1: Prepare a publicly available remote sensing image dataset. Manually label the remote sensing images containing roads to separate the background and roads (or use the ground truth set included in the dataset). Standardize the size to 448×448 and divide them into training, validation, and test sets in an 8:1:1 ratio.

[0045] Step 2: Augment the training and validation sets by performing image processing operations such as rotation, scaling, flipping, and mirroring to double the size of the dataset.

[0046] Step 3: Input the data-augmented training and validation sets into the network model proposed in this invention for training: the total number of training iterations is 80,000, stochastic gradient descent (SGD) is used as the optimizer, the batch size is 4, and the initial learning rate starts from 1e. -2 As the number of iterations decays to 1e -6 A weighted loss function is used. The output size is the same as the graph size of the ground truth set, which is 448×448.

[0047] Furthermore, the weighted loss function includes the standard cross-entropy loss function and the Dice loss function, and its weighted loss function L is defined as follows:

[0048] L = L Cross +3L Dice

[0049] Furthermore, the network model proposed in this invention, such as Figure 1 As shown, it contains 9 layers connected in series, where:

[0050] The first layer is a module consisting of a spatial domain convolutional layer, a frequency domain convolutional layer, and a downsampling layer connected in series.

[0051] The spatial domain convolutional layer has a 3×3 kernel size and a stride of 1, using 1-pixel zero padding. Its main purpose is to upscale the image, integrating its feature information into the channel data. The frequency domain convolutional layer has kernel sizes of 1×3 and 3×1, with a stride of 1. The downsampling layer uses max pooling with a 2×2 kernel size. The input image size for the first layer is 448×448×3, and the output feature data size is 224×224×64.

[0052] The second and third layers are modules composed of two frequency-domain convolutional layers and a downsampling layer connected in series. The kernel sizes of the two frequency-domain convolutional layers are 1×3 and 3×1, respectively, with a stride of 1. The downsampling layer uses max pooling with a kernel size of 2×2. The input feature map sizes of the second and third layers are 224×224×64 and 112×112×128, respectively, and the output feature map sizes are 112×112×128 and 56×56×256, respectively.

[0053] The fourth layer is a module consisting of two spatial domain convolutional layers and a downsampling layer connected in series. The two spatial domain convolutional layers each have a 3×3 kernel size, a stride of 1, and use 1-pixel zero padding. The downsampling layer uses max pooling with a 2×2 kernel size. The input feature map size of this layer is 56×56×256, and the output feature map size is 28×28×512.

[0054] The fifth layer is a module composed of two frequency domain convolutional layers directly cascaded together, with kernel sizes of 1×3 and 3×1, and a stride of 1. This layer primarily serves to weight the deep channel feature information in one step, reducing the interference of cluttered background features on segmentation. The input feature map size for this layer is 28×28×512, and the output feature map size is 28×28×1024; the length and width dimensions of the feature map are not modified.

[0055] The sixth layer consists of an upsampling layer and two spatial domain convolutional layers connected in series. Its input feature map size is 28×28×1024, and its output feature map size is 56×56×512. The convolutional kernel size is 3×3, and the stride is 2. The upsampling layer uses transposed convolution with a 2×2 kernel. It's important to note that after upsampling, this layer needs to be concatenated with the feature maps of the same dimensions in the fourth layer, and then they are passed together through two spatial domain convolutional layers.

[0056] Layers 7 and 8 are modules consisting of an upsampling layer and two frequency-domain convolutional layers connected in series. The kernel sizes of the two frequency-domain convolutional layers are 1×3 and 3×1, respectively, with a stride of 1. The upsampling layer uses transposed convolution with a kernel size of 2×2. Similar to the above, after upsampling, layers 7 and 8 need to be concatenated with the feature maps of the same dimensions in layers 2 and 3, and then passed together through two frequency-domain convolutional layers. The input feature map sizes are 56×56×512 and 112×112×256, respectively, and the output feature map sizes are 112×112×256 and 224×224×128, respectively.

[0057] The ninth layer is a module consisting of an upsampling layer, two frequency-domain convolutional layers, and a sigmoid layer connected in series. The settings for the upsampling and frequency-domain convolutional layers are the same as those for the seventh and eighth layers. The final sigmoid layer consists of a 1×1 spatial domain convolutional kernel and a sigmoid activation function. The former reduces the feature map dimension to the output dimension, while the latter separates the background and road. The input feature map size is 224×224×128, and the output predicted map size is 448×448, which is the same as the size of manually labeled (or the ground truth set in a public dataset). After upsampling, the ninth layer needs to be concatenated with the feature maps of the same dimensions in the first layer before entering the frequency-domain convolutional layer.

[0058] like Figure 2 As shown, the spatial domain convolutional layers described above are all composed of a single-connected spatial domain 3×3 convolutional module, batch normalization, and a GeLu activation function module.

[0059] like Figure 3 As shown, the frequency domain convolutional layer described above consists of frequency domain convolution (FC), fully connected layer (FCL), local linear transformation (LLT), and layer normalization (LN), which can be expressed as:

[0060] H′=FC(LN(H i-1 ))+LLT(LN(H i-1 ))

[0061] H i =FCL(LN(H′))

[0062] Where: H i H represents the feature mapping of the i-th layer, and H′ represents the intermediate feature mapping result of the previous layer after frequency domain convolution.

[0063] like Figure 4 As shown, the calculation process of the frequency domain convolution described above can be expressed as follows:

[0064] First, a Fast Fourier Transform (FFT) is used to transform the features of H in the real domain to the complex domain, i.e., H = FFT(H′) = H. R +iH I It contains the real part H R and the imaginary part H I However, they can be regarded as two logically different real-value entities.

[0065] Then, by defining two real-valued convolution kernels, the effect of simulating a complex convolution kernel W is achieved, i.e., W = W R +iW IBecause the convolution operator has a distributive property, complex convolution (frequency domain convolution) can be expressed by the following formula:

[0066] W*H=(W R *H R -W I *H I )+i(W R *H I +W I *H R )

[0067] In the formula: * represents a convolution operation, and the convolution operation used in this invention is the separable convolution method mentioned below;

[0068] Finally, by using the inverse fast Fourier transform, the feature map generated by the convolution can be converted from the complex domain back to the real domain.

[0069] like Figure 5 As shown, the above-mentioned separable convolution method consists of horizontal convolution (ConvH), vertical convolution (ConvV), global average pooling (GAP), and an attention mechanism layer. Its process can be described as follows:

[0070] The convolution operation used in this invention is the separable convolution method mentioned below.

[0071] First, the complex feature map of size C×H×W is subjected to horizontal pooling and vertical pooling respectively, which replaces the complete feature map to reduce spectral features in the frequency dimension.

[0072] Subsequently, asymmetric convolutions are performed in their respective directions to effectively extract representative features. This process can be represented as follows:

[0073]

[0074]

[0075] In the formula: The dimensions are C×1×W. The dimensions are C×H×1, representing the two intermediate results obtained.

[0076] Meanwhile, the complex feature map of size C×H×W needs to be smoothed by global average pooling for all elements of the spectral features, which can be expressed mathematically as follows:

[0077]

[0078] Where: H GAP The dimensions are C×1×1, H ChwW represents the complex numerical element located at (h,w) in the feature map (complex feature map), while |W| represents the total number of elements in the complex feature map;

[0079] Subsequently, the smoothed feature maps need to enter the attention mechanism layer to output intermediate calculation results, which can be expressed as:

[0080]

[0081] In the formula: Conv(·) represents a regular convolution with a kernel size of 1×1, LN(·) is layer normalization, and σ refers to the Sigmoid activation function.

[0082] Finally, the three intermediate results calculated above are combined to complete the separable convolution operation, which can be described as follows:

[0083]

[0084] In the formula: ⊙ represents element-wise multiplication. yes The complex vector mapping after broadcasting (the feature maps after broadcasting have the same size), and H' F This is the output of the final separable convolution operation, with dimensions C×H×W, the same as the dimensions before convolution.

[0085] Step 4: Use the trained model as the prediction model, input the test set into the prediction model to segment the roads, and the segmentation result will be obtained.

[0086] To verify the method proposed in this invention, experiments were conducted using the DeepGlobe dataset provided by the 2018 CVPR Road Extraction Challenge. The proposed method was also compared with commonly used remote sensing segmentation network models such as Deeplabv3, U-Net, and Attention-UNet. Figure 6 This is a schematic diagram comparing the road segmentation results of two remote sensing images from the DeepGlobe dataset. As can be seen from the figure, the method proposed in this invention can better integrate local and non-local features compared with current common network models, and is clearer and more accurate in terms of detail preservation and overall contour coherence.

[0087] Table 1 details the segmentation quality evaluation and efficiency of the proposed network model method compared to other network models, and quantitatively analyzes the experimental results using accuracy, recall, mean intersection-over-union ratio, and number of model parameters. This invention also compares the prediction results of Deeplabv3, U-Net, and Attention-UNet.

[0088] Table 1. Segmentation quality evaluation and efficiency comparison of the network model method with other network models.

[0089]

[0090] As can be seen, compared with other common network models, the model proposed in this invention not only improves the accuracy, recall, and mean intersection-over-union ratio for road segmentation, but also has the fewest learnable parameters among all comparison methods. This lightweight feature not only facilitates rapid model learning and convergence, shortening training time, but also enables rapid inference, solving more practical tasks that require mobile or edge computing.

[0091] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the scope of protection of the present invention.

Claims

1. A road segmentation method for remote sensing images based on a U-shaped network model in the spatial and frequency domains, characterized in that, Includes the following steps: Step 1: Prepare a publicly available remote sensing image dataset. Manually label the remote sensing images containing roads to separate the background and roads, unify their size, and divide them into training, validation, and test sets according to the proportions. Step 2: Perform data augmentation on the training and validation sets; Step 3: Input the training set and validation set into the U-shaped network model for training; the U-shaped network model consists of 9 layers connected in series, specifically including: The first layer consists of a spatial domain convolutional layer, a frequency domain convolutional layer, and a downsampling layer connected in series. The second and third layers are both composed of two frequency domain convolutional layers and one downsampling layer connected in series. The fourth layer consists of two spatial domain convolutional layers and a downsampling layer connected in series. The fifth layer consists of two frequency domain convolutional layers directly connected in series; The sixth layer consists of an upsampling layer and two spatial domain convolutional layers connected in series. After upsampling, the feature maps with the same length and width dimensions as those in the fourth layer are concatenated and then passed together through two spatial domain convolutional layers. The seventh and eighth layers are both composed of an upsampling layer and two frequency domain convolutional layers connected in series. After upsampling, the seventh layer is concatenated with the feature maps of the same length and width in the third layer, and then they pass through two frequency domain convolutional layers together. After upsampling, the eighth layer is concatenated with the feature maps of the same length and width in the second layer, and then they pass through two frequency domain convolutional layers together. The ninth layer consists of an upsampling layer, two frequency domain convolutional layers, and a sigmoid layer connected in series. After upsampling, the feature maps with the same dimensions as those in the first layer are concatenated, and then the layers enter the frequency domain convolutional layer. Step 4: Use the trained U-shaped network model as the prediction model, input the test set into the prediction model to segment the roads, and you can get the segmentation results. The frequency domain convolutional layer includes: Frequency domain convolution (FC), layer normalization (LN), and fully connected layer (FCL) are connected sequentially. The output of the Local Linear Transform (LLT) is connected to the output of the Frequency Domain Convolution (FC). The frequency domain convolutional FC includes: horizontal convolution ConvH, vertical convolution ConvV, global average pooling GAP, and an attention mechanism layer; Complex feature maps of size C×H×W are subjected to horizontal pooling and vertical pooling respectively, and then asymmetric convolution is performed in their respective directions. Meanwhile, complex feature maps of size C×H×W are processed by global average pooling, and the smoothed feature maps then enter the attention mechanism layer. The intermediate results of the above calculations are combined to complete the separable convolution operation.

2. The remote sensing image road segmentation method based on a spatial and frequency domain U-shaped network model according to claim 1, characterized in that, Each spatial domain convolutional layer consists of a single-connected spatial domain 3×3 convolutional module, a batch normalization module, and a GeLu activation function module.

3. The remote sensing image road segmentation method based on a spatial and frequency domain U-shaped network model according to claim 1, characterized in that, The dataset is divided into training, validation, and test sets in an 8:1:1 ratio.