A ultra-high resolution remote sensing image segmentation method based on frequency domain information fusion

Abstract

The present invention provides an ultra-high resolution remote sensing image segmentation method based on frequency domain information fusion, comprising: step 1, obtaining Vaihingen and Potsdam remote sensing data sets; step 2, building an ultra-high resolution remote sensing image segmentation network integrating multivariate frequency domain information; step 3, inputting the Vaihingen training set into the ultra-high resolution remote sensing image segmentation network for training to obtain an optimal parameter model; step 4, inputting the Vaihingen test set into the optimal parameter model to output an accurate segmentation map of the remote sensing image; step 5, replacing the data set with a Potsdam training set, a Potsdam validation set and a Potsdam test set, and repeating the operations of steps 3 and 4. The model of the present invention improves the receptive field and feature representation capability of local branches by introducing the LKResNet‑18 network.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an ultra-high resolution remote sensing image segmentation method based on frequency domain information fusion. Background Art

[0002] Remote sensing semantic segmentation is a key computer vision task that aims to assign semantic labels to each pixel in remote sensing images to extract ground object information such as buildings, roads, vegetation, and water bodies. With the continuous advancement of remote sensing technology, especially the innovation of optical sensors, drones, and satellite technologies, it has become easier to obtain ultra-high-resolution remote sensing images. These high-resolution images provide valuable information for many fields such as disaster assessment, environmental monitoring, urban planning, and agricultural management.

[0003] With the development of deep learning, convolutional neural networks (CNNs) have been widely used in the field of image processing due to their powerful feature extraction capabilities, providing a strong technical foundation for semantic segmentation. Many scholars have proposed CNN-based segmentation networks. Although CNN-based methods have made great progress in the field of remote sensing image segmentation, due to the inherent limitations of convolution, CNN networks have limited perception of long-distance contextual information and lack a global understanding of images. In contrast, the self-attention mechanism of Transformer can effectively solve this problem. After introducing Transformer into the field of vision, researchers demonstrated its powerful global information encoding capabilities. However, the high computational complexity of Transformer poses a huge challenge to the efficiency and memory usage of the network. Therefore, researchers proposed combining CNN with Transformer. Geng et al. constructed a dual-path network using Swin Transformer and CNN (J. Geng, S. Song, and W. Jiang, “Dual-Path Feature Aware Network for Remote Sensing Image SemanticSegmentation,” IEEE Trans. Circ. Syst. Video Tech., vol. 34, no. 5, pp. 3674–3686, May 2024., doi:DOI: 10.1109 / TCSVT.2023.3317937.) to improve the expression ability of remote sensing target features through an adaptive feature fusion mechanism. CAGNet (S. Wang, Q. Hu, S. Wang, P. Zhao, J. Li, and M. Ai, Category attention guided network for semantic segmentation of Fine-Resolution remote sensing images, International Journal of Applied Earth Observation and Geoinformation, Volume 127, 2024, 103661, ISSN 1569-8432, https: / / doi.org / DOI: 10.1016 / j.jag.2024.103661) proposed by Wang et al. optimizes the extraction of semantic segmentation information through the category attention mechanism.Chen et al. effectively integrated the advantages of CNN and Transformer through a carefully designed feature extractor and bidirectional decoder (C. Lu, X. Zhang, K. Du, H. Xu and G. Liu, "CTCFNet: CNN-TransformerComplementary and Fusion Network for High-Resolution Remote Sensing ImageSemantic Segmentation," in IEEE Transactions on Geoscience and RemoteSensing, vol. 62, pp. 1-17, 2024, Art no. 5408717, DOI: 10.1109 / TGRS.2024.3458446) to obtain accurate segmentation results. Nevertheless, the hybrid-based model still has the quadratic complexity of the self-attention mechanism, so there are still certain challenges when facing UHR remote sensing images. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide an ultra-high resolution remote sensing image segmentation method based on frequency domain information fusion in view of the shortcomings of the prior art. The present invention can more efficiently integrate local features and global features, perform feature extraction and fusion at multiple levels and scales, thereby improving the segmentation accuracy of ultra-high resolution remote sensing images and providing a new solution for this field.

[0005] The method of the present invention comprises the following steps: Step 1, obtaining Vaihingen and Potsdam remote sensing data sets, wherein the data sets include original images and corresponding label images; the Vaihingen data set includes 33 ultra-high resolution images with an average size of 2496×2064 pixels, and the Potsdam data set includes 38 ultra-high resolution images with an average size of 6000×6000 pixels.

[0006] After preprocessing, the Vaihingen remote sensing dataset is divided into Vaihingen training set, Vaihingen validation set and Vaihingen test set according to the set ratio of 5:2:3.

[0007] After preprocessing, the Potsdam remote sensing dataset is divided into Potsdam training set, Potsdam validation set and Potsdam test set according to the set ratio of 5:2:3.

[0008] Step 2: Build an ultra-high-resolution remote sensing image segmentation network that integrates multi-frequency domain information, including a large kernel residual network (Large Kernel ResNet-18, LKResNet-18), a discriminative spectral enhancement Mamba (Discriminative Spectral Enhancement Mamba, DSE-Mamba) network, a spectral fusion wavelet downsampling (Spectral Fusion Wavelet Downsampling, SFWD) module, and a multiscale aggregation enhancement (Multiscale Aggregation-enhanced Wavelet Upsampling, MAWU) module.

[0009] The large kernel residual network and the spectrum fusion wavelet downsampling module constitute the local branch of the ultra-high resolution remote sensing image segmentation network.

[0010] The discriminative spectrum enhancement Mamba network includes a pre-processing Stem module and a discriminative spectrum enhancement module.

[0011] The discriminative spectrally enhanced Mamba network constitutes the global branch of the ultra-high resolution remote sensing image segmentation network.

[0012] The local branch and the global branch constitute a dual-branch encoder part of an ultra-high resolution remote sensing image segmentation network.

[0013] The multi-scale aggregation enhancement module constitutes the decoder part of the ultra-high resolution remote sensing image segmentation network.

[0014] The ultra-high resolution remote sensing image segmentation network constructed in this step is specially designed for ultra-high resolution remote sensing images, and can effectively solve the problems of large intra-class variance, small inter-class variance, and large target scale differences in remote sensing images.

[0015] Step 3: Input the Vaihingen training set into the ultra-high resolution remote sensing image segmentation network for training, calculate the loss function and perform back propagation, update the network parameters, and obtain the optimal parameter model; after each training for a certain number of times, input the Vaihingen validation set into the ultra-high resolution remote sensing image segmentation network for verification, observe the training effect and record it.

[0016] Step 4: Input the Vaihingen test set in step 3 into the optimal parameter model trained in step 3, and output the accurate segmentation map of the remote sensing image.

[0017] Step 5: Replace the data set with the Potsdam training set, Potsdam validation set, and Potsdam test set, and repeat steps 3 to 4.

[0018] Step 1 includes: obtaining a Vaihingen remote sensing dataset, wherein the Vaihingen remote sensing dataset includes five foreground categories and one background category, as well as corresponding label images, wherein the five foreground categories include low vegetation, trees, impervious surfaces, cars, and buildings.

[0019] The Potsdam remote sensing dataset is obtained, where the Potsdam remote sensing dataset contains six categories and corresponding label maps, where the six categories include low vegetation, trees, impervious surfaces, cars, buildings, and clutter.

[0020] According to the hardware conditions of the experimental equipment, the original images and the corresponding label images in the Vaihingen remote sensing dataset were cropped into images of 256×256 pixels in sequence, and the farthest point random cropping data enhancement method was used to obtain the remote sensing original images and the corresponding label images.

[0021] According to the hardware conditions of the experimental equipment, the original images and the corresponding label images in the Potsdam remote sensing dataset were cropped into images of 256×256 pixels in order, and image enhancement techniques such as random rotation and folding were used to obtain the original remote sensing images and the corresponding label images.

[0022] The Vaihingen remote sensing dataset is randomly divided into Vaihingen training set, Vaihingen validation set and Vaihingen test set; the Potsdam remote sensing dataset is randomly divided into Potsdam training set, Potsdam validation set and Potsdam test set. For consistency, the training set, validation set and test set used in the following steps are all from the Vaihingen dataset.

[0023] Step 2 includes: Step 2.1: The local branch includes 5 stages. The remote sensing image JPG original image in the Vaihingen training set is input into the local branch to obtain the output feature map of the 5 stages of the local branch.

[0024] Step 2.2: The global branch includes 5 stages. The JPG original remote sensing image in the Vaihingen training set is input into the global branch to obtain the output feature maps of the 5 stages of the global branch.

[0025] Step 2.3: Input the detail feature maps of the second to fifth stages obtained in step 2.1 and the global feature maps of the second to fifth stages obtained in step 2.2 into the multi-scale aggregation enhancement module, and output the final remote sensing image segmentation map.

[0026] Step 2.1 includes: among the 5 stages of the local branch, the first stage is composed of the residual preprocessing ResStem module, the second stage is composed of the large kernel residual block LKResblock, and the third, fourth and fifth stages are composed of the large kernel residual block LKResblock and the spectrum fusion wavelet downsampling module.

[0027] Through convolution, batch normalization function BN normalization, activation function, fast Fourier transform and wavelet transform operations, the extraction of remote sensing image from low-level detail features to high-level detail features is realized, specifically including: inputting the JPG original image into the first stage of the local branch to obtain the first stage detail feature map with a resolution of 1 / 4 of the original input image, inputting the first stage detail feature map into the second stage of the local branch to obtain the second stage detail feature map with a resolution of 1 / 4 of the original input image, inputting the second stage detail feature map into the third stage of the local branch to obtain the third stage detail feature map with a resolution of 1 / 8 of the original input image, inputting the third stage detail feature map into the fourth stage of the local branch to obtain the fourth stage detail feature map with a resolution of 1 / 16 of the original input image, and inputting the fourth stage detail feature map into the fifth stage of the local branch to obtain the fifth stage detail feature map with a resolution of 1 / 32 of the original input image.

[0028] The residual preprocessing ResStem module in the first stage includes convolution, batch normalization function BN normalization and pooling layer. The JPG original image is subjected to 3×3 convolution with a step size of 2, batch normalization function BN normalization and maximum pooling operation with a pooling kernel of 3 and a step size of 2 in sequence to obtain the first stage detail feature map with a resolution of 1 / 4 of the original input image.

[0029] The large kernel residual block LKResblock of the second stage consists of three branches, including a main branch, a sub-branch and a residual branch. In the main branch, the JPG original image is subjected to 7×7 depth convolution, batch normalization function BN normalization, ReLU activation operation, 7×7 depth convolution and BN normalization in turn to obtain the main branch feature map; in the sub-branch, the JPG original image is subjected to batch normalization function BN normalization and 3×3 depth convolution in turn to obtain the sub-branch feature map; the residual branch does not perform any operation on the original input; finally, the feature maps obtained by the main branch, sub-branch and residual branch are added and ReLU activation operation is performed to obtain the second-stage detail feature map with a resolution of 1 / 4 of the original input image.

[0030] The large kernel residual block LKResblock of the third stage has the same structure as the large kernel residual block LKResblock of the second stage. The spectrum fusion wavelet downsampling module of the third stage takes the output of the large kernel residual block LKResblock of the third stage as input. The spectrum fusion wavelet downsampling module includes a wavelet transform branch and a fast Fourier transform branch. In the wavelet branch, the input is firstly subjected to a discrete wavelet transform to obtain a low-frequency component and three high-frequency components, with a resolution of 1 / 2 of the input resolution. Then, a low-frequency component and three high-frequency components are concatenated in the channel dimension, and then 1×1 convolution, batch normalization function BN normalization and ReLU activation operations are performed in sequence to obtain a wavelet transform branch feature map. In the fast Fourier transform branch, the input is subjected to a discrete wavelet transform to obtain a low-frequency component and three high-frequency components, with a resolution of 1 / 2 of the input resolution. Then, a low-frequency component and three high-frequency components are concatenated in the channel dimension, and then 1×1 convolution, batch normalization function BN normalization and ReLU activation operations are performed in sequence to obtain a wavelet transform branch feature map. In the Rieger transform branch, the input is first subjected to a fast Fourier transform to obtain an amplitude spectrum and a phase spectrum. Then, the amplitude spectrum and the phase spectrum are subjected to a 1×1 convolution, a LeakyReLU activation, a channel attention SE layer weighting operation, and a 1×1 convolution operation, respectively. The processed amplitude spectrum and phase spectrum are then used to obtain the spectrum information through complex synthesis. The spectrum information is converted back to the spatial domain through an inverse Fourier transform to obtain a fast Fourier transform branch feature map with a resolution of 1 / 2 of the input. Finally, the wavelet transform branch feature map and the fast Fourier transform branch feature map are added and subjected to a 1×1 convolution, a batch normalization function BN normalization, and a GELU activation operation in sequence to obtain a third-stage detail feature map with a resolution of 1 / 8 of the original input image.

[0031] The channel attention SE layer includes global average pooling, a fully connected layer and an activation function operation, specifically including: firstly performing global average pooling on the input feature map, then sequentially passing through a dimensionality-reduced Linear fully connected layer, a ReLU activation, a dimensionality-increased Linear fully connected layer and a Sigmoid activation function, and finally multiplying the generated weight coefficient with the input feature map to realize a weighted operation.

[0032] The large kernel residual block LKResblock and spectrum fusion wavelet downsampling module of the fourth stage have the same structure as those of the large kernel residual block LKResblock and spectrum fusion wavelet downsampling module of the third stage. The detail feature map of the third stage is input into the fourth stage to obtain the detail feature map of the fourth stage with a resolution of 1 / 16 of the original input image.

[0033] The large kernel residual block LKResblock and spectrum fusion wavelet downsampling module of the fifth stage have the same structure as those of the large kernel residual block LKResblock and spectrum fusion wavelet downsampling module of the third stage. The fourth stage detail feature map is input into the fifth stage to obtain the fifth stage detail feature map with a resolution of 1 / 32 of the original input image.

[0034] Step 2.2 includes: among the 5 stages of the global branch, the first stage is composed of a preprocessing Stem module, and the second stage, the third stage, the fourth stage and the fifth stage are composed of a discriminant spectrum enhancement network. Through convolution, layer normalization function LN normalization, activation function, and fast Fourier transform operations, the remote sensing image is extracted from low-level global features to high-level global features; specifically including: inputting the JPG original image into the first stage of the global branch to obtain a first-stage global feature map with a resolution of 1 / 2 of the original input image, inputting the first-stage global feature map into the second stage of the global branch to obtain a second-stage global feature map with a resolution of 1 / 4 of the original input image, inputting the second-stage global feature map into the third stage of the global branch to obtain a third-stage global feature map with a resolution of 1 / 8 of the original input image, inputting the third-stage global feature map into the fourth stage of the global branch to obtain a fourth-stage global feature map with a resolution of 1 / 16 of the original input image, and inputting the fourth-stage global feature map into the fifth stage of the global branch to obtain a fifth-stage global feature map with a resolution of 1 / 32 of the original input image.

[0035] The preprocessing Stem module in the first stage includes convolution and instance normalization. Specifically, a 7×7 convolution with a step size of 2 and instance normalization operations are performed on the JPG original image to obtain the first stage global feature map with a resolution of 1 / 2 of the original input image.

[0036] The discriminant spectrum enhancement network of the second stage includes a spectrum enhancement part and a visual state space module (VSS Block). The spectrum enhancement part specifically includes: using 1×1 convolution on the input to map the number of channels to a higher dimension, then reconstructing the dimension and dividing it into two or more small blocks, performing a two-dimensional real Fourier transform on each small block to obtain frequency domain features; defining a learnable parameter for filtering, multiplying the learnable parameter with the frequency domain feature, and then performing an inverse Fourier transform to convert the frequency domain feature back to the time domain and reconstruct it back to the original dimension; then using a deep separable convolution on the features after spectrum filtering to divide the output into a first part and a second part along the channel dimension, using a GELU activation function to perform a nonlinear transformation on the first part, and then multiplying it with the second part, and finally restoring the feature dimension to the original input dimension through a 1×1 convolution to obtain a spectrum enhancement feature; the visual state space module specifically includes: using LN normalization on the input, and then using Linear The fully connected layer doubles the number of feature channels and divides the features into the first part of features and the second part of features from the channel dimension, performs deep convolution and SiLU activation operations on the first part of features in sequence, and then inputs the processed first part of features into the computing unit state space model SSM, and outputs 4 features; the SSM is the underlying core computing unit of the existing Mamba network, and the 4 features are fixed outputs of the SSM; the 4 features are added, the feature dimension is transformed using the transpose operation and the LN normalization operation is performed to obtain the output after state learning; the second part of features is nonlinearly transformed using the SiLU activation function, and then multiplied with the output after state learning to obtain a new feature; the new feature is restored to the original dimension using the Linear fully connected layer, and finally the feature is regularized using the Dropout function to obtain the final output feature, and the resolution remains unchanged.

[0037] The discriminative spectrum enhancement network specifically includes: using 2×2 convolution to perform block processing on the input, then passing the block-processed features into the spectrum enhancement part to obtain spectrum enhancement features, and then inputting the processed features into the visual state space module VSSBlock to obtain processed features, and finally dividing the processed features into 4 parts from the height and width dimensions and splicing them along the channel dimension, performing LN normalization and Linear full connection layer, and outputting the second-stage global feature map with a resolution of 1 / 4 of the original input image.

[0038] The discriminant spectrum enhancement network of the third stage has the same structure as the discriminant spectrum enhancement network of the second stage. The global feature map of the second stage is input into the third stage to obtain the global feature map of the third stage, and the resolution of the global feature map is 1 / 8 of the original input image.

[0039] The discriminant spectrum enhancement network of the fourth stage has the same structure as the discriminant spectrum enhancement network of the second stage. The global feature map of the third stage is input into the fourth stage to obtain the global feature map of the fourth stage, and the resolution of the global feature map is 1 / 16 of the original input image.

[0040] The discriminant spectrum enhancement network of the fifth stage has the same structure as the discriminant spectrum enhancement network of the second stage. The global feature map of the fourth stage is input into the fifth stage to obtain the global branch fifth stage global feature map, and the resolution is 1 / 32 of the original input image.

[0041] Step 2.3 includes: the multi-scale aggregation enhancement module includes 4 feature aggregation parts with consistent structures and 4 upsampling parts, the 4 feature aggregation parts with consistent structures are respectively the first feature aggregation part, the second feature aggregation part, the third feature aggregation part and the fourth feature aggregation part, and the outputs of the second, third, fourth and fifth stages of step 2.1 and step 2.2 are respectively used as input; the 4 upsampling parts are respectively the first upsampling part, the second upsampling part, the third upsampling part and the fourth upsampling part.

[0042] The feature aggregation part includes a preprocessing link, a multi-head attention link and a channel optimization link; the preprocessing link receives two input parameters, first concatenates the two input parameters in the channel dimension to obtain a new feature, and then performs deep convolution, BN normalization, GELU activation function, point convolution, BN normalization and GELU activation function operations on the new feature in turn to obtain the preprocessed feature; the multi-head attention link specifically includes: taking the output of the preprocessing link as input, using linear transformation on the input feature to obtain a value vector V, then reshaping the linear transformation and feature dimension to obtain a key vector K, and then using deep separable convolution to perform the key vector of each attention head Feature extraction is performed to obtain features, and the attention head is a module initialization parameter; the features are processed, including expanding the channel dimension, normalization, activation and dimensionality reduction, and then the features are weighted with a value vector, and finally the dimension is reduced through the projection layer and the neuron random dropout method is applied to obtain the output; the channel optimization link consists of three branches, wherein the first branch contains a BN normalization operation; the second branch contains a depth convolution and a BN normalization operation; the third branch contains a point convolution and a BN normalization operation, and the input is passed into the three branches respectively, and the results of the last three branches are added and a GELU activation operation is performed to obtain the final output.

[0043] The specific operations of the feature aggregation part include: passing the input into the preprocessing stage, the multi-head attention stage and the channel optimization stage in sequence to obtain the feature aggregation output.

[0044] The upsampling part takes the output of two feature aggregation parts as input, and the outputs of the two feature aggregation parts are the first input and the second input respectively. First, the first input is subjected to wavelet transform to extract one low-frequency component and three high-frequency components, and the high-frequency components are spliced ​​along the channel dimension and processed using a convolutional layer with a residual connection to obtain high-frequency features; the low-frequency components are spliced ​​with the second input along the channel dimension and processed using a convolutional layer with a residual connection, and the processed features are spliced ​​with the high-frequency features along the channel dimension, and finally an inverse wavelet transform is performed to obtain the output.

[0045] The specific operation of the multi-scale aggregation enhancement module includes: using the fifth-stage detail feature map and the fifth-stage global feature map as inputs of the fourth feature aggregation part to obtain a fourth feature aggregation output, using the fourth-stage detail feature map and the fourth-stage global feature map as inputs of the third feature aggregation part to obtain a third feature aggregation output, using the third-stage detail feature map and the third-stage global feature map as inputs of the second feature aggregation part to obtain a second feature aggregation output, and using the second-stage detail feature map and the second-stage global feature map as inputs of the first feature aggregation part to obtain a first feature aggregation output.

[0046] The fourth feature aggregation output and the third feature aggregation output are used as the input of the fourth upsampling part to obtain the third stage output; the second feature aggregation output and the third stage output are used as the input of the third upsampling part to obtain the second stage output; the first feature aggregation output and the second stage output are used as the input of the second upsampling part to obtain the first stage output; finally, the DySample upsampling function is used on the first stage output to obtain the final accurate remote sensing image segmentation map.

[0047] Step 3 includes: Step 3.1: Randomly initialize the parameters of the ultra-high resolution remote sensing image segmentation network, input the training set and validation set data into the high-resolution remote sensing image segmentation network, generate a semantic segmentation probability map, and calculate the cross entropy loss and Dice loss .

[0048] Step 3.2: Loss back propagation, update network parameters, minimize the loss function as the optimization goal, obtain the optimal parameter model and save it.

[0049] In step 3.1, the cross entropy loss The calculation formula is:

[0050] ,

[0051] Where N is the number of pixels, C is the number of categories, is the predicted probability of the kth category in the ith pixel, It represents the value in the one-hot encoding of the i-th pixel belonging to category k. If the i-th pixel belongs to category k, is 1, otherwise is 0.

[0052] The Dice loss The calculation formula is:

[0053] ,

[0054] Among them, TP is the number of true positives, that is, the number of pixels whose predicted and true labels are both positive; FP is the number of false positives, that is, the number of pixels predicted to be positive but actually negative; FN is the number of false negatives, that is, the number of pixels predicted to be negative but actually positive.

[0055] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0056] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0057] Beneficial effects: The model of the present invention improves the receptive field and feature representation ability of the local branch by introducing the LKResNet-18 network. A spectrum fusion wavelet downsampling module is added to the local branch, and the network can retain more effective features in the downsampling stage through the organic combination of fast Fourier transform and wavelet transform. The model introduces Mamba into the global branch, replacing the traditional Transformer, providing a new solution for global feature extraction. At the same time, it introduces FFT into Mamba, proposes discriminant spectrum enhancement Mamba, and uses frequency domain weighted information to assist spatial domain feature perception, thereby improving the global feature extraction ability of the network. The model designs a MAWU decoding module, which can fuse local information and global information at multiple scales and realize information lossless upsampling through wavelet transform. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0059] Figure 1 A schematic diagram of the process flow of an ultra-high resolution remote sensing image segmentation network that integrates multivariate frequency domain information provided in an embodiment.

[0060] Figure 2 This is a structural diagram of an ultra-high-resolution remote sensing image segmentation network that integrates multi-frequency domain information provided in an embodiment.

[0061] Figure 3 A structural diagram of the ResStem module provided in the embodiment.

[0062] Figure 4 A structural diagram of the Stem module provided for the embodiment.

[0063] Figure 5 A structural diagram of the LKResblock module provided in the embodiment.

[0064] Figure 6 This is a structural diagram of the spectrum fusion wavelet downsampling module provided in the embodiment.

[0065] Figure 7 This is a structural diagram of the DSE-Mamba module provided in the embodiment.

[0066] Figure 8 A structural diagram of a multi-scale aggregation enhancement module provided in an embodiment.

[0067] Fig. 9 This is a comparison chart of indicators of the ultra-high resolution remote sensing image segmentation network that integrates multivariate frequency domain information on the Vaihingen dataset provided in the embodiment.

[0068] Fig.10 This is a comparison chart of indicators of the ultra-high resolution remote sensing image segmentation network that integrates multivariate frequency domain information on the Potsdam dataset provided in the embodiment.

[0069] Fig.11 A schematic diagram of the segmentation results of the ultra-high resolution remote sensing image segmentation network integrating multivariate frequency domain information provided in the embodiment on the Vaihingen dataset.

[0070] Fig.12 A schematic diagram of the segmentation results of the ultra-high resolution remote sensing image segmentation network integrating multivariate frequency domain information provided in the embodiment on the Potsdam dataset. DETAILED DESCRIPTION

[0071] The embodiment of the present invention provides an ultra-high resolution remote sensing image segmentation method based on frequency domain information fusion, such as Figure 1 , Figure 2 As shown, the following steps are included: Step 1: Obtain the Vaihingen and Potsdam remote sensing datasets, including the original image and the corresponding label image, and divide the dataset into a training set, a validation set, and a test set according to a set ratio after preprocessing.

[0072] Step 2: Build an ultra-high resolution remote sensing image segmentation network that integrates multi-frequency domain information, including LKResNet-18 network, DSE-Mamba network, spectrum fusion wavelet downsampling module, and multi-scale aggregation enhancement module. The LKResNet-18 network and spectrum fusion wavelet downsampling module constitute the local branches of the ultra-high resolution remote sensing image segmentation network, the DSE-Mamba network constitutes the global branch of the ultra-high resolution remote sensing image segmentation network, the local branch and the global branch constitute the dual-branch encoder part of the ultra-high resolution remote sensing image segmentation network, and the multi-scale aggregation enhancement module constitutes the decoder part of the ultra-high resolution remote sensing image segmentation network.

[0073] Step 3: Input the preprocessed Vaihingen training set in step 1 into the ultra-high resolution remote sensing image segmentation network in step 2 for training, calculate the loss function and perform back propagation, update the network parameters, and obtain the optimal parameter model. After a certain number of training times (for example, 10 times), input the preprocessed Vaihingen validation set in step 1 into the ultra-high resolution remote sensing image segmentation network in step 2 for verification, observe the training effect and record it.

[0074] Step 4: Input the preprocessed Vaihingen test set in step 1 into the optimal parameter model trained in step 3, and output the accurate segmentation map of the remote sensing image.

[0075] Step 5: Change the dataset to the Potsdam dataset and repeat steps 3 and 4.

[0076] The steps of acquiring a data set, preprocessing a data set, and dividing a data set in step 1 are as follows: the steps of acquiring a data set are as follows: acquiring a Vaihingen remote sensing data set, which includes five foreground categories (short vegetation, trees, impervious surfaces, cars, and buildings) and one background category (clutter) and corresponding label maps, with a total of 33 high-resolution remote sensing images with an average of 2496×2496 pixels; acquiring a Potsdam remote sensing data set, which includes six categories (short vegetation, trees, impervious surfaces, cars, buildings, and clutter) and corresponding label maps, with a total of 38 high-resolution remote sensing images with an average of 6000×6000 pixels.

[0077] The dataset preprocessing is specifically as follows: the original images and the corresponding label images in the Vaihingen dataset are sequentially cropped into images of 256×256 pixels in size. In order to prevent overfitting and improve the spatial information content of the dataset, the farthest point random cropping data enhancement method is adopted, and finally 19335 remote sensing original images and corresponding label images are obtained; the original images and the corresponding label images in the Potsdam dataset are sequentially cropped into images of 256×256 pixels in size, and a variety of image enhancement techniques such as flipping, translation, scaling and shearing are adopted, and finally 20102 remote sensing original images and corresponding label images are obtained.

[0078] The data set is divided specifically as follows: a data set of 256×256 size is randomly divided into a training set, a validation set, and a test set in a ratio of 5:2:3.

[0079] The steps of building an ultra-high resolution remote sensing image segmentation network integrating multivariate frequency domain information in step 2 are as follows: Step 2.1: Input the remote sensing image JPG original image into the local branch to obtain the output feature map of the 5 stages of the local branch. The local branch is mainly composed of 5 stages, the first stage is composed of the ResStem module, the second stage is composed of the LKResblock, and the third to fifth stages are composed of the LKResblock and the spectrum fusion wavelet downsampling module. Through operations such as convolution, BN normalization, activation function, fast Fourier transform, wavelet transform, etc., efficient extraction of remote sensing images from low-level detail features to high-level detail features is achieved. The steps are specifically as follows: input the JPG original image into the first stage of the local branch, obtain the first stage detail feature map with a resolution of 1 / 4 of the original input image, and then input the first stage detail feature map into the second stage of the local branch to obtain the second stage detail feature map with a resolution of 1 / 4 of the original input image, and so on to obtain the third stage detail feature map, the fourth stage detail feature map and the fifth stage detail feature map of the local branch in sequence, and the feature map resolutions are 1 / 8, 1 / 16 and 1 / 32 of the original input image in sequence. The ResStem module of the first stage is composed of convolution, BN normalization and pooling layers, specifically: perform 3×3 convolution with a step size of 2, BN normalization and maximum pooling operations with a pooling kernel of 3 and a step size of 2 on the JPG original image in sequence to obtain the first stage detail features of the local branch with a resolution of 1 / 4 of the original input image. The LKResblock of the second stage is composed of three branches, including a main branch, a secondary branch and a residual branch. Specifically: in the main branch, the original JPG image is subjected to 7×7 depth convolution, BN normalization, ReLU activation operation, 7×7 depth convolution and BN normalization in turn to obtain the main branch feature map; in the secondary branch, the original JPG image is subjected to BN normalization and 3×3 depth convolution in turn to obtain the secondary branch feature map; the residual branch does not perform any operation on the original input; finally, the feature maps obtained by the main branch, secondary branch and residual branch are added and ReLU activation operation is performed to obtain the second stage detail features of the local branch with a resolution of 1 / 4 of the original input image. The LKResblock of the third stage has the same structure as the LKResblock of the second stage. The spectrum fusion wavelet downsampling module takes the output of the LKResblock of the same three stages as input. The module consists of two branches, including a wavelet transform branch and a fast Fourier transform branch. Specifically: in the wavelet branch, the input is first subjected to discrete wavelet transform to obtain a low-frequency component and three high-frequency components, whose resolutions are 1 / 2 of the input resolution respectively. Then the four components are concatenated in the channel dimension, and then 1×1 convolution, BN normalization and ReLU activation operations are performed in sequence to obtain the wavelet transform branch feature map.In the fast Fourier transform branch, the input is first subjected to fast Fourier transform to obtain the amplitude spectrum and phase spectrum. Then, the amplitude spectrum and phase spectrum are subjected to 1×1 convolution, LeakyReLU activation, SE layer weighting operation and 1×1 convolution operation respectively; then, the processed amplitude spectrum and phase spectrum are used to obtain the spectrum information through complex synthesis; the spectrum information is converted back to the spatial domain through inverse Fourier transform to obtain the enhanced output image with a resolution of 1 / 2 of the input. Finally, the wavelet transform branch feature map and the fast Fourier transform branch feature map are added and 1×1 convolution, BN normalization and GELU activation operations are performed in sequence to obtain the third stage detail features of the local branch with a resolution of 1 / 8 of the original input image. The SE layer includes global average pooling, fully connected layer and activation function operation, specifically: first, the input feature map is subjected to global average pooling, then it is subjected to the dimensionality reduction Linear fully connected layer, ReLU activation, dimensionality increase Linear fully connected layer and Sigmoid activation function in sequence, and finally the generated weight coefficient is multiplied by the input feature map to realize the weighted operation. The LKResblock and spectrum fusion wavelet downsampling module of the fourth stage have the same structure as the LKResblock and spectrum fusion wavelet downsampling module of the third stage, and the detail features of the third stage are input into the fourth stage to obtain the local branch fourth stage detail features with a resolution of 1 / 16 of the original input image. The LKResblock and spectrum fusion wavelet downsampling module of the fifth stage have the same structure as the LKResblock and spectrum fusion wavelet downsampling module of the third stage, and the detail features of the fourth stage are input into the fifth stage to obtain the local branch fifth stage detail features with a resolution of 1 / 32 of the original input image.

[0080] Step 2.2: Input the JPG original image of the remote sensing image into the global branch to obtain the output feature map of the 5 stages of the global branch. The global branch is mainly composed of 5 stages, the first stage is composed of the Stem module, and the second to fifth stages are composed of the DSE-Mamba module. Through operations such as convolution, LN normalization, activation function, and fast Fourier transform, efficient extraction of remote sensing images from low-level global features to high-level global features is achieved. The specific steps are: input the JPG original image into the first stage of the global branch, obtain the first stage global feature map with a resolution of 1 / 2 of the original input image, and then input the first stage global feature map into the second stage of the global branch to obtain the second stage global feature map with a resolution of 1 / 4 of the original input image, and so on to obtain the third stage global feature map of the global branch, the fourth stage global feature map, and the fifth stage global feature map, and the feature map resolution is 1 / 8, 1 / 16, and 1 / 32 of the original input image. The Stem module of the first stage consists of convolution and instance normalization, specifically: 7×7 convolution with a step size of 2 and instance normalization operations are performed on the JPG original image in sequence to obtain the global features of the first stage of the global branch, with a resolution of 1 / 2 of the original input image. The DSE-Mamba of the second stage consists of structures such as the spectrum enhancement part and the VSS Block. The spectrum enhancement part is specifically as follows: first, a 1×1 convolution is used on the input to map its channel number to a higher dimension, then the dimension is reconstructed and divided into multiple small blocks, and a two-dimensional real Fourier transform is performed on each small block to obtain frequency domain features. A learnable parameter for filtering is defined, and the parameter is multiplied with the frequency domain feature, and then an inverse Fourier transform is performed to convert the frequency domain feature back to the time domain and reconstruct it back to the original dimension. Then, a deep separable convolution is used on the features after spectrum filtering, and the output is divided into two parts along the channel dimension. The first part is nonlinearly transformed using the GELU activation function, and then multiplied with the second part. Finally, the feature dimension is restored to the original input dimension through a 1×1 convolution to obtain the spectrum enhancement feature. The VSS Block is specifically as follows: first, LN normalize the input, then use the Linear fully connected layer to double the number of feature channels and split the features into two parts from the channel dimension. Perform deep convolution and SiLU activation operations on the first part of the features in sequence, then input the processed first part of the features into the calculation unit SSM, and output 4 features. The SSM is the underlying core calculation unit of the existing Mamba network. Add the 4 features, use the transpose operation to transform the feature dimension and perform the LN normalization operation. Use the SiLU activation function to perform a nonlinear transformation on the second part of the features, and then multiply them with the features obtained in the previous step to obtain a new feature. Use the Linear fully connected layer to restore the original dimension of the feature, and finally use the Dropout function to regularize the feature to obtain the final output feature, and the resolution remains unchanged.The DSE-Mamba is specifically as follows: first, the input is processed in blocks using 2×2 convolution, and then the features are passed into the spectrum enhancement part to obtain the spectrum enhancement features, and then the VSS Block is input to obtain the processed features. Finally, the processed features are divided into 4 parts and spliced ​​along the channel dimension, and then LN normalization and Linear full connection layer are performed to output the global branch second stage global feature map, with a resolution of 1 / 4 of the original input image. The DSE-Mamba in the third stage has the same structure as the DSE-Mamba in the second stage. The second stage global feature map is input into the third stage to obtain the global branch third stage global feature map, with a resolution of 1 / 8 of the original input image. The DSE-Mamba in the fourth stage has the same structure as the DSE-Mamba in the second stage. The third stage global feature map is input into the fourth stage to obtain the global branch fourth stage global feature map, with a resolution of 1 / 16 of the original input image. The DSE-Mamba of the fifth stage has the same structure as the DSE-Mamba of the second stage. The global feature map of the fourth stage is input into the fifth stage to obtain the global branch fifth stage global feature map, and the resolution is 1 / 32 of the original input image.

[0081] Step 2.3: Input the detail feature maps of the second to fifth stages obtained in step 2.1 and the global feature maps of the second to fifth stages obtained in step 2.2 into the decoder multi-scale aggregation enhancement module, and output the final accurate remote sensing image segmentation map. The multi-scale aggregation enhancement module includes 4 feature aggregation parts with consistent structures and 4 upsampling parts. The feature aggregation parts are the first feature aggregation part, the second feature aggregation part, the third feature aggregation part and the fourth feature aggregation part, and the outputs of the second, third, fourth and fifth stages of step 2.1 and step 2.2 are used as inputs respectively; the four upsampling parts are the first upsampling part, the second upsampling part, the third upsampling part and the fourth upsampling part; the feature aggregation part consists of a preprocessing link, a multi-head attention link and a channel optimization link. The preprocessing link receives two input parameters, first concatenates the two inputs in the channel dimension to obtain new features, and then performs depth convolution, BN normalization, GELU activation function, point convolution, BN normalization and GELU activation function operations on them in turn to obtain preprocessed features. The multi-head attention link is specifically as follows: a value vector (V) is obtained by linear transformation of the input features, and then a key vector (K) is obtained by linear transformation and feature dimension reshaping, and then a depthwise separable convolution is used to extract features from the key vector of each attention head, where the attention head is the module initialization parameter. The convolution output is processed, the channel dimension is expanded, normalized, activated, and dimension is reduced, and then the features are weighted by the value vector, and finally the dimension is reduced and dropout is applied through the projection layer to obtain the output. The channel optimization link consists of three branches, specifically: the first branch contains a BN normalization operation; the second branch contains a depthwise convolution and a BN normalization operation; and the third branch contains a pointwise convolution and a BN normalization operation. The input is passed to the three branches respectively, and the results of the last three branches are added and a GELU activation operation is performed to obtain the final output. The feature aggregation part is specifically operated as follows: the input is passed to the preprocessing link, the multi-head attention link, and the channel optimization link in sequence to obtain a feature aggregation output. The upsampling part takes the outputs of the two feature aggregation parts as input. First, perform a wavelet transform on input 1 to extract one low-frequency component and three high-frequency components. Concatenate the high-frequency components along the channel dimension and process them using a convolutional layer with residual connections to obtain high-frequency features. Concatenate the low-frequency components with input 2 along the channel dimension and process them using a convolutional layer with residual connections. Concatenate the processed features with the high-frequency features along the channel dimension, and finally perform an inverse wavelet transform to obtain the output.The specific operation of the multi-scale aggregation enhancement module includes: using the fifth stage detail feature map and the fifth stage global feature map as inputs of the fourth feature aggregation part to obtain the fourth feature aggregation output, using the fourth stage detail feature map and the fourth stage global feature map as inputs of the third feature aggregation part to obtain the third feature aggregation output, using the third stage detail feature map and the third stage global feature map as inputs of the second feature aggregation part to obtain the second feature aggregation output, using the second stage detail feature map and the second stage global feature map as inputs of the first feature aggregation part to obtain the first feature aggregation output; using the fourth feature aggregation output and the third feature aggregation output as inputs of the fourth upsampling part to obtain the third stage output; using the second feature aggregation output and the third stage output as inputs of the third upsampling part to obtain the second stage output; using the first feature aggregation output and the second stage output as inputs of the second upsampling part to obtain the first stage output; finally, using the DySample upsampling function on the first stage output to obtain the final accurate remote sensing image segmentation map.

[0082] The step 3 specifically includes: step 3.1: randomly initializing the parameters of the ultra-high resolution remote sensing image segmentation network, inputting the preprocessed training set and validation set data in step 1 into the ultra-high resolution remote sensing image segmentation network in step 2, generating a semantic segmentation probability map, and calculating the cross entropy loss and Dice loss.

[0083] Step 3.2: Loss back propagation, update network parameters, minimize the loss function as the optimization goal, obtain the optimal parameter model and save it.

[0084] The loss functions used in step 3 for training the ultra-high resolution remote sensing image segmentation network are the cross entropy loss function and the Dice loss function. The calculation formula of the cross entropy loss function is:

[0085] ,

[0086] Where N is the number of pixels, C is the number of categories, is the predicted probability of the kth category in the ith pixel, It represents the value in the one-hot encoding of the i-th pixel belonging to category k (if the i-th pixel belongs to category k, it is 1, otherwise it is 0).

[0087] The calculation formula of the Dice loss function is:

[0088] ,

[0089] Among them, TP is the number of true positives, that is, the number of pixels whose predicted and true labels are both positive; FP is the number of false positives, that is, the number of pixels predicted to be positive but actually negative; FN is the number of false negatives, that is, the number of pixels predicted to be negative but actually positive.

[0090] The following is a further description of the structure of each module provided in the embodiment: Figure 3 As shown: the first layer of the ResStem module is a 3×3 ordinary convolution with a step size of 2, the second layer is BN normalization, and the third layer is a maximum pooling with a pooling kernel size of 3 and a step size of 2.

[0091] The JPG original image in step 2.1 is used as the input of the ResStem module. The JPG original image is subjected to a 3×3 convolution operation with a step size of 2, BN normalization, and maximum pooling with a pooling kernel size of 3 and a step size of 2, and a local branch first-stage detail feature map with a resolution of 1 / 4 of the original input image is output.

[0092] The structure of LKResblock provided in the embodiment is as follows Figure 5 As shown in the figure: The LKResblock module consists of three branches, including a main branch, a sub-branch and a residual branch. The first layer of the main branch is a 7×7 depth convolution with a step size of 1, the second layer is BN normalization, the third layer is a ReLU activation function, the fourth layer is a 7×7 depth convolution with a step size of 1, and the fifth layer is BN normalization; the first layer of the sub-branch is BN normalization, the second layer is a 3×3 ordinary convolution with a step size of 1; the residual branch does not perform any operation. Finally, the results of the three branches are added and passed through a layer of ReLU activation function.

[0093] Taking the second stage of the local branch described in step 2.1 as an example, the LKResblock module takes the detailed features of the first stage of the local branch described in step 2.1 as input, and passes the input into the three branches to perform operations respectively, and obtains the main branch feature map, the secondary branch feature map and the residual branch feature map respectively, adds the three feature maps, and then performs the ReLU activation operation to output the local branch second stage feature map with a resolution of 1 / 4 of the original input image.

[0094] The structure of the spectrum fusion wavelet downsampling module provided in the embodiment is as follows: Figure 6As shown: the first layer of the spectrum fusion wavelet downsampling module consists of two branches, including a wavelet transform branch and a fast Fourier transform branch. The first layer of the wavelet transform branch is a discrete wavelet transform, the second layer is a splicing operation, and the third layer is a composite convolution layer, including 1×1 ordinary convolution, BN normalization and ReLU activation function. The first layer of the fast Fourier transform branch is a discrete Fourier transform, and the second layer contains two parts. The first part is responsible for calculating and processing the amplitude part of the feature, wherein the first layer calculates the amplitude of the feature; the second layer is a composite convolution layer, including 1×1 ordinary convolution, LeakyReLU activation function, SE attention layer and 1×1 ordinary convolution, and the SE attention layer includes an average pooling layer, a Linear fully connected layer, a ReLU activation function, a Linear fully connected layer and a Sigmoid activation function layer; the third layer performs a feature addition operation, adding the features of the first and second layers; the fourth layer calculates the real part of the feature. The second part of the second layer of the fast Fourier transform is responsible for calculating and processing the phase part of the feature, wherein the first layer calculates the phase of the feature; the second layer is a composite convolution layer, and the structure is the same as the composite convolution layer of the second layer of the first part of the second layer of the fast Fourier transform; the third layer performs a feature addition operation, adding the features of the first and second layers; the fourth layer calculates the imaginary part of the feature. The third layer of the fast Fourier transform branch performs a complex number synthesis operation, and the fourth layer is an inverse discrete Fourier transform. The second layer of the spectrum fusion wavelet downsampling module is a composite convolution layer, including 1×1 ordinary convolution, BN normalization, and GELU activation function.

[0095] Taking the third stage of the local branch described in step 2.1 as an example, the spectrum fusion wavelet downsampling module uses the feature map of the second stage of the local branch as input, and passes the input to the two branches of the first layer of the spectrum fusion wavelet downsampling module respectively, and obtains the wavelet transform feature and the fast Fourier transform feature respectively. Then the above two features are passed to the second layer of the spectrum fusion wavelet downsampling module, the features are added, and then a composite convolution operation is performed to output a feature map with a resolution of 1 / 8 of the original input image. This feature map will be used as the input of the third stage LKResblock of the local branch described in step 2.1.

[0096] The structure of the Stem module provided in the embodiment is as follows Figure 4 As shown: the first layer of the Stem module is a 7×7 ordinary convolution with a step size of 2, and the second layer is LN normalization.

[0097] In step 2.1, the JPG original image is used as the input of the Stem module, and a 7×7 convolution operation with a step size of 2 and LN normalization are performed on the JPG original image to output the global branch first-stage global feature map with a resolution of 1 / 2 of the original input image.

[0098] The structure of the DSE-Mamba module provided in the embodiment is as follows Figure 7As shown in the figure: the first layer of the DSE-Mamba module is a 2×2 ordinary convolution with a step size of 2; the second layer contains a 1×1 ordinary convolution with a step size of 1 and a dimension reorganization operation; the third layer is a two-dimensional real Fourier transform; the fourth layer is a feature multiplication operation, multiplying the third layer result with the weight matrix initialized by the module; the fifth layer includes an inverse Fourier transform and a feature reorganization operation; the sixth layer contains a 3×3 depth convolution with a step size of 1 and a feature segmentation operation, which divides the feature into two parts; the seventh layer contains a GELU activation function and a feature multiplication operation, and the eighth layer is a 1×1 ordinary convolution with a step size of 1. The second to eighth layers are the spectrum enhancement part of the DSE-Mamba module. The ninth layer is LN normalization; the tenth layer contains a Linear fully connected layer and a feature segmentation operation, which divides the feature into two parts; the eleventh layer is a 3×3 depth convolution with a step size of 1; the twelfth layer is a Silu activation function; the thirteenth layer is the underlying core computing unit SSM of the existing Mamba network; the fourteenth layer is an LN normalization layer; the fifteenth layer contains a Silu activation function and a feature multiplication operation; the sixteenth layer is a Linear fully connected layer; the seventeenth layer is a Dropout activation function layer; the eighteenth layer is a feature addition operation; the nineteenth layer is a composite layer, which includes a feature reorganization and splicing operation, an LN normalization layer, and a Linear fully connected layer. The ninth to eighteenth layers are the VSS Block part of the DSE-Mamba module.

[0099] Taking the second stage of the global branch described in step 2.2 as an example, the DSE-Mamba module uses the output of the Stem module as input, first passes the input into the first layer for feature dimension adjustment, and then passes it into the spectrum enhancement part of the DSE-Mamba module to obtain the spectrum weighted feature. Among them, the seventh layer is specifically: perform a GELU activation operation on the first part of the output of the sixth layer, and then multiply it with the second part of the output of the sixth layer to obtain a weighted result. Then, the spectrum weighted feature is passed into the VSS Block of the DSE-Mamba module to obtain a global feature. Among them, the fifteenth layer is specifically: perform a Silu activation operation on the second part of the output of the tenth layer, and then multiply it with the output of the fourteenth layer to obtain a weighted result. Finally, the global feature is input into the nineteenth layer, and a global branch second stage feature map with a resolution of 1 / 4 of the original input image is output.

[0100] The structure of the multi-scale aggregation enhancement module provided in the embodiment is as follows: Figure 8As shown: the multi-scale aggregation enhancement module includes a feature aggregation part and an upsampling part. The feature aggregation part has 4 parts, each with the same structure. The first layer is feature splicing; the second layer is a composite convolution layer, including a 3×3 depth convolution with a step size of 1, a BN normalization layer, a GELU activation function layer, a 1×1 point convolution with a step size of 1, a BN normalization layer, and a GELU activation function layer; the third layer is a Linear fully connected layer; the fourth layer includes a Linear fully connected layer and a dimension reorganization operation; the fifth layer includes 4 layers of parallel depth convolution, with convolution kernel sizes of 3, 5, 7, 9 and feature splicing operations; the sixth layer is a composite layer, including a 1×1 dimensional up-convolution with a step size of 1, BN normalization, a GELU activation function, and a 1×1 dimensional down-convolution with a step size of 1; the seventh layer is feature multiplication; the eighth layer includes a Linear fully connected layer and a Dropout activation function. The third to eighth layers are the multi-head attention links of the feature aggregation part. The ninth layer contains three branches. The first branch is a BN normalization layer. The second branch contains a 3×3 depth convolution with a stride of 1 and a BN normalization layer. The third branch contains a 1×1 point convolution with a stride of 1 and a BN normalization layer. Finally, the results of the three branches are added together and the GELU activation function is used. The tenth layer is a composite convolution layer, which contains a 1×1 point convolution with a stride of 1, a BN normalization layer, and a GELU activation function layer. The upsampling part has 4 parts in total. The second to fourth parts have the same structure. The first layer is discrete wavelet transform; the second layer is feature splicing operation; the third layer is a composite convolution layer, including 1×1 point convolution with a step size of 1, 3×3 ordinary convolution with a step size of 1, GELU activation function, 3×3 ordinary convolution with a step size of 1, GELU activation function, residual connection and 1×1 point convolution with a step size of 1; the fourth layer is feature splicing operation; the fifth layer is a composite convolution layer, including 1×1 point convolution with a step size of 1, 3×3 ordinary convolution with a step size of 1, GELU activation function, 3×3 ordinary convolution with a step size of 1, GELU activation function, residual connection and 1×1 point convolution with a step size of 1; the sixth layer includes feature splicing operation and inverse discrete wavelet transform. The first part of the upsampling part is the DySample upsampling function, and the upsampling ratio is 4.

[0101] The multi-scale aggregation enhancement module uses the detail feature map of the second to fifth stages of the local branch in step 2.1 and the global feature map of the second to fifth stages of the global branch in step 2.2 as input. Specifically, the fifth stage detail feature map and the fifth stage global feature map are used as inputs of the fourth feature aggregation part to obtain the fourth feature aggregation output, and the fourth stage detail feature map and the fourth stage global feature map are used as inputs of the third feature aggregation part to obtain the third feature aggregation output, and so on to obtain the second feature aggregation output and the first feature aggregation output. The fourth feature aggregation output and the third feature aggregation output are used as inputs of the fourth upsampling part to obtain the third stage output; the second feature aggregation output and the third stage output are used as inputs of the third upsampling part to obtain the second stage output; the first feature aggregation output and the second stage output are used as inputs of the second upsampling part to obtain the first stage output; finally, the DySample upsampling function is used on the first stage output to obtain the final accurate remote sensing image segmentation map.

[0102] The results of the ultra-high resolution remote sensing image segmentation network that integrates multivariate frequency domain information provided by the embodiment are compared as follows: Fig. 9 , Fig.10 , Fig.11 , Fig.12 shown.

[0103] In order to prove the effectiveness of the ultra-high resolution remote sensing image segmentation network integrating multivariate frequency domain information provided in this embodiment, the Vaihingen and Potsdam remote sensing datasets are used to train, verify and test the model. Fig. 9 , 10 It can be seen that the evaluation indicators of this embodiment are higher than those of the existing segmentation network, and the segmentation effect is closest to the original image.

[0104] The segmentation results of the ultra-high resolution remote sensing image segmentation network integrating multivariate frequency domain information provided in the embodiment on the Vaihingen dataset are compared as follows: Fig.11 shown. Fig.11 Figures a, f, and g in the figure show the characteristics of high inter-class similarity in remote sensing images. Taking Figure a as an example, the texture and size of the car and the background in the figure are highly similar. The traditional CNN network UNet has a simple network structure and insufficient utilization of feature extraction, so it completely misidentifies the background. EIGNet segmentation is correct, but the edge segmentation is not clear. UnetFormer, CMLFormer, MSGCNet, and CTCFNet have good segmentation effects, but misjudgments and missed detections still occur. In contrast, this embodiment combines frequency domain information and spatial domain information, and can effectively cope with the challenge of texture similarity. Therefore, the segmentation effect is the best, surpassing the compared methods. Fig.11Figures b to e in the middle show scenes with large grayscale changes in the dataset, such as shadow occlusion or rich edge information. Fig.11 Take b as an example, Fig.11 The right side of middle b is an area composed of low vegetation, trees and opaque surfaces, in which the shadow of the building covers most of the low vegetation, resulting in the loss of spatial domain information. UNet produced the most segmentation errors; CMLFormer and CTCFNet also produced many segmentation errors, segmenting most of the opaque surfaces as buildings, and identifying the low vegetation not covered by the shadow as trees; UnetFormer and MSGCNet had good segmentation effects, only identifying a small part of the opaque surface and low vegetation as buildings and trees; EIGNet had a better segmentation effect, only segmenting a small part of the opaque surface as buildings, not identifying the low vegetation as trees, and the segmentation edges were good. The segmentation results of this embodiment are the most accurate, all categories are correctly identified, no segmentation errors occur, and the segmentation edges are complete, without disconnection jagged edges, etc., which is closest to the label image.

[0105] like Fig.12 As shown in FIG. 1 , this embodiment also shows the segmentation results of some advanced networks on the Potsdam dataset. Fig.12 From the first to the fourth and sixth rows in the figure, it can be observed that the scale of objects in remote sensing images varies greatly. The scales of buildings in Figures a, b, and f are very different, and there are differences in texture and color. The scales of low vegetation in Figures c and d are also different. The segmentation results of the advanced network are not good, and there are still difficulties in segmenting multi-scale targets in remote sensing images. Compared with other models, this embodiment has the best segmentation effect, especially in the area within the orange frame.

[0106] from Fig.12 It can be observed from the fifth and seventh rows that, compared with all pure spatial domain segmentation networks, this embodiment performs better in segmenting areas with significant grayscale changes, such as shadow changes in Figure e and object edges in Figure g. In the case of grayscale changes, all the compared methods have problems of missed detection and rough edges, while this embodiment uses spectral information to make up for the lack of perception in the spatial domain, allowing the network to capture complete target features and good edge information, thereby improving the accuracy of remote sensing image segmentation.

[0107] The present invention provides an ultra-high resolution remote sensing image segmentation method based on frequency domain information fusion. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A method for ultra-high resolution remote sensing image segmentation based on frequency domain information fusion, characterized in that: The following steps are involved: Step 1: Get the Vaihingen and Potsdam remote sensing datasets, which contain the original images and the corresponding label images. After preprocessing the remote sensing data set, the remote sensing data set is divided into a training set, a validation set and a test set according to a set ratio; Step 2: Build an ultra-high-resolution remote sensing image segmentation network that integrates multivariate frequency domain information, including a large kernel residual network, a discriminative spectrum enhancement Mamba network, a spectrum fusion wavelet downsampling module, and a multi-scale aggregation enhancement module; The large kernel residual network and the spectrum fusion wavelet downsampling module constitute a local branch of the ultra-high resolution remote sensing image segmentation network; The discriminative spectrum enhancement Mamba network includes a preprocessing Stem module and a discriminative spectrum enhancement module, which constitute a global branch of an ultra-high resolution remote sensing image segmentation network; The local branch and the global branch constitute a dual-branch encoder part of an ultra-high resolution remote sensing image segmentation network; The multi-scale aggregation enhancement module constitutes the decoder part of the ultra-high resolution remote sensing image segmentation network; Step 3: Input the Vaihingen training set into the ultra-high resolution remote sensing image segmentation network for training, calculate the loss function and perform back propagation, update the network parameters, and obtain the optimal parameter model; input the Vaihingen validation set into the ultra-high resolution remote sensing image segmentation network for verification, observe the training effect and record it; Step 4, input the Vaihingen test set in step 3 into the optimal parameter model trained in step 3, and output the accurate segmentation map of the remote sensing image; Step 5: Replace the data set with the Potsdam training set, Potsdam validation set, and Potsdam test set, and repeat steps 3 to 4.

2. The method according to claim 1, characterized in that: Step 1 includes: A Vaihingen remote sensing dataset is obtained, wherein the Vaihingen remote sensing dataset includes five foreground categories and one background category, and corresponding label maps, wherein the five foreground categories include low vegetation, trees, impervious surfaces, cars, and buildings; Obtain a Potsdam remote sensing dataset, wherein the Potsdam remote sensing dataset includes six categories and corresponding label maps, wherein the six categories include low vegetation, trees, impervious surfaces, cars, buildings, and clutter; The original image and the corresponding label image in the Vaihingen remote sensing dataset are cropped in sequence, and the farthest point random cropping data enhancement method is used to obtain the remote sensing original image and the corresponding label image; The original image and the corresponding label image in the Potsdam remote sensing dataset are cropped in order, and the random rotation and folding image enhancement technology are used to obtain the remote sensing original image and the corresponding label image; The Vaihingen remote sensing dataset is randomly divided into Vaihingen training set, Vaihingen validation set and Vaihingen test set according to proportion; the Potsdam remote sensing dataset is randomly divided into Potsdam training set, Potsdam validation set and Potsdam test set according to proportion.

3. The method according to claim 2, characterized in that Step 2 includes: Step 2.1: The local branch includes 5 stages. The remote sensing image JPG original image in the Vaihingen training set is input into the local branch to obtain the output feature map of the 5 stages of the local branch; Step 2.2: The global branch includes 5 stages. The remote sensing image JPG original image in the Vaihingen training set is input into the global branch to obtain the output feature map of the 5 stages of the global branch; Step 2.3: Input the detail feature maps of the second to fifth stages obtained in step 2.1 and the global feature maps of the second to fifth stages obtained in step 2.2 into the multi-scale aggregation enhancement module, and output the final remote sensing image segmentation map.

4. The method according to claim 3, characterized in that Step 2.1 includes: in the five stages of the local branch, the first stage is composed of a residual preprocessing ResStem module, the second stage is composed of a large kernel residual block LKResblock, and the third, fourth and fifth stages are composed of a large kernel residual block LKResblock and a spectrum fusion wavelet downsampling module; Through convolution, batch normalization function BN normalization, activation function, fast Fourier transform and wavelet transform operations, the remote sensing image is extracted from low-level detail features to high-level detail features, specifically including: inputting the JPG original image into the first stage of the local branch to obtain the first stage detail feature map with a resolution of 1 / 4 of the original input image, inputting the first stage detail feature map into the second stage of the local branch to obtain the second stage detail feature map with a resolution of 1 / 4 of the original input image, inputting the second stage detail feature map into the third stage of the local branch to obtain the third stage detail feature map with a resolution of 1 / 8 of the original input image, inputting the third stage detail feature map into the fourth stage of the local branch to obtain the fourth stage detail feature map with a resolution of 1 / 16 of the original input image, and inputting the fourth stage detail feature map into the fifth stage of the local branch to obtain the fifth stage detail feature map with a resolution of 1 / 32 of the original input image; The residual preprocessing ResStem module in the first stage includes convolution, batch normalization function BN normalization and pooling layer. The JPG original image is sequentially subjected to 3×3 convolution with a step size of 2, batch normalization function BN normalization and maximum pooling operation with a pooling kernel of 3 and a step size of 2 to obtain the first stage detail feature map with a resolution of 1 / 4 of the original input image. The large kernel residual block LKResblock of the second stage consists of three branches, including a main branch, a sub-branch and a residual branch. In the main branch, the original JPG image is subjected to 7×7 depth convolution, batch normalization function BN normalization, ReLU activation operation, 7×7 depth convolution and BN normalization in turn to obtain the main branch feature map; in the sub-branch, the original JPG image is subjected to batch normalization function BN normalization and 3×3 depth convolution in turn to obtain the sub-branch feature map; the residual branch does not perform any operation on the original input; finally, the feature maps obtained by the main branch, sub-branch and residual branch are added and ReLU activation operation is performed to obtain the second stage detail feature map with a resolution of 1 / 4 of the original input image; The large kernel residual block LKResblock of the third stage has the same structure as the large kernel residual block LKResblock of the second stage. The spectrum fusion wavelet downsampling module of the third stage takes the output of the large kernel residual block LKResblock of the third stage as input. The spectrum fusion wavelet downsampling module includes a wavelet transform branch and a fast Fourier transform branch. In the wavelet branch, the input is firstly subjected to a discrete wavelet transform to obtain a low-frequency component and three high-frequency components, with a resolution of 1 / 2 of the input resolution. Then, a low-frequency component and three high-frequency components are concatenated in the channel dimension, and then 1×1 convolution, batch normalization function BN normalization and ReLU activation operations are performed in sequence to obtain a wavelet transform branch feature map. In the fast Fourier transform branch, the input is subjected to a discrete wavelet transform to obtain a low-frequency component and three high-frequency components, with a resolution of 1 / 2 of the input resolution. Then, a low-frequency component and three high-frequency components are concatenated in the channel dimension, and then 1×1 convolution, batch normalization function BN normalization and ReLU activation operations are performed in sequence to obtain a wavelet transform branch feature map. In the Rieger transform branch, the input is first subjected to a fast Fourier transform to obtain an amplitude spectrum and a phase spectrum. Then, the amplitude spectrum and the phase spectrum are subjected to a 1×1 convolution, a LeakyReLU activation, a channel attention SE layer weighting operation, and a 1×1 convolution operation, respectively. The processed amplitude spectrum and phase spectrum are then used to obtain the spectrum information through complex synthesis. The spectrum information is converted back to the spatial domain through an inverse Fourier transform to obtain a fast Fourier transform branch feature map with a resolution of 1 / 2 of the input. Finally, the wavelet transform branch feature map and the fast Fourier transform branch feature map are added and sequentially subjected to a 1×1 convolution, a batch normalization function BN normalization, and a GELU activation operation to obtain a third-stage detail feature map with a resolution of 1 / 8 of the original input image. The channel attention SE layer includes global average pooling, a fully connected layer, and an activation function operation, specifically including: firstly, performing global average pooling on the input feature map, then sequentially passing through a dimensionality-reduced Linear fully connected layer, a ReLU activation, a dimensionality-increased Linear fully connected layer, and a Sigmoid activation function, and finally multiplying the generated weight coefficient with the input feature map to implement a weighted operation; The large kernel residual block LKResblock and the spectrum fusion wavelet downsampling module of the fourth stage have the same structure as the large kernel residual block LKResblock and the spectrum fusion wavelet downsampling module of the third stage. The detail feature map of the third stage is input into the fourth stage to obtain the detail feature map of the fourth stage with a resolution of 1 / 16 of the original input image. The large kernel residual block LKResblock and spectrum fusion wavelet downsampling module of the fifth stage have the same structure as those of the large kernel residual block LKResblock and spectrum fusion wavelet downsampling module of the third stage. The fourth stage detail feature map is input into the fifth stage to obtain the fifth stage detail feature map with a resolution of 1 / 32 of the original input image.

5. The method according to claim 4, characterized in that Step 2.2 includes: in the 5 stages of the global branch, the first stage is composed of a preprocessing Stem module, and the second stage, the third stage, the fourth stage and the fifth stage are composed of a discriminant spectrum enhancement network. Through convolution, layer normalization function LN normalization, activation function, and fast Fourier transform operations, the remote sensing image is extracted from low-level global features to high-level global features; specifically including: inputting the JPG original image into the first stage of the global branch to obtain a first-stage global feature map with a resolution of 1 / 2 of the original input image, inputting the first-stage global feature map into the second stage of the global branch to obtain a second-stage global feature map with a resolution of 1 / 4 of the original input image, inputting the second-stage global feature map into the third stage of the global branch to obtain a third-stage global feature map with a resolution of 1 / 8 of the original input image, inputting the third-stage global feature map into the fourth stage of the global branch to obtain a fourth-stage global feature map with a resolution of 1 / 16 of the original input image, and inputting the fourth-stage global feature map into the fifth stage of the global branch to obtain a fifth-stage global feature map with a resolution of 1 / 32 of the original input image; The first stage preprocessing Stem module includes convolution and instance normalization. Specifically, the JPG original image is subjected to 7×7 convolution with a step size of 2 and instance normalization operations in sequence to obtain the first stage global feature map with a resolution of 1 / 2 of the original input image. The discriminative spectrum enhancement network of the second stage includes a spectrum enhancement part and a visual state space module, and the spectrum enhancement part specifically includes: using 1×1 convolution on the input to map the number of channels to a higher dimension, then reconstructing the dimension and dividing it into two or more small blocks, performing a two-dimensional real Fourier transform on each small block to obtain frequency domain features; defining a learnable parameter for filtering, multiplying the learnable parameter with the frequency domain feature, and then performing an inverse Fourier transform to convert the frequency domain feature back to the time domain and reconstruct it back to the original dimension; then using a deep separable convolution on the features after spectrum filtering to divide the output into a first part and a second part along the channel dimension, using a GELU activation function to perform a nonlinear transformation on the first part, and then multiplying it with the second part, and finally restoring the feature dimension to the original input dimension through a 1×1 convolution to obtain a spectrum enhancement feature; The visual state space module specifically includes: using LN normalization on the input, then using the Linear fully connected layer to double the number of feature channels and split the features into the first part of features and the second part of features from the channel dimension, performing deep convolution and SiLU activation operations on the first part of features in sequence, and then inputting the processed first part of features into the computing unit state space model SSM, outputting 4 features; adding the 4 features, using the transpose operation to transform the feature dimension and performing the LN normalization operation to obtain the output after state learning; using the SiLU activation function to perform nonlinear transformation on the second part of features, and then multiplying them with the output after state learning to obtain new features; using the Linear fully connected layer to restore the original dimension of the new features, and finally using the Dropout function to perform feature regularization to obtain the final output features, with the resolution remaining unchanged; The discriminative spectrum enhancement network specifically includes: using 2×2 convolution to process the input in blocks, then passing the block-processed features into the spectrum enhancement part to obtain the spectrum enhancement features, and then inputting the processed features into the visual state space module to obtain the processed features, and finally dividing the processed features into 4 parts from the height and width dimensions and splicing them along the channel dimension, performing LN normalization and Linear full connection layer, and then outputting the second-stage global feature map with a resolution of 1 / 4 of the original input image; The discriminant spectrum enhancement network of the third stage has the same structure as the discriminant spectrum enhancement network of the second stage, and the global feature map of the second stage is input into the third stage to obtain the global feature map of the third stage, and the resolution is 1 / 8 of the original input image; The discriminant spectrum enhancement network of the fourth stage has the same structure as the discriminant spectrum enhancement network of the second stage, and the global feature map of the third stage is input into the fourth stage to obtain the global feature map of the fourth stage, and the resolution of the global feature map of the fourth stage is 1 / 16 of the original input image; The discriminant spectrum enhancement network of the fifth stage has the same structure as the discriminant spectrum enhancement network of the second stage. The global feature map of the fourth stage is input into the fifth stage to obtain the global branch fifth stage global feature map, and the resolution is 1 / 32 of the original input image.

6. The method according to claim 5, characterized in that Step 2.3 includes: the multi-scale aggregation enhancement module includes 4 feature aggregation parts with consistent structures and 4 upsampling parts, the 4 feature aggregation parts with consistent structures are respectively a first feature aggregation part, a second feature aggregation part, a third feature aggregation part and a fourth feature aggregation part, and the outputs of the second, third, fourth and fifth stages of step 2.1 and step 2.2 are respectively used as inputs; the 4 upsampling parts are respectively a first upsampling part, a second upsampling part, a third upsampling part and a fourth upsampling part; The feature aggregation part includes a preprocessing step, a multi-head attention step and a channel optimization step; the preprocessing step receives two input parameters, first concatenates the two input parameters in the channel dimension to obtain a new feature, and then performs deep convolution, BN normalization, GELU activation function, point convolution, BN normalization and GELU activation function operations on the new feature in turn to obtain the preprocessed feature; the multi-head attention step specifically includes: taking the output of the preprocessing step as input, using linear transformation on the input feature to obtain a value vector V, then reshaping the key vector K through linear transformation and feature dimension reshaping, and then using deep separable convolution for each attention The key vector of the head is used for feature extraction to obtain features, and the attention head is a module initialization parameter; the features are processed, including expanding the channel dimension, normalization, activation and dimension reduction, and then the features are weighted with the value vector, and finally the dimension is reduced through the projection layer and the neuron random drop method is applied to obtain the output; the channel optimization link consists of three branches, wherein the first branch contains a BN normalization operation; the second branch contains a depth convolution and a BN normalization operation; the third branch contains a point convolution and a BN normalization operation, and the input is passed into the three branches respectively, and the results of the last three branches are added and a GELU activation operation is performed to obtain the final output; The specific operation of the feature aggregation part includes: passing the input to the preprocessing stage, the multi-head attention stage and the channel optimization stage in sequence to obtain the feature aggregation output; The upsampling part takes the outputs of the two feature aggregation parts as inputs, the outputs of the two feature aggregation parts are the first input and the second input respectively, firstly performs wavelet transform on the first input, extracts one low-frequency component and three high-frequency components, concatenates the high-frequency components along the channel dimension and processes them using a convolutional layer with a residual connection to obtain high-frequency features; concatenates the low-frequency components with the second input along the channel dimension and processes them using a convolutional layer with a residual connection, concatenates the processed features with the high-frequency features along the channel dimension, and finally performs inverse wavelet transform to obtain output; The specific operation of the multi-scale aggregation enhancement module includes: using the fifth stage detail feature map and the fifth stage global feature map as inputs of the fourth feature aggregation part to obtain a fourth feature aggregation output, using the fourth stage detail feature map and the fourth stage global feature map as inputs of the third feature aggregation part to obtain a third feature aggregation output, using the third stage detail feature map and the third stage global feature map as inputs of the second feature aggregation part to obtain a second feature aggregation output, using the second stage detail feature map and the second stage global feature map as inputs of the first feature aggregation part to obtain a first feature aggregation output; The fourth feature aggregation output and the third feature aggregation output are used as the input of the fourth upsampling part to obtain the third stage output; the second feature aggregation output and the third stage output are used as the input of the third upsampling part to obtain the second stage output; the first feature aggregation output and the second stage output are used as the input of the second upsampling part to obtain the first stage output; finally, the DySample upsampling function is used on the first stage output to obtain the final accurate remote sensing image segmentation map.

7. The method according to claim 6, characterized in that Step 3 includes: Step 3.1: Randomly initialize the parameters of the ultra-high resolution remote sensing image segmentation network, input the training set and validation set data into the high-resolution remote sensing image segmentation network, generate a semantic segmentation probability map, and calculate the cross entropy loss L CE and Dice loss L dice ; Step 3.2: Loss back propagation, update network parameters, minimize the loss function as the optimization goal, obtain the optimal parameter model and save it.

8. The method according to claim 7, characterized in that In step 3.1, the cross entropy loss L CE The calculation formula is: Among them, N is the number of pixels, C is the number of categories, and p ik is the predicted probability of the kth category in the ith pixel, y ik It represents the value of the one-hot encoding of the i-th pixel belonging to category k. If the i-th pixel belongs to category k, y ik is 1, otherwise y ik is 0; The Dice loss L dice The calculation formula is: Among them, TP is the number of true positives; FP is the number of false positives; FN is the number of false negatives.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 8 are executed.

Images