A frequency-space dual-domain joint fine-tuning visual base model network method for remote sensing image domain generalization semantic segmentation

Abstract

The application discloses a kind of frequency-space dual-domain joint fine-tuning visual basic model network methods for remote sensing image domain generalization semantic segmentation, which enhances the consistency of cross-domain similar features by adaptive frequency selection in the middle features of the frequency domain fine-tuned model, and improves the discriminant ability of the feature cluster boundary in the deep features of the spatial domain fine-tuned model to achieve robust domain generalization semantic segmentation performance. Through four experimental settings of three public datasets Potsdam, Vaihingen and LoveDA, the mIoU index of the present method is improved by 1.64% and 1.09% on average compared with the most advanced method, and the model parameter amount is only 4.22M, which balances the accuracy and computational efficiency.

Description

Technical Field

[0001] This invention relates to a method for generalized semantic segmentation of remote sensing image domains, involving joint fine-tuning of the frequency-space dual-domain visual fundamental model network, and belongs to the field of remote sensing image processing. Background Technology

[0002] Remote sensing semantic segmentation is a fundamental task in the field of Earth observation. Its core objective is to assign a corresponding semantic label to each pixel in an image, and it is widely used in many important scenarios such as land type classification, urban planning, and environmental monitoring. With the rapid development of deep learning technology, various remote sensing semantic segmentation methods have emerged and been iteratively optimized. However, most existing methods assume that the training data and inference data follow an independent and identically distributed (i.i.d.) pattern. In real-world applications, this assumption often fails to hold due to factors such as wavelength differences, geographical span, and variations in spatial resolution. This phenomenon, known as domain shift, poses a significant challenge to the performance of deep learning models—models that perform well on training data often experience a significant performance decline when deployed to unfamiliar inference data. This problem severely restricts the large-scale application of remote sensing semantic segmentation technology in real-world scenarios.

[0003] To address the domain offset problem, domain adaptation and domain generalization have become two core solutions. Domain adaptation aligns the distributions of the source and target domains by combining labeled source domain data with unlabeled target domain data, thereby improving model performance in the target domain. However, domain adaptation methods rely on access permissions to the target domain data, which is often difficult or extremely costly to obtain in remote sensing scenarios, significantly limiting its applicability in known target domain scenarios. In contrast, domain generalization relies solely on source domain data for model training. Its core objective is to learn cross-domain universal domain-invariant features, enabling the model to effectively generalize to unknown target domains, thus alleviating the problem of limited access to target domain data. Existing domain generalization methods can be categorized into three main strategies: data processing, representation learning, and learning strategies. Data processing strategies enhance model robustness by enriching the diversity of training data distributions; representation learning strategies focus on extracting cross-domain consistent domain-invariant features; and learning strategies enhance the model's generalization ability by optimizing the training process (such as meta-learning). However, when the amount of training data is limited, these methods often struggle to achieve sufficient generalization performance.

[0004] The rise of visual foundation models has provided a new perspective for domain-generalized semantic segmentation. These models, based on large-scale datasets and utilizing self-supervised or semi-supervised pre-training paradigms, can extract general visual features with excellent generalization capabilities and flexibly adapt to various data distribution scenarios. Against this backdrop, visual foundation models have shown broad application prospects in remote sensing image domain-generalized semantic segmentation tasks, especially with their efficient parameter fine-tuning techniques, which can effectively help models achieve cross-domain generalization even in scenarios with scarce labeled data. Currently, domain-generalized semantic segmentation methods based on visual foundation models mostly focus on designing and employing efficient parameter fine-tuning strategies in the spatial domain dimension; however, existing methods for efficient spatial domain parameter fine-tuning of visual foundation models still struggle to effectively address the prominent cross-domain intra-class discrepancies in remote sensing scenarios. Summary of the Invention

[0005] The technical solution of this invention is: A method for generalized semantic segmentation in the remote sensing image domain, involving joint fine-tuning of the frequency-space dual-domain visual foundation model network, is proposed. The specific implementation steps of this method include: The first step involves inputting the remote sensing image into the backbone network of the visual basic model for feature extraction. For the features output from each intermediate layer in the backbone network, a frequency-domain adaptive sensing adapter is used for fine-tuning in the frequency domain before the fine-tuned features are input into the next layer of the backbone network for further propagation. The specific structure of the frequency-domain adaptive sensing adapter is as follows: Figure 2 As shown.

[0006] Specifically, given features extracted from the nth layer of the visual base model backbone network As input, to reduce computational overhead, the frequency-domain adaptive sensing adapter first performs a downsampling operation on the input features, and then performs a two-dimensional Fourier transform on the downsampled features to convert them from the spatial domain to the frequency domain. The transformation process can be represented as follows: , Where (x,y) and (u,v) represent spatial and frequency domain coordinates, respectively, and j represents the imaginary unit.

[0007] To enable the model to adaptively weight frequency components corresponding to different network depths during training and improve cross-domain feature adaptation capability, this paper introduces a learnable frequency-domain adaptive mask. This mask can dynamically weight the frequency components of each layer of features. This weighting process can be specifically represented as: , in, This indicates element-wise multiplication.

[0008] Frequency domain adaptive mask Based on global frequency statistics, the specific process is as follows: First, calculate the frequency domain features. amplitude Global average pooling (GAP) and global max pooling (GMP) operations are performed along the channel dimension to capture complementary frequency response features. The two pooling results are then concatenated, fused through a lightweight convolutional layer, and finally activated by a sigmoid activation function to generate a frequency-domain adaptive mask. The specific process is shown in the table below: , Among them, | | indicates taking the frequency domain amplitude, [ , [] represents the feature concatenation operation. This represents the sigmoid activation function.

[0009] Features obtained after frequency weighting It needs to be mapped back to the spatial domain through a two-dimensional inverse Fourier transform. This inverse transform process can be expressed as: , Subsequently, the features were transformed back to the spatial domain. Perform an upsampling operation to project it back to the channel dimension of the original features; finally, introduce a learnable scaling parameter. The residual connections are used to preserve the effective information of the original input features, and finally the output features are finely tuned in the frequency domain.

[0010]

[0011] Output features The input is propagated to the next layer of the visual basic model backbone network. By embedding frequency domain adaptive perception adapters in multiple intermediate layers of the model, the model can adaptively weight various frequency components at different network depths, thereby selectively strengthening cross-domain consistent frequency features and improving feature consistency within categories in cross-domain scenarios.

[0012] The second step involves fine-tuning the final layer of features in the visual base model using a frequency-adaptive perceptual adapter, followed by further fine-tuning in the spatial domain using a spatial multi-prototype adapter. This operation compresses the distances of easily confused features at class cluster boundaries, improving the discriminative ability of such features. The specific structure of the spatial multi-prototype adapter is as follows: Figure 3 As shown.

[0013] Input characteristics of this module This represents the output features of the final layer of the visual foundation model, fine-tuned by a frequency-adaptive perceptual adapter. To fully model intra-category diversity, we introduce a set of non-learnable multi-prototype libraries. These prototypes are determined solely by sub-centers of features within each category. Specifically, each category... From M prototypes It indicates that the prototype It is defined as the center of the m-th sub-cluster of training pixel samples belonging to class k.

[0014] To calculate the similarity between pixel features and individual prototypes, the input features are first... Flatten the image into a set of pixel features, and denote the feature vector corresponding to the i-th pixel as:

[0015] Each pixel feature and prototypes of all categories Calculate cosine similarity to measure how well the pixel matches different categories and their intra-class patterns.

[0016]

[0017] in and All have been L2 normalized. This represents the similarity between the i-th pixel feature and the m-th prototype of category k.

[0018] For each pixel, we select the most similar prototype by taking the maximum similarity value, perform weighted pixel feature calculation, and then feed it into the decoder to complete the subsequent processing and obtain the prediction result.

[0019] For the multi-prototype library, an online clustering strategy is employed for prototype selection and allocation: during model training, pixel-level features within the same semantic category are assigned to the corresponding prototype set based on sample labels, and the prototype representation is continuously updated according to the feature allocation results. This clustering and updating process encourages the model to further explore and characterize separable discriminative patterns within categories, thereby better modeling intra-class diversity. It is important to note that the multi-prototype library is only updated during the training phase and remains frozen during the inference phase to ensure the stability and efficiency of inference.

[0020] Beneficial effects This invention proposes a frequency-space dual-domain joint fine-tuning visual base model network method for generalized semantic segmentation in the remote sensing image domain. Four different experimental settings were designed and conducted on three publicly available remote sensing datasets: Potsdam, Vaihingen, and Loveda. The average intersection-over-union (IoU) ratios under each setting are as follows: a) Potsdam->Vaihingen: The model was trained on the Potsdam dataset and inference was performed on the Vaihingen dataset test set, achieving an average intersection-union ratio of 73.49%. b) Vaihingen->Potsdam: The model was trained on the Vaihingen dataset and inference was performed on the Potsdam dataset test set, with an average intersection-union ratio of 61.97%. c) Rural->Urban: The model was trained using the Rural subset of the Loveda dataset as the training set and tested on the Urban subset of the same dataset as the validation set, achieving an average intersection-over-union ratio of 59.28%. d) Urban -> Rural: The model was trained using the Urban subset of the Loveda dataset as the training set and tested on the Rural subset of the same dataset as the validation set, achieving an average intersection-union ratio of 48.41%.

[0021] The experimental results above show that, compared with existing related methods, the network method proposed in this invention has achieved improvements in all experimental settings, verifying its effectiveness.

[0022] The model has 4.22M parameters, which is relatively lightweight among existing methods, saving resources and improving computational efficiency. Attached Figure Description

[0023] Figure 1 Overall framework diagram of the invention; Figure 2 Frequency adaptive sensing adapter framework diagram; Figure 3 Spatial multi-prototype adapter framework diagram. Detailed Implementation

[0024] The present invention will be further described below with reference to the embodiments.

[0025] Example A frequency-space dual-domain joint fine-tuning visual base model network method for generalized semantic segmentation in the remote sensing image domain is proposed. Taking the practical application scenario of generalized semantic segmentation in the optical remote sensing image domain as an example, the overall framework of this method is as follows: Figure 1 As shown, the steps of this method include: The first step involves inputting the optical remote sensing image into the backbone network of the visual basic model for feature extraction. For the features output from each intermediate layer in the backbone network, they are first fine-tuned in the frequency domain using a frequency-domain adaptive sensing adapter, and then the fine-tuned features are input into the next layer of the backbone network for further propagation. The specific structure of the frequency-domain adaptive sensing adapter is as follows: Figure 2 As shown.

[0026] Specifically, given features extracted from the nth layer of the visual base model backbone network As input, to reduce computational overhead, the frequency-domain adaptive sensing adapter first performs a downsampling operation on the input features, and then performs a two-dimensional Fourier transform on the downsampled features to convert them from the spatial domain to the frequency domain. The transformation process can be represented as follows: , Where (x,y) and (u,v) represent spatial and frequency domain coordinates, respectively, and j represents the imaginary unit.

[0027] To enable the model to adaptively weight frequency components corresponding to different network depths during training and improve cross-domain feature adaptation capability, this paper introduces a learnable frequency-domain adaptive mask. This mask can dynamically weight the frequency components of each layer of features. This weighting process can be specifically represented as: , in, This indicates element-wise multiplication.

[0028] Frequency domain adaptive mask Based on global frequency statistics, the specific process is as follows: First, calculate the frequency domain features. amplitude Global average pooling (GAP) and global max pooling (GMP) operations are performed along the channel dimension to capture complementary frequency response features. The two pooling results are then concatenated, fused through a lightweight convolutional layer, and finally activated by a sigmoid activation function to generate a frequency-domain adaptive mask. The specific process is shown in the table below: , Among them, | | indicates taking the frequency domain amplitude, [ , [] represents the feature concatenation operation. This represents the sigmoid activation function.

[0029] Features obtained after frequency weighting It needs to be mapped back to the spatial domain through a two-dimensional inverse Fourier transform. This inverse transform process can be expressed as: , Subsequently, the features were transformed back to the spatial domain. Perform an upsampling operation to project it back to the channel dimension of the original features; finally, introduce a learnable scaling parameter. The residual connections are used to preserve the effective information of the original input features, and finally the output features are finely tuned in the frequency domain.

[0030]

[0031] Output features The input is propagated to the next layer of the visual basic model backbone network. By embedding frequency domain adaptive perception adapters in multiple intermediate layers of the model, the model can adaptively weight various frequency components at different network depths, thereby selectively strengthening cross-domain consistent frequency features and improving feature consistency within categories in cross-domain scenarios.

[0032] The second step involves fine-tuning the final layer of features in the visual base model using a frequency-adaptive perceptual adapter, followed by further fine-tuning in the spatial domain using a spatial multi-prototype adapter. This operation compresses the distances of easily confused features at class cluster boundaries, improving the discriminative ability of such features. The specific structure of the spatial multi-prototype adapter is as follows: Figure 3 As shown.

[0033] Input characteristics of this module This represents the output features of the final layer of the visual foundation model, fine-tuned by a frequency-adaptive perceptual adapter. To fully model intra-category diversity, we introduce a set of non-learnable multi-prototype libraries. These prototypes are determined solely by sub-centers of features within each category. Specifically, each category... From M prototypes It indicates that the prototype It is defined as the center of the m-th sub-cluster of training pixel samples belonging to class k.

[0034] To calculate the similarity between pixel features and individual prototypes, the input features are first... Flatten the image into a set of pixel features, and denote the feature vector corresponding to the i-th pixel as:

[0035] Each pixel feature and prototypes of all categories Calculate cosine similarity to measure how well the pixel matches different categories and their intra-class patterns.

[0036]

[0037] in and All have been L2 normalized. This represents the similarity between the i-th pixel feature and the m-th prototype of category k.

[0038] For each pixel, we select the most similar prototype by taking the maximum similarity value, perform weighted pixel feature calculation, and then feed it into the decoder to complete the subsequent processing, thereby obtaining the prediction result.

[0039] Taking the practical application scenario of generalized semantic segmentation in the optical remote sensing image domain as an example, under the experimental setting of training the model on the Potsdam dataset as the training set and performing inference on the Vaihingen dataset as the test set, a comparison is made with various existing generalized semantic segmentation methods in the remote sensing image domain. As shown in Table 1, it can be seen that the method proposed in the embodiment has a higher average intersection-union ratio.

[0040] Table 1. Comparison of performance metrics between various existing remote sensing image domain generalized semantic segmentation methods and the proposed method.

[0041] In summary, the above are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generalized semantic segmentation of remote sensing image domains, involving joint fine-tuning of the frequency-space dual-domain visual fundamental model network, characterized in that... include: The first step is to input the remote sensing images into the backbone network of the visual basic model for feature extraction; For the features output by each intermediate layer in the backbone network, they are first fine-tuned in the frequency domain through a frequency domain adaptive sensing adapter, and then the fine-tuned features are input to the next layer of the backbone network to continue propagation. The second step involves fine-tuning the features of the last layer of the visual basic model through a frequency-adaptive perception adapter, and then sending them to a spatial multi-prototype adapter for fine-tuning in the spatial domain. This operation compresses the distance of easily confused features at the boundaries, thereby improving the ability to distinguish easily confused features at the boundaries of category clusters.

2. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network as described in claim 1, is characterized in that... The frequency domain adaptive perception adapter includes: performing frequency domain transformation on the features extracted from the visual basic model, and dynamically selecting frequency features of different frequency bands through a learnable adaptive frequency domain mask, thereby enhancing the consistency of features within categories in cross-domain scenarios.

3. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network, as described in claim 1, is characterized in that... This invention introduces a learnable frequency-domain adaptive mask. This mask can dynamically weight the frequency components of each layer of features. This weighting process can be specifically represented as: ； in, This indicates element-wise multiplication.

4. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network, as described in claim 1, is characterized in that... In the first step, frequency domain adaptive mask Based on global frequency statistics, the specific process is as follows: First, calculate the frequency domain features. amplitude Global average pooling (GAP) and global max pooling (GMP) operations are performed along the channel dimension to capture complementary frequency response features; The two pooling results are then concatenated, fused through a lightweight convolutional layer, and finally activated by the sigmoid activation function to generate a frequency-domain adaptive mask. The specific process is shown in the table below: ； Among them, | | indicates taking the frequency domain amplitude, [ , [] represents the feature concatenation operation. This represents the sigmoid activation function.

5. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network as described in claim 1, is characterized in that... In the first step, the features obtained after frequency weighting It needs to be mapped back to the spatial domain through a two-dimensional inverse Fourier transform. This inverse transform process can be expressed as: ； Subsequently, the features were transformed back to the spatial domain. Perform an upsampling operation to project it back to the channel dimension of the original features; finally, introduce a learnable scaling parameter. The residual connections are used to preserve the effective information of the original input features, and finally the output features are fine-tuned in the frequency domain. ； Output features It is input into the next layer of the visual basic model backbone network for propagation.

6. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network as described in claim 1, is characterized in that... The spatial multi-prototype adapter includes: constructing multiple intra-class feature prototypes for each land cover category, and weighting the pixel-level land cover features according to the similarity between the pixel-level land cover features and the intra-class feature prototypes, so as to compress the intra-class distance of easily confused features at the feature cluster boundary, thereby improving the discrimination ability of easily confused features at the feature cluster boundary.

7. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network as described in claim 1, is characterized in that... In the second step, the input features of the module The output features of the last layer of the visual basic model are fine-tuned by a frequency-adaptive perceptual adapter; to fully model the diversity within categories, a set of non-learnable multi-prototype libraries is introduced. These prototypes are determined solely by subcenters of the features of each category; each category From M prototypes It indicates that the prototype It is defined as the center of the m-th sub-cluster of training pixel samples belonging to class k.

8. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network as described in claim 1, is characterized in that... In the second step, to calculate the similarity between pixel features and individual prototypes, the input features are first... Flatten the image into a set of pixel features, and denote the feature vector corresponding to the i-th pixel as: ； Each pixel feature and prototypes of all categories Calculate cosine similarity to measure how well the pixel matches different categories and their intra-class patterns: ； in and All have been L2 normalized. This represents the similarity between the i-th pixel feature and the m-th prototype of category k.

9. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of the visual fundamental model network as described in claim 1, is characterized in that... In the second step, for each pixel, the most similar prototype is selected by taking the maximum similarity value, and the pixel features are weighted and then sent to the decoder to complete the subsequent processing and obtain the prediction result.

10. The method for generalized semantic segmentation of remote sensing image domains, frequency-space dual-domain joint fine-tuning of visual fundamental model networks as described in claim 1, is characterized in that... In the second step, for the multi-prototype library, an online clustering strategy is adopted for prototype selection and allocation: during the model training phase, pixel-level features within the same semantic category are allocated to the prototype set corresponding to that category by using sample labels, and the prototype representation is continuously updated based on the feature allocation results.

Images