A Large Model Adaptation and Fine-Tuning Remote Sensing Land Cover Classification Method Based on Fourier Transform

Abstract

This invention relates to the field of remote sensing image processing, specifically to a large-model adaptation and fine-tuning method for remote sensing land cover classification based on Fourier transform. It combines Fourier transform with multi-feature adaptation and fine-tuning of SAM (Simulated Aspect-Oriented Mapping), constructing a SAM remote sensing land cover classification model coupled with Fourier transform and multi-feature adaptation and fine-tuning. The method includes the following steps: data acquisition and preprocessing, construction of a weight adaptation mechanism, Fourier transform module design, construction of a multi-feature adapter, model structure design, model training, and model performance evaluation. This invention significantly improves the model's ability to represent complex scenes and multi-scale features, effectively adapting to the differences in features at different scales and the complexity of scenes in remote sensing images. Furthermore, by fully utilizing the pre-training capabilities of SAM, the model can adapt to various remote sensing image resolutions and input sizes, greatly improving its applicability and scalability, and providing an efficient and practical new method for remote sensing land cover classification.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, specifically a remote sensing land cover classification method based on large model adaptation and fine-tuning using Fourier transform. Background Technology

[0002] Land cover classification using remote sensing imagery plays a crucial role in modern geographic information systems and environmental monitoring. It provides precise data for land resource management, supporting rational planning and efficient land use, and effectively monitoring land use change. In ecological environmental protection, by identifying ecosystem types and monitoring their dynamic changes, it provides a scientific basis for ecological protection and restoration, contributing to biodiversity conservation. Simultaneously, land cover classification plays a vital role in agriculture, identifying agricultural land types, supporting precision agriculture management, and ensuring food security. Furthermore, it provides critical information for urban planning, disaster early warning, and resource development, promoting sustainable socio-economic development.

[0003] In the early stages of remote sensing technology development, land cover classification primarily relied on spectral features. Commonly used methods included supervised and unsupervised classification. These methods classified based on the spectral reflectance characteristics of land features, but their ability to distinguish spectrally similar features was limited. With the improvement of remote sensing image resolution, object-oriented classification methods gradually emerged. These methods segmented images into multiple objects (such as land parcels, buildings, etc.) and combined spectral, texture, and shape features for classification. In the 21st century, statistical models and machine learning methods began to be introduced into remote sensing image classification. Algorithms such as random forests and support vector machines gradually became mainstream methods due to their powerful classification capabilities and adaptability to complex data. In recent years, the rapid development of deep learning technology has brought new breakthroughs to remote sensing image land cover classification. Convolutional neural networks can effectively handle the complexity of high-resolution images by automatically learning image features. Furthermore, improved network architectures (such as incorporating attention mechanisms and multi-scale loss functions) have further improved classification performance. Although the above-mentioned remote sensing image land cover classification methods have made significant progress, some shortcomings and challenges remain. The features of remote sensing images vary due to factors such as time, region, sensor type, and climate, making it difficult for models to adapt to diverse data and resulting in insufficient generalization ability. Furthermore, deep learning models are typically trained on fixed datasets, making it difficult for them to automatically adapt to data drift.

[0004] As an innovative and general segmentation model, SAM (Separate Aspects of Imagery) demonstrates outstanding generalization ability and high adaptability to data drift, bringing new technological breakthroughs to land cover classification tasks. Through large-scale pre-training, SAM deeply mines and masters rich and extensive visual features, enabling it to understand and analyze various complex visual scenes. This powerful feature learning capability allows SAM to effectively adapt to different types of remote sensing image data. When faced with image data acquired in different seasons or by different sensors, significant data drift often exists. However, SAM, with its unique model architecture and flexible training strategy, can quickly adapt to new data distributions through fine-tuning. By fine-tuning on a small amount of target data, SAM can rapidly adjust model parameters, allowing the model to better adapt to the characteristics of image data in specific scenarios while maintaining its understanding of general visual features, thereby achieving high-precision segmentation and classification in land cover classification tasks.

[0005] While SAM (Search Engine Analysis) demonstrates significant advantages in land cover classification tasks using remote sensing imagery, it still suffers from some undeniable shortcomings and limitations. First, its adaptability to remote sensing imagery is insufficient. Trained on large datasets of natural images, SAM's performance on remote sensing images is limited because these images differ significantly from natural images in terms of viewpoint, color, texture, and layout. Second, SAM's output is a binary mask, primarily focusing on segmenting target regions and lacking direct representation of category information. This prevents it from providing pixel-level category labeling for land cover classification tasks, which require precise differentiation of different land cover categories—a characteristic that makes it difficult for SAM to perform independently. Third, SAM performs poorly when faced with fine-grained classification tasks. Remote sensing applications often require fine-grained classification of similar land cover features, such as distinguishing different types of crops or subdividing urban buildings. While SAM possesses powerful segmentation capabilities, its ability to differentiate complex land cover details is limited, easily leading to misclassification and affecting the accuracy and reliability of the classification results. Based on this, the present invention designs a SAM semantic segmentation remote sensing land cover classification method coupled with Fourier transform and multi-feature adaptation fine-tuning to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a large-model adaptation and fine-tuning remote sensing land cover classification method based on Fourier transform, so as to solve the technical problems mentioned in the background art.

[0007] Based on the above ideas, the present invention provides the following technical solution:

[0008] The method for adapting and fine-tuning a large model based on Fourier transform for remote sensing land cover classification includes the following steps:

[0009] S1. Data Acquisition and Preprocessing: This invention uses the remote sensing semantic segmentation dataset GID-5 to train the model. The GID-5 dataset contains 150 remote sensing images, each with a resolution of 6800×7200 pixels. During the data partitioning process, 125 images were randomly selected as the training set, 5 images as the validation set, and 20 images for testing, to ensure a reasonable distribution between training and validation.

[0010] Preferably, due to limitations in computer video memory, it is not possible to directly input full-size images into the network for training. Therefore, the image is first sliced ​​into small blocks of 448×448 pixels. Simultaneously, to ensure data integrity and consistency in training, insufficient portions of the slices are padded with background to meet the network's input size requirements.

[0011] S2. Constructing a Weight Adaptation Mechanism: When training the model using the pre-trained weights of the SAM, since this method employs the image encoder part of the SAM model, accurately selecting the weights of the image encoder part in the SAM pre-trained weights becomes a crucial step. First, the model's size parameters are calculated, including key parameters such as the input image size, patch size, and window size, to ensure the accuracy of subsequent weight adaptation. After completing the parameter calculation, the model weights are traversed layer by layer to select the weights belonging to the image encoder part; these weights are the core part of the model capturing image features.

[0012] Preferably, when using SAM pre-trained weights for model training, since the image size used in the SAM pre-training process is 1024×1024, while the image size input by this method is 448×448, this difference makes it necessary to adjust the positional encoding in the weights to ensure the performance of the model under the new input size.

[0013] Preferably, the adjustment of positional encoding mainly involves two key parts: absolute positional encoding and relative positional encoding. Absolute positional encoding plays a crucial role in accurately representing the spatial relationships of features within the model. Given the change in image size, the original absolute positional encoding needs to be interpolated based on the current image size and patch size. Specifically, appropriate interpolation algorithms, such as bilinear interpolation or bicubic interpolation, are used to recalculate and adjust the original encoding. Relative positional encoding is primarily responsible for expressing the model's ability to capture relative positional dependencies. Due to changes in window size and image size, the range of relative positions needs to be recalculated. Based on this, the relative positional encoding values ​​are interpolated to match the current window size. After adjusting the positional encoding, the adapted weights are loaded into the model to ensure efficient operation at the new size.

[0014] S3. Fourier Transform Module Design: A two-dimensional Fast Fourier Transform (FFT) is performed on the input data to transform it from the spatial domain to the frequency domain. The real and imaginary parts are concatenated along the channel dimension to obtain a frequency domain feature map. Two concatenated frequency domain features are then applied to two convolutional layers and processed using the ReLU activation function. A two-dimensional Inverse Fast Fourier Transform (IFT) is performed on the convolutional frequency domain feature map to restore it to its spatial domain format. In the IFT output, only the real part is taken as the final output of the frequency domain branch. This is because the real part can more effectively reflect the key features of the data in certain application scenarios, while avoiding interference from the imaginary part. Regarding the spatial domain path, the input feature map passes through two 3×3 convolutional layers, denoted as conv3 and conv4. In these two convolutional layers, through carefully designed kernel parameters and convolution operations, deep feature extraction is performed on the input feature map to uncover local features and spatial structure information of the input data in the spatial domain, thus obtaining a feature map processed by the spatial domain path. Finally, the outputs of the frequency domain branch, the spatial domain path, and the original input feature map are added element-wise. This fusion method fully integrates the feature information of the data in different domains, achieving the organic fusion of multi-dimensional features.

[0015] S4. Constructing a Multi-Feature Adapter: Since the SAM design does not explicitly distinguish between high-level and low-level features and lacks hierarchical output, while multi-scale features are required later, a multi-feature adapter needs to be constructed. This adapter temporarily reuses the SAM output directly as two types of features. The final high-level output features are fed into Spatial Pyramid Pooling (ASPP) for multi-scale feature extraction, and then upsampled and fused with the earlier low-level features. This effectively integrates high-level semantic information and low-level detail information, thereby fully leveraging the advantages of both, enhancing the model's ability to capture multi-scale ground features in complex scenes, and ultimately improving segmentation accuracy and robustness.

[0016] S5. Model Structure Design: The overall model structure includes a weight adaptation mechanism module, a Fourier transform module, a fine-tuned ViT encoder, a multi-feature adapter, and a classification head. The weight adaptation mechanism module, as the initial layer of the model, plays a crucial role in model construction. Given the potential differences in size and other aspects between the pre-trained model weights and the target model, this module uses a carefully designed algorithm to adapt the weights of the pre-trained model, ensuring a precise match between the weight parameters and the structural dimensions of the target model. Following this is the Fourier transform module, designed to simultaneously capture both local spatial features and global frequency information of the image. After obtaining the output of the Fourier transform module, it is input into the pre-fine-tuned ViT encoder. The ViT encoder utilizes techniques such as self-attention to perform deep processing of the input features, establishing connections between different feature dimensions, thereby performing a more comprehensive and in-depth encoding operation on the image. Next, the multi-feature adapter module converts the features output by the ViT encoder into the form required by the classification head. As the final layer of the model, the classification head bears the important task of generating the final classification result. The classification head utilizes the Spatial Pyramid Pooling (ASPP) module to sample features at different scales, effectively extracting multi-scale contextual information. This module can expand the receptive field and capture target information at different scales in the image without increasing computational load excessively. Simultaneously, the classification head fuses low-level and high-level features, fully utilizing the detailed information in low-level features and the semantic information in high-level features to generate pixel-level classification results, achieving accurate classification of various objects in the image.

[0017] Step 6: Model Training: Input the preprocessed training images into the constructed segmentation network model. The following are the key parameter settings during the training process. The total number of training iterations is set to 150,000. This value was determined after comprehensively considering factors such as model complexity, dataset size, and computing resources, aiming to ensure that the model can fully learn the features and patterns in the data and achieve good convergence results.

[0018] Preferably, the initial learning rate is set to 0.01. The learning rate, as a crucial hyperparameter controlling the step size of model parameter updates, has a key impact on the model's training speed and convergence. To dynamically adjust the learning rate during training to balance the model's convergence speed and accuracy, this method employs a polynomial decay learning rate scheduling strategy. During training, the batch size is set to 6. Through multiple experiments and analyses, a batch size of 6 has been determined to achieve a good balance between computational resources and model performance.

[0019] Preferably, this method uses the cross-entropy loss function to measure the difference between the model's predicted results and the true labels. Its function formula is:

[0020]

[0021] Where y i Let x be the probability that the true label x belongs to the i-th category. Predict the probability that x belongs to the i-th category for the model.

[0022] Preferably, the network parameters and weights are updated using a backpropagation algorithm and optimizer to minimize the loss function, thereby improving the network's segmentation performance. During training, we use images from the validation set for real-time validation to evaluate the model's segmentation performance and adjust the network parameters based on the validation results. Whenever the model performs better on the validation set than before, we save the current model weights to ensure we obtain the optimal network model. Through continuous iteration of this process, the finally trained model exhibits the most accurate and stable segmentation performance on the test set.

[0023] S7. Model Performance Evaluation: The trained network model is used for semantic segmentation of remote sensing images to test its segmentation performance. Evaluation metrics selected in this process include mean Intersection over Union (mIoU), overall Intersection over Union (oIoU), overall accuracy (OA), and F1 score (F1Score). mIoU comprehensively considers the segmentation accuracy of each class, reflecting the model's overall performance in multi-class segmentation tasks, especially sensitive to the segmentation of minority classes and complex backgrounds. oIoU provides the model's global segmentation performance across the entire dataset but does not distinguish between the contributions of different classes, making it suitable for quickly evaluating the model's overall segmentation performance. OA is the proportion of correctly classified pixels out of the total number of pixels, reflecting the model's classification accuracy globally. F1Score comprehensively evaluates the model's precision and recall, particularly suitable for handling class imbalance problems, balancing the model's predictive ability for minority classes.

[0024] The beneficial effects of this invention are:

[0025] This invention utilizes Fourier transform to collaboratively extract features in both the frequency and spatial domains, enabling global-local joint modeling and improving the model's ability to capture image details and global structure. Furthermore, the self-attention mechanism more efficiently captures global information and long-range dependencies in images, thus more accurately modeling spatial relationships and global features among land features. To further enhance the model's ability to perceive image contextual information, this invention employs a dilated spatial pyramid pooling (ASPP) structure. The ASPP structure, through dilated convolutions with different sampling rates, can extract image features at different scales, effectively improving the model's ability to represent complex scenes and multi-scale land features. This allows the model to better adapt to the differences in land features at different scales and the complexity of scenes in remote sensing imagery, significantly improving the accuracy of land cover classification.

[0026] Meanwhile, this invention features flexible size adaptation capabilities and a smooth feature transformation mechanism in its technical implementation, enabling it to adapt to various remote sensing image resolutions and input sizes, thus improving the model's applicability and scalability. Through a weight adaptation mechanism, this invention fully utilizes the pre-training capabilities of SAM, effectively shortening model training time while significantly improving model performance and generalization ability. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a method for adapting and fine-tuning remote sensing land cover classification based on Fourier transform for large model adaptation according to the present invention.

[0028] Figure 2 This is a model architecture diagram of a large-scale model adaptation and fine-tuning remote sensing land cover classification method based on Fourier transform according to the present invention.

[0029] Figure 3 This is a diagram of the Fourier transform module for adapting and fine-tuning a remote sensing land cover classification method based on Fourier transform in this invention. Detailed Implementation

[0030] Example 1

[0031] Please see Figure 1-3 This invention provides a technical solution: a large-model adaptation and fine-tuning remote sensing land cover classification method based on Fourier transform, the specific steps of which are as follows:

[0032] Phase 1: Data Acquisition and Preprocessing

[0033] Step 1-1: This invention uses the remote sensing semantic segmentation dataset GID-5 to train the algorithm. The GID-5 dataset contains 150 remote sensing images, each with a resolution of 6800×7200 pixels. During the data partitioning process, 125 images are randomly selected as the training set, 5 images as the validation set, and 20 images for testing, to ensure a reasonable distribution of training and validation.

[0034] Steps 1-2: Due to limitations in computer video memory, it's impossible to directly input a full-size image into the network for training. Therefore, the image is first sliced ​​into small blocks of 448×448 pixels. Simultaneously, to ensure data integrity and consistency in training, insufficient portions of the slices are padded with background to meet the network's input size requirements.

[0035] Phase Two: Constructing a Weight Adaptation Mechanism

[0036] Step 2-1: When training the model using the pre-trained weights of SAM, accurately selecting the weights of the image encoder part of the SAM pre-trained weights is a crucial step, since this method uses the image encoder part of the SAM model. First, calculate the model's size parameters, including key parameters such as input image size, patch size, and window size, to ensure the accuracy of subsequent weight adaptation. After completing the parameter calculation, iterate through the model weights layer by layer to select the weights belonging to the image encoder part; these weights are the core part of the model capturing image features.

[0037] Step 2-2: When using SAM pre-trained weights for model training, the image size used in the SAM pre-training process is 1024×1024, while the image size input by this method is 448×448. This difference makes it necessary to adjust the positional encoding in the weights to ensure the performance of the model under the new input size.

[0038] In terms of positional encoding adjustment, two key parts are involved: absolute positional encoding and relative positional encoding. Absolute positional encoding plays a crucial role in accurately representing the spatial relationships of features within the model. Given the change in image size, the original absolute positional encoding needs to be interpolated based on the current image size and patch size. Specifically, appropriate interpolation algorithms, such as bilinear or bicubic interpolation, are used to recalculate and adjust the original encoding. Relative positional encoding is primarily responsible for expressing the model's ability to capture relative positional dependencies. Due to changes in window size and image size, the range of relative positions needs to be recalculated. Based on this, the relative positional encoding values ​​are interpolated to match the current window size. After adjusting the positional encoding, the adapted weights are loaded into the model to ensure efficient operation at the new dimensions.

[0039] Phase 3: Fourier Transform Module Design

[0040] Step 3-1, as follows Figure 3 As shown, a two-dimensional Fast Fourier Transform (FFT) is performed on the input data to transform it from the spatial domain to the frequency domain. The real and imaginary parts are concatenated along the channel dimension to obtain a frequency domain feature map. Two concatenated frequency domain features are then applied to two convolutional layers and processed using the ReLU activation function. A two-dimensional Inverse Fast Fourier Transform (IFT) is performed on the convolutional frequency domain feature map to restore it to its spatial domain format. In the IFT output, only the real part is taken as the final output of the frequency domain branch. This is because the real part can more effectively reflect the key features of the data in certain application scenarios, while avoiding interference from the imaginary part. Regarding the spatial domain path, the input feature map passes through two 3×3 convolutional layers, denoted as conv3 and conv4. In these two convolutional layers, through carefully designed kernel parameters and convolution operations, deep feature extraction is performed on the input feature map to uncover local features and spatial structure information of the input data in the spatial domain, thus obtaining a feature map processed by the spatial domain path. Finally, the outputs of the frequency domain branch, the spatial domain path, and the original input feature map are added element-wise. This fusion method fully integrates the feature information of the data in different domains, achieving the organic fusion of multi-dimensional features.

[0041] Phase 4: Constructing a multi-feature adapter

[0042] Step 4-1: Since the SAM design does not explicitly distinguish between "high-level" and "low-level" features and lacks hierarchical output, while the subsequent classification head requires multi-scale features, a multi-feature adapter needs to be constructed. This adapter temporarily reuses the SAM output directly as two types of features. The final high-level output features are then fed into Spatial Pyramid Pooling (ASPP) for multi-scale feature extraction, followed by upsampling and fusion with the earlier low-level features. This effectively integrates high-level semantic information with low-level detail information, thereby fully leveraging the advantages of both, enhancing the model's ability to capture multi-scale ground features in complex scenes, and ultimately improving segmentation accuracy and robustness.

[0043] Phase 5: Model Structure Design

[0044] Step 5-1: The overall model structure includes a weight adaptation mechanism module, a Fourier transform module, a fine-tuned ViT encoder, a multi-feature adapter, and a classification head. The weight adaptation mechanism module, as the initial layer of the model, plays a crucial role in model construction. Given the potential differences in size and other aspects between the pre-trained model weights and the target model, this module uses a carefully designed algorithm to adapt the weights of the pre-trained model, ensuring a precise match between the weight parameters and the structural dimensions of the target model. Following this is the Fourier transform module, designed to simultaneously capture both local spatial features and global frequency information of the image. After obtaining the output of the Fourier transform module, it is input into the pre-fine-tuned ViT encoder. The ViT encoder utilizes techniques such as self-attention to perform deep processing of the input features, establishing connections between different feature dimensions, thereby enabling a more comprehensive and in-depth encoding operation of the image. Next, the multi-feature adapter module converts the features output by the ViT encoder into the form required by the classification head. As the final layer of the model, the classification head bears the important task of generating the final classification result. The classification head utilizes the Spatial Pyramid Pooling (ASPP) module to sample features at different scales, effectively extracting multi-scale contextual information. This module can expand the receptive field and capture target information at different scales in the image without increasing computational load excessively. Simultaneously, the classification head fuses low-level and high-level features, fully utilizing the detailed information in low-level features and the semantic information in high-level features to generate pixel-level classification results, achieving accurate classification of various objects in the image.

[0045] Phase 6: Model Training

[0046] Step 6-1: Input the preprocessed training images into the constructed segmentation network model. The following are the key parameter settings during the training process. The total number of training iterations is set to 150,000. This value was determined after comprehensively considering factors such as model complexity, dataset size, and computing resources, aiming to ensure that the model can fully learn the features and patterns in the data and achieve good convergence results.

[0047] The initial learning rate was set to 0.01. As a crucial hyperparameter controlling the step size of model parameter updates, the learning rate has a significant impact on the model's training speed and convergence. To dynamically adjust the learning rate during training to balance convergence speed and accuracy, this method employs a polynomial decay learning rate scheduling strategy. During training, the batch size was set to 6. Through multiple experiments and analyses, a batch size of 6 was determined to achieve a good balance between computational resources and model performance.

[0048] This method uses the cross-entropy loss function to measure the difference between the model's predictions and the true labels. Its formula is:

[0049]

[0050] Where y i Let x be the probability that the true label x belongs to the i-th category. Predict the probability that x belongs to the i-th category for the model.

[0051] Step 6-2: Update the network parameters and weights using the backpropagation algorithm and optimizer to minimize the loss function, thereby improving the network's segmentation performance. During training, we use images from the validation set for real-time validation to evaluate the model's segmentation performance and adjust the network parameters based on the validation results. Whenever the model performs better on the validation set than before, we save the current model weights to ensure we obtain the optimal network model. Through continuous iteration of this process, the finally trained model exhibits the most accurate and stable segmentation performance on the test set.

[0052] Phase 7: Model Testing

[0053] The trained network model was used for semantic segmentation of remote sensing images to test its segmentation performance. The evaluation metrics selected included mean Intersection over Union (mIoU), overall Intersection over Union (oIoU), overall accuracy (OA), and F1 score. mIoU comprehensively considers the segmentation accuracy of each class, reflecting the model's overall performance in multi-class segmentation tasks, and is particularly sensitive to the segmentation of minority classes and complex backgrounds. oIoU provides the model's global segmentation performance across the entire dataset, but does not distinguish the contributions of different classes, making it suitable for quickly evaluating the model's overall segmentation performance. OA is the proportion of correctly classified pixels out of the total number of pixels, reflecting the model's classification accuracy globally. The F1 score comprehensively evaluates the model's precision and recall, and is particularly suitable for handling class imbalance problems, balancing the model's predictive ability for minority classes.

[0054] Example 2

[0055] In Example 1, the fusion of spatial and frequency domain features was achieved through the Fourier transform module, significantly improving the segmentation effect for complex features and multi-scale targets. However, in actual testing, for remote sensing scenes with rich edge details or high noise levels, there are still problems such as insufficient local precision and inadequate cross-scale feature interaction. Therefore, based on steps three (Fourier transform module design) and four (multi-feature adaptation) of Example 1, this example conducts further parameterization research and improves the fusion strategy, aiming to further enhance the model's segmentation performance for edges and details, solve the feature confusion problem under noise interference, and thus achieve higher segmentation accuracy and better robustness.

[0056] This includes a multi-scale weighted Fourier fusion algorithm, which introduces an adjustable multi-scale weighting factor in the parallel frequency and spatial domain branches of Example 1 to further balance the importance of high-frequency and low-frequency information at different scales and optimize edge detection and noise suppression of complex features.

[0057] The steps include:

[0058] In the original Fourier transform module, a 2D FFT is performed on the input feature map to obtain a complex frequency domain tensor. This tensor is then divided into blocks or filtered according to different frequency bands to distinguish between high-frequency and low-frequency components.

[0059] Let α and β be weighting factors for fusing high-frequency and low-frequency components. α and β are automatically learned during training to adapt to the balance between edge and overall information in different scenarios.

[0060] Finally, the weighted high-frequency components and low-frequency components are combined into a new frequency domain representation, which is then added element-wise to the spatial domain branch in subsequent steps.

[0061] A more flexible balance is achieved between low-frequency (overall semantics) and high-frequency (edge ​​details) through a multi-scale weighted fusion formula, resulting in more robust semantic segmentation, especially in noisy environments or scenarios rich in edge features. This includes:

[0062] Freq fusion (k,l)=α·Freq low (k,l)+β·Freq high (k,l)

[0063] in:

[0064] Freq low (k,l) represents the low-frequency component, Freq high (k,l) represents the high-frequency component; α and β are weighted hyperparameters introduced in this algorithm, which are dynamically learned during training and satisfy the constraint α+β=1; (k,l) is the frequency domain coordinate index.

[0065] It also includes: Based on the multi-scale weighted Fourier fusion algorithm, in order to address the problem that different image regions (such as flat regions and textured regions) have different requirements for high and low frequency fusion, the "adaptive spatial-frequency coupling algorithm" is further proposed. According to the texture complexity of the current features in the spatial domain, the weighting factors α and β in the multi-scale weighted Fourier fusion algorithm are adaptively adjusted.

[0066] The steps include:

[0067] In the spatial domain, gradient / variance measures are applied to the input feature map or its local regions to obtain the texture complexity metric T.

[0068] An adaptive correction coefficient γ is generated based on the fact that T is within the range of [0, T_{max}]. The more complex the texture, the larger γ becomes, in order to increase the proportion of high frequencies.

[0069] The weighting factors are updated and fusion is performed. The α and β in the multi-scale weighted Fourier fusion algorithm are modified to obtain new α′ and β′, which further enhances the high-frequency weights in regions with complex textures.

[0070] The ASFC coupling formula reuses the parameters α and β from the multi-scale weighted Fourier fusion algorithm and wraps them with a new parameter γ, as follows:

[0071] α′=α+γ·(β-α)

[0072] β′=β+γ·(β-α)

[0073] in:

[0074] α and β are defined in the same way as in the multi-scale weighted Fourier fusion algorithm;

[0075] γ∈[0,1] is the correction coefficient obtained by adaptive learning based on spatial domain texture complexity;

[0076] When the value of γ is large, it will increase the actual role of β in the fusion, thereby increasing the weight of high-frequency components.

[0077] This adaptive strategy can automatically adjust the frequency domain fusion method according to the complexity of regional features, allowing the model to obtain richer high-frequency information in regions with complex textures or object edges, thereby improving the segmentation effect.

[0078] It also includes a multi-level linkage fusion algorithm. After the above algorithms have respectively implemented multi-scale weighted fusion and adaptive spatial-frequency domain coupling, a multi-level linkage fusion algorithm (MLSF) is further designed to convert the Freq output by the multi-scale weighted fusion algorithm into a single step. fusion Combined with α′ and β′ obtained by the adaptive spatial-frequency domain coupling algorithm, they are unified into a multi-level linkage framework, which improves the overall deep collaboration effect between the frequency domain and the spatial domain.

[0079] The steps include:

[0080] The Freq output by the multi-scale weighted fusion algorithm fusion (k,l) serves as the basic frequency domain feature input.

[0081] The α′ and β obtained by introducing an adaptive spatial-frequency coupling algorithm are used to adjust Freq in different spatial locations or blocks. fusion (k,l) is further weighted and refined.

[0082] After spatial domain branch convolution in Example 1, an adaptive selection layer is performed based on the texture complexity metric T of the adaptive spatial-frequency domain coupling algorithm to filter and highlight the high frequencies of key regions.

[0083] The multi-level linkage and fusion formula is expressed as follows:

[0084] in:

[0085] Freqfusion(k,l) is the frequency domain fusion result;

[0086] α′ and β′ are calculated by an adaptive spatial-frequency domain coupling algorithm;

[0087] S space This represents the feature map corresponding to the convolutional branch in the spatial domain, in Φ(S space Under this operation, important regions (such as high gradient or high texture locations) can be extracted;

[0088] This indicates an element-wise Hadamard multiplication or convolution operation, used to achieve deep coupling between frequency domain features and key regions in the spatial domain.

[0089] By linking the results of the multi-scale weighted fusion algorithm and the adaptive spatial-frequency coupling algorithm, the algorithm first performs multi-scale weighting and then adaptive coupling, and finally performs multiplicative coupling with the spatial domain high-texture features, thereby enhancing the edge fidelity and the ability to capture small targets in complex scenes.

[0090] Final output Freq MLSF (k,l) can be restored to the spatial domain by inverse FFT or further fused with the spatial domain branch in subsequent steps of Example 1 to effectively improve the overall segmentation performance.

[0091] By introducing multi-scale weighted fusion and adaptive coupling, high- and low-frequency components are regionalized and reweighted in the frequency domain, overcoming the edge loss problem caused by noise. α and β are dynamically adjusted based on local texture complexity, enabling targeted frequency domain information enhancement for different regions.

[0092] By unifying the fused features and adaptive weights, and achieving a deeper level of complementarity with spatial domain features, more subtle differences in ground feature boundaries and targets can be captured in high-resolution remote sensing images, further improving the performance of Example 1.

[0093] During testing, the edge pixels of ground features became more finely distinguishable, and the misclassification rate decreased.

[0094] At the same time, high-frequency details are enhanced, and a balance between details and global semantics is achieved through multi-level linkage.

[0095] Improve overall mIoU, F1, and other metrics: Under the same hardware and training strategy, the improvements in Example 2 can further improve the final model in terms of mIoU, oIoU, F1, and other metrics compared to Example 1.

Claims

1. A large-model adaptation and fine-tuning remote sensing land cover classification method based on Fourier transform, characterized in that, Includes the following steps: S1. Data acquisition and preprocessing, including: using the remote sensing image dataset, randomly selecting several images as training set, validation set and test set; cropping large images into smaller images; and filling the background in the insufficient parts to meet the size requirements of the network input. S2. Construct a weight adaptation mechanism, including: selecting the weights belonging to the image encoder part in the model pre-training weights; and interpolating the absolute position encoding and relative position encoding in the weights based on the difference between the original image size and the size of the small image to adapt to the input size. S3, Fourier Transform Module Design, includes performing a two-dimensional fast Fourier transform on the input features, concatenating the real and imaginary parts in the channel dimension and sequentially passing them through convolution and corrected linear activation, and then performing a two-dimensional inverse fast Fourier transform, retaining only the real part output; at the same time, two convolutions are used in the spatial domain path to extract depth features, and finally added element-wise with the frequency domain branch and the original input features; S4. Construct a multi-feature adapter, including: temporarily reusing the output as two types of features, high-level and low-level; inputting the high-level features into the hollow spatial pyramid pooling for multi-scale feature extraction and then upsampling, and fusing them with the low-level features to enhance the model's ability to capture multi-scale land cover features; S5. Model structure design, including: weight adaptation mechanism module, Fourier transform module, fine-tuned ViT encoder, multi-feature adapter and classification head; the classification head extracts contextual information at different scales through dilated spatial pyramid pooling and fuses low-level and high-level features to achieve pixel-level fine segmentation; S6. Model training, including: inputting the preprocessed training images and backpropagating using the cross-entropy loss function, and using a multinomial decay strategy; S7. Model performance evaluation, including: evaluating the model using the mean crossover ratio, overall crossover ratio, overall accuracy, and F1 score.

2. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 1, characterized in that, The dataset of remote sensing images in S1 contains 150 remote sensing images, each with a resolution of 6800×7200 pixels. During the data partitioning process, 125 images are randomly selected as the training set, 5 images as the validation set, and 20 images for testing.

3. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 2, characterized in that, In step S1, the image is sliced ​​and cropped into small image blocks of 448×448 pixels.

4. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 1, characterized in that, In step S2, when training the model using the pre-trained weights, the image encoder part of the model is used to first calculate the model's size parameters, including the input image size, small image size, and window size. After the parameter calculation is completed, the model weights are traversed layer by layer to select the weights belonging to the image encoder part.

5. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 4, characterized in that, In step S2, when training the model using pre-trained weights, since the image size used in the pre-training process is 1024×1024 and the input image size is 448×448, in order to ensure the model's performance under the new input size, the positional encoding in the pre-trained weights needs to be adjusted. The positional encoding adjustment includes absolute positional encoding and relative positional encoding. The absolute positional encoding is interpolated based on the current image size and the size of the small image blocks. The relative positional encoding recalculates the range of relative positions and interpolates the relative positional encoding values ​​to match the current window size. After the positional encoding adjustment is completed, the adapted weights are loaded into the model.

6. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 1, characterized in that, In step S3, a two-dimensional fast Fourier transform is performed on the input data to convert the data from the spatial domain to the frequency domain, and the real and imaginary parts are concatenated along the channel dimension to generate a frequency domain feature map. Two convolutional layers are applied sequentially to the frequency domain feature map, and after processing with a modified linear activation function, a two-dimensional inverse fast Fourier transform is performed to restore the feature map to the spatial domain format, retaining only the real part of the two-dimensional inverse fast Fourier transform output as the final output of the frequency domain branch. In the spatial domain path, two 3×3 convolutional layers are applied sequentially to the input feature map to extract local features and spatial structure information to generate a spatial domain path feature map. Finally, the frequency domain branch output, the spatial domain path output, and the original input feature map are added element-wise to achieve multi-dimensional feature fusion.

7. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 1, characterized in that, In S5, the overall model structure includes a weight adaptation mechanism module, a Fourier transform module, a fine-tuned ViT encoder, a multi-feature adapter, and a classification head. The weight adaptation mechanism module, as the initial layer of the model, adapts the weights of the pre-trained model to match the structural dimensions of the target model. The Fourier transform module aims to simultaneously capture the local spatial features and global frequency information of the image. After obtaining the output of this module, it is input into the fine-tuned ViT encoder, which establishes connections between different feature dimensions through a self-attention mechanism, thereby encoding the image more deeply. Subsequently, the multi-feature adapter converts the features output by the ViT encoder into the form required by the classification head. The classification head obtains contextual information from different scales through dilated spatial pyramid pooling and fuses low-level and high-level features to generate pixel-level classification results, achieving accurate classification of various objects in the image.

8. The method for large-scale model adaptation and fine-tuning remote sensing land cover classification based on Fourier transform according to claim 1, characterized in that, In step S6, the preprocessed training image is input into the constructed segmentation network model, and the total number of training iterations is set to 150,000. The initial learning rate was set to 0.

01. As an important hyperparameter controlling the step size of model parameter updates, the learning rate has a key impact on the training speed and convergence of the model. To dynamically adjust the learning rate during training to balance the model's convergence speed and accuracy, a multinomial decay learning rate scheduling strategy is adopted, including setting the batch size to 6 during training. When measuring the difference between the model's predicted results and the true labels, the cross-entropy loss function is used, and its formula is: ； in Let x be the probability that the true label x belongs to the i-th category. Predict the probability that x belongs to the i-th category for the model; The network parameters and weights are updated using the backpropagation algorithm and optimizer to minimize the loss function, thereby improving the network's segmentation performance. During training, images from the validation set are used for real-time validation to evaluate the model's segmentation performance, and the network parameters are adjusted based on the validation results. Whenever the model performs better on the validation set than in the previous state, the current model weights are saved to obtain the optimal network model.

Images