Remote sensing image semantic segmentation method and device based on parameter efficient adapter

Abstract

This invention relates to the field of remote sensing image processing technology, and in particular to a method and apparatus for semantic segmentation of remote sensing images based on a parameter-efficient adapter. This invention avoids the high computational overhead and catastrophic forgetting risk associated with full fine-tuning by freezing all parameters of the pre-trained visual backbone network, effectively preserving the general visual feature representation capabilities accumulated during the pre-training phase. A lightweight trainable adapter module is inserted after the preset intermediate layer of the backbone network, achieving feature transfer from the general visual domain to the remote sensing domain using only a small number of parameters from the adapter module, significantly reducing the computational resources and memory overhead required for training. Furthermore, a segmentation head transforms the pixel-level feature representation after removing classification labels into category prediction, and then upsamples it to restore the original spatial size, achieving end-to-end pixel-level semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method and apparatus for semantic segmentation of remote sensing images based on a parameter-efficient adapter. Background Technology

[0002] Semantic segmentation of remote sensing images is a core task in Earth observation and geographic information systems, aiming to assign predefined semantic category labels to each pixel in a remote sensing image. In recent years, large-scale vision-language pre-trained models, represented by CLIP (Contrastive Language-Image Pre-training), have demonstrated excellent generalization ability in visual understanding tasks. Works such as RemoteCLIP have further extended this paradigm to the field of remote sensing.

[0003] However, when transferring pre-trained visual-language models to remote sensing semantic segmentation tasks, the following major technical problems exist: the parameter scale of pre-trained models is usually in the hundreds of millions (e.g., ViT-Large contains about 304M parameters). Full fine-tuning not only requires enormous computing resources and GPU (Graphics Processing Unit) memory overhead, but also faces the risk of catastrophic forgetting, that is, the general visual knowledge accumulated in the pre-training stage is covered by the limited labeled data of the downstream task; while the solution of simply freezing the backbone network and training only the linear classification head is unable to bridge the feature distribution differences between the pre-training data domain and the remote sensing data domain due to insufficient adaptability, resulting in extremely low segmentation accuracy. Existing technologies lack a parameter-efficient adaptation method that can both retain the general feature representation ability of the pre-trained model and achieve efficient transfer from the general visual domain to the remote sensing domain with a very small number of trainable parameters.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to achieve efficient feature transfer from the general visual domain to the remote sensing domain with a very small number of trainable parameters while maintaining the general feature representation capability of large-scale pre-trained visual models, thereby balancing segmentation accuracy and training efficiency.

[0006] The present invention adopts the following technical solution: Firstly, a remote sensing image semantic segmentation method based on a parameter-efficient adapter is provided, including: The remote sensing image to be segmented is obtained, and the remote sensing image is segmented into a sequence of non-overlapping image patches through an image patch embedding layer. A feature vector is obtained by linear projection on each image patch. A learnable classification label is added to the beginning of the feature vector, and an interpolated and adapted position code is superimposed to obtain an input label sequence. The input labeled sequence is input into a pre-trained visual backbone network with frozen parameters for layer-by-layer feature extraction. After the preset intermediate layer of the backbone network, a trainable adapter module is inserted to perform domain adaptation processing on the output features of that layer, resulting in a domain-adapted feature sequence. All pre-trained parameters of the backbone network are kept frozen during training, and only the parameters of the adapter module participate in training updates. The classification label is removed from the feature sequence after processing by all layers of the backbone network, and the image patch label is retained as a pixel-level feature representation. The pixel-level feature representation is input into a trainable segmentation head to generate the category prediction result for each image patch position. The category prediction results are then upsampled to restore the spatial dimensions of the original remote sensing image, resulting in the final pixel-level semantic segmentation result.

[0007] Preferably, the domain adaptation process of the adapter module includes: The input features are projected onto a low-dimensional hidden space through a bottleneck structure to obtain the projected features; The projected features are interacted through a multi-head cross-attention mechanism to obtain attention-enhanced features; After concatenating the projection features and the attention enhancement features in the channel dimension, the original feature dimension is restored through a multilayer perceptron to obtain the fused features. The fused features are residually concatenated with the original input features through a learnable residual scaling factor to obtain a feature sequence adapted to the longitudinal domain.

[0008] Preferably, the step of projecting the input features onto a low-dimensional hidden space through a bottleneck structure to obtain projected features specifically includes: The input features of the transformer block containing the adapter module are passed through a dimension reduction linear layer to compress the feature dimension from the original dimension D to the hidden dimension d, where d < D; The dimensionality-reduced features are sequentially normalized to stabilize the feature distribution, then nonlinear expression is introduced through the GELU nonlinear activation function, and finally overfitting is suppressed by Dropout regularization, resulting in a projected feature of dimension d.

[0009] Preferably, the step of performing feature interaction on the projected features through a multi-head cross-attention mechanism to obtain attention-enhanced features specifically includes: The labeled sequence of the projected features is reshaped into a two-dimensional spatial grid. Local region features are extracted from the two-dimensional spatial grid, and sine-cosine position coding is added to the local region features to preserve spatial position information, thus obtaining a local feature representation. The projection features are used as query vectors, and the local feature representations are used as key vectors and value vectors, respectively. The adaptive attention response of each image patch label to the local region features is calculated through multi-head cross attention, so that each image patch label aggregates semantic information from the relevant target region to obtain attention output. After performing layer normalization on the attention output, a residual connection is made with the projected features to obtain the attention enhancement features.

[0010] Preferably, the step of concatenating the projected features and the attention-enhanced features along the channel dimension, and then restoring them to the original feature dimension using a multilayer perceptron to obtain the fused features, specifically includes: The projection feature and the attention enhancement feature are concatenated along the channel dimension to obtain the concatenated feature, the dimension of which is twice that of the hidden dimension. The concatenated features are input into the first linear layer, which maps the feature dimension from twice the hidden dimension to the hidden dimension. After processing by the GELU nonlinear activation function, the features are input into the second linear layer to restore the feature dimension from the hidden dimension to the original feature dimension, resulting in the fused feature x_fused. x_fused=Linear_2(GELU(Linear_1([x_proj;x_attended])))∈R^(B×N×D); In this layer, Linear_1 is the first linear layer, Linear_2 is the second linear layer, x_proj is the projection feature, and x_attended is the attention-enhanced feature.

[0011] Preferably, the fused features are residually concatenated with the original input features through a learnable residual scaling factor to obtain a domain-adapted feature sequence x_out. The feature sequence x_out is calculated as follows: x_out = x + α × x_fused; Where x is the original input feature of the layer, x_fused is the fused feature, and α is a learnable residual scaling factor. α is initialized to a preset value in the range of 0.05 to 0.15. α is adaptively updated during training through backpropagation gradients.

[0012] Preferably, the predetermined intermediate layer of the backbone network is determined as follows: The backbone network contains L cascaded converter blocks, which are divided into shallow, intermediate, and deep regions according to depth. The adapter module is inserted only at multiple evenly spaced locations within the intermediate layer region, including after the L / 4, 3L / 8, and L / 2 converter blocks; The intermediate layer region corresponds to the key stage of the backbone network's transition from low-level texture features to mid-level semantic features. Injecting domain adaptation signals at this stage can effectively guide the migration of features from the general visual domain to the remote sensing domain.

[0013] Preferably, when training the adapter module and the segmentation head, a combined loss function L_total, which combines focus loss and Dice loss, is used: L_total = L_Focal + L_Dice; The focus loss L_Focal is defined as: L_Focal=-α_t×(1-p_t)^γ×log(p_t); The Dice loss L_Dice is defined as: L_Dice=1-(2|P∩G|+ε) / (|P|+|G|+ε); Where α_t is the preset weight coefficient for each category, p_t is the predicted probability of the segmentation head outputting the true category, γ is the focusing parameter; P and G represent the pixel sets of the predicted and true labels, respectively, ε=1.0 is a smoothing term used to prevent the denominator from being zero. The Dice loss is calculated independently for each category and then averaged to ensure that the gradient signal of the minority class is not overwhelmed by the majority class.

[0014] Secondly, a remote sensing image semantic segmentation device based on a parameter-efficient adapter is provided, the remote sensing image semantic segmentation device based on a parameter-efficient adapter includes: a processor and a memory for storing processor-executable instructions; The processor is configured to execute the remote sensing image semantic segmentation method based on a parameter-efficient adapter.

[0015] Thirdly, a non-volatile computer storage medium is provided, the computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the remote sensing image semantic segmentation method based on a parameter-efficient adapter as described in the first aspect.

[0016] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in a memory to perform the remote sensing image semantic segmentation method based on a parameter-efficient adapter as described in the first aspect.

[0017] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the remote sensing image semantic segmentation method based on a parameter-efficient adapter as described in the first aspect.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention avoids the high computational overhead and catastrophic forgetting risk associated with full fine-tuning by freezing all parameters of the pre-trained visual backbone network, effectively preserving the general visual feature representation capabilities accumulated during the pre-training phase. Simultaneously, a lightweight trainable adapter module is inserted after the preset intermediate layer of the backbone network, achieving feature transfer from the general visual domain to the remote sensing domain using only a small number of parameters, significantly reducing the computational resources and GPU memory overhead required for training. Furthermore, a segmentation head transforms the pixel-level feature representation after removing classification labels into category predictions, and then upsamples them to restore the original spatial dimensions, achieving end-to-end pixel-level semantic segmentation. This invention maintains the generalization ability of the pre-trained model while effectively bridging the feature distribution differences between the pre-training data domain and the remote sensing data domain with a minimal number of trainable parameters, balancing segmentation accuracy and parameter efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the principle of a remote sensing image semantic segmentation method based on a parameter-efficient adapter provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a remote sensing image semantic segmentation method based on a parameter-efficient adapter provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the domain adaptation process of an adapter module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an ablation experiment design provided by an embodiment of the present invention; Figure 5 This is a schematic diagram of the main experimental results on the LoveDA and Vaihingen datasets provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the category-wise IoU results of the LoveDA dataset provided in an embodiment of the present invention; Figure 7This is a schematic diagram of the category-wise IoU results of the Vaihingen dataset provided in an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the comparison results of parameter efficiency provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a remote sensing image semantic segmentation device based on a parameter-efficient adapter provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0023] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0024] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0025] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Example 1: This embodiment provides a remote sensing image semantic segmentation method based on a parameter-efficient adapter, comprising: acquiring a remote sensing image to be segmented; segmenting the remote sensing image into a sequence of non-overlapping image patches through an image patch embedding layer; linearly projecting each image patch to obtain a feature vector; adding a learnable classification label at the beginning of the feature vector and superimposing an interpolated and adapted positional code to obtain an input label sequence; inputting the input label sequence into a pre-trained visual backbone network with frozen parameters for layer-by-layer feature extraction; inserting a trainable adapter module after a preset intermediate layer of the backbone network to perform domain adaptation processing on the output features of that layer to obtain a domain-adapted feature sequence; wherein all pre-trained parameters of the backbone network are kept frozen during training, and only the parameters of the adapter module participate in training updates; removing the classification label from the feature sequence processed by all layers of the backbone network, retaining the image patch label as a pixel-level feature representation; inputting the pixel-level feature representation into a trainable segmentation head to generate a category prediction result for each image patch position; and restoring the category prediction result to the spatial size of the original remote sensing image through an upsampling operation to obtain the final pixel-level semantic segmentation result.

[0027] The core idea of ​​this method is to completely freeze all parameters of the pre-trained visual backbone network and insert trainable adapter modules only after its preset intermediate layer transformer blocks. Efficient transfer from the general visual domain to the remote sensing domain is achieved through efficient parameter fine-tuning. This method avoids the high computational cost and catastrophic forgetting risk of full fine-tuning, while achieving effective domain adaptation through a carefully designed adapter structure.

[0028] In one implementation, such as Figure 1As shown, the overall architecture of the method is as follows: First, the input 1024×1024 remote sensing image is scaled to a preset input size of 336×336. Then, the image is divided into 24×24=576 non-overlapping image patches of 14×14 size by an image patch embedding layer. A linear projection is performed on each image patch to obtain a 1024-dimensional feature vector. Simultaneously, the pre-trained positional encoding is adapted to the size using bicubic interpolation and then superimposed onto the feature vector sequence to obtain the input label sequence. Subsequently, the input label sequence is fed into a frozen pre-trained visual backbone network containing 24 transformer blocks for layer-by-layer feature processing. The architecture extracts features by inserting a lightweight trainable adapter module after each of the 6th, 9th, and 12th transformer blocks. Each adapter module employs a bottleneck structure to compress the feature dimension from 1024 to 128 and then back to 1024. It then uses a learnable residual scaling factor to perform residual connections with the original input features, adapting pre-trained general visual features to the remote sensing domain. Finally, a trainable segmentation head (containing a two-layer multilayer perceptron (1024→512→C) and bilinear upsampling operations) restores the pixel-level feature representation after removing classification labels to its original spatial dimensions, outputting the final pixel-level semantic segmentation result. In this architecture, only the parameters of the adapter modules and the segmentation head participate in training and updates (approximately 1.82M, accounting for 0.6% of the total parameters), while all pre-trained parameters of the backbone network remain frozen (approximately 304M, accounting for 99.4%), thus achieving efficient feature transfer from the general visual domain to the remote sensing domain with extremely low training overhead.

[0029] In one implementation, such as Figure 2 As shown, the remote sensing image semantic segmentation method based on a parameter-efficient adapter described in this embodiment specifically includes: Step 101: Obtain the remote sensing image to be segmented, segment the remote sensing image into a sequence of non-overlapping image patches through an image patch embedding layer, and obtain a feature vector by linear projection on each image patch. Add a learnable classification label at the beginning of the feature vector and superimpose the interpolated and adapted positional code to obtain the input label sequence.

[0030] Specifically, this embodiment uses RemoteCLIP ViT-L-14 as a pre-trained visual backbone network as an example. This model contains 24 Transformer Blocks, each with a hidden dimension of 1024, 16 attention heads, and a feedforward network dimension of 4096. The image patch size of ViT-L-14 is 14×14 pixels.

[0031] First, the remote sensing image to be segmented is acquired. This image can be taken by high-resolution remote sensing satellites (such as the WorldView and GaoFen series) or by a UAV remote sensing platform. Typical images are 1024×1024 pixels or larger. In this embodiment, the input remote sensing image is scaled to a preset input size of 336×336 pixels to fit the input requirements of the backbone network.

[0032] The remote sensing image is segmented into a non-overlapping sequence of image patches using a patch embedding layer. Specifically, the 336×336 pixel input image is segmented into non-overlapping patches of size 14×14, resulting in 24×24=576 image patches. Each image patch is linearly projected into a 1024-dimensional feature vector; that is, each 14×14×3 image patch is mapped to a 1024-dimensional feature vector through a linear projection layer.

[0033] A learnable classification token ([CLS] token) is added to the beginning of the feature vector sequence. This token is a trainable 1024-dimensional vector used to aggregate global semantic information during backbone network processing. After adding the classification token, the total length of the transformer sequence is 576 + 1 = 577.

[0034] Finally, the interpolated and adapted positional codes are superimposed to obtain the input label sequence. Since the input size during backbone network pre-training is 224×224 (corresponding to 16×16=256 image patches), while the input size in this embodiment is 336×336 (corresponding to 24×24=576 image patches), interpolation adaptation of the pre-trained positional codes is necessary. Specifically, the image patch portion of the pre-trained positional codes is reshaped from a 16×16 two-dimensional grid and then enlarged to a target size of 24×24 using bicubic interpolation, while the positional codes of the classification labels remain unchanged. The interpolated positional codes are then superimposed onto the feature vector sequence to obtain the final input label sequence.

[0035] Step 102: The input labeled sequence is input into the pre-trained visual backbone network with frozen parameters for layer-by-layer feature extraction. After the preset intermediate layer of the backbone network, a trainable adapter module is inserted to perform domain adaptation processing on the output features of the layer, so as to obtain the domain-adapted feature sequence. All pre-trained parameters of the backbone network are kept frozen during the training process, and only the parameters of the adapter module participate in the training update.

[0036] The backbone network comprises L cascaded transformer blocks, which are divided into shallow, intermediate, and deep regions. The adapter modules are inserted only at evenly spaced intervals within the intermediate region, including after the L / 4, 3L / 8, and L / 2 transformer blocks. The intermediate region corresponds to the critical stage in the backbone network's transition from low-level texture features to mid-level semantic features. Injecting domain adaptation signals at this stage can effectively guide the migration of features from the general visual domain to the remote sensing domain.

[0037] Specifically, the RemoteCLIP ViT-L-14 backbone network used in this embodiment contains 24 cascaded transformer blocks (i.e., L=24), and its pre-trained weights come from the model checkpoints released by RemoteCLIP. During training, all parameters of the backbone network (including patch embedding layers, 24 transformer blocks, and LayerNorm layers) are completely frozen and do not participate in gradient updates. The parameter size of RemoteCLIP ViT-L-14 is approximately 304M (millions). The complete freezing strategy avoids the problem that full fine-tuning not only requires huge computational resources and GPU memory overhead, but also avoids the risk of catastrophic forgetting, that is, the general visual knowledge accumulated in the pre-training stage is covered by the limited labeled data of downstream tasks.

[0038] The 24 converter blocks are divided into shallow regions (layers 1-5), intermediate regions (layers 6-12), and deep regions (layers 13-24). The adapter modules are inserted only at evenly spaced locations within the intermediate region, including after converter blocks L / 4=6, L / 8=9, and L / 2=12. The design motivation for choosing these three locations is as follows: (1) The shallow regions (layers 1-5) are mainly responsible for extracting low-level visual features (such as edges and textures). These features have strong universality and do not require domain adaptation.

[0039] (2) The intermediate layer region (layers 6-12) corresponds to the key stage of the backbone network transitioning from low-level texture features to intermediate-level semantic features. It is crucial for encoding the texture and shape features of remotely sensed objects. Injecting domain adaptation signals at this stage can effectively guide the features to migrate from the general visual domain to the remote sensing domain.

[0040] (3) The deep regions (layers 13-24) are responsible for advanced semantic reasoning and global context integration. Keeping them frozen can preserve the semantic alignment ability of the pre-trained model and avoid overfitting to the limited samples of the downstream dataset.

[0041] In one implementation, the input labeled sequence is processed sequentially through 24 transformer blocks. After the sequence passes through the 6th transformer block, its output features are input to the first adapter module for domain adaptation; after the 9th transformer block, its output features are input to the second adapter module for domain adaptation; and after the 12th transformer block, its output features are input to the third adapter module for domain adaptation. The output of each adapter module is then passed as the input to the next transformer block.

[0042] In summary, the trainable components include only three adapter modules (approximately 1.59M parameters) and a segmentation head (approximately 0.23M parameters). The total trainable parameters are approximately 1.82M, representing only 0.6% of the total model parameters (approximately 304M). This strategy significantly reduces training memory requirements and computational overhead, enabling efficient training on a single consumer-grade GPU (e.g., an NVIDIA GeForce RTX 4090 with 24GB of VRAM).

[0043] In one implementation, the domain adaptation process of the adapter module is as follows: Figure 3 As shown, the domain adaptation process of the adapter module includes: Step 1021: Project the input features into the low-dimensional hidden space through the bottleneck structure to obtain the projected features.

[0044] Specifically, the input features x∈R^(B×N×D) (where B is the batch size, N=577 is the sequence length, and D=1024 is the feature dimension) of the transformer block containing the adapter module are compressed from the original dimension D=1024 to the hidden dimension d=128 through a dimensionality reduction linear layer, where d<D. This bottleneck structure reduces the number of parameters from D×D≈1M to D×d+d×D≈0.26M, achieving efficient utilization of parameters.

[0045] The dimensionality-reduced features are sequentially subjected to LayerNorm to stabilize the feature distribution, then nonlinear expressive power is introduced through the GELU nonlinear activation function, and finally overfitting is suppressed by Dropout regularization (Dropout rate of 0.1), resulting in a projected feature x_proj with dimension d=128. The calculation process is as follows: x_proj=Dropout(GELU(LayerNorm(Linear(x))))∈R^(B×N×d) Among them, LayerNorm ensures the stability of the feature distribution, the GELU activation function introduces the necessary non-linear expression flexibility, and Dropout(0.1) provides regularization effect, effectively preventing the adapter from overfitting on limited training data.

[0046] Step 1022: The projected features are interacted through a multi-head cross-attention mechanism to obtain attention-enhanced features.

[0047] Specifically, the label sequence of the projection features is reshaped into a two-dimensional spatial grid. Since the 576 image patch labels correspond to the 24×24 spatial grid layout of the original image, the image patch label portion of the projection features (after removing the classification labels) is reshaped from a one-dimensional sequence into a 24×24 two-dimensional spatial grid.

[0048] Local region features are extracted from the two-dimensional spatial grid. In this embodiment, image patch markers at corresponding locations are extracted from the two-dimensional grid as local region features based on the spatial extent of the target region. When there is no explicit candidate bounding box input, this step degenerates into a direct transfer of global features, i.e., all image patch markers are used as local features. Sine-cosine positional encoding is added to the local region features to preserve spatial location information, resulting in a local feature representation. Sine-cosine positional encoding provides a unique location identifier for different spatial locations, enabling subsequent attention calculations to distinguish features from different spatial locations.

[0049] The projected features are used as the query vector, and the local feature representations are used as the key and value vectors, respectively. Multi-head cross-attention is used to calculate the adaptive attention response of each image patch label to the local region features. This embodiment uses four attention heads, each with a dimension of d / 4 = 32. The multi-head cross-attention mechanism enables each image patch label to adaptively aggregate semantic information from relevant target regions to obtain the attention output. This cross-attention mechanism allows for sufficient information interaction between global patch features and local target features, enhancing the model's ability to perceive remotely sensed ground objects.

[0050] Finally, after layer normalization, the attention output is residually connected to the projected features to obtain the attention-enhanced feature x_attended. The residual connection ensures that even if the attention module fails to provide effective information, the original projected features can still be fully transmitted, guaranteeing the stability of training.

[0051] Step 1023: After concatenating the projection features and the attention enhancement features in the channel dimension, restore them to the original feature dimension through a multilayer perceptron to obtain the fused features.

[0052] Specifically, the projection feature x_proj and the attention enhancement feature x_attended are concatenated along the channel dimension to obtain the concatenated feature. The dimension of the concatenated feature is twice that of the hidden dimension, i.e., dim=128×2=256.

[0053] The concatenated features are input into the first linear layer (Linear_1), mapping the feature dimension from twice the hidden dimension (256) to the hidden dimension (128). After processing by the GELU nonlinear activation function, they are input into the second linear layer (Linear_2) to restore the feature dimension from the hidden dimension (128) to the original feature dimension (D=1024), resulting in the fused feature x_fused. Its calculation formula is: x_fused=Linear_2(GELU(Linear_1([x_proj;x_attended])))∈R^(B×N×D); In this architecture, Linear_1 is the first linear layer (input dimension 256, output dimension 128), Linear_2 is the second linear layer (input dimension 128, output dimension 1024), x_proj is the projection feature, and x_attended is the attention-enhanced feature. Through the fusion operation of these two MLP layers, the complementary information in the projection and attention-enhanced features is effectively integrated and restored to the original feature dimensions for subsequent fusion with the backbone network features.

[0054] Step 1024: The fused features are residually concatenated with the original input features through a learnable residual scaling factor to obtain a feature sequence adapted to the longitudinal domain.

[0055] Specifically, the fused feature x_fused is residually concatenated with the original input feature x through a learnable residual scaling factor α, resulting in a domain-adapted feature sequence x_out. Its calculation method is as follows: x_out = x + α × x_fused; Here, x represents the original input feature of the layer, x_fused represents the fused feature, and α is a learnable residual scaling factor, initialized to a preset value within the range of 0.05 to 0.15. α is adaptively updated during training via backpropagation gradients. The small initial value of α ensures that the adapter minimizes perturbation to the pre-trained features in the early stages of training. As training progresses, α adaptively adjusts, gradually increasing the adaptation strength. This conservative-then-adaptive strategy helps stabilize the training process, avoiding the destruction of semantic information of the pre-trained features in the early stages of training and ensuring the stable preservation of pre-trained knowledge.

[0056] Each adapter module has approximately 0.53M parameters, and the three adapters together have approximately 1.59M parameters. The bottleneck design of the adapters (1024→128→1024) forces the model to transmit features through a low-dimensional information bottleneck. This process naturally filters out noise and domain-specific information, retaining the most discriminative feature components, while also providing a regularization effect to prevent overfitting on limited remote sensing labeled data. Step 103: Remove the classification label from the feature sequence after processing by all layers of the backbone network, retain the image patch label as pixel-level feature representation, and input the pixel-level feature representation into the trainable segmentation head to generate the category prediction result for each image patch position.

[0057] Specifically, after processing by all 24 transformer blocks and the three inserted adapter modules, a feature sequence of length 577 is obtained. The classification token ([CLS] token) at the beginning of the sequence is discarded, and 576 image patch tokens are retained as pixel-level feature representations. These 576 image patch tokens correspond to the 24×24 spatial grid layout of the original image, and each token encodes the semantic information of the corresponding image patch region.

[0058] The pixel-level feature representations are input into a trainable segmentation head to generate class predictions for each image patch location. The segmentation head employs a two-layer MLP structure, mapping the 1024-dimensional features labeled for each image patch to predictions in the number of class dimensions (logits). The specific structure of the segmentation head is as follows: logits=Dropout(0.1)(Linear_2(GELU(LayerNorm(Linear_1(x))))); Where Linear_1: 1024→512, Linear_2: 512→C (C is the number of categories, LoveDA is 7, Vaihingen is 6). The logits generated by the segmentation head are 24×24 in size (i.e., the patch grid size), which need to be upsampled to the original input size (336×336) by bilinear interpolation.

[0059] Step 104: The category prediction results are upsampled to restore the spatial size of the original remote sensing image to obtain the final pixel-level semantic segmentation results.

[0060] Specifically, the spatial size of the category prediction results (logits) generated by the segmentation head is 24×24 (i.e., the image patch grid size), which needs to be restored to the spatial size of the original remote sensing image through upsampling. This embodiment uses bilinear interpolation to upsample the 24×24 prediction results to an input image size of 336×336. Alternatively, two layers of transposed convolution (ConvTranspose2d, kernel_size=2, stride=2) can be included as an optional upsampling module to progressively enlarge the 24×24 feature map to 96×96, and then complete the final upsampling through bilinear interpolation.

[0061] When the size of the original remote sensing image is greater than 336×336 (e.g., 1024×1024), the upsampled prediction results are further restored to the spatial size of the original remote sensing image through bilinear interpolation to obtain the final pixel-level semantic segmentation result. The final output is a segmentation map of the same size as the original remote sensing image, where each pixel is assigned a predefined semantic category label, achieving end-to-end pixel-level semantic segmentation.

[0062] The following training strategy is used when training the adapter module and the segmentation head: In one implementation, this embodiment addresses the severe class imbalance prevalent in remote sensing datasets (e.g., the Barren class accounts for only about 4.25% of the total pixels in the LoveDA dataset). When training the adapter module and the segmentation head, a combined loss function L_total, combining focal loss and Dice loss, is used. L_total = L_Focal + L_Dice; The core idea of ​​focus loss L_Focal is to reduce the loss contribution of easily separable samples by using a modulation factor, so that the training process pays more attention to difficult samples. Focus loss L_Focal is defined as: L_Focal=-α_t×(1-p_t)^γ×log(p_t); Where p_t is the predicted probability of the segmentation head outputting the true class, and γ is the focusing parameter, which controls the weight difference between easy and difficult samples. In this embodiment, γ=2.0. With γ=2.0, for easy samples with a predicted probability p_t=0.9, the focus weight is (1-0.9)^2=0.01, which is almost negligible; while for difficult samples with p_t=0.3, the focus weight is (1-0.3)^2=0.49, which is 49 times that of easy samples. This makes the gradient signal during training mainly come from difficult pixels, effectively alleviating the dominant role of the majority class in the gradient.

[0063] α_t represents the preset weight coefficients for each category. For the seven categories of the LoveDA dataset [Background, Building, Road, Water, Barren, Forest, Agriculture], the weights are set to [0.5, 1.0, 1.5, 0.8, 8.0, 2.5, 5.0]. Barren and Agriculture received the highest weights of 8.0 and 5.0 respectively, reflecting their extremely low pixel proportion in the training set, which causes the model to impose a greater penalty on prediction errors for these categories during optimization.

[0064] The Dice loss L_Dice directly optimizes the Dice coefficient between the predicted and the true labels. The Dice loss L_Dice is defined as follows: L_Dice=1-(2|P∩G|+ε) / (|P|+|G|+ε); Where P and G represent the pixel sets of the predicted and true labels, respectively, ε=1.0 is a smoothing term used to prevent the denominator from being zero. The Dice loss is calculated independently for each class and then averaged to ensure that the gradient signal of the minority class is not overwhelmed by the majority class.

[0065] Dice loss is inherently unaffected by differences in class sample size because it is calculated independently for each class and then averaged, ensuring that the gradient signals of the minority classes are not overwhelmed by the majority classes. Focus loss addresses pixel-by-pixel optimization of hard-to-distinguish pixels, while Dice loss focuses on optimizing the overlap of the overall region. The combination of these two methods can synergistically alleviate class imbalance from different perspectives, achieving complementary optimization effects.

[0066] In one implementation, considering the high resolution and variable target scale of remote sensing images, this embodiment adopts the following data augmentation strategy: Random ResizedCrop: A region is cropped from the original image at a random scaling ratio of 0.5 to 1.0 and then scaled to the model input size (336×336). Through the local magnification effect, small objects that originally occupy only a few pixels in the global view (such as wasteland areas or small buildings) may occupy a higher proportion of pixels in the cropped local view, thereby enhancing the model's learning signal for these objects.

[0067] Color Jitter: Randomly perturbs brightness (brightness=0.3), contrast (contrast=0.3), and saturation (saturation=0.3) to simulate spectral differences caused by different lighting conditions and seasonal changes, thereby enhancing the model's robustness to lighting changes.

[0068] Geometric transformations include random horizontal flips (probability p=0.5) and random vertical flips (probability p=0.5), which take advantage of the orientation-independent nature of remote sensing images (rotation of images from a top-down view does not change their semantics) to increase the diversity of training samples.

[0069] Furthermore, the AdamW optimizer (β1=0.9, β2=0.999) was used, with an initial learning rate of 1×10⁻⁶. -4 The weight decay coefficient is 1×10 -4 The learning rate scheduling uses a cosine annealing strategy, with a minimum learning rate of 1×10⁻⁶. -6 The total training duration is 50 epochs. The training batch size is 2, and the evaluation batch size is 4. Mixed precision training (AMP) is used to accelerate training and reduce GPU memory usage.

[0070] In summary, this embodiment avoids the high computational overhead and catastrophic forgetting risk associated with full fine-tuning by freezing all parameters of the pre-trained visual backbone network, effectively preserving the general visual feature representation capabilities accumulated during the pre-training phase. Simultaneously, a lightweight trainable adapter module is inserted after the preset intermediate layer of the backbone network, achieving feature transfer from the general visual domain to the remote sensing domain using only a small number of parameters from the adapter module, significantly reducing the computational resources and GPU memory overhead required for training. Furthermore, the segmentation head transforms the pixel-level feature representation after removing classification labels into category predictions, and then upsamples them to restore the original spatial dimensions, achieving end-to-end pixel-level semantic segmentation. This invention maintains the generalization ability of the pre-trained model while effectively bridging the feature distribution differences between the pre-training data domain and the remote sensing data domain with a very small number of trainable parameters, balancing segmentation accuracy and parameter efficiency.

[0071] Example 2: To verify the effectiveness of the method described in Example 1, this example conducted a systematic experimental evaluation on two publicly available remote sensing semantic segmentation benchmark datasets, LoveDA and ISPRS Vaihingen.

[0072] LoveDA (Land-cover Domain Adaptive Semantic Segmentation) is a large-scale remote sensing semantic segmentation benchmark dataset released by Wuhan University. This dataset covers high-resolution Google Earth imagery (spatial resolution approximately 30cm) from multiple cities in China, including both urban and rural scenes. It defines seven semantic categories: Background, Building, Road, Water, Barren, Forest, and Agriculture. The dataset contains 5987 1024×1024 pixel images, divided into a training set (2520 images), a validation set (1036 images), and a test set (2431 images) in a 5:2:3 ratio. A significant characteristic of the LoveDA dataset is severe class imbalance. In urban scenes, Building and Road occupy a large number of pixels, while Agriculture is almost non-existent; in rural scenes, Forest and Agriculture dominate, while Barren is extremely rare. Overall, the Barren category accounts for only about 4.25% of the total pixels, making it one of the most challenging categories. Furthermore, the domain differences between urban and rural scenes also present additional challenges to the model's generalization ability.

[0073] The ISPRS Vaihingen 2D semantic annotation dataset, provided by the German Society for Photogrammetry and Remote Sensing (ISPRS), is one of the most classic benchmarks in the field of remote sensing image analysis. Covering an area of ​​approximately 0.5 square kilometers in Vaihingen, Germany, with a spatial resolution of up to 9 cm, it includes a Digital Surface Model (DSM) and TrueOrthophotos. The dataset defines six semantic categories: Impervious Surfaces (including roads and squares), Buildings, Low Vegetation, Trees, Cars, and Clutter. The key features of the Vaihingen dataset are its extremely high spatial resolution (9 cm, approximately three times that of LoveDA) and the challenge of identifying small targets. A typical car target is only about 10 × 20 pixels, accounting for less than 1% of the total pixels in the dataset, making it a classic test case for small target segmentation. Since Vaihingen was not involved in any pre-training process, it was used as a zero-shot transfer testing platform to evaluate the model's generalization ability.

[0074] For the selection of the backbone network and pre-trained weights, this embodiment uses RemoteCLIP ViT-L-14 as the backbone network, and its pre-trained weights are derived from the model checkpoints released by RemoteCLIP. Positional encoding is adapted from the original 224×224 input size to 336×336 using bicubic interpolation. All backbone network parameters are frozen during training.

[0075] Optimizer and learning rate scheduling. The AdamW optimizer (β1=0.9, β2=0.999) was used, with an initial learning rate of 1×10⁻⁶. -4 The weight decay coefficient is 1×10 -4 The learning rate scheduling uses a cosine annealing strategy, with a minimum learning rate of 1×10⁻⁶. -6 The total training period is 50 epochs.

[0076] The training batch size was 2 (limited by GPU memory), and the evaluation batch size was 4. Mixed precision training was used to accelerate training and reduce memory usage. All experiments were performed on a single NVIDIA GeForce RTX 4090 (24GB VRAM) GPU, using Ubuntu 22.04 as the operating system and PyTorch 2.0+ as the deep learning framework.

[0077] The following standard evaluation indicators are used in this embodiment: (1) Intersection over Union (IoU): IoU = TP / (TP + FP + FN), where TP, FP, and FN are the number of true positives, false positives, and false negatives, respectively; (2) Mean Intersection over Union (mIoU): the arithmetic mean of IoU across all classes; (3) Precision: Precision = TP / (TP + FP); (4) Recall: Recall = TP / (TP + FN); (5) F1 score: F1 = 2 × Precision × Recall / (Precision + Recall). Unless otherwise specified, all mIoU report values ​​do not include the background class (Background / Clutter).

[0078] In order to systematically evaluate the contribution of each component, such as Figure 4 As shown, this embodiment designed four progressive ablation experiments.

[0079] Ablation Experiment 1 (Baseline / Linear Probe): Only the segmentation head was trained, without inserting adapters, and without using special data augmentation or loss functions. This baseline represents the lower limit of performance for using the frozen backbone directly for segmentation. The trainable parameters are only about 0.53M.

[0080] Ablation Experiment 2 (+Adapter): An adapter was inserted after layers 6, 9, and 12, but without RRCrop enhancement and Focal Loss. The loss function was standard CE+Dice. This experiment isolated the contribution of the adapter itself. Approximately 1.82M trainable parameters were obtained.

[0081] Ablation Experiment 3 (+RRCrop): RandomResizedCrop data augmentation was added to ablation experiment 2. This experiment verified the marginal contribution of data augmentation strategies on the adapter basis.

[0082] Ablation Experiment 4 (Full Ours): Full configuration, including adapter, RRCrop, and Focal Loss + class weights. This experiment evaluates the final performance of all components working together.

[0083] The main experimental results include: like Figure 5 The table summarizes the mIoU results of four ablation experiments on the LoveDA and Vaihingen datasets (excluding the background class). Among them: (1) The decisive role of the adapter. From ablation experiment 1 to ablation experiment 2, with only an increase of about 1.29M parameters (from 0.53M to 1.82M), mIoU jumped from 0.2074 to 0.4304 (+22.3%) on LoveDA and from 0.0281 to 0.6643 (+63.6%) on Vaihingen. This huge improvement demonstrates that the pre-trained RemoteCLIP model has encoded rich visual knowledge, which can be effectively transferred to remote sensing segmentation tasks with only a very small number of adaptation parameters. The adapter helps the model convert the semantic understanding ability of natural images into discriminative features of remote sensing objects by injecting domain adaptation signals in the intermediate layer.

[0084] (2) Dataset dependency of RRCrop. From ablation experiment 2 to ablation experiment 3, the introduction of RRCrop had almost no positive impact on LoveDA (0.4304→0.4285, a slight decrease of 0.19%), but led to a more significant performance degradation on Vaihingen (0.6643→0.6523, a decrease of 1.8%). This counterintuitive result reveals the limitations of local enhancement strategies: when the original image already has sufficiently high resolution and target visibility, random cropping may truncate the integrity of the target, thereby impairing the model's global perception ability.

[0085] (3) Targeted improvement of FocalLoss + class weights. From ablation experiment 3 to ablation experiment 4 (FullOurs), FocalLoss and class weights improved mIoU on LoveDA from 0.4285 to 0.4368 (+0.83%), reaching the highest value among all configurations. This improvement mainly came from a few difficult classes, indicating that the reweighting strategy of the loss function effectively alleviated the class imbalance problem.

[0086] The LoveDA dataset results by category include: like Figure 6 The image shows the class-by-class IoU results for each experiment on the LoveDA dataset across the seven categories. Figure 6 The following detailed analysis can be derived from this: (1) Agriculture (Agricultural land). This category best reflects the effect of Focal Loss + category weights. From 0.4245 in ablation experiment 2 to 0.4946 in ablation experiment 4, the IoU increased by 7.0% (absolute value), the largest increase among all categories. This is mainly due to the higher loss weight (5.0) given to the Agriculture category, which makes the model pay more attention to the prediction errors of this category during training. In terms of Precision and Recall, Full Ours' Agriculture Precision is 0.6829 and Recall is 0.6419, which are significantly improved compared to ablation experiment 2 (Precision 0.6210, Recall 0.5985), indicating that Focal Loss not only improves the detection rate of this category, but also reduces false detections.

[0087] (2) Barren. Barren is the category with the lowest pixel percentage (4.25%) in LoveDA and is also the most challenging category. The baseline Barren IoU is only 0.0341 (almost completely unrecognizable), which is improved to 0.2739 by the adapter (ablation experiment 2) and further improved to 0.2996 by Full Ours. Although the absolute IoU is still low, considering the extreme imbalance of this category, this result has demonstrated the effectiveness of the adapter and loss function in jointly addressing class imbalance. The Barren Precision of Full Ours is 0.4415 and the Recall is 0.4824. The higher Recall than Precision indicates that the model tends to predict some boundary regions as Barren, which is a reasonable trade-off in pursuing coverage under imbalance conditions.

[0088] (3) Water. Water achieved the highest IoU across all configurations (Baseline 0.3607, FullOurs 0.5960), attributed to the unique spectral characteristics of water bodies in remote sensing images (strong absorption in the near-infrared band), resulting in clear distinctions between water and other land features. FullOurs showed the highest Water Precision (0.7551) and Recall (0.7388) among all categories, reflecting the high separability of water body features.

[0089] (4) Building. The IoU for Building improved from 0.2698 at the baseline to 0.4740 at Full Ours. Notably, the Building Recall (0.7477) at Full Ours was significantly higher than that at Precision (0.5643), indicating that the model could detect building areas well, but there were some false positives—some non-building areas (such as the edges of impermeable surfaces) were incorrectly classified as buildings. This phenomenon is related to the visual similarity between Building and Impervious Surfaces.

[0090] (5) Road. The IoU for the Road category steadily increased from 0.2102 at the baseline to 0.4098 at the full range. The linear structural features and regular geometry of roads make them relatively easy targets for adapters to learn. However, the confusion between the boundaries between roads and buildings remains a major source of error.

[0091] The category-wise results of the Vaihingen dataset include: like Figure 7The figure shows the class-by-class IoU results for each experiment on the Vaihingen dataset across six categories. The Vaihingen results reveal the following important findings: (1) The complete failure of the baseline and the rescue effect of the adapter. The performance of the baseline on Vaihingen degraded to the level of random guessing (mIoU=0.0281), and the model predicted all pixels as Impervious. The root cause of this phenomenon is the huge difference between LoveDA and Vaihingen in terms of class definition, data distribution and resolution - the linear classifier head trained on LoveDA cannot generalize to completely different data domains. The adapter, by performing domain adaptation at the feature level, successfully improved the performance to 0.6643 (ablation experiment 2), proving that feature-level domain adaptation is far superior to decision-level adaptation.

[0092] (2) Excellent segmentation of Car. Car is the most challenging small object category in Vaihingen, with a typical size of only about 10×20 pixels (at 9cm resolution). Adapter-only (ablation experiment 2) achieved an IoU of up to 0.8831 on Car (Precision 0.9345, Recall 0.9436), which is remarkable—high-precision small object segmentation can be achieved even without a dedicated candidate box generation module (such as YOLO / SAM), using only the intermediate layer adapter. This shows that the adapter enhances the domain adaptation capability of the intermediate layer, enabling the model to better utilize the global context modeling capability of the pre-trained ViT to locate and segment small objects.

[0093] (3) Negative impact of RRCrop on small targets. From ablation experiment 2 to ablation experiment 3, the IoU of Car decreased from 0.8831 to 0.8349 (-4.8%), which is the most direct evidence of the negative effect of RRCrop on Vaihingen. Analysis suggests that the image resolution of Vaihingen is extremely high (9cm), and the car target already has sufficient pixel representation in the global view (about 10×20 pixels). Random cropping may cause some car targets to be truncated or completely excluded from the training samples, which weakens the model's learning signal. This finding has important implications for the design of remote sensing data augmentation strategies: augmentation strategies need to be customized according to the resolution characteristics of the dataset and the target scale distribution. Blindly applying augmentation methods from the natural image domain may be counterproductive.

[0094] (4) The continued difficulty of Clutter. Clutter achieved the lowest IoU across all configurations (highest 0.3515, ablation experiment 2). The Clutter category includes various features that are difficult to classify into other categories (such as fences, swimming pools, and small sheds), with ambiguous semantic definitions and diverse visual appearances, making it difficult for the model to learn consistent discriminative features. Improvements in this category may require more refined semantic annotations or additional contextual information.

[0095] Furthermore, the adapter module is the decisive factor in the performance improvement of the method in this embodiment, contributing far more than other components. On LoveDA, the adapter alone contributed approximately 22.3% of the mIoU improvement (0.2074→0.4304), while the combined contribution of RRCrop and Focal Loss was only about 0.64% (0.4304→0.4368). On Vaihingen, the adapter's contribution is even more significant, contributing approximately 63.6% of the mIoU improvement alone (0.0281→0.6643).

[0096] The effectiveness of an adapter can be understood from the following perspectives: (1) The bridging role of domain adaptation. The ViT backbone of pre-trained RemoteCLIP has been pre-trained for contrastive learning on large-scale remote sensing data. Its intermediate layer features have encoded the basic structural information of the remote sensing scene (such as the texture patterns and spatial distribution patterns of ground objects). The adapter provides the last-mile adaptation from general remote sensing features to specific segmentation tasks by injecting a small number of trainable parameters into the intermediate layer. The number of parameters required for this process is much smaller than that required for training from scratch or full fine-tuning.

[0097] (2) The regularization effect of the bottleneck structure. The bottleneck design of the adapter (1024→128→1024) forces the model to pass features through the low-dimensional information bottleneck. This process naturally filters out noise and domain-specific information, and retains the most discriminative feature components.

[0098] (3) Knowledge protection of residual connections. The initial value of the learnable scaling factor α is 0.1, which ensures that the adapter has little perturbation to the pre-trained features in the early stage of training. As training progresses, the adaptation strength is gradually increased. This progressive adaptation strategy effectively avoids destroying the pre-trained knowledge.

[0099] The experimental results of RRCrop reveal the complex behavior of data augmentation strategies under different dataset characteristics. We observed that RRCrop did not bring positive effects on either dataset, and even produced a significant negative impact (-1.8% mIoU) on Vaihingen. The following is an in-depth analysis of this phenomenon: (1) Resolution dependence. The core assumption of RRCrop is that there is information loss in the original image due to resolution compression, which can be recovered by local magnification. However, when the resolution of the original image is already high enough (such as Vaihingen's 9cm), and the target already has sufficient pixel representation in the global view, further magnification not only fails to provide additional information, but also destroys the global context - the model cannot see the target and its surrounding environment at the same time, resulting in the loss of spatial relationship information.

[0100] (2) Target truncation effect. Random pruning inevitably truncates targets located near the pruning boundary. For small targets (such as cars), even with a low pruning probability, a single pruning may result in the complete loss of the target in the training samples. In Vaihingen's training set, the number of car targets was already limited, and the reduction in samples caused by pruning further exacerbated this problem.

[0101] (3) Category-specific benefits on LoveDA. Although RRCrop has almost no impact on the overall mIoU on LoveDA, it has an indirect positive impact on the Agriculture category when combined with Focal Loss (ablation experiment 4). The analysis suggests that the local amplification of agricultural areas by RRCrop allows the model to learn more refined crop texture features, an effect that is amplified under the guidance of Focal Loss.

[0102] The combination of Focal Loss and class weights showed a targeted improvement on LoveDA, primarily in the Agriculture (+7.0%) and Barren (+2.5%) hard cases categories. The following is a quantitative analysis of this effect: (1) FocalLoss's hard sample focusing effect. With γ=2.0, for easily distinguishable samples with a prediction probability p_t=0.9, the Focal weight is (1-0.9)^2=0.01, which is almost negligible; while for hard samples with p_t=0.3, the Focal weight is (1-0.3)^2=0.49, which is 49 times that of easily distinguishable samples. This makes the gradient signal during training mainly come from hard-to-distinguish pixels, effectively alleviating the dominant role of the majority class in the gradient.

[0103] (2) Priority guidance of class weights. In LoveDA's class weights [0.5, 1.0, 1.5, 0.8, 8.0, 2.5, 5.0], Barren (8.0) and Agriculture (5.0) received significantly higher weights than other classes. This design imposes a greater penalty on the model's prediction errors for these classes during optimization, forcing the model to invest more learning power in these classes. Experimental results validated the effectiveness of this design: the IoU of Agriculture increased from 0.4245 in ablation experiment 2 to 0.4946 in ablation experiment 4, and that of Barren increased from 0.2739 to 0.2996.

[0104] (3) Cost of Simple Classes. It is worth noting that while FocalLoss+ class weights improve the performance of difficult classes, they also lead to a slight degradation of some simple classes. For example, the IoU of Water slightly increased from 0.5804 in ablation experiment 2 to 0.5873 in ablation experiment 3, but further increased to 0.5960 in ablation experiment 4, indicating that Focal Loss' focus on difficult samples did not harm the learning of simple classes. However, the IoU of Building improved from 0.4438 in ablation experiment 3 to 0.4740 in ablation experiment 4 (+3.0%). This unexpected positive effect may be due to the enhanced attention Focal Loss pays to the boundary pixels of Building.

[0105] Experimental results on the Vaihingen dataset provide strong evidence for the model's generalization ability. It's important to emphasize that Vaihingen was not involved in any pre-training or training process—the model was trained solely on LoveDA and then evaluated directly on Vaihingen. Therefore, the performance on Vaihingen directly reflects the model's zero-shot transfer ability from LoveDA to Vaihingen.

[0106] The baseline model achieved an mIoU of only 0.0281 on Vaihingen, degenerating to predict all pixels as the Impervious class. The root cause of this failure was that the Linear Probe's classification head, learned on the LoveDA class definition and feature distribution, failed to generalize to Vaihingen's different class systems. In contrast, the adapter, through domain adaptation at the feature level, enabled the model to leverage the generality of the RemoteCLIP pre-trained features, achieving an mIoU of 0.6643 on Vaihingen.

[0107] Of particular note is the segmentation performance for the Car category. On Vaihingen, the adapter-only approach (ablation experiment 2) achieved a Car IoU of 0.8831 (Precision 0.9345, Recall 0.9436), a result that approaches or even surpasses some complex methods specifically designed for small objects. This demonstrates that, through appropriate domain adaptation, the global context modeling capability of pre-trained ViT can be effectively activated, thereby achieving high-precision small object segmentation without the need for explicit small object detection modules (such as YOLO and SAM).

[0108] However, generalization ability also has limitations. Full Ours (ablation experiment 4) showed a slightly lower mIoU (0.6498) on Vaihingen than Adapter-only (ablation experiment 2, 0.6643). This is because the Focal Loss and class weights designed for class imbalance in LoveDA are not suitable for the class distribution of Vaihingen. This finding indicates that the adjustment of the loss function needs to be customized for the specific distribution of the target dataset, and direct transfer may lead to suboptimal results. Exploring adaptive loss function strategies will be a valuable direction for future work.

[0109] The parameter efficiency of the method in this embodiment is one of its core advantages. For example... Figure 8 The diagram shows a comparison of parameter efficiency under different configurations. From Figure 8 As can be seen, the RemoteCLIP-Adapter achieves a more than 2x improvement in mIoU (0.2074 → 0.4368) with only 0.6% more trainable parameters (1.82M vs. 304M total parameters). Compared to full fine-tuning, the adapter's parameter count is reduced by approximately 167 times, while avoiding the memory bottleneck (full fine-tuning of ViT-Large typically requires more than 40GB of GPU memory, exceeding the 24GB capacity of a single RTX4090) and the risk of catastrophic forgetting that come with full fine-tuning. This high parameter efficiency makes the proposed method particularly suitable for deployment in computationally limited scenarios.

[0110] In summary, this embodiment avoids the high computational overhead and catastrophic forgetting risk associated with full fine-tuning by freezing all parameters of the pre-trained visual backbone network, effectively preserving the general visual feature representation capabilities accumulated during the pre-training phase. Simultaneously, a lightweight trainable adapter module is inserted after the preset intermediate layer of the backbone network, achieving feature transfer from the general visual domain to the remote sensing domain using only a small number of parameters from the adapter module, significantly reducing the computational resources and GPU memory overhead required for training. Furthermore, the segmentation head transforms the pixel-level feature representation after removing classification labels into category predictions, and then upsamples them to restore the original spatial dimensions, achieving end-to-end pixel-level semantic segmentation. This invention maintains the generalization ability of the pre-trained model while effectively bridging the feature distribution differences between the pre-training data domain and the remote sensing data domain with a very small number of trainable parameters, balancing segmentation accuracy and parameter efficiency.

[0111] Example 3: In Embodiment 1, a remote sensing image semantic segmentation method based on a parameter-efficient adapter was provided. In this embodiment, a remote sensing image semantic segmentation device based on a parameter-efficient adapter will be proposed. The remote sensing image semantic segmentation device based on a parameter-efficient adapter includes: a processor and a memory for storing processor-executable instructions; wherein, the processor is configured to execute the remote sensing image semantic segmentation method based on a parameter-efficient adapter as described in Embodiment 1.

[0112] like Figure 9 As shown, the remote sensing image semantic segmentation device based on a parameter-efficient adapter includes a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can be connected by a bus or other means.

[0113] Processor 21 can be a Central Processing Unit (CPU). Processor 21 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0114] The memory 22, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the remote sensing image semantic segmentation method based on a parameter-efficient adapter in Embodiment 1 of this invention. The processor executes various functional applications and training processes by running the non-transitory software programs, instructions, and modules stored in the memory.

[0115] The memory 22 may include a program storage area and a training storage area. The program storage area may store the operating system and applications required for at least one function; the training storage area may store training data created by the processor. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory, or other non-transitory solid-state storage device. In some embodiments, the memory 22 may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The one or more modules stored in the memory 22, when executed by the processor 21, perform functions such as... Figure 1 The remote sensing image semantic segmentation method based on a parameter-efficient adapter is shown in Example 1. For specific details of the above-described remote sensing image semantic segmentation method based on a parameter-efficient adapter, please refer to the relevant documentation. Figure 1 , Figure 2 and Figure 3 The relevant descriptions and effects in the embodiments shown are for reference only and will not be repeated here.

[0116] This embodiment also provides a computer storage medium storing a computer program that can be executed by a processor to perform the remote sensing image semantic segmentation method based on a parameter-efficient adapter described in Embodiment 1.

[0117] The computer storage medium stores computer-executable instructions, which can execute the remote sensing image semantic segmentation method based on a parameter-efficient adapter in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0118] The specific steps of the remote sensing image semantic segmentation method based on the parameter-efficient adapter are described in Example 1, and will not be repeated in this example.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on a parameter-efficient adapter, characterized in that, include: The remote sensing image to be segmented is obtained, and the remote sensing image is segmented into a sequence of non-overlapping image patches through an image patch embedding layer. A feature vector is obtained by linear projection on each image patch. A learnable classification label is added to the beginning of the feature vector, and an interpolated and adapted position code is superimposed to obtain an input label sequence. The input labeled sequence is input into a pre-trained visual backbone network with frozen parameters for layer-by-layer feature extraction. After the preset intermediate layer of the backbone network, a trainable adapter module is inserted to perform domain adaptation processing on the output features of that layer, resulting in a domain-adapted feature sequence. All pre-trained parameters of the backbone network are kept frozen during training, and only the parameters of the adapter module participate in training updates; The classification label is removed from the feature sequence after processing by all layers of the backbone network, and the image patch label is retained as a pixel-level feature representation. The pixel-level feature representation is input into a trainable segmentation head to generate the category prediction result for each image patch position. The category prediction results are then upsampled to restore the spatial dimensions of the original remote sensing image, resulting in the final pixel-level semantic segmentation result.

2. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 1, characterized in that, The domain adaptation process of the adapter module includes: The input features are projected onto a low-dimensional hidden space through a bottleneck structure to obtain the projected features; The projected features are interacted through a multi-head cross-attention mechanism to obtain attention-enhanced features; After concatenating the projection features and the attention enhancement features in the channel dimension, the original feature dimension is restored through a multilayer perceptron to obtain the fused features. The fused features are residually concatenated with the original input features through a learnable residual scaling factor to obtain a feature sequence adapted to the longitudinal domain.

3. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 2, characterized in that, The step of projecting the input features onto a low-dimensional hidden space through a bottleneck structure to obtain projected features specifically includes: The input features of the transformer block containing the adapter module are passed through a dimension reduction linear layer to compress the feature dimension from the original dimension D to the hidden dimension d, where d < D; The dimensionality-reduced features are sequentially normalized to stabilize the feature distribution, then nonlinear expression is introduced through the GELU nonlinear activation function, and finally overfitting is suppressed by Dropout regularization, resulting in a projected feature of dimension d.

4. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 2, characterized in that, The step of performing feature interaction on the projected features through a multi-head cross-attention mechanism to obtain attention-enhanced features specifically includes: The labeled sequence of the projected features is reshaped into a two-dimensional spatial grid. Local region features are extracted from the two-dimensional spatial grid, and sine-cosine position coding is added to the local region features to preserve spatial position information, thus obtaining a local feature representation. The projection features are used as query vectors, and the local feature representations are used as key vectors and value vectors, respectively. The adaptive attention response of each image patch label to the local region features is calculated through multi-head cross attention, so that each image patch label aggregates semantic information from the relevant target region to obtain attention output. After performing layer normalization on the attention output, a residual connection is made with the projected features to obtain the attention enhancement features.

5. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 2, characterized in that, The step of concatenating the projection features and the attention enhancement features along the channel dimension, and then restoring them to the original feature dimension through a multilayer perceptron to obtain the fused features, specifically includes: The projection feature and the attention enhancement feature are concatenated along the channel dimension to obtain the concatenated feature, the dimension of which is twice that of the hidden dimension. The concatenated features are input into the first linear layer, which maps the feature dimension from twice the hidden dimension to the hidden dimension. After processing by the GELU nonlinear activation function, the features are input into the second linear layer to restore the feature dimension from the hidden dimension to the original feature dimension, resulting in the fused feature x_fused. x_fused=Linear_2(GELU(Linear_1([x_proj;x_attended])))∈R^(B×N×D); In this layer, Linear_1 is the first linear layer, Linear_2 is the second linear layer, x_proj is the projection feature, and x_attended is the attention-enhanced feature.

6. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 2, characterized in that, The fused features are residually concatenated with the original input features through a learnable residual scaling factor to obtain a domain-adapted feature sequence x_out. The feature sequence x_out is calculated as follows: x_out = x + α × x_fused; Where x is the original input feature of the layer, x_fused is the fused feature, and α is a learnable residual scaling factor. α is initialized to a preset value in the range of 0.05 to 0.

15. α is adaptively updated during training through backpropagation gradients.

7. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 1, characterized in that, The method for determining the preset intermediate layer of the backbone network is as follows: The backbone network contains L cascaded converter blocks, which are divided into shallow, intermediate, and deep regions according to depth. The adapter module is inserted only at multiple evenly spaced locations within the intermediate layer region, including after the L / 4, 3L / 8, and L / 2 converter blocks; The intermediate layer region corresponds to the key stage of the backbone network's transition from low-level texture features to mid-level semantic features. Injecting domain adaptation signals at this stage can effectively guide the migration of features from the general visual domain to the remote sensing domain.

8. The remote sensing image semantic segmentation method based on a parameter-efficient adapter according to claim 1, characterized in that, The adapter module and the segmentation head are trained using a combined loss function L_total, which combines focus loss and Dice loss. L_total = L_Focal + L_Dice; The focus loss L_Focal is defined as: L_Focal=-α_t×(1-p_t)^γ×log(p_t); The Dice loss L_Dice is defined as: L_Dice=1-(2|P∩G|+ε) / (|P|+|G|+ε); Where α_t is the preset weight coefficient for each category, p_t is the predicted probability of the segmentation head outputting the true category, γ is the focusing parameter; P and G represent the pixel sets of the predicted and true labels, respectively, ε=1.0 is a smoothing term used to prevent the denominator from being zero. The Dice loss is calculated independently for each category and then averaged to ensure that the gradient signal of the minority class is not overwhelmed by the majority class.

9. A remote sensing image semantic segmentation device based on a parameter-efficient adapter, characterized in that, The remote sensing image semantic segmentation device based on a parameter-efficient adapter includes: a processor and a memory for storing processor-executable instructions; The processor is configured to execute the remote sensing image semantic segmentation method based on a parameter-efficient adapter as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the remote sensing image semantic segmentation method based on a parameter-efficient adapter as described in any one of claims 1-8.

Images