Large model adaptation method for small sample remote sensing interpretation task

Abstract

The invention relates to the technical field of deep learning, and provides a small sample remote sensing interpretation task-oriented large model adaptation method, which comprises the following steps of: determining a small sample remote sensing image, carrying out cross-domain feature calibration on the small sample remote sensing image, and determining a first processed remote sensing image; performing sample expansion based on the first processed remote sensing image by using a dynamic support set optimization strategy to obtain a second processed remote sensing image; constructing an adaptive loss function; and in the process of performing model training on the large model through a sample based on the second processed remote sensing image, performing model adaptability adjustment by using an adaptive loss function in combination with a lightweight adaptation strategy to obtain a target large model. According to the method, feature alignment, sample optimization and adaptive learning are taken as cores, efficient adaptability of a large model under a small sample condition is realized through organic fusion of cross-domain feature calibration, a dynamic support set optimization strategy and an adaptive loss function, and the precision and efficiency of remote sensing interpretation are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a method for adapting large models for small-sample remote sensing interpretation tasks. Background Technology

[0002] With the development of deep learning technology, large models based on pre-training and fine-tuning have achieved significant results in multiple vision tasks. However, in the field of remote sensing image interpretation, especially in scenarios with few samples, existing methods still have the following problems: Natural images and remote sensing images differ significantly in terms of data distribution, perspective, texture, and spectral features. Features extracted by pre-trained models are difficult to directly transfer to remote sensing tasks, leading to feature transfer bias and affecting model performance.

[0003] The utilization efficiency of limited samples is low. When expanding the samples, the enhancement of remote sensing images is constrained by geographic realism. Over-enhancement will lead to sample distortion and reduce model performance. Moreover, when meta-learning methods use models such as ProtoNet and MAML to simulate small sample scenarios through scenario training based on training and testing tasks, they lack specificity in constructing support sets and feature fusion, and cannot fully explore the semantic associations and multi-source information between samples.

[0004] Remote sensing images come from diverse sources, and imaging conditions are affected by factors such as season, lighting, and sensor type. However, existing technical solutions lack effective domain adaptation mechanisms, making it difficult for models trained in one scenario to quickly adapt to other scenarios, resulting in insufficient generalization ability and robustness of the models in different scenarios.

[0005] While applying large models to small-sample remote sensing interpretation tasks can improve model performance to some extent, it requires a lot of computing resources and time, making it difficult to meet the real-time requirements of remote sensing application scenarios, and increasing the computational overhead and inference latency of the model.

[0006] It is evident that large models are poorly adapted to small-sample remote sensing interpretation tasks, which in turn affects the accuracy and efficiency of remote sensing interpretation. Therefore, a new method for adapting large models to small-sample remote sensing interpretation tasks is urgently needed to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to provide a large model adaptation method for small-sample remote sensing interpretation tasks. With feature alignment, sample optimization and adaptive learning as the core, it achieves efficient adaptation of large models under small-sample conditions by organically integrating cross-domain feature calibration, dynamic support set optimization strategy and adaptive loss function, which significantly improves the accuracy and efficiency of remote sensing interpretation.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for adapting large models to small-sample remote sensing interpretation tasks, comprising: Identify small sample remote sensing images and perform cross-domain feature calibration on the small sample remote sensing images to determine the first processed remote sensing images; The second-processed remote sensing image is obtained by using a dynamic support set optimization strategy to augment the sample based on the first-processed remote sensing image. Construct an adaptive loss function; During the training of the large model based on samples from second-processed remote sensing images, an adaptive loss function combined with a lightweight adaptation strategy is used to adjust the model's adaptability, resulting in the target large model.

[0009] Preferably, cross-domain feature calibration is performed on small sample remote sensing images, including: Cross-domain alignment of spectral features is performed on small sample remote sensing images to obtain the first processing result; Spatial feature cross-domain alignment is performed on small sample remote sensing images to obtain the second processing result; The first processed remote sensing image is obtained by fusing features based on the first and second processing results.

[0010] Preferably, cross-domain alignment of spectral features for small sample remote sensing images includes: Spectral attention weighting is applied to small sample remote sensing images to obtain the weighted spectral features; The weighted spectral features are then subjected to nonlinear transformation and domain-aligned projection through a spectral feature mapping network to obtain the first processing result.

[0011] Preferably, cross-domain alignment of spatial features for small sample remote sensing images includes: Multi-scale spatial feature extraction is performed on small sample remote sensing images to obtain the multi-scale spatial features of small sample remote sensing images; Cross-domain feature comparison is performed based on multi-scale spatial features to obtain the second processing result.

[0012] Preferably, when using the dynamic support set optimization strategy to expand samples based on the first processed remote sensing image, the support set is screened for small samples based on the first processed remote sensing image using an adaptive screening mechanism. Multi-scale feature fusion is performed according to the support set and a fusion strategy to construct a multi-modal feature representation. The support set is combined with the multi-modal feature representation to determine the initial prototype feature vector. The prototype features are dynamically updated according to the initial prototype feature vector to obtain an optimized prototype. At the same time, feedback and optimization are performed on the adaptive screening mechanism and the fusion strategy. Then, the expanded samples are obtained according to the optimized prototype, and the expanded remote sensing image is determined.

[0013] Preferably, when constructing the adaptive loss function, the classification loss, domain adaptation loss, and consistency loss are fused to obtain the adaptive loss function.

[0014] Preferably, during the large model training process based on samples from second-processed remote sensing images, an adaptive loss function combined with a lightweight adaptation strategy is used to adjust model fitness, including: The loss data of the large model is analyzed and calculated during the model training process based on samples from the second-processed remote sensing images, and the total loss data is calculated through an adaptive loss function. Backpropagation and gradient calculation are performed based on the total loss data; The large model parameters are updated according to the gradient based on the lightweight adaptation strategy to determine the updated large model. Based on the updated large model, determine whether to continue the next round of model training and obtain the first analysis and judgment result; If the first analysis and judgment result is yes, the updated large model is used to continue model training; if the first analysis and judgment result is no, the model is evaluated and the loss weight coefficients are dynamically adjusted and updated through the validation set. By combining the termination conditions, a second analysis and judgment result is obtained to determine whether to terminate the model training. If the result of the second analysis is negative, the model continues to be trained based on the updated adaptive loss function. If the result of the second analysis is positive, the optimal values ​​of the model parameters are obtained, and the optimized large model is obtained.

[0015] Preferably, after obtaining the optimized large model, model distillation is performed on the optimized large model using model distillation technology to obtain a lightweight adapted model.

[0016] Preferably, the method further includes: during the application deployment phase, combining the lightweight adaptation model with the dynamic inference path selection mechanism and deploying it in the application environment of the small sample remote sensing interpretation task, so that during the inference process of the lightweight adaptation model for small sample remote sensing images, the target inference path is determined in combination with the scene complexity, and the remote sensing interpretation of the small sample remote sensing images is realized based on the target inference path.

[0017] Preferably, when deploying applications for large target models, a security defense mechanism is also incorporated to conduct security supervision during the inference process of lightweight adaptation models for small sample remote sensing images.

[0018] The present invention has achieved the following beneficial effects: This invention reduces inter-domain differences at the source through cross-domain feature calibration, quickly aligning the general feature space of a large model to the remote sensing interpretation task domain. This lays the foundation for subsequent analysis and processing, preventing the large model from losing its way in the initial stage and causing significant deviations in later steps, thus ensuring the adaptability of the large model. By utilizing a dynamic support set optimization strategy to augment samples based on the first-processed remote sensing image, the problem of insufficient small sample data can be effectively alleviated, improving the sample utilization rate in remote sensing images, increasing the diversity of samples in remote sensing images, avoiding distortion of augmented samples, and enabling the augmented samples to provide richer information for the training of the large model, thereby improving the generalization ability and robustness of the large model. By using an adaptive loss function, the weights or form of the loss function can be dynamically adjusted according to the actual situation during the training process of the large model. This allows the large model to learn key features more efficiently, improving training performance. This, in turn, guides the optimization of the large model during training, enabling it to better handle complex scenes and diverse land cover types in remote sensing interpretation tasks, thus improving the accuracy and reliability of remote sensing interpretation. Furthermore, combining this with a lightweight adaptation strategy for model adaptation not only improves the efficiency of parameter adjustment and model optimization but also ensures efficient adaptation of the large model to remote sensing imagery under small sample conditions. Even when faced with a large number of small sample remote sensing images, it reduces the computational overhead and inference latency of the large model, enabling rapid completion of remote sensing interpretation tasks. This reduces the computational resource consumption of the large model in practical applications, improves its inference speed, and ensures the accuracy of the large model in remote sensing interpretation tasks with small sample sizes.

[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the application.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram illustrating the steps of the large model adaptation method described in this invention; Figure 2 This is a schematic diagram of some steps in step one of the large model adaptation method described in this invention; Figure 3 This is a schematic diagram of step two in the large model adaptation method described in this invention; Figure 4This is a schematic diagram of step four in the large model adaptation method described in this invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] like Figure 1 As shown, this embodiment of the invention provides a method for adapting large models to small-sample remote sensing interpretation tasks, including: Step 1: Identify small sample remote sensing images and perform cross-domain feature calibration on the small sample remote sensing images to determine the first processed remote sensing images.

[0024] In this step, small-sample remote sensing images refer to remote sensing images with a limited amount of labeled data. When performing cross-domain feature calibration on small-sample remote sensing images, the spectral and spatial feature representations of the remote sensing images are optimized according to spectral and spatial features respectively, reducing inter-domain differences from the source. The first processed remote sensing image is the small-sample remote sensing image after spectral feature enhancement and spatial feature calibration.

[0025] Step 2: Using a dynamic support set optimization strategy, expand the samples based on the first-processed remote sensing image to obtain the second-processed remote sensing image.

[0026] In this step, the dynamic support set optimization strategy fully explores the semantic information and complementary features of the limited samples in the first-processed remote sensing image through support set selection, multi-scale feature fusion, and prototype feature dynamic updating, thereby improving the utilization rate of small samples in the remote sensing image, increasing the number and diversity of samples in the remote sensing image, and thus providing richer data information for the training of large models.

[0027] Step 3: Construct an adaptive loss function.

[0028] In this step, when constructing the adaptive loss function, an adaptive loss function that integrates classification loss, domain adaptation loss, and consistency loss is constructed to achieve multi-objective collaborative optimization.

[0029] Step 4: During the training of the large model based on the second-processed remote sensing images using samples, the adaptive loss function combined with a lightweight adaptation strategy is used to adjust the model's adaptability, resulting in the target large model.

[0030] In this step, the large model typically refers to a deep learning model pre-trained on a large-scale dataset, such as: Transformer architecture models (Vision Transformer, Swing Transformer, Bidirectional Encoder Representations from Transformers, etc.); Convolutional Neural Network (CNN) architecture models (Residual Networks, EfficientNet, Densely Connected Convolutional Networks, etc.); and hybrid architecture models (ConvNeXt, ViT-Adapter, etc.). The target large model is the large model adapted to the small-sample remote sensing interpretation task. When adjusting model adaptability using an adaptive loss function combined with a lightweight adaptation strategy, the adaptive loss function analyzes the loss during model training, and based on the loss, the parameters are adjusted and the model is optimized according to the lightweight adaptation strategy to determine the large model adapted to the small-sample remote sensing interpretation task, thus obtaining the target large model.

[0031] The aforementioned technical solution focuses on feature alignment, sample optimization, and adaptive learning for large-scale model adaptation. By organically integrating cross-domain feature calibration, dynamic support set optimization strategies, and adaptive loss functions, it achieves efficient adaptation of large models under small-sample conditions, significantly improving the accuracy and efficiency of remote sensing interpretation. Cross-domain feature calibration reduces inter-domain differences at the source, quickly aligning the general feature space of the large model to the remote sensing interpretation task domain, laying the foundation for subsequent analysis and processing, and preventing the large model from losing its way in the initial stage, which could lead to significant deviations in subsequent steps, thus ensuring the adaptability of the large model. By utilizing dynamic support set optimization strategies to augment samples based on the first-processed remote sensing image, it effectively alleviates the problem of insufficient small-sample data, improves the sample utilization rate in remote sensing images, increases the diversity of samples in remote sensing images, avoids distortion of augmented samples, and allows the augmented samples to provide richer information for large-scale model training, thereby improving the generalization ability and robustness of the large model. By using an adaptive loss function, the weights or form of the loss function can be dynamically adjusted according to the actual situation during the training process of the large model. This allows the large model to learn key features more efficiently, improving training performance. This, in turn, guides the optimization of the large model during training, enabling it to better handle complex scenes and diverse land cover types in remote sensing interpretation tasks, thus improving the accuracy and reliability of remote sensing interpretation. Furthermore, combining this with a lightweight adaptation strategy for model adaptation not only improves the efficiency of parameter adjustment and model optimization but also ensures efficient adaptation of the large model to remote sensing imagery under small sample conditions. Even when faced with a large number of small sample remote sensing images, it reduces the computational overhead and inference latency of the large model, enabling rapid completion of remote sensing interpretation tasks. This reduces the computational resource consumption of the large model in practical applications, improves the inference speed of the large model, and ensures the remote sensing interpretation accuracy of the large model for small sample remote sensing interpretation tasks.

[0032] In one embodiment provided by the present invention, such as Figure 2 As shown, cross-domain feature calibration for small sample remote sensing images includes: A1. Perform cross-domain alignment of spectral features on small sample remote sensing images to obtain the first processing result.

[0033] In this step, a spectral attention mechanism is introduced at the input stage of the large model. The spectral attention mechanism is combined with the feature space of the large model to perform cross-domain alignment of spectral features, and the first processing result is obtained.

[0034] A2. Perform cross-domain alignment of spatial features for small sample remote sensing images to obtain the second processing result.

[0035] In this step, a dynamic receptive field adjustment module is constructed by using a multi-scale attention mechanism based on a large model. The dynamic receptive field adjustment module is then combined with the feature space of the large model to perform cross-domain alignment of spatial features, resulting in the second processing result.

[0036] A3. Based on the first processing result and the second processing result, feature fusion is performed to obtain the first processed remote sensing image.

[0037] In this step, when performing feature fusion based on the first and second processing results, the domain-aligned spectral features from the first processing result are stitched together with the domain-invariant spatial features from the second processing result. An adaptive weight generator calculates weights for the spectral and spatial features, generating spectral feature fusion weights and spatial feature fusion weights. Weighted fusion is then performed based on the stitched result according to these weights, outputting a small-sample remote sensing image with cross-domain feature calibration, resulting in the first-processed remote sensing image. The sum of the spectral feature fusion weights and spatial feature fusion weights is 1. When performing weighted fusion based on the stitched result according to these weights, the attention mechanism is dynamically calculated using the following formula: In the above formula, To produce a weighted fusion output, For adaptive weighting coefficients, For the domain-aligned spectral features in the first processing result, This refers to the spatial characteristics that remain unchanged in the second processing result.

[0038] The above technical solution reduces the inter-domain differences of small-sample remote sensing images from the source by performing cross-domain feature calibration on small-sample remote sensing images, improving the transferability and consistency of features. This allows large models to directly transfer features from the first-processed remote sensing images to remote sensing tasks, reducing feature transfer bias, lowering the risk of overfitting, ensuring the model's performance on new data, and thus guaranteeing the performance of large models. This reduces the probability of misclassification due to feature differences and improves the accuracy of large models in interpreting remote sensing images. Cross-domain alignment of spectral and spatial features for small-sample remote sensing images can mitigate the shortcomings of insufficient capture of key information such as spectral features and spatial topological relationships of ground objects in remote sensing images, improve the discriminative power of remote sensing features, and thus ensure the accuracy of remote sensing interpretation. Furthermore, cross-domain alignment of spectral features ensures better consistency in spectral features across different domains (e.g., sensor, time, region), reducing spectral feature drift caused by sensor differences and environmental changes, thereby improving feature transferability. Similarly, cross-domain alignment of spatial features ensures consistency in spatial features (e.g., shape, texture, and edge information of ground objects) across different domains, reducing inconsistencies caused by differences in image resolution and imaging angles, further improving feature transferability. Feature fusion based on the first and second processing results not only preserves all spectral and spatial feature information but also ensures consistency of these features across different domains through cross-domain alignment. This allows large models to better learn sample features during model training, enhancing the model's generalization ability to data from different domains. Furthermore, when performing feature fusion based on the first and second processing results, an adaptive weight generator is used to calculate the weights for spectral and spatial features, thereby achieving dynamic determination of the weights, effectively calibrating inter-domain feature differences, reducing feature transfer bias, and ensuring the performance of large models.

[0039] In one embodiment of the present invention, cross-domain alignment of spectral features for small sample remote sensing images includes: Spectral attention weighting is applied to small sample remote sensing images to obtain the weighted spectral features.

[0040] In this step, when performing spectral attention weighting on small sample remote sensing images, band features are determined for the small sample images, and the information entropy of each band and the correlation between bands are calculated to obtain the entropy and correlation calculation results. Based on the entropy and correlation calculation results, a set of weight vectors is adaptively generated through a lightweight neural network. Then, the weight vectors are combined to apply weights to the band features, highlighting spectral features that are important for ground cover classification and suppressing the interference of noisy bands, resulting in weighted spectral features. Here, the lightweight neural network is usually an MLP (Multilayer Perceptron).

[0041] The weighted spectral features are then subjected to nonlinear transformation and domain-aligned projection through a spectral feature mapping network to obtain the first processing result.

[0042] In this step, the spectral feature mapping network is usually a nonlinear mapping network composed of fully connected layers and activation functions. The weighted spectral features are nonlinearly transformed by the spectral feature mapping network, and then the spectral features are mapped from their original distribution, which is significantly different from that of natural images, to a distribution that corresponds to the feature space of the large model, to obtain the first processing result, which directly reduces the difference in spectral distribution between domains from the spectral dimension.

[0043] The aforementioned technical solution employs spectral attention weighting for small-sample remote sensing images, using information entropy to measure band uncertainty. This allows for the identification of which bands are more helpful for classification. Furthermore, calculating the correlation between bands avoids interference from redundant information, reducing redundancy and improving the independence and effectiveness of band features. A lightweight neural network adaptively generates weight vectors, dynamically adjusting weights based on specific data to ensure the weighted spectral features are more discriminative and thus more accurately reflect the spectral characteristics of ground objects. By performing nonlinear transformation and domain-aligned projection on the weighted spectral features through a spectral feature mapping network, the network captures the complex nonlinear relationships of spectral features, making them more expressive in the new space. Mapping spectral features from their original distribution, which differs significantly from natural images, to a distribution corresponding to the feature space of the large model directly reduces the differences in spectral distributions between domains. This allows the spectral features of small-sample remote sensing images to better adapt to the requirements of large models, improving the generalization ability of large models to data from different domains.

[0044] In one embodiment of the present invention, spatial feature cross-domain alignment is performed on small sample remote sensing images, including: Multi-scale spatial feature extraction is performed on small sample remote sensing images to obtain the multi-scale spatial features of small sample remote sensing images.

[0045] In this step, when extracting multi-scale spatial features from small-sample remote sensing images, a dynamic receptive field adjustment module adaptively adjusts the shape of the convolution kernel and the size of the receptive field according to the scale changes of remotely sensed features, thereby achieving effective extraction of multi-scale spatial features and obtaining the multi-scale spatial features of the small-sample remote sensing images. Here, the scale changes of remotely sensed features refer to the changes in the size of features such as buildings, roads, and vegetation in the spatial location of the small-sample remote sensing images. The dynamic receptive field adjustment module is constructed based on the multi-scale attention mechanism of a large model.

[0046] Cross-domain feature comparison is performed based on multi-scale spatial features to obtain the second processing result.

[0047] In this step, when performing cross-domain feature comparison based on multi-scale spatial features, spatial features are determined for natural images, and the spatial features of natural images are used as anchor points. Cross-domain feature comparison learning is performed on multi-scale spatial features based on anchor points to find multi-scale spatial features of remote sensing images that belong to the same semantic category as the anchor point features. Through a loss function, their feature representations are brought closer to each other in the embedding space. Anchor point features and multi-scale spatial features of remote sensing images of different semantic categories are combined into negative sample pairs, and through a loss function, their feature representations are moved further apart in the embedding space. This process extracts domain-invariant spatial features that are insensitive to domain changes and are only related to semantic content. The spatial feature transfer bias caused by differences in viewpoint, texture, etc., is calibrated, thereby achieving cross-domain alignment of spatial features and obtaining the second processing result.

[0048] The aforementioned technical solution, when extracting multi-scale spatial features from small-sample remote sensing images, utilizes a dynamic receptive field adjustment module to adaptively adjust the shape of the convolution kernel and the size of the receptive field according to the scale changes of remotely sensed objects. This better captures features of objects at different scales, flexibly handles objects at various scales, and improves the robustness of feature extraction. Thus, it can effectively extract multi-scale spatial features from small-sample remote sensing images, ensuring the representativeness of these features based on the small-sample images. When performing cross-domain feature comparison based on multi-scale spatial features, a loss function is used to bring features of the same semantic category closer together in the embedding space, while features of different semantic categories are kept further apart. This ensures semantic consistency of the extracted spatial features, enabling large models to better handle remote sensing images from different domains. Furthermore, contrastive learning can calibrate spatial feature transfer biases caused by differences in viewpoint, texture, etc., reducing spatial feature differences due to changes in imaging conditions. This improves the adaptability of large models to data from different domains, laying the foundation for model training.

[0049] In one embodiment provided by the present invention, such as Figure 3As shown, when using the dynamic support set optimization strategy to expand samples based on the first-processed remote sensing image, the support set is screened for small samples using an adaptive screening mechanism based on the first-processed remote sensing image. Multi-scale feature fusion is performed according to the support set and a fusion strategy to construct a multi-modal feature representation. The support set is combined with the multi-modal feature representation to determine the initial prototype feature vector, and the prototype features are dynamically updated according to the initial prototype feature vector to obtain the optimized prototype. At the same time, feedback and optimization are performed on the adaptive screening mechanism and the fusion strategy. Then, the expanded samples are obtained according to the optimized prototype, and the expanded remote sensing image is determined.

[0050] Specifically, when selecting a support set for small samples based on the first-processed remote sensing image using an adaptive screening mechanism, the adaptive screening mechanism for selecting a support set for small samples in the first-processed remote sensing image includes: performing feature vector analysis and determination on small samples in the first-processed remote sensing image, and calculating information entropy based on the feature vectors to obtain sample feature entropy; calculating the semantic correlation between samples based on the feature vectors using cosine similarity; performing adaptive screening based on a similarity threshold, determining whether the semantic correlation between samples is greater than the similarity threshold, and if it is greater than the similarity threshold, removing samples based on their feature entropy, retaining samples with higher feature entropy, removing another redundant sample, and removing samples with extremely low feature entropy to obtain the first screening result; and dynamically selecting samples with high feature entropy and low semantic correlation between samples from the first screening result according to task requirements to obtain the second screening result, thus determining the support set.

[0051] Based on the support set, multi-scale feature fusion is performed according to the fusion strategy, including: extracting feature maps from different intermediate layers of the large model for the samples in the support set to obtain feature maps from different levels; then, by constructing a feature pyramid, combining shallow detailed features with deep semantic features of the feature maps from different levels, and introducing an attention weight allocation mechanism to adaptively adjust the adaptive contribution of features at different scales; then, multi-scale feature fusion is performed on the feature maps from different levels according to the adaptive contribution of features at different scales to obtain multi-modal feature representations.

[0052] The process involves determining the initial prototype feature vector by combining the support set with multimodal feature representation, and then dynamically updating the prototype features based on this initial feature vector to obtain an optimized prototype. Simultaneously, feedback and optimization are performed on the adaptive selection mechanism and fusion strategy. This is conducted within a meta-learning framework for contextual training. When determining the initial prototype feature vector by combining the support set with multimodal feature representation, the samples in the support set are augmented using multimodal feature representation, and mean analysis is performed according to category to obtain the initial prototype feature vector. During the dynamic update of the prototype features based on the initial prototype feature vector, the distance between the query sample and the initial prototype is calculated through the interaction feedback between the query sample and the prototype features. High-confidence query samples are identified based on this distance, and their features are then fused into the corresponding initial prototype to obtain the optimized prototype. Simultaneously, feedback information is obtained regarding support set selection and multi-scale feature fusion. This feedback information is used to optimize and adjust the adaptive selection mechanism in support set selection and the fusion strategy in multi-scale feature fusion.

[0053] The aforementioned technical solution establishes a tightly collaborative and feedback-driven dynamic support set optimization strategy through support set selection, multi-scale feature fusion, and dynamic updating of prototype features. This enables a complete processing flow encompassing preprocessing, enhancement, and iterative optimization, fully leveraging semantic relationships and complementary information between samples. This improves the utilization rate of small samples, enhances sample expansion, avoids sample distortion during optimization, and ultimately ensures the performance of large models. Support set selection identifies samples rich in feature information and redundant samples, eliminating those containing less useful information and improving sample quality. Using cosine similarity to calculate semantic relationships between samples, combined with similarity thresholds, removes redundant samples, retains more representative ones, avoids excessive similarity, and increases sample diversity. Dynamically selecting samples with high feature entropy and low semantic relationships based on task requirements further optimizes the support set quality. Flexible adjustments based on specific task needs ensure the selected samples are best suited for the current task. Multi-scale feature fusion extracts feature maps from different intermediate layers of the large model, capturing multi-scale features from shallow details to deep semantics, providing a more comprehensive feature representation and enhancing sample feature richness. By introducing an attention weight allocation mechanism, the contribution of features at different scales can be adaptively adjusted, enabling large models to dynamically adjust the importance of features according to specific task requirements, further optimizing the feature fusion effect. Through initial prototype feature vector determination and dynamic updating of prototype features, not only can a more stable and richer initial prototype be provided, but the large model can also be continuously optimized during training, improving its adaptability to new samples. Simultaneously, during the process of obtaining the optimized prototype, feedback optimization is performed on the adaptive selection mechanism in the support set selection, allowing the selection mechanism to continuously adjust based on feedback information during training, improving the accuracy and efficiency of selection. Furthermore, the fusion strategy in multi-scale feature fusion is optimized and adjusted, dynamically improving the feature fusion effect based on feedback information during training, enhancing the model's generalization ability, reducing training instability caused by fixed feature representations, and making the large model more stable and converge better during training. In addition, by conducting scenario training within the framework of meta-learning, the learning process of the large model in different tasks can be simulated, improving the model's rapid adaptability, thereby enabling faster learning of general features and reducing training time and computational resource consumption.

[0054] In one embodiment of the present invention, when constructing the adaptive loss function, the classification loss, domain adaptation loss and consistency loss are fused to obtain the adaptive loss function.

[0055] In this section, to alleviate class imbalance in small sample scenarios, an improved focal loss function is employed. This function introduces a sample difficulty coefficient, causing the large model to focus more on samples that are difficult to classify correctly during training, thereby reducing classification bias caused by class imbalance. Furthermore, by dynamically adjusting parameters to adapt to different levels of class imbalance in various scenarios, the loss function can better adapt to diverse small sample situations, improving the model's ability to classify minority classes. The specific form of the classification loss is as follows: ; in, For classification loss; The parameters are adjusted to control the weights of difficult-to-classify samples, and are dynamically adjusted to adapt to different scenarios. For the large model to the first The predicted probability of each sample; For the first The true label of each sample; The total number of samples.

[0056] Domain adaptation loss is used to minimize the feature distribution difference between the source domain (pre-training data) and the target domain (remote sensing samples). This helps reduce feature distribution differences caused by different data sources, improves the performance of large models on the target domain (remote sensing samples), and enhances the model's domain adaptability. Maximum mean difference (MMD) loss is used to calculate the distance between the features of two domains in the reproducing kernel Hilbert space, effectively capturing the difference in feature distribution, guiding the model to learn domain-invariant features, and minimizing this difference allows the model to better adapt to the target domain.

[0057] Consistency loss is used to constrain the model's prediction consistency under different data augmentation methods. Using consistency loss significantly improves the model's stability in the face of noise and scene changes, enabling it to better cope with changes in remote sensing images caused by multiple factors (such as lighting conditions and sensor noise), and improve the reliability of interpretation results. Slight spectral perturbations and spatial transformations are applied to the support set samples, and the difference in the model's prediction results between the original and augmented samples is calculated. By minimizing this difference, the model's robustness to noise and scene changes is improved.

[0058] The adaptive loss function introduces loss weight coefficients that are dynamically adjusted using the validation set. This allows for a flexible balance of the importance of different loss terms based on specific task requirements and data characteristics. This enables large models to better balance classification performance, domain adaptability, and robustness during training, thereby improving overall performance. Its specific representation is as follows: ; in, Total loss; For classification loss; Domain adaptation loss; For consistency loss, and These are the loss weight coefficients, which are dynamically adjusted through the validation set to achieve collaborative optimization of different loss terms.

[0059] The above technical solution integrates classification loss, domain adaptation loss, and consistency loss, enabling the adaptive loss function to simultaneously optimize multiple key objectives. This allows large models to better learn general features under small sample conditions, improve their generalization ability to different tasks and data, and better adapt to various small sample scenarios and task requirements, thereby enhancing the adaptability and reliability of the model in practical applications.

[0060] In one embodiment provided by the present invention, such as Figure 4 As shown, during the large model training process based on samples from second-processed remote sensing images, an adaptive loss function combined with a lightweight adaptation strategy is used to adjust model fitness, including: B1. Analyze and calculate the loss data of the large model based on the second-processed remote sensing image during the model training process, and calculate the total loss data through the adaptive loss function.

[0061] In this step, when analyzing and calculating the loss data of the large model during the model training process based on the second-processed remote sensing image through samples, the process data information of the large model performing model analysis on the samples is obtained. Based on the process data information, the classification loss, domain adaptation loss and consistency loss are analyzed and calculated separately to obtain the classification loss data, domain adaptation loss data and consistency loss data. Then, the total loss data is calculated according to the adaptive loss function based on the classification loss data, domain adaptation loss data and consistency loss data.

[0062] B2. Perform backpropagation and gradient calculation based on the total loss data.

[0063] In this step, when backpropagating and calculating gradients based on the total loss data, backpropagation is performed on the total loss to calculate the gradients of all trainable parameters of the model.

[0064] B3. Update the parameters of the large model according to the gradient based on the lightweight adaptation strategy, and determine the updated large model.

[0065] In this step, when updating the parameters of the large model according to the gradient based on the lightweight adaptation strategy, the large model is divided into three layers (bottom layer, middle layer and top layer) according to the network depth. The layered fine-tuning scheme is used to adjust the parameters of the large model, including: freezing the bottom layer parameters, fine-tuning the middle layer parameters, and fully fine-tuning the top layer parameters.

[0066] B4. Based on the updated large model, determine whether to continue the next round of model training and obtain the first analysis and judgment result.

[0067] In this step, when determining whether to continue the next round of model training based on the updated large model, the model training cycle threshold is used to determine whether the next round of model training needs to be carried out.

[0068] B5. When the first analysis and judgment result is yes, the updated large model is used to continue model training. When the first analysis and judgment result is no, the model is evaluated and the loss weight coefficients are dynamically adjusted and updated through the validation set.

[0069] In this step, when evaluating the model using the validation set, dynamically adjusting the loss weight coefficients, and updating the adaptive loss function, the samples in the validation set are used to perform model analysis on the current large model to obtain model validation analysis data. Based on the model validation analysis data, the classification accuracy, generalization, and robustness of the large model are evaluated to obtain the validation analysis results. Then, the loss weight coefficients are dynamically adjusted based on the validation analysis results; for example, if the classification accuracy is insufficient, the weights are reduced. and Insufficient generalization will increase Insufficient robustness will increase Thus, an updated adaptive loss function is obtained based on the dynamically adjusted loss weight coefficients.

[0070] B6. Combine the termination conditions to analyze and determine whether to terminate the model training, and obtain the second analysis and judgment result.

[0071] In this step, the termination conditions include: model training time threshold, model training cycle threshold, accuracy standard, etc. When analyzing and determining whether to terminate model training based on the termination conditions, model training can be terminated if any one of the termination conditions is met.

[0072] B7. When the result of the second analysis is negative, continue model training based on the updated adaptive loss function. When the result of the second analysis is positive, obtain the optimal values ​​of the model parameters and obtain the optimized large model.

[0073] In this step, when the model continues to be trained based on the updated adaptive loss function, the updated adaptive loss function is used for loss analysis and calculation during the training of the large model based on the second-processed remote sensing images through samples.

[0074] The aforementioned technical solution, through adaptive loss function collaborative guidance of large model parameter updates, not only comprehensively evaluates the model's performance in different aspects and achieves optimization of multiple key objectives, but also flexibly adjusts the optimization direction according to specific task requirements and data characteristics. This allows the optimized large model to better handle complex scenes and diverse land cover types in remote sensing interpretation tasks with small sample sizes, improving the accuracy and reliability of remote sensing interpretation. Combined with a lightweight adaptation strategy, the large model is divided into bottom, middle, and top layers according to network depth for hierarchical parameter adjustment, reducing computational resource consumption while ensuring that key model parameters are fully optimized. Freezing the bottom layer parameters preserves the general feature extraction capabilities of the pre-trained model, avoiding excessive adjustment of bottom layer parameters under small sample conditions that could lead to performance degradation and reducing the amount of large model parameter updates. Fine-tuning the middle and top layer parameters not only improves the efficiency of large model parameter updates but also enables the large model to better adapt to specific tasks and data, improving the model's classification accuracy and generalization ability. Furthermore, by dynamically adjusting and updating the loss weight coefficients during training, the overall performance of the model is continuously improved, enabling the model to better adapt to task requirements in subsequent training. Thus, when facing remote sensing interpretation tasks with small sample sizes, the optimized large model can better handle complex scenes and diverse land cover types in remote sensing interpretation tasks, thereby improving the accuracy and reliability of remote sensing interpretation.

[0075] In one embodiment of the present invention, after obtaining the optimized large model, model distillation is performed on the optimized large model using model distillation technology to obtain a lightweight adapted model.

[0076] Among these, model distillation is performed on the optimized large model using model distillation techniques, including: Construct a teacher-student model pair, using the optimized large model as the teacher model and selecting a compact, computationally inefficient lightweight model architecture as the student model. Student models can be, for example, MobileNet series, ShuffleNet series, EfficientNet-Lite, or a miniaturized Transformer (such as TinyViT). Moreover, the number of parameters and computational complexity of the student model should be significantly lower than those of the teacher model, typically 1 / 10 or even 1 / 100.

[0077] The samples based on the second-processed remote sensing images are used to train and optimize the teacher model and the student model, obtain the optimized student model, and obtain the lightweight adaptation model. Here, the samples based on the second-processed remote sensing images are the same as the samples used for training the large model.

[0078] The aforementioned technical solution transforms the optimized large model into a lightweight, adaptable model through model distillation, reducing the demand for hardware resources such as CPUs and GPUs, decreasing computation time and energy consumption. Furthermore, the lightweight adaptable model's low computational complexity and small size allow it to better adapt to various application scenarios. Even when faced with a large number of small-sample remote sensing images, it reduces the computational overhead and inference latency of the target large model, enabling rapid completion of remote sensing interpretation tasks. This reduces the computational resource consumption of the target large model in practical applications, improves its inference speed, and enhances its maintainability and scalability. Moreover, the teacher model guides the student model, providing richer information and helping the student model learn smoother decision boundaries, thereby enhancing its generalization ability. This allows the student model to better capture key features of unseen data, improving its adaptability to different scenarios and data variations.

[0079] In one embodiment of the present invention, the method further includes: during the application deployment phase, combining the lightweight adaptation model with the dynamic inference path selection mechanism and deploying it in the application environment of the small sample remote sensing interpretation task, so that during the inference process of the lightweight adaptation model for small sample remote sensing images, the target inference path is determined in combination with the scene complexity, and the remote sensing interpretation of the small sample remote sensing images is realized based on the target inference path.

[0080] In the application deployment phase, the target large model is deployed by combining a lightweight adaptation model with a dynamic inference path selection mechanism in a small-sample remote sensing interpretation task application environment. During the inference process of the lightweight adaptation model for small-sample remote sensing images, the target inference path is determined based on the scene complexity. This includes: performing scene recognition analysis on the small-sample remote sensing images to identify the corresponding scenes, calculating the complexity based on the scenes, and then determining whether the scene is simple or complex based on the complexity. When the scene is simple, a shallow feature extraction path is used; when the scene is complex, a deep feature extraction path is used. Here, the shallow feature extraction path mainly focuses on the macroscopic features of the image, significantly improving inference speed with low computational complexity, quickly providing remote sensing interpretation results, and improving the efficiency of remote sensing interpretation. Deep feature extraction paths can capture more details and deeper features of images, more accurately identify and classify ground features, improve the reliability of interpretation results, and ensure interpretation accuracy. Therefore, in complex scenarios, deep feature extraction paths can be selected first to ensure that interpretation accuracy is not affected. While ensuring efficiency, interpretation accuracy is not sacrificed, so that it can perform well in different scenarios.

[0081] The aforementioned technical solution not only effectively addresses the balance between efficiency and accuracy in remote sensing image interpretation tasks under small sample conditions, promoting the widespread application of the model in more scenarios, but also significantly improves inference efficiency and interpretation accuracy, enhances the adaptability and flexibility of the target large model, ensures the performance of the target large model in practical applications, reduces resource consumption, and improves the reliability of the large model for small sample remote sensing interpretation tasks. By combining a lightweight adaptable model with a dynamic inference path selection mechanism and deploying it in the application environment of small sample remote sensing interpretation tasks, the problem of imbalance between model complexity and inference efficiency is effectively solved. This allows for the automatic selection of the most suitable inference path for efficient remote sensing interpretation of different types of image data and images of varying complexity, enhancing the adaptability and flexibility of the target large model. Furthermore, by dynamically selecting the inference path, resource allocation can be flexibly adjusted according to scene complexity, reducing misjudgments caused by improper path selection. This enables efficient operation in different scenarios, significantly reducing the operating cost of the target large model and improving its practicality and reliability.

[0082] In one embodiment of the present invention, when deploying applications for a large target model, a security defense mechanism is also combined to conduct security supervision during the inference process of the lightweight adaptation model for small sample remote sensing images.

[0083] Specifically, when combining security defense mechanisms with the lightweight adaptation model for inference of small sample remote sensing images, the system monitors the process based on the security defense mechanisms, obtains real-time monitoring information, and analyzes and judges whether there is a risk of data leakage based on the real-time monitoring information. When there is a risk of data leakage, the system identifies the location of the potential data leakage, implements security defenses for the potential data leakage location, and issues risk warnings.

[0084] The aforementioned technical solution employs a security defense mechanism to monitor the application process of large-scale target models, ensuring the security of small-sample remote sensing imagery and remote sensing interpretation information. This avoids legal risks arising from data leakage when small-sample remote sensing imagery involves sensitive information or areas, ensuring data security and privacy, and enabling safe application within the scope of national and regional laws and regulations. Furthermore, real-time monitoring and rapid response enable effective defensive measures to be taken before or in the early stages of data leakage events, reducing the likelihood and scope of data breaches.

[0085] Those skilled in the art should understand that the terms "first" and "second" in this invention merely refer to different application stages.

[0086] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0087] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for adapting large models to small-sample remote sensing interpretation tasks, characterized in that, include: Identify small sample remote sensing images and perform cross-domain feature calibration on the small sample remote sensing images to determine the first processed remote sensing images; The second-processed remote sensing image is obtained by using a dynamic support set optimization strategy to augment the sample based on the first-processed remote sensing image. Construct an adaptive loss function; During the training of the large model based on samples from second-processed remote sensing images, an adaptive loss function combined with a lightweight adaptation strategy is used to adjust the model's adaptability, resulting in the target large model.

2. The large model adaptation method according to claim 1, characterized in that, Cross-domain feature calibration for small sample remote sensing images includes: Cross-domain alignment of spectral features is performed on small sample remote sensing images to obtain the first processing result; Spatial feature cross-domain alignment is performed on small sample remote sensing images to obtain the second processing result; The first processed remote sensing image is obtained by fusing features based on the first and second processing results.

3. The large model adaptation method according to claim 2, characterized in that, Cross-domain alignment of spectral features for small sample remote sensing images, including: Spectral attention weighting is applied to small sample remote sensing images to obtain the weighted spectral features; The weighted spectral features are then subjected to nonlinear transformation and domain-aligned projection through a spectral feature mapping network to obtain the first processing result.

4. The large model adaptation method according to claim 2, characterized in that, Cross-domain alignment of spatial features for small sample remote sensing images, including: Multi-scale spatial feature extraction is performed on small sample remote sensing images to obtain the multi-scale spatial features of small sample remote sensing images; Cross-domain feature comparison is performed based on multi-scale spatial features to obtain the second processing result.

5. The large model adaptation method according to claim 1, characterized in that, When expanding samples based on the first-processed remote sensing image using a dynamic support set optimization strategy, the support set is selected based on the first-processed remote sensing image for small samples using an adaptive screening mechanism. Multi-scale feature fusion is performed according to the support set and a fusion strategy to construct a multi-modal feature representation. The support set is combined with the multi-modal feature representation to determine the initial prototype feature vector, and the prototype features are dynamically updated based on the initial prototype feature vector to obtain an optimized prototype. At the same time, feedback and optimization are performed on the adaptive screening mechanism and the fusion strategy. Then, the expanded samples are obtained based on the optimized prototype, and the expanded remote sensing image is determined.

6. The large model adaptation method according to claim 1, characterized in that, When constructing the adaptive loss function, the classification loss, domain adaptation loss, and consistency loss are combined to obtain the adaptive loss function.

7. The large model adaptation method according to claim 6, characterized in that, During the large model training process based on samples from second-processed remote sensing images, an adaptive loss function combined with a lightweight adaptation strategy is used to adjust model fitness, including: The loss data of the large model is analyzed and calculated during the model training process based on samples from the second-processed remote sensing images, and the total loss data is calculated through an adaptive loss function. Backpropagation and gradient calculation are performed based on the total loss data; The large model parameters are updated according to the gradient based on the lightweight adaptation strategy to determine the updated large model. Based on the updated large model, determine whether to continue the next round of model training and obtain the first analysis and judgment result; If the first analysis and judgment result is yes, the updated large model is used to continue model training; if the first analysis and judgment result is no, the model is evaluated and the loss weight coefficients are dynamically adjusted and updated through the validation set. By combining the termination conditions, a second analysis and judgment result is obtained to determine whether to terminate the model training. If the result of the second analysis is negative, the model continues to be trained based on the updated adaptive loss function. If the result of the second analysis is positive, the optimal values ​​of the model parameters are obtained, and the optimized large model is obtained.

8. The large model adaptation method according to claim 7, characterized in that, After obtaining the optimized large model, model distillation is performed on the optimized large model to obtain a lightweight adapted model.

9. The large model adaptation method according to claim 8, characterized in that, Also includes: During the application deployment phase, the lightweight adaptation model is combined with the dynamic inference path selection mechanism and deployed to the application environment of small sample remote sensing interpretation tasks. In this way, during the inference process of the lightweight adaptation model for small sample remote sensing images, the target inference path is determined based on the complexity of the scene, and the remote sensing interpretation of small sample remote sensing images is realized based on the target inference path.

10. The large model adaptation method according to claim 9, characterized in that, When deploying applications for large target models, security defense mechanisms are also incorporated to conduct security oversight during the inference process of lightweight adaptation models for small sample remote sensing images.

Images