Multi-source remote sensing image incremental learning method based on prompt fine tuning

Abstract

The invention belongs to the technical field of remote sensing image processing. The invention provides a multi-source remote sensing image class incremental learning method based on prompt fine tuning. According to the embodiment of the invention, a mode-specific prompt pool is constructed, an instance-based prompt query mechanism is designed, and proper prompts are dynamically selected for different input characteristics; a learnable modal sharing global prompt is added to each attention layer of the frozen Vision Transform network, and modal public information specific to a task is extracted; carrying out cross-modal prompt conversion and fusion; freezing a part of the modal mapping network to reserve a modal conversion relation of the old task; establishing a prompt-guided knowledge aggregator module, freezing a knowledge aggregation token in an old task network, and jointly guiding the knowledge aggregation token, a knowledge aggregation token of a new task and a modal specific prompt to guide the aggregator module to learn image features; and training the established incremental learning network model.

Description

Technical Field

[0001] This disclosure relates to the field of remote sensing image processing technology, and in particular to a multi-source remote sensing image class incremental learning method based on cue-based fine-tuning. Background Technology

[0002] In the vast field of remote sensing science, multi-source remote sensing classification is a core research direction, dedicated to integrating data resources from different sensors (including but not limited to hyperspectral images, LiDAR data, SAR images, etc.) to achieve high-precision classification of surface targets. With the widespread application of new sensors and the continuous expansion of observation range, the categories of remote sensing data are gradually increasing. Traditional classification methods typically require retraining the model when adding new categories, which not only consumes a lot of time and resources but may also cause the model to forget previously learned knowledge, leading to catastrophic forgetting and significantly reducing overall classification performance. In light of this, incremental class learning provides an efficient solution, enabling the model to progressively learn new categories without accessing the original data, while retaining the ability to recognize old categories.

[0003] Many scholars have proposed deep learning-based methods for incremental learning of remote sensing image classes. These methods can be divided into four categories: knowledge distillation, knowledge replay, prototype representation, and dynamic model expansion. Knowledge distillation-based incremental learning methods typically employ distillation loss based on Kullback–Leibler (KL) divergence to transfer the predicted distributions of the old model on complex land cover types (such as buildings and farmland) to the new model, mitigating catastrophic forgetting during class expansion. Knowledge replay-based incremental learning methods use sample replay or feature replay strategies, combined with a small number of old class samples or intermediate features, to model and remember the distribution changes of high-resolution, multi-scale remote sensing image data. Prototype representation-based incremental learning methods utilize the construction of prototype representations and nearest neighbor matching mechanisms to classify based on feature centers of land cover types (such as lakes and roads), thereby enhancing robustness to high intra-class variability and low inter-class variability. Dynamic model expansion-based methods introduce dynamic classifier expansion or incremental subnetworks to dynamically increase model capacity when introducing new land cover types (such as new buildings or disaster areas), adapting to the challenges brought by the complexity of remote sensing image scenes and the continuous growth of categories.

[0004] In recent years, pre-trained models have demonstrated strong generalization capabilities, showing good representation of features from both new and old classes. However, in the field of remote sensing, due to the diverse types of remote sensing images, it is difficult to find pre-trained models with good generalization ability. Therefore, the trade-off between the stability and plasticity of incremental models remains a core issue. Furthermore, most remote sensing incremental learning tasks still focus on processing single-source data, which clearly cannot meet the dynamic and diverse needs of current remote sensing applications. At the same time, due to the significant heterogeneity of multi-source remote sensing data, the complementarity of features between different modalities is crucial for classification tasks. In each incremental training stage, the modal relationships between multimodal data are not entirely the same. The modal relationships of new class data may not be applicable to old class data. This leads to the inability to effectively utilize complementary features between modalities when classifying old data. Conflicts between new and old modal relationships may cause a decrease in fusion performance, affecting classification results.

[0005] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] The purpose of this disclosure is to provide a multi-source remote sensing image incremental learning method based on cue-based fine-tuning, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0008] According to embodiments of this disclosure, a multi-source remote sensing image class incremental learning method based on cue-based fine-tuning is provided, including: The constructed incremental learning network model includes a multimodal prompting learning network, a modality mapping network, and a knowledge aggregator. During the incremental learning phase, an independent cue pool is built for each modality, and each cue pool stores a set of learnable cue vectors; The Vision Transformer network is frozen, and only modality-specific cues are trained. Input sample features are used, and the N selected cue vectors most relevant to the sample features are dynamically selected from the cue pool through a cue query mechanism to obtain the original modality cue. In this process, learnable modality-shared global cues are added to each multi-head attention layer of the frozen Vision Transformer network. When a new task is performed, the network parameters of the modality mapping network for the old task are frozen, and a new trainable lightweight MLP is added to update the modality mapping network. Input multi-source data, use the updated modality mapping network to perform cross-modal transformation on modality-specific cues, and fuse the original modality cues and the cross-modal transformed cues to obtain modality fusion cues; where multi-source data includes new task data and old task model samples; Knowledge Aggregation Token for Freezing Old Tasks Initialize the knowledge aggregation token for the new task. Aggregate knowledge tokens from old tasks Knowledge aggregation tokens for new tasks The concatenated, joint modality fusion prompt serves as the Query input for the cross-attention mechanism of the knowledge aggregator. The image feature output of the old task is aggregated as the Key and Value input of the cross-attention mechanism to output the image feature of the new task. Finally, it is input into the incremental classifier to obtain the incremental classification result.

[0009] Furthermore, an independent cue pool is built for each modality:

[0010] in, To indicate the maximum capacity of the pool, For the first in the prompt pool A prompt vector, To indicate the length of the vector, This indicates the feature dimension of the hint vector.

[0011] Furthermore, the process of freezing the Vision Transformer network, training only modality-specific cues, and dynamically selecting N most relevant cue vectors from the cue pool through a cue query mechanism to obtain the original modality cue includes: Design an instance-based suggestion query mechanism that matches each suggestion vector with a series of learnable keys; Use a frozen Vision Transformer network as a query function to find one or more key-value pairs that are most relevant to the features of the current sample. The sample features are encoded into query features. By minimizing the cosine similarity distance between the key-value pairs and the query features, the N most relevant suggestion vectors are selected to obtain the selected suggestion vector. The calculation process, which minimizes the cosine similarity distance between key-value pairs and query features, is as follows:

[0012] in, For cosine similarity loss, For the selected k most similar hints, For sample features, These are the key values ​​corresponding to the top k most similar hints.

[0013] The selected cue vector is concatenated with the sample features and then input into the subsequent network.

[0014] Furthermore, learnable modality-sharing global cues are added to each multi-head attention layer of the frozen Vision Transformer network:

[0015] in, To add global suggestions, For each layer of the Vision Transformer network, add a cue prefix to the feature output. For modal sharing global hints, Output image features for each layer of the Vision Transformer network; In a multi-head self-attention layer, global cues are appended to the Query and Value, and the task-specific information contained in the global cues guides the model to learn feature representations.

[0016] in, This is a multi-head self-attention mechanism. The mapping of the query to the sample features. Mapping of sample features to the Key. The mapping of the value of the sample feature. For global hints attached to K, A global hint attached to V.

[0017] Furthermore, the steps of inputting multi-source data, using the updated modality mapping network to perform cross-modal transformation on modality-specific cues, and fusing the original modality cues and the cross-modal transformed cues to obtain modality-fused cues include: Initialize an independent cue pool for multi-source data of each modality; Bidirectional conversion of cross-modal cues is achieved using an updated modality mapping network; By introducing a maximum mutual information constraint, the complementarity between the transformed prompt and the original prompt is maximized to obtain a modal fusion prompt.

[0018] Furthermore, the steps of freezing the knowledge aggregation token of the old task, initializing the knowledge aggregation token of the new task, concatenating the knowledge aggregation tokens of the old and new tasks, using the joint modality fusion prompt as the query input of the cross-attention mechanism of the knowledge aggregator, aggregating the image feature output of the old task as the key and value input of the cross-attention mechanism, and outputting the image features of the new task include: Freeze the knowledge aggregation token of the modality mapping network for old tasks; Based on the shared learnable tokens and modality fusion hints of the updated modality mapping network, a knowledge aggregation token for the new task is generated and used as the query input for the cross-attention mechanism. The image features output from the old task are used as the key and value inputs of the cross-attention mechanism and aggregated to obtain the image features for the new task.

[0019] Furthermore, the method also includes: In each incremental learning phase, features extracted from multi-source data of the new task and model samples of the old task are loaded into the classification network, and the predicted classification result is output; among them, the classification loss is calculated using cross-entropy loss:

[0020] in, Cross-entropy classification loss, For category labels, For the predicted results; An auxiliary classifier is used to assign different label values ​​to the multi-source data of the new task and the model samples of the old task to perform an auxiliary binary classification task:

[0021] in, To assist in binary classification labels, This is currently multi-source data that includes new task multi-source data and old task model samples.

[0022] Furthermore, the overall loss is:

[0023] in, For maximum mutual information constraints, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient.

[0024] Furthermore, the training process uses the Adam optimizer with a batch size of 32, an initial learning rate of 0.001, 50 training rounds per stage, and saves the optimal parameters at the end.

[0025] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the above-described incremental learning method for multi-source remote sensing images based on cue fine-tuning not only dynamically generates modality-specific cues to distinguish between new and old tasks, but also preserves cross-modal transformation relationships from previous tasks, thereby enhancing the complementary ability of multimodal knowledge in the continuous learning model. Furthermore, by using the fused multimodal cues as query labels, the most discriminative feature portions of the feature map are selectively focused on, effectively improving the model's plasticity. Additionally, the auxiliary classifier assigns different labels only to new and old tasks, performing binary classification to maximize the discriminative power of features between new and old samples. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0027] Figure 1 This diagram illustrates the steps of a multi-source remote sensing image class incremental learning method based on cue-based fine-tuning in an exemplary embodiment of this disclosure. Figure 2 This diagram illustrates a specific flowchart of the incremental learning method for multi-source remote sensing image classes based on prompting fine-tuning in an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates the prompting transition performed by the prompting fine-tuning module in an exemplary embodiment of this disclosure; Figure 4 A comparison diagram showing the results of the comparison method and the method of this application in an exemplary embodiment of this disclosure is presented. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0030] This example implementation provides a multi-source remote sensing image class incremental learning method based on cue-based fine-tuning. (Reference) Figure 1 As shown, this cue-based fine-tuning multi-source remote sensing image class incremental learning method may include: Step S101: Constructing an incremental learning network model; wherein, the incremental learning network model includes a multimodal cue learning network, a modality mapping network, and a knowledge aggregator; Step S102: In the incremental learning phase, an independent cue pool is constructed for each modality, and each cue pool stores a set of learnable cue vectors; Step S103: Freeze the Vision Transformer network and train only modality-specific cues. Input sample features and dynamically select N selected cue vectors most relevant to the sample features from the cue pool through a cue query mechanism to obtain the original modality cue. Among them, learnable modality-shared global cues are added to each multi-head attention layer of the frozen Vision Transformer network. Step S104: When performing a new task, freeze the network parameters of the modality mapping network for the old task and add a new trainable lightweight MLP to update the modality mapping network. Step S105: Input multi-source data, use the updated modality mapping network to perform cross-modal transformation on modality-specific cues, and fuse the original modality cues and the cross-modal transformed cues to obtain modality fusion cues; wherein, the multi-source data includes new task data and old task model samples; Step S106: Freeze the knowledge aggregation token of the old task, initialize the knowledge aggregation token of the new task, concatenate the knowledge aggregation tokens of the old task and the new task, use the joint modality fusion prompt as the Query input of the cross-attention mechanism of the knowledge aggregator, use the image feature output of the old task as the Key and Value input of the cross-attention mechanism for aggregation, output the image features of the new task, and finally input them into the incremental classifier to obtain the incremental classification result.

[0031] The aforementioned incremental learning method for multi-source remote sensing images based on cue fine-tuning not only dynamically generates modality-specific cues to distinguish between new and old tasks, but also preserves cross-modal transformation relationships from previous tasks, thereby enhancing the complementary ability of multimodal knowledge in the continuous learning model. Furthermore, by using the fused multimodal cues as query labels, the most discriminative feature portions of the feature map are selectively focused on, effectively improving the model's plasticity. Additionally, the auxiliary classifier assigns different labels only to new and old tasks, performing binary classification to maximize the discriminative power of features between new and old samples.

[0032] Below, we will refer to Figures 1 to 4 The steps of the above-described incremental learning method for multi-source remote sensing image classes based on cue-based fine-tuning in this example embodiment will be described in more detail.

[0033] In step S101, an incremental learning network model is constructed; wherein, the incremental learning network model includes a multimodal cue learning network, a modality mapping network, and a knowledge aggregator.

[0034] like Figure 2 The diagram shows a flowchart illustrating the process of incremental learning for multi-source remote sensing images using a cue-based fine-tuning network model.

[0035] In steps S102 and S103, during the incremental learning phase, an independent cue pool is constructed for each modality, and each cue pool stores a set of learnable cue vectors. The Vision Transformer network is frozen, and only modality-specific cues are trained. Input sample features are used, and the N selected cue vectors most relevant to the sample features are dynamically selected from the cue pool through a cue query mechanism to obtain the original modality cue. In this process, learnable modality-shared global cues are added to each multi-head attention layer of the frozen Vision Transformer network.

[0036] Specifically, in the incremental learning phase, a modality-specific cue pool is constructed, and an instance-based cue query mechanism is designed to dynamically select appropriate cuees for different input features. The Vision Transformer network from the old task is frozen, and only cuees specific to modality features are trained.

[0037] More specifically, a modality-specific cue pool is first used to store a series of cue vectors, defined as:

[0038] in, This indicates the maximum capacity of the prompt pool. Indicates the first in the prompt pool A prompt vector, It indicates the length of the hint vector. This is the feature dimension of the cue vector. The cue pool encourages the transfer and sharing of modality-specific knowledge across different incremental tasks.

[0039] Then, an instance-based suggestion query mechanism is designed, which matches each suggestion with a series of learnable keys, as shown below:

[0040] Use a query function Perform query-key matching to find one or more key-value pairs that are most relevant to the features of the current sample. Query function It is a deterministic function. Here, the VisionTransformer network frozen from the old task is used as the query function. It encodes each block of the input sample into the same feature dimension as the key. The input features are represented as:

[0041] Next, an optimization objective is set to identify the top k hints that are most similar to the features of the current sample. Cosine distance is used to measure the similarity between the key and the query, and the cosine distance between the key and the query is minimized.

[0042] Based on the above formula, the top N keys most relevant to the current sample features can be identified. Then, these N cue vectors are concatenated with the sample features and fed into the subsequent network for processing.

[0043] Then, learnable modality-sharing global cues are added to each attention layer of the frozen Vision Transformer network to extract task-specific modality common information.

[0044] Introducing learnable inter-modal shared global cues It is attached to the multi-head attention layer in each layer:

[0045] in Defined as a method to attach global hints.

[0046] In a multi-head self-attention layer, global cues are appended to the key and value. Based on the task-specific information contained in the global cues, the model is guided to learn more effective feature representations. The specific formula is as follows:

[0047] in This indicates a multi-head self-attention mechanism. , , These are the mappings of query, key, and value for sample features.

[0048] In steps S104 and S105, when a new task is performed, the network parameters of the modality mapping network for the old task are frozen, and a new trainable lightweight MLP is added to update the modality mapping network. Multi-source data is input, and the updated modality mapping network is used to perform cross-modal transformation on modality-specific cues. The original modality cues and the cross-modal transformed cues are then fused to obtain a modality-fused cues. The multi-source data includes new task data and old task model samples.

[0049] Specifically, a cue fine-tuning module (i.e., a modal mapping network) is designed to perform cross-modal cue conversion and fusion by constraining the mutual information of modal-specific cues. For example... Figure 3 The image shows a schematic diagram illustrating the prompt transition performed by the prompt fine-tuning module.

[0050] More specifically, a modality-specific cue pool is initialized for each type of data. Taking hyperspectral data and LiDAR data as examples, the two modality cue pools are defined as follows:

[0051] To enhance cross-modal cue interaction, a lightweight modality mapping network is introduced to achieve bidirectional transformation of modality-specific cues between two modalities. This transformation can be represented as:

[0052] in This indicates that the hyperspectral modal cue will be converted to a LiDAR modal cue. Indicates the first Each incremental learning phase. A constraint based on maximum mutual information is introduced to ensure that the transformed prompts have maximum complementarity. The formula for calculating the maximum mutual information is as follows:

[0053] in It is a neural network that provides The approximate probability distribution is calculated using the maximum mutual information formula, which is then optimized during training as part of the overall network loss function.

[0054] Design a task-specific modality cueing relationship preservation strategy, which preserves the modality transformation relationships of old tasks by freezing part of the modality mapping network.

[0055] When a new incremental task arrives, the modality mapping network parameters of the old task are frozen, and a new trainable lightweight network is initialized to learn the modality transformation relationship of the new task to extend the original network. This application uses a three-layer MLP (Multilayer Perceptron) as the trainable lightweight mapping network for each incremental stage, as shown in the following formula:

[0056] in Indicates the first The extended lightweight modality mapping network of MLP in each incremental task, network parameters It is frozen during updates; only the newly added modality mapping network will be trained in incremental tasks.

[0057] In step S106, the knowledge aggregation token of the old task is frozen. Initialize the knowledge aggregation token for the new task. Aggregate knowledge tokens from old tasks Knowledge aggregation tokens for new tasks The concatenated, joint modality fusion prompt serves as the Query input for the cross-attention mechanism of the knowledge aggregator. The image feature output of the old task is aggregated as the Key and Value input of the cross-attention mechanism to output the image feature of the new task. Finally, it is input into the incremental classifier to obtain the incremental classification result.

[0058] Specifically, a prompt-guided knowledge aggregator module is established, the knowledge aggregator tokens in the old task network are frozen, and the knowledge aggregator tokens of the new task and modality-specific prompts are used together to guide the aggregator module to learn image features and gradually learn the key information in the feature map.

[0059] The design incorporates a knowledge aggregation module guided by prompts. Simultaneously, for each incremental task, a new set of task-specific shared learnable tokens is initialized and input along with the prompt vector into the cross-attention mechanism as query input. Upon the arrival of each new incremental task, the shared tokens from the previous stage are frozen, ensuring that information from the old task can be transferred to the new task. Let the image features output from the previous stage be... The final feature output after passing through the knowledge aggregator is So in the first The image representation output of the knowledge aggregation module in each incremental stage is:

[0060] in, , This is a modal fusion hint from the previous level. The previous stage, These are the mapping weights for queries, keys, and values, respectively.

[0061] In addition, a multi-source data joint incremental learning model is built using multiple designed modules. The network model is trained to obtain network parameters suitable for the model, thereby achieving incremental classification.

[0062] In each incremental learning phase, multi-source data for the new task and a very small number of model samples from the old task are loaded into the classification network to be trained, and the predicted classification results are output. During testing, the model's performance is evaluated against samples from all known classes. The classification loss is calculated using cross-entropy loss, and the calculation process is as follows:

[0063] in It's a category label. This is the prediction result. We also need to constrain the similarity between the prompts in the prompt pool and the sample features, i.e., minimize the cosine distance. The calculation process is as follows:

[0064] in These are the selected k most similar cues. Maximum mutual information constraint during training:

[0065] In addition, to better distinguish between the features of new and old tasks, an auxiliary classifier is introduced. This classifier assigns different label values ​​to samples from both tasks, performing an auxiliary binary classification task. The calculation process is as follows:

[0066] in, , The current training set contains both new data and some old data. Ultimately, the overall loss of the incremental model is expressed as:

[0067] in , , These are weighting coefficients; During training in each incremental phase, the old sample storage space is dynamically updated. The most representative old model sample is selected based on the mean of sample features and added to the current old sample storage space. The Adam optimizer is used to update the network parameters until the network converges. The batch size is set to 32, and the number of training epochs per incremental phase is set to 50. The learning rate is initialized to 0.001. After training in each incremental phase, the optimal network parameters for that phase are saved.

[0068] In a specific embodiment, the technical effects of this application are explained in detail with reference to simulation experiments: Simulation experimental conditions The hardware platform for the simulation experiment in this application is: Nvidia GeForce RTX 3090, Intel(R) Core(TM) i9-10900X CPU@3.70GHz.

[0069] The software platform for the simulation experiment in this application is: Linux 18.04 operating system, Python 3.7, PyTorch 1.13.1 and DGL 2.0.0.

[0070] The hyperspectral and lidar data used in the simulation experiments of this application are from the Houston2013 dataset. This dataset was collected by the National Airborne Laser Mapping Center (CASI) using a Compact Airborne Spectrographic Imager (CASI) in the University of Houston campus and surrounding urban areas. The dataset consists of hyperspectral images and digital surface models (DSMs) based on lidar data, both with a size of 349 × 1905 pixels and a spatial resolution of 2.5 meters. The hyperspectral images contain 144 spectral bands, with wavelengths ranging from 0.38 to 1.05 μm. The dataset contains 15,029 ground condition samples, categorized into 15 classes, including natural objects (e.g., water, soil, trees, and grass) and man-made objects (e.g., parking lots, railways, highways, and roads).

[0071] Evaluation indicators In the experiments, overall accuracy (OA), average accuracy (aa), and kappa coefficient were used to evaluate model performance. These metrics are widely used in the field of hyperspectral image and LiDAR data fusion classification, and help to comprehensively evaluate the network model from an objective perspective. In addition, average performance decrease was used to measure the average degree of forgetting at each incremental stage, defined as follows:

[0072] in and Let N represent the accuracy of category i in incremental tasks T and T-1, and let N represent the total number of categories currently known.

[0073] Experimental content and results analysis To verify the validity of this application, five common incremental learning methods were selected: iCarl, Podnet, WA, Foster, and Beef.

[0074] Table 1. Performance Indicators Comparison Between This Application and Comparative Algorithms

[0075] The objective classification results of different methods are shown in Table 1. As can be seen from the comparison results in Table 1, the performance of this application on the Houston 2013 dataset is significantly better than the five comparison methods. As incremental learning continues, the performance of each model will decrease to some extent. However, the anti-forgetting ability of this application has a significant advantage over other methods. In the last incremental learning stage, the OA, AA, and Kappa coefficients reached 77.39%, 82.08%, and 75.56%, respectively. In contrast, the best performing method among the five comparison methods, Foster, achieved OA, AA, and Kappa coefficients of 67.46%, 71.85%, and 64.89% in the last incremental learning stage, which is significantly inferior to this application.

[0076] like Figure 4 The figures shown are comparison diagrams of the results of the comparative method and the method of this application; where (a) is a ground condition diagram; (b) is a result diagram of the iCarl method; (c) is a result diagram of the Podnet method; (d) is a result diagram of the WA method; (e) is a result diagram of the Foster method; (f) is a result diagram of the Beef method; and (g) is a result diagram of the method of this application.

[0077] The experimental results in Table 1 verify the effectiveness of the method proposed in this application. Figure 4 The same conclusion can be drawn from this.

[0078] The technical effects of this application will be described in detail below with reference to experiments.

[0079] To demonstrate the effectiveness of the cross-modal prompting fine-tuning module designed in this application, this experiment uses only single-modal prompts for fine-tuning, rather than cross-modal prompting interaction. The experimental results are as follows: Table 2 Comparison of performance metrics between this application and applications using only a single modal cue.

[0080] The experimental results are shown in Table 2. Compared with the prompts using only a single modality, the final performance of the prompts using a fused multimodal modality is worse, regardless of the modality used. This proves the effectiveness of the cross-modal interactive prompt module in this application.

[0081] The aforementioned incremental learning method for multi-source remote sensing images based on cue fine-tuning not only dynamically generates modality-specific cues to distinguish between new and old tasks, but also preserves cross-modal transformation relationships from previous tasks, thereby enhancing the complementary ability of multimodal knowledge in the continuous learning model. Furthermore, by using the fused multimodal cues as query labels, the most discriminative feature portions of the feature map are selectively focused on, effectively improving the model's plasticity. Additionally, the auxiliary classifier assigns different labels only to new and old tasks, performing binary classification to maximize the discriminative power of features between new and old samples.

[0082] It should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., in the above description indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this disclosure.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0084] In the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0085] In embodiments of this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0087] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A multi-source remote sensing image class incremental learning method based on cue-based fine-tuning, characterized in that, The method includes: The constructed incremental learning network model includes a multimodal prompting learning network, a modality mapping network, and a knowledge aggregator. During the incremental learning phase, an independent cue pool is built for each modality, and each cue pool stores a set of learnable cue vectors; The Vision Transformer network is frozen, and only modality-specific cues are trained. Input sample features are used, and the N selected cue vectors most relevant to the sample features are dynamically selected from the cue pool through a cue query mechanism to obtain the original modality cue. In this process, learnable modality-shared global cues are added to each multi-head attention layer of the frozen Vision Transformer network. When a new task is performed, the network parameters of the modality mapping network for the old task are frozen, and a new trainable lightweight MLP is added to update the modality mapping network. Input multi-source data, use the updated modality mapping network to perform cross-modal transformation on modality-specific cues, and fuse the original modality cues and the cross-modal transformed cues to obtain modality fusion cues; where multi-source data includes new task data and old task model samples; Freeze the knowledge aggregation token of the old task, initialize the knowledge aggregation token of the new task, concatenate the knowledge aggregation tokens of the old task and the knowledge aggregation token of the new task, use the joint modality fusion prompt as the Query input of the cross-attention mechanism of the knowledge aggregator, use the image feature output of the old task as the Key and Value input of the cross-attention mechanism for aggregation, output the image features of the new task, and finally input them into the incremental classifier to obtain the incremental classification result.

2. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 1, characterized in that, Build an independent cue pool for each modality: in, To indicate the maximum capacity of the pool, For the first in the prompt pool A prompt vector, To indicate the length of the vector, This indicates the feature dimension of the hint vector.

3. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 2, characterized in that, The steps of freezing the Vision Transformer network, training only modality-specific cues, taking input sample features, and dynamically selecting N most relevant cue vectors from the cue pool through a cue query mechanism to obtain the original modality cue include: Design an instance-based suggestion query mechanism that matches each suggestion vector with a series of learnable keys; Use a frozen Vision Transformer network as a query function to find one or more key-value pairs that are most relevant to the features of the current sample. The sample features are encoded into query features. By minimizing the cosine similarity distance between the key-value pairs and the query features, the N most relevant suggestion vectors are selected to obtain the selected suggestion vector. The calculation process, which minimizes the cosine similarity distance between key-value pairs and query features, is as follows: in, For cosine similarity loss, For the selected k most similar hints, For sample features, These are the key values ​​corresponding to the top k most similar hints. The selected cue vector is concatenated with the sample features and then input into the subsequent network.

4. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 3, characterized in that, Add learnable modality-sharing global cues to each multi-head attention layer of the frozen Vision Transformer network: in, To add global suggestions, For each layer of the Vision Transformer network, add a cue prefix to the feature output. For modal sharing global hints, Output image features for each layer of the Vision Transformer network; In a multi-head self-attention layer, global cues are appended to the Query and Value, and the task-specific information contained in the global cues guides the model to learn feature representations. in, This is a multi-head self-attention mechanism. The mapping of the query to the sample features. Mapping of sample features to the Key. The mapping of the value of the sample feature. For global hints attached to K, A global hint attached to V.

5. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 4, characterized in that, The steps of inputting multi-source data, using an updated modality mapping network to perform cross-modal transformation on modality-specific cues, and fusing the original modality cues and the cross-modal transformed cues to obtain modality-fused cues include: Initialize an independent cue pool for multi-source data of each modality; Bidirectional conversion of cross-modal cues is achieved using an updated modality mapping network; By introducing a maximum mutual information constraint, the complementarity between the transformed prompt and the original prompt is maximized to obtain a modal fusion prompt.

6. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 5, characterized in that, The steps of freezing the knowledge aggregation token for the old task, initializing the knowledge aggregation token for the new task, concatenating the knowledge aggregation tokens for the old and new tasks, using joint modal fusion prompts as the query input for the cross-attention mechanism of the knowledge aggregator, aggregating the image feature output of the old task as the key and value input for the cross-attention mechanism, and outputting the image features of the new task include: Freeze the knowledge aggregation token of the modality mapping network for old tasks; Based on the shared learnable tokens and modality fusion hints of the updated modality mapping network, a knowledge aggregation token for the new task is generated and used as the query input for the cross-attention mechanism. The image features output from the old task are used as the key and value inputs of the cross-attention mechanism and aggregated to obtain the image features for the new task.

7. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 6, characterized in that, The method also includes: In each incremental learning phase, features extracted from multi-source data of the new task and model samples of the old task are loaded into the classification network, and the predicted classification result is output; among them, the classification loss is calculated using cross-entropy loss: in, Cross-entropy classification loss, For category labels, For the predicted results; An auxiliary classifier is used to assign different label values ​​to the multi-source data of the new task and the model samples of the old task to perform an auxiliary binary classification task: in, To assist in binary classification labels, This is currently multi-source data that includes new task multi-source data and old task model samples.

8. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 7, characterized in that, The overall loss is: in, For maximum mutual information constraints, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient.

9. The multi-source remote sensing image class incremental learning method based on cue-based fine-tuning according to claim 7, characterized in that, The training process uses the Adam optimizer with a batch size of 32, an initial learning rate of 0.001, and 50 training iterations per stage. The optimal parameters are saved at the end of the training.

Images