An online incremental learning method for large remote sensing image models based on ViT

Abstract

The present invention discloses an online incremental learning method for a large remote sensing image model based on ViT, which belongs to the field of image processing technology. The method comprises the following steps: obtaining a data set; constructing a dual-branch codec network, comprising a spatial-spectral dual-branch encoder, a hybrid connection encoder HCE, and a hybrid connection decoder HCD; constructing a loss function L of the dual-branch codec network, training the network to convergence by minimizing L using a remote sensing image data set D, obtaining a first basic model, and freezing its parameters; constructing a second basic model based on the first basic model and a classifier, and performing online incremental learning using the second basic model. The present invention can achieve deep interaction and enhancement of cross-modal features; and performs online incremental learning and network parameter updates based on the LoRA method, and monitors the loss changes of online classification tasks in real time to freeze and merge LoRA parameters. A regularization term is introduced into the LoRA module to avoid catastrophic forgetting, thereby achieving efficient, scalable, and task-independent online incremental learning.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an online incremental learning method for a large remote sensing image model based on ViT. Background Art

[0002] With the rapid development of remote sensing technology, the continuous acquisition of massive, dynamic remote sensing image data has provided unprecedented opportunities for refined Earth observation. Large-scale pre-trained models such as ViT (Vision Transformer) have shown great potential in remote sensing image interpretation tasks due to their powerful feature extraction capabilities. However, in practical applications, remote sensing data often arrives in the form of a continuous data stream, which may contain new ground object categories, changes in sensor characteristics, or data distribution drift caused by different phases and regions. This requires the model to have online incremental learning capabilities, that is, to continuously learn from new data without accessing historical data, while not forgetting the learned knowledge. Direct online fine-tuning of large ViT models often leads to catastrophic forgetting, which severely limits its application in dynamic remote sensing scenarios.

[0003] Existing online incremental learning methods primarily include strategies based on experience replay, regularization, and parameter isolation. Experience replay methods require the storage of some historical data, which leads to storage overhead and data privacy issues, and their effectiveness is limited when the data distribution changes dramatically. Regularization-based methods mitigate forgetting by penalizing modifications to important parameters of old tasks. However, the overhead of calculating global parameter importance on large ViT models is significant, making them difficult to adapt to the demands of online real-time updates. Parameter isolation-based methods typically require explicit task boundary information, which is often inapplicable in online remote sensing data streams where task boundaries are ambiguous or unknown. Therefore, there is an urgent need to design an efficient, scalable, and task-independent online incremental learning method for large remote sensing ViT models.

[0004] ViT is an encoder-decoder architecture. The encoder is a Transformer encoder, consisting of multiple stacked Transformer layers. Each Transformer layer includes multi-head self-attention and a feed-forward network. ViT can be used with decoders for a variety of different functions, such as image classification and data reconstruction. Its main function is to decode the encoder output. The Transformer decoder is a common decoder, also consisting of multiple stacked Transformer layers.

[0005] InfoNCE loss (Information Noise Contrastive Estimation) is a contrastive learning loss function widely used in unsupervised and self-supervised learning tasks. Its goal is to learn an effective representation of the data by reducing the distance between positive pairs of examples while increasing the distance between negative pairs of examples. The core idea of the symmetric InfoNCE loss function is to symmetrize the InfoNCE loss function, that is, to consider similarity in both directions simultaneously.

[0006] LoRA (Low-Rank Adaptation) is an algorithm for efficient fine-tuning based on pre-trained models. Its core is to perform low-rank decomposition of the model parameter update ΔW into two low-rank matrices A and B. Model fine-tuning can be achieved by adjusting only A and B, thereby significantly reducing the computational cost and storage requirements of fine-tuning while maintaining the original capabilities of the pre-trained model. Summary of the Invention

[0007] The purpose of the present invention is to provide an online incremental learning method for a large remote sensing image model based on ViT, which solves the problem of catastrophic forgetting in the above-mentioned ViT large model online incremental learning scenario.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows: an online incremental learning method of a large remote sensing image model based on ViT, comprising the following steps:

[0009] S1, obtain remote sensing image dataset D, D={X1,X2,…,X i ,…,X N}, X i is the i-th image block labeled with category in D, , C is the number of spectral channels, P×P is the image block size, and N is the total number of image blocks;

[0010] S2, constructing a dual-branch encoding and decoding network, including a spatial-spectral dual-branch encoder, a hybrid connection encoder HCE and a hybrid connection decoder HCD, the construction method including S21 to S23;

[0011] S21, constructing a spatial-spectral dual-branch encoder, including a spatial encoder and a spectral encoder;

[0012] The spatial encoder is used to i Perform random masking and spatial encoding to obtain spatial features e i,s , the spectral encoder is used to i Perform random masking and spectral encoding to obtain the spectral feature e i,c ;

[0013] S22, constructing a hybrid connection encoder HCE;

[0014] Get two Transformer encoders with the same structure, each consisting of Z stacked Transformer layers;

[0015] Connect the input of a Transformer encoder to the output of the spatial encoder, where the zth Transformer layer is marked as E sz , the other Transformer encoder input is connected to the output of the spectral encoder, where the zth Transformer layer is marked as E cz , 1≤z≤Z;

[0016] E sz The multi-head self-attention layer in the middle is modified to the first hybrid connection attention module HCA1 to obtain the spatial encoding branch, and E cz The multi-head self-attention layer is modified to the second hybrid connection attention module HCA2 to obtain the spectral encoding branch, and the two encoding branches constitute a hybrid connection encoder;

[0017] The output features HCA(c, s) of HCA1 and HCA(s, c) of HCA2 are obtained according to the following formulas:

[0018] HCA(c,s)=Attention(Q c ,K c ,V c )+Attention(Q c ,K s ,V s ),

[0019] HCA(s,c)=Attention(Q s ,K s ,V s )+Attention(Q s ,K c ,V c ),

[0020] Where Attention(∙) is the self-attention operation, Q c , K c 、V c E cz The Q matrix, K matrix and V matrix in Q s , K s 、V s E sz The Q matrix, K matrix and V matrix in;

[0021] S23, construct a hybrid connection decoder HCD, including a spatial decoding branch and a spectral decoding branch. The spatial decoding branch is used to predict the mask area of the spatial encoder to obtain a spatial reconstruction image The spectrum decoding branch is used to predict the mask area of the spectrum encoder to obtain the spectrum reconstruction map ;

[0022] S3, construct the loss function L of the dual-branch encoder-decoder network, use the remote sensing image dataset D to minimize L and train until convergence, obtain the first basic model and freeze its parameters;

[0023] S4, deleting HCD from the first basic model, adding the two outputs of HCE and feeding them into a classifier to obtain a second basic model, wherein the second basic model is used to input the image block and output the corresponding classification result;

[0024] S5, online incremental learning of the second base model;

[0025] S51, introducing a LoRA module into each of the spatial coding branch and the spectral coding branch of the second basic model;

[0026] S52, using the second basic model to online detect the data stream composed of image blocks, using the data stream to update the LoRA module parameters based on the LoRA method, and detecting whether the data distribution of the data stream has changed;

[0027] S53: If changed, the LoRA module parameters are incorporated into the second basic model and S51 to S52 are repeated; otherwise, no processing is performed.

[0028] Preferably, in S23, the structure of the hybrid connection decoder HCD is:

[0029] Get two Transformer decoders with the same structure, each consisting of Z stacked Transformer layers;

[0030] Connect the input of a Transformer decoder to the output of the spatial encoding branch, where the zth Transformer layer is marked as D sz , the other Transformer decoder input is connected to the output of the spectral encoding branch, where the zth Transformer layer is marked as D cz ;

[0031] D sz The multi-head self-attention layer in the middle is modified to the third hybrid connection attention module HCA3 to obtain the spatial decoding branch, and D czThe multi-head self-attention layer is modified to the fourth hybrid connection attention module HCA4 to obtain the spectral decoding branch. The two decoding branches constitute a hybrid connection decoder, and the spatial decoder is used to predict the mask area of the spatial encoder to obtain the spatial reconstruction map. , the spectral decoder is used to predict the mask area of the spectral encoder to obtain the spectral reconstruction image ;

[0032] The output features HCA(sD, sE) of HCA3 and HCA(cD, cE) of HCA4 are obtained according to the following formulas:

[0033] HCA(sD,sE)=Attention(Q sD ,K sD ,V sD )+Attention(Q sD ,K sE ,V sE ),

[0034] HCA(cD,cE)=Attention(Q cD ,K cD ,V cD )+Attention(Q cD ,K cE ,V cE ),

[0035] Where Q sD , K sD 、V sD D sz The Q matrix, K matrix and V matrix in sE 、V sE E sz The Q matrix and K matrix in Q cD , K cD 、V cD D cz The Q matrix, K matrix and V matrix in cE 、V cE E cz The Q matrix and K matrix in .

[0036] Preferably, in S3, the loss function L is calculated according to the following formula:

[0037] ,

[0038] 、 are spatial reconstruction loss and spectral reconstruction loss respectively, L SSCis the symmetric InfoNCE loss between spatial features and spectral features, and λ is L SSC The weight of .

[0039] As a preference,

[0040] ,

[0041] ,

[0042] Where, |M s | is the mask region M of the spatial encoder s Total number of pixels in X s (p) and X i and The pixel value at position p in |M c | is the mask area M of the spectral encoder c Total number of pixels in X c (p) and X i and Spectral vector of position p in ;

[0043] L SSC According to the following formula:

[0044] ,

[0045] Where sim(∙,∙) is the cosine similarity, exp(∙) is the exponential function, τ is the temperature hyperparameter, and e q,s 、e q,c are the spatial features and spectral features corresponding to the qth image block in the remote sensing image dataset D, respectively.

[0046] Preferably, S51 is specifically to select one layer of Q matrix, V matrix and K matrix in the spatial coding branch, introduce the LoRA module, and select one layer of Q matrix, V matrix and K matrix in the spectral coding branch, and introduce the LoRA module.

[0047] Preferably, in S52, detecting whether the data distribution of the data stream has changed specifically comprises:

[0048] Sequentially divide the data stream into multiple batches, calculate the classification loss value of each batch, and sample the batches using a sliding window. If the current t-th batch satisfies the following formula, the data distribution of the t-th batch is judged to have changed;

[0049] ,

[0050] Where k is the sliding window size, l j is the classification loss value of the jth batch of data stream.

[0051] Preferably, the spatial encoder includes a spatial mask layer, a first normalization layer, a two-dimensional convolution layer, an attention layer, a first addition layer, a second normalization layer, a multi-layer perceptron, and a second addition layer connected in sequence, and the output of the spatial mask layer is jump-connected to the first addition layer, and the output of the first addition layer is jump-connected to the output of the second addition layer;

[0052] The spatial mask layer is used to randomly i Apply a C×1×1 mask to the pixels in the image to get the first spatial feature , and only keep the annotations of the unmasked areas;

[0053] The second spatial feature is obtained by the first normalization layer, the two-dimensional convolution layer, and the attention layer. ,

[0054] described After the first addition layer and Add element by element to get the third space feature ;

[0055] described The fourth spatial feature is obtained by the second normalization layer and the multi-layer perceptron. ;

[0056] described After the second addition layer and Add element by element to get the spatial feature e i,s。

[0057] Preferably, the attention layer is a self-attention layer or a spatial attention layer;

[0058] The spatial attention layer performs spatial attention operations according to the following formula;

[0059] ,

[0060] Where Attention1(Q,K,V) is the spatial attention operation, Softmax(∙) is the Softmax function, Q, K, and V are the Q matrix, K matrix, and V matrix obtained by mapping the output of the two-dimensional convolutional layer; d k is the scaling factor.

[0061] Preferably, the spectral encoder includes a spectral mask layer, a first normalization layer, a linear projection layer, an attention layer, a first addition layer, a second normalization layer, a multilayer perceptron, and a second addition layer connected in sequence, and the output of the spatial mask layer is jump-connected to the first addition layer, and the output of the first addition layer is jump-connected to the output of the second addition layer;

[0062] The spectral mask layer is used to randomlyi Apply a 1×P×P mask to the band to obtain the first spectral feature , and only retain the unmasked band information;

[0063] The second spectral feature is obtained by the first normalization layer, linear projection layer, and attention layer in sequence. ,

[0064] described After the first addition layer and Add element by element to get the third spectral feature ;

[0065] described The fourth spectral feature is obtained by the second normalization layer and the multi-layer perceptron. ;

[0066] described After the second addition layer and The spectral characteristics e are obtained by adding the elements i,c。

[0067] Preferably, the optimization method of the LoRA module is:

[0068] Sa1, decompose the LoRA module into two low-rank matrices A and B;

[0069] Sa2, maintains a hard buffer with a capacity of 4, which is used to store the four image blocks with the highest classification loss values;

[0070] Sa3, calculate the parameter importance Ω of the low-rank matrices A and B according to the following formula A and Ω B ;

[0071] ,

[0072] ,

[0073] Where x k is the kth image block in the hard buffer, 1≤k≤4, θ is the network parameter of the current second basic model, p(x k |θ) is the current second basic model for x k The classification result, ∇ is the gradient operation, ∘ is the element-by-element product operation, is the weight matrix W of A A The element in row i and column j in Ω A The element in row i and column j, is the weight matrix W of B B The element in row i and column j, Ω BThe element in row i and column j;

[0074] Sa4, calculate the regularization value r and regularization term L of the current LoRA module according to the following formula LoRA ;

[0075] ,

[0076] ,

[0077] Where R is the set of regularization values of all current LoRA modules, and λ is the regularization coefficient;

[0078] Sa5, construct the loss function L1 of the LoRA module and optimize the parameters of the LoRA module by minimizing L1;

[0079] ;

[0080] L cls (,∙,) is the classification loss, F(X;θ) is the predicted classification result of the current second basic model for a batch of image blocks, Y is the true category label of the batch of samples, F(X B ;θ) is the classification result of the current second basic model for the samples in the hard buffer, Y B is the true category label of the image block in the hard buffer.

[0081] The idea of the present invention is to first construct a space-spectrum dual-branch encoder, a hybrid connection encoder HCE and a hybrid connection decoder HCD to form a dual-branch encoding and decoding network, and train the dual-branch encoding and decoding network to obtain a first basic model such as Figure 1 As shown. A second basic model is constructed based on the first basic model and the classifier for classifying and predicting image blocks in online data streams. Two LoRA modules are introduced into the second basic model. The parameters of the LoRA modules are continuously trained using the online input data stream until a change in the data distribution of the data stream is detected. The parameters of the LoRA modules are then incorporated into the network parameters of the second basic model. Two more LoRA modules are introduced and learning continues until a change in the data distribution of the data stream is detected again.

[0082] Regarding the spatial-spectral dual-branch encoder: It is used to fully capture the inherent spatial contextual information and fine-grained spectral discriminative features of image patches. The spatial encoder and spectral encoder extract features from image patches in the spatial and spectral dimensions, respectively, and enhance their ability to perceive key features through a self-masking pre-training strategy. To achieve more powerful feature representation, the attention mechanism used in the spatial encoder and spectral encoder can adopt a multi-head attention mechanism, splitting the Q, K, and V matrices into multiple independent attention heads along the feature channel dimension. This allows for the parallel learning of different attention modes and captures richer feature interaction information.

[0083] About the hybrid connection encoder HCE: It includes a spatial coding branch and a spectral coding branch. Both are based on the basic structure of the ViT encoder, both belong to the Transformer encoder, and are stacked with Z layers of Transformer layers. The Transformer layer itself includes a multi-head self-attention layer, and its input is the Q matrix, K matrix and V matrix generated by the previous stage output mapping. Since the present invention is a dual-branch structure, the multi-head self-attention layer in the Transformer layer is modified to a hybrid connection attention module. In the present invention, the multi-head self-attention layer in the spatial coding branch is replaced by the first hybrid connection attention module HCA1, and the multi-head self-attention layer in the spectral coding branch is modified to the second hybrid connection attention module HCA2. According to the formula HCA(c,s)=Attention(Q c ,K c ,V c )+Attention(Q c ,K s ,V s ) It can be seen that the input of HCA1 is 5, namely Q s , K s 、V s , K c 、V c The first three come from the spectral encoding branch, and the last two come from the spatial encoding branch itself. Their purpose is to guide the features of the spectral encoding branch to flow and aggregate toward the spatial encoding branch. Similarly, HCA2 guides the features of the spatial encoding branch to flow and aggregate toward the spectral encoding branch. Since the two branches are symmetrical, this creates a symmetric information flow within the hybrid connection encoder (HCE), enabling cross-modal information enhancement from spatial to spectral features and improving feature representation.

[0084] About the hybrid connection decoder HCD: The overall architecture is the same as the hybrid connection encoder HCE, but the features processed by the hybrid connection attention module are different. The hybrid connection decoder HCD also has a dual-branch structure, including a spatial decoding branch and a spectral decoding branch, which are connected one-to-one with the spatial encoding branch and the spectral encoding branch. More specifically, the spatial decoding branch structure is the same as the spatial encoding branch, but HCA3 replaces HCA1 in the spatial encoding branch, and the Q generated by the upper output map is fused according to the formula of HCA(sD,sE) sD , K sD 、V sD , and K in the spatial coding branch sE ,V sE Through this operation, we can make full use of the feature information of different levels of the spatial coding branch in HCE and effectively guide the reconstruction process of the spatial decoding branch. In addition, the matrices introduced in the formula HCA (sD, sE) are the features of the corresponding layers of the spatial coding branch and the spatial decoding branch, such as K of the fifth layer HCA3. sD 、V sD Derived from the fifth layer, HCA1, this correspondence allows the spatial decoding branch to directly utilize feature details captured at the same abstraction level in the spatial encoding branch during reconstruction, significantly enhancing its ability to capture fine features. The spectral decoding branch follows a similar structure, but HCA4 replaces HCA2 in the spectral encoding branch. Feature fusion is performed according to the HCA(cD, cE) formula.

[0085] Regarding incremental learning: In learning scenarios, the distribution of data streams may change at any time without clear task boundary signals. To enable the model to automatically adapt to such changes, the model adopts an adaptive model expansion strategy based on loss dynamics. The classification loss value of the model is monitored in real time when processing continuously input batches of remote sensing samples. The classification loss value within the sliding window is used to detect whether the data distribution of the data stream has changed. If a change occurs, the current LoRA module is frozen and its weight parameters are incorporated into the second basic model to avoid the continuous growth of parameters caused by the infinite accumulation of LoRA modules. Then, two new LoRA modules are reinitialized to learn new knowledge that may appear in the subsequent data stream or adapt to the new data distribution.

[0086] Compared with existing technologies, the advantages of this invention lie in proposing a novel dual-branch encoder-decoder network and introducing a multi-level feature connection mechanism. This approach utilizes a parallel spatial-spectral dual-branch encoder to extract spatial and spectral features, employs a hybrid connection attention (HCA) module to achieve deep cross-modal feature interaction and enhancement, and combines masked autoencoder reconstruction with spatial-spectral contrastive learning during pre-training to enable the model to learn a robust and discriminative joint spatial-spectral feature representation. This design aims to provide a high-quality, decoupled, and fused feature foundation for subsequent online incremental learning.

[0087] A new adaptive model expansion strategy is proposed: first, based on the first basic model obtained by training the dual-branch codec network, a second basic model is constructed through a classifier. Second, the LoRA method is introduced into the second basic model for online incremental learning and network parameter update. This update method realizes efficient online model adjustment of parameters, that is, only a small number of LoRA parameters need to be trained, while the huge backbone network is frozen. Third, the present invention adopts an adaptive model expansion strategy based on batch classification loss values. By monitoring the loss changes of online classification tasks in real time, when changes are found, the currently learned LoRA parameters are automatically frozen and their weights are merged into the backbone. At the same time, new LoRA parameters are initialized to learn subsequent data, thereby dynamically adapting to changes in data streams without the need for task boundary information.

[0088] A regularization term is introduced into the LoRA module to mitigate the catastrophic forgetting of previously learned knowledge when learning a new LoRA module. This term penalizes changes in important parameters. This enables the model to continuously learn new knowledge from the continuous stream of remote sensing data while effectively suppressing the forgetting of previously learned knowledge, achieving robust online incremental learning.

[0089] In summary, the present invention can achieve efficient, scalable and task-independent online incremental learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 This is a diagram of the dual-branch encoding and decoding network architecture;

[0091] Figure 2 A schematic diagram of constructing a second basic model and incremental learning based on the first basic model;

[0092] Figure 3 Schematic diagram of the connection between HCA1 in the spatial coding branch and HCA2 in the spectral spatial coding branch;

[0093] Figure 4 Schematic diagram of the structure of the first hybrid connection attention module HCA1;

[0094] Figure 5 Schematic diagram of the spatial encoder structure in the space-spectrum dual-branch encoder;

[0095] Figure 6 Schematic diagram of the spectral encoder structure in the space-spectral dual-branch encoder;

[0096] Figure 7 This is a comparison chart of the evaluation indicators of the present invention and the ViT image classification model;

[0097] Figure 8 This is a comparison chart of the classification results of the present invention and the ViT image classification model. DETAILED DESCRIPTION

[0098] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0099] Example 1: See Figure 1 Arrival Figure 6 ,An online incremental learning method for a large remote sensing image model based on ViT, includes the following steps;

[0100] S1, obtain remote sensing image dataset D, D={X1,X2,…,X i ,…,X N}, X i is the i-th image block labeled with category in D, , C is the number of spectral channels, P×P is the image block size, and N is the total number of image blocks;

[0101] S2, constructing a dual-branch encoding and decoding network, including a spatial-spectral dual-branch encoder, a hybrid connection encoder HCE and a hybrid connection decoder HCD, the construction method including S21 to S23;

[0102] S21, constructing a spatial-spectral dual-branch encoder, including a spatial encoder and a spectral encoder;

[0103] The spatial encoder is used to i Perform random masking and spatial encoding to obtain spatial features e i,s , the spectral encoder is used to i Perform random masking and spectral encoding to obtain the spectral feature e i,c ;

[0104] S22, constructing a hybrid connection encoder HCE;

[0105] Get two Transformer encoders with the same structure, each consisting of Z stacked Transformer layers;

[0106] Connect the input of a Transformer encoder to the output of the spatial encoder, where the zth Transformer layer is marked as E sz , the other Transformer encoder input is connected to the output of the spectral encoder, where the zth Transformer layer is marked as E cz , 1≤z≤Z;

[0107] E sz The multi-head self-attention layer in the middle is modified to the first hybrid connection attention module HCA1 to obtain the spatial encoding branch, and E cz The multi-head self-attention layer is modified to the second hybrid connection attention module HCA2 to obtain the spectral encoding branch, and the two encoding branches constitute a hybrid connection encoder;

[0108] The output features HCA(c, s) of HCA1 and HCA(s, c) of HCA2 are obtained according to the following formulas:

[0109] HCA(c,s)=Attention(Q c ,K c ,V c )+Attention(Q c ,K s ,V s ),

[0110] HCA(s,c)=Attention(Q s ,K s ,V s )+Attention(Q s ,K c ,V c ),

[0111] Where Attention(∙) is the self-attention operation, Q c , K c 、V c E cz The Q matrix, K matrix and V matrix in Q s , K s 、V s E sz The Q matrix, K matrix and V matrix in;

[0112] S23, construct a hybrid connection decoder HCD, including a spatial decoding branch and a spectral decoding branch. The spatial decoding branch is used to predict the mask area of the spatial encoder to obtain a spatial reconstruction image The spectrum decoding branch is used to predict the mask area of the spectrum encoder to obtain the spectrum reconstruction map ;

[0113] S3, construct the loss function L of the dual-branch encoder-decoder network, use the remote sensing image dataset D to minimize L and train until convergence, obtain the first basic model and freeze its parameters;

[0114] S4, deleting HCD from the first basic model, adding the two outputs of HCE and feeding them into a classifier to obtain a second basic model, wherein the second basic model is used to input the image block and output the corresponding classification result;

[0115] S5, online incremental learning of the second base model;

[0116] S51, introducing a LoRA module into each of the spatial coding branch and the spectral coding branch of the second basic model;

[0117] S52, using the second basic model to online detect the data stream composed of image blocks, using the data stream to update the LoRA module parameters based on the LoRA method, and detecting whether the data distribution of the data stream has changed;

[0118] S53: If changed, the LoRA module parameters are incorporated into the second basic model and S51 to S52 are repeated; otherwise, no processing is performed.

[0119] See also Figure 3 , Figure 3 Shows the zth Transformer layer E in the two spatial encoding branches sz and the zth Transformer layer E in the spectral encoding branch cz Mixed connection mode. sz Therefore, the multi-head self-attention layer of the left Transformer layer is replaced with HCA1 and the right one is E cz Therefore, the multi-head self-attention layer of the right Transformer layer is replaced with HCA2 during the improvement, and the input is increased from 3 to 5, coming from 2 encoding branches respectively, thus realizing the interaction and enhancement of features. Figure 3 The remaining layers in the same structure as the existing Transformer layers. Figure 4 Taking HCA1 as an example, the input increases from 3 to 5 channels, and 2 self-attention layers are needed. Figure 3 The connection method of HCA1 is composed of two self-attention layers, one on the left and one on the right, and the outputs of the two self-attention layers are added through the addition layer.

[0120] Example 2: See Figures 1 to 6 ,Based on Example 1, we provide a method for constructing a hybrid connection decoder HCD: ,obtain two Transformer decoders with the same structure, both of which are composed of Z ,Transformer layers stacked;

[0121] Connect the input of a Transformer decoder to the output of the spatial encoding branch, where the zth Transformer layer is marked as D sz , the other Transformer decoder input is connected to the output of the spectral encoding branch, where the zth Transformer layer is marked as D cz ;

[0122] D sz The multi-head self-attention layer in the middle is modified to the third hybrid connection attention module HCA3 to obtain the spatial decoding branch, and D cz The multi-head self-attention layer is modified to the fourth hybrid connection attention module HCA4 to obtain the spectral decoding branch. The two decoding branches constitute a hybrid connection decoder, and the spatial decoder is used to predict the mask area of the spatial encoder to obtain the spatial reconstruction map. , the spectral decoder is used to predict the mask area of the spectral encoder to obtain the spectral reconstruction image ;

[0123] The output features HCA(sD, sE) of HCA3 and HCA(cD, cE) of HCA4 are obtained according to the following formulas:

[0124] HCA(sD,sE)=Attention(Q sD ,K sD ,V sD )+Attention(Q sD ,K sE ,V sE ),

[0125] HCA(cD,cE)=Attention(Q cD ,K cD ,V cD )+Attention(Q cD ,K cE ,V cE ),

[0126] Where Q sD , K sD 、V sD D sz The Q matrix, K matrix and V matrix in sE 、V sE E sz The Q matrix and K matrix in Q cD , K cD 、V cD D cz The Q matrix, K matrix and V matrix incE 、V cE E cz The rest of the embodiment 2 is the same as that of the embodiment 1.

[0127] Example 3: See Figures 1 to 6 ,In S3, the loss function L is calculated according to the following formula;

[0128] ,

[0129] 、 are spatial reconstruction loss and spectral reconstruction loss respectively, L SSC is the symmetric InfoNCE loss between spatial features and spectral features, and λ is L SSC The weight of .

[0130] ,

[0131] ,

[0132] Where, |M s | is the mask region M of the spatial encoder s Total number of pixels in X s (p) and X i and The pixel value at position p in |M c | is the mask area M of the spectral encoder c Total number of pixels in X c (p) and X i and Spectral vector of position p in ;

[0133] L SSC According to the following formula:

[0134] ,

[0135] Where sim(∙,∙) is the cosine similarity, exp(∙) is the exponential function, τ is the temperature hyperparameter, and e q,s 、e q,c are the spatial features and spectral features corresponding to the qth image block in the remote sensing image dataset D. The rest is the same as in Example 1 and Example 2.

[0136] Example 4: See Figures 1 to 6Specifically, S51 includes selecting one layer of Q matrix, V matrix, and K matrix in the spatial coding branch and introducing the LoRA module. Furthermore, S52 includes selecting one layer of Q matrix, V matrix, and K matrix in the spectral coding branch and introducing the LoRA module. Specifically, S52 includes detecting whether the data distribution of the data stream has changed.

[0137] Sequentially divide the data stream into multiple batches, calculate the classification loss value of each batch, and sample the batches using a sliding window. If the current t-th batch satisfies the following formula, the data distribution of the t-th batch is judged to have changed;

[0138] ,

[0139] Where k is the sliding window size, l j is the classification loss value of the jth batch of data stream. The rest is the same as in Examples 1 to 3.

[0140] Example 5: See Figures 1 to 6 The spatial encoder includes a spatial mask layer, a first normalization layer, a two-dimensional convolution layer, an attention layer, a first addition layer, a second normalization layer, a multi-layer perceptron, and a second addition layer connected in sequence, and the output of the spatial mask layer is skip-connected to the first addition layer, and the output of the first addition layer is skip-connected to the output of the second addition layer;

[0141] The spatial mask layer is used to randomly i Apply a C×1×1 mask to the pixels in the image to get the first spatial feature , and only keep the annotations of the unmasked areas;

[0142] The second spatial feature is obtained by the first normalization layer, the two-dimensional convolution layer, and the attention layer. ,

[0143] described After the first addition layer and Add element by element to get the third space feature ;

[0144] described The fourth spatial feature is obtained by the second normalization layer and the multi-layer perceptron. ;

[0145] described After the second addition layer and Add element by element to get the spatial feature e i,s。

[0146] The attention layer is a self-attention layer or a spatial attention layer, and the spatial attention layer performs a spatial attention operation according to the following formula:

[0147] ,

[0148] Where Attention1(Q,K,V) is the spatial attention operation, Softmax(∙) is the Softmax function, Q, K, and V are the Q matrix, K matrix, and V matrix obtained by mapping the output of the two-dimensional convolutional layer; d k is the scaling factor.

[0149] The spectral encoder includes a spectral mask layer, a first normalization layer, a linear projection layer, an attention layer, a first addition layer, a second normalization layer, a multilayer perceptron, and a second addition layer connected in sequence, and the output of the spatial mask layer is jump-connected to the first addition layer, and the output of the first addition layer is jump-connected to the output of the second addition layer;

[0150] The spectral mask layer is used to randomly i Apply a 1×P×P mask to the band to obtain the first spectral feature , and only retain the unmasked band information;

[0151] The second spectral feature is obtained by the first normalization layer, linear projection layer, and attention layer in sequence. ,

[0152] described After the first addition layer and Add element by element to get the third spectral feature ;

[0153] described The fourth spectral feature is obtained by the second normalization layer and the multi-layer perceptron. ;

[0154] described After the second addition layer and The spectral characteristics e are obtained by adding the elements i,c The rest is the same as in Example 1 to Example 4.

[0155] Example 6: See Figures 1 to 6 , the optimization method of LoRA module is;

[0156] Sa1, decompose the LoRA module into two low-rank matrices A and B;

[0157] Sa2, maintains a hard buffer with a capacity of 4, which is used to store the four image blocks with the highest classification loss values;

[0158] Sa3, calculate the parameter importance Ω of the low-rank matrices A and B according to the following formula A and Ω B ;

[0159] ,

[0160] ,

[0161] Where x k is the kth image block in the hard buffer, 1≤k≤4, θ is the network parameter of the current second basic model, p(x k |θ) is the current second basic model for x k The classification result, ∇ is the gradient operation, ∘ is the element-by-element product operation, is the weight matrix W of A A The element in row i and column j in Ω A The element in row i and column j, is the weight matrix W of B B The element in row i and column j, Ω B The element in row i and column j;

[0162] Sa4, calculate the regularization value r and regularization term L of the current LoRA module according to the following formula LoRA ;

[0163] ,

[0164] ,

[0165] Where R is the set of regularization values of all current LoRA modules, and λ is the regularization coefficient;

[0166] Sa5, construct the loss function L1 of the LoRA module and optimize the parameters of the LoRA module by minimizing L1;

[0167] ;

[0168] L cls (,∙,) is the classification loss, F(X;θ) is the predicted classification result of the current second basic model for a batch of image blocks, Y is the true category label of the batch of samples, F(X B ;θ) is the classification result of the current second basic model for the samples in the hard buffer, Y B is the true category label of the image block in the hard buffer.

[0169] Example 7: See Figure 7 and Figure 8 ,This embodiment selects three evaluation indicators commonly used in the field of image block classification: ,overall classification accuracy OA, average accuracy AA and Kappa coefficient.

[0170] Overall classification accuracy OA, English Overall Accuracy, represents the ratio of the total number of correctly classified samples to the total number of test samples, reflecting the overall classification performance of the model on the entire dataset. The formula is , where T TP represents the number of samples correctly classified as positive examples, T TN Indicates the number of samples correctly classified as negative examples, F FP Indicates the number of samples that are misclassified as positive examples, F FN Indicates the number of samples that are misclassified as negative examples.

[0171] Average Accuracy (AA): represents the arithmetic mean of the classification accuracy of all categories, which can more fairly evaluate the performance of the model under the condition of imbalanced class samples. Its calculation formula is: , where n represents the total number of categories, A n Indicates the The accuracy of each category.

[0172] Kappa Coefficient: measures the degree of consistency between the model classification results and the random classification results, reducing the impact of random consistency. Its calculation formula is: , where: p o The sum of the number of correctly classified samples in each category divided by the total number of samples, that is, the overall classification accuracy, p e is random consistency. Assume that the number of real samples of each type of sample is a1, a2, ⋯, a c , and the predicted number of samples of each category are b1,b2,⋯,b c , the total number of samples is N, then .

[0173] Based on the above three evaluation indicators, the second basic model of the present invention and the ViT image classification model are used to classify the Berlin dataset respectively, and the comparison chart of the evaluation indicators is as follows: Figure 7 As shown. Among them, the second basic model of the present invention is based on Figures 1 to 6 The optimal model is constructed. Figure 7 It can be seen that the present invention is superior to the ViT image classification model in terms of overall classification accuracy OA, average accuracy AA and Kappa coefficient.

[0174] In order to more intuitively express the classification results of the present invention and the ViT image classification model, a local enlarged area of a sample is selected from the Berlin dataset and classified using the second basic model and the ViT image classification model, and a comparison diagram is obtained as follows: Figure 8 shown. Figure 8(a), (b), and (c) are the true value map, the classification results of the present invention, and the classification results of the ViT image classification model respectively. The true value map represents the actual situation of the remote sensing image object category and is the benchmark for evaluating the performance of the classification algorithm. Figure 8 It can be seen that the classification result of the present invention is closer to the true value graph than the ViT classification result, and has better performance.

[0175] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An online incremental learning method for a large remote sensing image model based on ViT, characterized in that: The following steps are included: S1, obtain remote sensing image dataset D, D={X1,X2,…,X i ,…,X N }, X i is the i-th image block labeled by category in D, , C is the number of spectral channels, P×P is the image block size, and N is the total number of image blocks; S2, constructing a dual-branch encoding and decoding network, including a spatial-spectral dual-branch encoder, a hybrid connection encoder HCE and a hybrid connection decoder HCD, the construction method including S21 to S23; S21, constructing a spatial-spectral dual-branch encoder, including a spatial encoder and a spectral encoder; The spatial encoder is used to i Perform random masking and spatial encoding to obtain spatial features e i,s , the spectral encoder is used to i Perform random masking and spectral encoding to obtain the spectral feature e i,c ; S22, constructing a hybrid connection encoder HCE; Get two Transformer encoders with the same structure, each consisting of Z stacked Transformer layers; Connect the input of a Transformer encoder to the output of the spatial encoder, where the zth Transformer layer is marked as E sz , the other Transformer encoder input is connected to the output of the spectral encoder, where the zth Transformer layer is marked as E cz , 1≤z≤Z; E sz The multi-head self-attention layer in the middle is modified to the first hybrid connection attention module HCA1 to obtain the spatial encoding branch, and E cz The multi-head self-attention layer is modified to the second hybrid connection attention module HCA2 to obtain the spectral encoding branch, and the two encoding branches constitute a hybrid connection encoder; The output features HCA(c, s) of HCA1 and HCA(s, c) of HCA2 are obtained according to the following formulas: HCA(c,s)=Attention(Q c ,K c ,V c )+Attention(Q c ,K s ,V s ), HCA(s,c)=Attention(Q s ,K s ,V s )+Attention(Q s ,K c ,V c ), Where Attention(∙) is the self-attention operation, Q c , K c 、V c E cz The Q matrix, K matrix and V matrix in Q s , K s 、V s E sz The Q matrix, K matrix and V matrix in; S23, construct a hybrid connection decoder HCD, including a spatial decoding branch and a spectral decoding branch. The spatial decoding branch is used to predict the mask area of the spatial encoder to obtain a spatial reconstruction image The spectrum decoding branch is used to predict the mask area of the spectrum encoder to obtain the spectrum reconstruction map ; S3, construct the loss function L of the dual-branch encoder-decoder network, use the remote sensing image dataset D to minimize L and train until convergence, obtain the first basic model and freeze its parameters; S4, deleting HCD from the first basic model, adding the two outputs of HCE and feeding them into a classifier to obtain a second basic model, wherein the second basic model is used to input the image block and output the corresponding classification result; S5, online incremental learning of the second base model; S51, introducing a LoRA module into each of the spatial coding branch and the spectral coding branch of the second basic model; S52, using the second basic model to online detect the data stream composed of image blocks, using the data stream to update the LoRA module parameters based on the LoRA method, and detecting whether the data distribution of the data stream has changed; S53: If changed, the LoRA module parameters are incorporated into the second basic model and S51 to S52 are repeated; otherwise, no processing is performed.

2. The online incremental learning method for a large remote sensing image model based on ViT according to claim 1 is characterized in that: In S23, the structure of the hybrid connection decoder HCD is: Get two Transformer decoders with the same structure, each consisting of Z stacked Transformer layers; Connect the input of a Transformer decoder to the output of the spatial encoding branch, where the zth Transformer layer is marked as D sz , the other Transformer decoder input is connected to the output of the spectral encoding branch, where the zth Transformer layer is marked as D cz ; D sz The multi-head self-attention layer in the middle is modified to the third hybrid connection attention module HCA3 to obtain the spatial decoding branch, and D cz The multi-head self-attention layer is modified to the fourth hybrid connection attention module HCA4 to obtain the spectral decoding branch. The two decoding branches constitute a hybrid connection decoder, and the spatial decoder is used to predict the mask area of the spatial encoder to obtain the spatial reconstruction map. , the spectral decoder is used to predict the mask area of the spectral encoder to obtain the spectral reconstruction image ; The output features HCA(sD, sE) of HCA3 and HCA(cD, cE) of HCA4 are obtained according to the following formulas: HCA(sD,sE)=Attention(Q sD ,K sD ,V sD )+Attention(Q sD ,K sE ,V sE ), HCA(cD,cE)=Attention(Q cD ,K cD ,V cD )+Attention(Q cD ,K cE ,V cE ), Where Q sD , K sD 、V sD D sz The Q matrix, K matrix and V matrix in sE 、V sE E sz The Q matrix and K matrix in Q cD , K cD 、V cD D cz The Q matrix, K matrix and V matrix in cE 、V cE E cz The Q matrix and K matrix in .

3. The online incremental learning method for a large remote sensing image model based on ViT according to claim 1 is characterized in that: In S3, the loss function L is calculated according to the following formula; ， 、 are spatial reconstruction loss and spectral reconstruction loss respectively, L SSC is the symmetric InfoNCE loss between spatial features and spectral features, and λ is L SSC The weight of .

4. The online incremental learning method for a large remote sensing image model based on ViT according to claim 3 is characterized in that: ， ， Where, |M s | is the mask region M of the spatial encoder s Total number of pixels in X s (p) and X i and The pixel value at position p in |M c | is the mask region M of the spectral encoder c Total number of pixels in X c (p) and X i and Spectral vector of position p in ; L SSC According to the following formula: ， Where sim(∙,∙) is the cosine similarity, exp(∙) is the exponential function, τ is the temperature hyperparameter, and e q,s 、e q,c are the spatial features and spectral features corresponding to the qth image block in the remote sensing image dataset D, respectively.

5. The online incremental learning method for a large remote sensing image model based on ViT according to claim 1, characterized in that: Specifically, S51 is to select one layer of Q matrix, V matrix and K matrix in the spatial coding branch, introduce the LoRA module, and select one layer of Q matrix, V matrix and K matrix in the spectral coding branch, and introduce the LoRA module.

6. The online incremental learning method for a large remote sensing image model based on ViT according to claim 1, characterized in that: In S52, detecting whether the data distribution of the data stream has changed is specifically as follows: Sequentially divide the data stream into multiple batches, calculate the classification loss value of each batch, and sample the batches using a sliding window. If the current t-th batch satisfies the following formula, the data distribution of the t-th batch is judged to have changed; ， Where k is the sliding window size, l j is the classification loss value of the jth batch of data stream.

7. The online incremental learning method for a large remote sensing image model based on ViT according to claim 1, characterized in that: The spatial encoder includes a spatial mask layer, a first normalization layer, a two-dimensional convolution layer, an attention layer, a first addition layer, a second normalization layer, a multi-layer perceptron, and a second addition layer connected in sequence, and the output of the spatial mask layer is jump-connected to the first addition layer, and the output of the first addition layer is jump-connected to the output of the second addition layer; The spatial mask layer is used to randomly i Apply a C×1×1 mask to the pixels in the image to get the first spatial feature , and only keep the annotations of the unmasked areas; The second spatial feature is obtained by the first normalization layer, the two-dimensional convolution layer, and the attention layer. , described After the first addition layer and Add element by element to get the third space feature ; described The fourth spatial feature is obtained by the second normalization layer and the multi-layer perceptron. ; described After the second addition layer and Add element by element to get the spatial feature e i,s。 8. The online incremental learning method of a large remote sensing image model based on ViT according to claim 7 is characterized in that: The attention layer is a self-attention layer or a spatial attention layer; The spatial attention layer performs spatial attention operations according to the following formula; ， Where Attention1(Q,K,V) is the spatial attention operation, Softmax(∙) is the Softmax function, Q, K, and V are the Q matrix, K matrix, and V matrix obtained by mapping the output of the two-dimensional convolutional layer; d k is the scaling factor.

9. The online incremental learning method for a large remote sensing image model based on ViT according to claim 1, characterized in that: The spectral encoder includes a spectral mask layer, a first normalization layer, a linear projection layer, an attention layer, a first addition layer, a second normalization layer, a multilayer perceptron, and a second addition layer connected in sequence, and the output of the spatial mask layer is jump-connected to the first addition layer, and the output of the first addition layer is jump-connected to the output of the second addition layer; The spectral mask layer is used to randomly i Apply a 1×P×P mask to the band to obtain the first spectral feature , and only retain the unmasked band information; The second spectral feature is obtained by the first normalization layer, linear projection layer, and attention layer in sequence. , described After the first addition layer and Add element by element to get the third spectral feature ; described The fourth spectral feature is obtained by the second normalization layer and the multi-layer perceptron. ; described After the second addition layer and The spectral characteristics e are obtained by adding the elements i,c。 10. The online incremental learning method of a large remote sensing image model based on ViT according to claim 1, characterized in that: The optimization method of the LoRA module is: Sa1, decompose the LoRA module into two low-rank matrices A and B; Sa2, maintains a hard buffer with a capacity of 4, which is used to store the four image blocks with the highest classification loss values; Sa3, calculate the parameter importance Ω of the low-rank matrices A and B according to the following formula A and Ω B ; ， ， Where x k is the kth image block in the hard buffer, 1≤k≤4, θ is the network parameter of the current second basic model, p(x k |θ) is the current second basic model for x k The classification result, ∇ is the gradient operation, ∘ is the element-by-element product operation, is the weight matrix W of A A The element in row i and column j in Ω A The element in row i and column j, is the weight matrix W of B B The element in row i and column j, Ω B The element in row i and column j; Sa4, calculate the regularization value r and regularization term L of the current LoRA module according to the following formula LoRA ; ， ， Where R is the set of regularization values of all current LoRA modules, and λ is the regularization coefficient; Sa5, construct the loss function L1 of the LoRA module and optimize the parameters of the LoRA module by minimizing L1; ； L cls (,∙,) is the classification loss, F(X;θ) is the predicted classification result of the current second basic model for a batch of image blocks, Y is the true category label of the batch of samples, F(X B ;θ) is the classification result of the current second basic model for the samples in the hard buffer, Y B is the true category label of the image block in the hard buffer.

Images