SAR and multispectral fusion method based on low-rank adaptive fine tuning

Abstract

An SAR and multispectral fusion method based on low-rank adaptive fine tuning comprises the steps that a low-rank adaptive layer is constructed, a model is initialized, in initialization, weight pre-training is conducted on newly-added low-rank matrix parameters through a remote sensing visual language model RemoteCLIP, and pre-training parameters are frozen; the SAR image and the multispectral image are subjected to wave band grouping through a wave band grouping and prompt guiding mechanism based on physical attributes, the SAR image is divided into a VV group, a VH group and a PolSAR group through grouping, and multiple spectrums are divided into an RGB group, a VRE group, an NIR group and an SWIR group through grouping; respectively extracting image features and semantic prompt text features corresponding to each wave band group through an image encoder and a text encoder to obtain corresponding image feature vectors and text feature vectors; calculating the similarity between the image features and the text features to obtain predicted image category labels; calculating classification loss and comparison loss, carrying out network training, and repeating training until the model converges; and the model obtained by training is used for reasoning.

Description

Technical Field

[0001] This invention specifically relates to a SAR and multispectral fusion method based on low-rank adaptive fine-tuning. Background Technology

[0002] Remote sensing imagery plays a crucial role in Earth observation missions and is widely used in various fields such as environmental monitoring, urban planning, and land cover and use classification. With the continuous advancement of remote sensing technology, the types and numbers of remote sensing platforms and sensors have grown exponentially, enabling the acquisition of multi-resolution, multispectral, and multi-temporal multi-source remote sensing data from different sensors. Compared to single-source remote sensing images, multi-source images carry richer information about ground features and exhibit redundancy and complementarity at the data level. For example, SAR images can penetrate clouds and haze, providing stable surface information under all-weather conditions, while multispectral images can capture subtle spectral features of different ground features. Therefore, multi-source remote sensing image fusion has become one of the key research directions in the field of remote sensing. Its core objective is to fully explore and integrate the complementary information of different modalities to improve the ability to identify, classify, and understand ground features. Against this backdrop, how to efficiently and accurately fuse multi-source remote sensing data and fully leverage the advantages of various sensors has become an important research topic in the field of remote sensing image analysis, with broad application prospects and research value.

[0003] However, in existing technologies, due to the different imaging principles of SAR and multispectral images—SAR is based on microwave reflection and multispectral on spectral reflection—significant differences in physical properties lead to difficulties in aligning features between modes. Existing methods lack targeted modeling of the physical characteristics of bands, resulting in poor cross-modal semantic consistency and easy loss of key ground feature information in the fusion results, such as vegetation red edge features and SAR polarization characteristics. In addition, existing fusion methods mostly use pre-trained models from the natural image domain, such as CLIP and BERT initialization networks. These models do not include prior knowledge from the remote sensing domain, such as "VV polarization corresponds to surface roughness" and "SWIR bands are sensitive to soil moisture," making it difficult to understand remote sensing-specific semantics and resulting in poor alignment between text prompts and image features. Furthermore, fine-tuning the full parameters of large-scale pre-trained models requires updating hundreds of millions of parameters, leading to high memory consumption, high computational costs, long training times, and difficulty in deploying edge computing devices, thus limiting their application in practical remote sensing scenarios.

[0004] Therefore, there is an urgent need for a SAR and multispectral fusion method to solve the above problems. Summary of the Invention

[0005] To address the problems mentioned in the background section, the present invention aims to provide a SAR and multispectral fusion method based on low-rank adaptive fine-tuning, comprising the following steps:

[0006] S1. Construct a low-rank adaptive layer and initialize the model. The initialization is performed by pre-training weights for the newly added low-rank matrix parameters through the RemoteCLIP remote sensing visual language model, and then freezing the pre-trained parameters.

[0007] S2. Using a physical attribute-based band grouping and prompting guidance mechanism, SAR images and multispectral images are grouped by band, wherein the grouping divides SAR images into VV, VH, and PolSAR groups, and multispectral images into RGB, VRE, NIR, and SWIR groups.

[0008] S3. Extract image features and semantic prompt text features corresponding to each band group through image encoder and text encoder respectively to obtain the corresponding image feature vector and text feature vector;

[0009] S4. Calculate the similarity between image features and text features to obtain the predicted image category label;

[0010] S5. Calculate the classification loss and contrastive loss and train the network. Repeat steps S1 to S5 to train the classification model.

[0011] S6. Use the trained model for inference.

[0012] Preferably, step S1 specifically includes: the weights of the low-rank adaptive layer consist of model initialization weights and newly added trainable low-rank matrix parameters, wherein the pre-training of the newly added low-rank matrix parameters through the remote sensing visual language model RemoteCLIP includes initializing the text encoder with pre-trained weights of RemoteCLIP ViT-B / 32, freezing the initialization weights in the text encoder, and updating the newly added low-rank matrix parameters in the low-rank adaptive layer.

[0013] Preferably, step S1 further includes:

[0014] Low-rank adaptive layers are constructed in layers 8, 9, 10, and 11 of the text encoder, and W in each low-rank adaptive layer is adjusted. q W v W k The matrix is ​​adjusted, where the low-rank matrix consists of trainable matrix A and trainable matrix B. Matrix A is initialized using a Gaussian function, and matrix B is initialized to 0. The rank of both matrices is 2.

[0015] Preferably, S2 specifically includes: SAR image band grouping as follows: Where X VV For VV polarization group, X VH For VH polarization group, X POL The polarization synthesis group; the multispectral image band grouping is as follows: Where XRGB For RGB red, green and blue groups, X VRE For VRE vegetation red-edged group, X NIR For NIR and X SWIR SWIR shortwave infrared group.

[0016] Preferably, S3 specifically includes:

[0017] S31. Obtain the image feature vector of the remote sensing image through the image encoder, wherein the network structure of the image encoder is the Inception structure;

[0018] S32. Extract feature vectors of semantic prompt texts constructed for each band group using a text encoder. The network structure of the text encoder is a BERT structure, and a prompt template T is constructed for each group. i Define semantic embedding:

[0019] T i =E t (t i ), i = 1, 2, ..., N g

[0020] In the above formula, N g Indicates the number of band groups.

[0021] Preferably, in S4, calculating the similarity between image features and text features to obtain the predicted image category label includes: calculating image features f(x) i ) and text features g(t) for each category j The similarity between the images is used to calculate the probability that a remote sensing image belongs to a specific category. The category with the highest probability value is taken as the predicted image category label:

[0022]

[0023] In the above formula, <·> represents cosine similarity, f(·) represents image encoder, g(·) represents text encoder, K represents the number of categories in the dataset, and τ represents temperature hyperparameter.

[0024] Preferably, S5 specifically includes:

[0025] S51, Based on the actual category label y i and the predicted image category labels obtained in step S4 Calculate the classification loss L cls :

[0026]

[0027] In the above formula, B represents the size of the batch;

[0028] S52. Calculate the contrast loss L based on the image features and text features obtained in step S3. con ;

[0029] S53, Constructing the total loss L total =L cls +λL con Where λ represents the balance coefficient, L cls L represents classification loss. con This represents the loss in comparison.

[0030] Preferably, step S52 specifically includes:

[0031] S521, Based on the true category label y of the remote sensing image i Construct a supervision matrix gt, where gt is a B×B matrix, B is the size of the batch, and its elements are:

[0032]

[0033] In the above formula, the range of i and j is [0, B-1];

[0034] S522. Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on image features and text features;

[0035] S523. Calculate the contrast loss L based on the text similarity matrix i2t_logit, the text-image similarity matrix t2i_logit, and the supervision matrix gt. con =(L i2t +L t2i ) / 2, where L i2t and L t2i Both use the BCEWithLogitsLoss loss function:

[0036] BCEWithLogitsLoss={l1,...,l N},l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))]

[0037] In the above formula, N represents the number of batches, n represents the number of labels predicted in each batch, and σ(·) represents the Sigmoid function.

[0038] Preferably, calculating the image-text similarity matrix i2t_logit includes:

[0039] i2t_logit=I·T Τ

[0040] In the above formula, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing images, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing images.

[0041] Calculating the text-image similarity matrix t2i_logit includes:

[0042] i2t_logit=T·I Τ

[0043] In the above formula, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing images, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing images.

[0044] Preferably, in S523, the contrast loss L is calculated based on the text similarity matrix i2t_logit, the text-image similarity matrix t2i_logit, and the supervision matrix gt. con include:

[0045] The contrast loss L is calculated based on the supervision matrix gt and the image-text similarity matrix i2t_logit. i2t :

[0046] L i2t =BCEWithLogitsLoss(i2t_logit,gt);

[0047] The contrast loss L is calculated based on the step supervision matrix gt and the text-image similarity matrix t2i_logit. t2i :

[0048] L t2i =BCEWithLogitsLoss(t2i_logit,gt).

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] I. A band grouping mechanism based on physical attributes groups bands according to imaging physical characteristics and generates semantic prompts for each group. It constructs image-text alignment supervision to centrally express band features with similar physical characteristics, reduce cross-band interference, and improve feature recognition. The semantic prompt guidance mechanism enhances the semantic consistency between modalities through image-text alignment, improves the accuracy of ground feature classification, and enhances the accuracy and semantic consistency of cross-modal fusion. The universal design of the band grouping and prompting mechanism allows it to be extended to fusion tasks of other remote sensing modalities, and has good transferability.

[0051] Second, the text encoder is initialized with ViT-B / 32 weights from RemoteCLIP. Its weights contain a large amount of prior knowledge from remote sensing samples, enabling it to accurately capture specific semantics such as "polarization characteristics" and "spectral reflectance" during text feature extraction, thereby improving the semantic matching ability between text prompts and remote sensing features. At the same time, the binding of band groups with remote sensing land cover types makes the fusion results more in line with the needs of practical applications.

[0052] Third, a low-rank matrix is ​​inserted into the Transformer layer of the text encoder. The low-rank adaptive fine-tuning strategy only trains the parameters of the newly added low-rank matrix, such as only training matrices A and B, reducing the parameter size, reducing memory usage, and shortening the training time. At the same time, the performance of the pre-trained model is preserved while freezing the weights of the model. Under the premise of ensuring fusion accuracy, the model deployment threshold is significantly reduced, the training cost is greatly reduced, and lightweight deployment is achieved. The stability of the fusion method in complex environments is better than that of traditional methods.

[0053] This invention employs a collaborative strategy of "remote sensing pre-training initialization + physical attribute grouping + low-rank fine-tuning" to improve the accuracy of SAR and multispectral fusion while addressing the high cost and low adaptability issues of existing technologies. This provides a feasible path for the efficient fusion and practical application of multi-source remote sensing data. Experimental results demonstrate that, in terms of pre-training initialization and semantic alignment, RemoteCLIP weights improve overall classification accuracy by approximately 3%–4% compared to random initialization, with significant improvement in the F1-score for semantically confused categories. Regarding band grouping and cue learning, OA is improved by approximately 2%–3%, and the Kappa coefficient is improved by approximately 0.02–0.03. Furthermore, feature visualization results show clearer inter-class distinctions after grouping. In terms of low-rank adaptive fine-tuning, the number of parameters is reduced by over 90%, memory usage is reduced by 60%–70%, and training time is halved, while maintaining near-or better accuracy. Attached Figure Description

[0054] For ease of explanation, the present invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0055] Figure 1 This is a schematic diagram of a low-rank adaptive layer;

[0056] Figure 2 The flowchart shows the model training process for the SAR and multispectral fusion method.

[0057] Figure 3 This is a schematic diagram of a SAR and multispectral fusion network model based on low-rank adaptive fine-tuning. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0059] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0060] Specific Implementation Method 1: This embodiment only provides a preferred implementation method, a SAR and multispectral fusion method based on low-rank adaptive fine-tuning, including the following steps:

[0061] S1. Construct a low-rank adaptive layer and initialize the model. The initialization is performed by pre-training weights for the newly added low-rank matrix parameters through the RemoteCLIP remote sensing visual language model, and then freezing the pre-trained parameters.

[0062] S2. Using a physical attribute-based band grouping and prompting guidance mechanism, SAR images and multispectral images are grouped by band, wherein the grouping divides SAR images into VV, VH, and PolSAR groups, and multispectral images into RGB, VRE, NIR, and SWIR groups.

[0063] S3. Extract image features and semantic prompt text features corresponding to each band group through image encoder and text encoder respectively to obtain the corresponding image feature vector and text feature vector;

[0064] S4. Calculate the similarity between image features and text features to obtain the predicted image category label;

[0065] S5. Calculate the classification loss and contrastive loss and train the network. Repeat the training until the model converges.

[0066] S6. Use the trained model for inference.

[0067] Specifically, such as Figure 1 and Figure 3 As shown, this invention uses RemoteCLIP pre-trained weights as the initial weights for the text encoder during the initialization process. By leveraging a pre-trained model in the remote sensing field, the model's ability to understand remote sensing semantics is enhanced, thereby improving the alignment between band information and semantic features and increasing the generalization ability in multi-source remote sensing image fusion tasks.

[0068] Secondly, to further enhance the semantic fusion capability between image and text modalities, this invention introduces a band grouping and semantic cueing mechanism based on physical attributes. This invention fully considers the significant differences in imaging physical characteristics between Synthetic Aperture Radar (SAR) images and multispectral images, meticulously dividing the input remote sensing images into several physically consistent band groups. Image grouping can employ various methods, such as according to spectral ranges or task requirements. Preferably, in SAR images, they can be divided into VV, VH, and PolSAR band groups; while in multispectral images, they can be divided into RGB, VRE, NIR, and SWIR band groups. Semantic text cueing related to the imaging attributes and target category is designed for each band group. Each band image and its corresponding cue text are semantically aligned after encoding by an image encoder and a text encoder, respectively. This guides the model to extract fusion features with physical priors and semantic orientation, improving cross-modal consistency and task discrimination capabilities, and enhancing feature recognition and classification accuracy. Therefore, combining physical attribute grouping with a cue learning mechanism creates a framework that enhances semantic consistency and cross-modal fusion performance.

[0069] This invention embeds a low-rank adaptive layer into the Transformer structure within the text encoder. The parameters of this structure consist of two parts: the initialization parameters of the original Transformer and the parameters of the LoRA (Low-Rank Adaptation) low-rank factorization matrix. During the model training phase, the weight parameters in the original Transformer are completely frozen and do not participate in backpropagation or gradient updates; only the newly added low-rank factorization matrix parameters in the LoRA are learned and updated. This approach not only avoids gradient calculations and memory usage for the original weight parameters but also reduces the number of trainable parameters, thereby lowering resource consumption during training. Furthermore, since the initialization parameters remain unchanged, this method can fully preserve the remote sensing visual language priors contained in RemoteCLIP, improving the model's semantic understanding ability. Furthermore, after initial extraction, image features from different band groups can be weighted and fused based on their physical attributes and semantic cues. The weighting coefficients are not fixed but dynamically adjusted according to the importance of the band group and the specific needs of the current task. This invention also provides an adaptive learning strategy to dynamically adjust the model's learning rate and training strategy. This strategy intelligently adjusts hyperparameters such as the learning rate and regularization strength based on changes in the loss function, gradient distribution, and performance feedback on the validation set during training, ensuring that the model learns in an optimal state at different stages, improving the model's convergence speed and generalization ability on unseen data. Classification and contrastive losses are calculated and the network is trained. The training of the classification model is repeated until the loss converges or the validation set performance reaches a preset threshold. The resulting classification model is the final version of the fusion network. It can perform fusion feature extraction and semantic alignment on input SAR and multispectral images, group the images to be inferred and generate corresponding cues, encode them, perform similarity matching with various cues, obtain category probabilities, and output predicted ground feature category labels.

[0070] Specific Implementation Method Two: This embodiment provides only a preferred implementation method. In step S1, building a low-rank adaptive layer and initializing the model, the specific steps include: introducing a low-rank adaptive mechanism into the text encoder to construct a low-rank adaptive layer. The text encoder adopts a 12-Transformer layer BERT-base architecture. The initial parameters of its embedding layer, attention layer, and feedforward network layer are all transferred from the ViT-B / 32 text encoder weights of RemoteCLIP. The weights of the low-rank adaptive layer consist of model initialization weights and newly added trainable low-rank matrix parameters. The pre-training of the newly added low-rank matrix parameters through the RemoteCLIP remote sensing visual language model includes initializing the text encoder with ViT-B / 32 pre-trained weights based on the VisionTransformer (ViT) structure in RemoteCLIP, freezing the initialization weights in the text encoder to ensure that the remote sensing prior knowledge is not destroyed by subsequent training, and updating the newly added low-rank matrix parameters in the low-rank adaptive layer.

[0071] Specific Implementation Method 3: This embodiment only provides a preferred implementation method. In S1, it further includes: low-rank adaptive layers are constructed in layers 8, 9, 10, and 11 of the text encoder, and W in each low-rank adaptive layer is adjusted. q W v W k The matrices are adjusted by adding new low-rank matrix parameters to their initialization parameters to obtain new weights. The low-rank matrix consists of trainable matrix A and trainable matrix B. Matrix A is initialized using a Gaussian function, and matrix B is initialized to 0. Both matrices have a rank of 2. Matrix A has dimensions (d_model, r), and matrix B has dimensions (r, d_model), where d_model is the dimension of the hidden layer of the text encoder, and r is the rank of the low-rank matrix. In this embodiment, a rank of 2 is preferred. Specifically, backpropagation training is performed only on the newly added low-rank matrices A and B. Matrix A is initialized with a Gaussian distribution N(0, 0.01) to avoid excessively large initial values ​​interfering with the feature representation of the basic weights. Matrix B is initialized with all zeros to ensure that the output in the early stages of training is close to the pre-trained model, improving stability.

[0072] Specific Implementation Method Four: This embodiment only provides a preferred implementation method, such as... Figure 3 As shown, the band grouping of SAR and multispectral images using a physical attribute-based band grouping and guidance mechanism in S2 specifically includes: grouping SAR images into bands according to physical attributes, wherein the band group is... Where X VVThis is the VV polarization group. VV polarization refers to horizontally polarized transmission and reception. Its echo intensity is sensitive to surface roughness; for example, hard surfaces such as bare land and roads reflect light strongly, while vegetation-covered areas reflect light weakly. It has unique advantages in monitoring arid regions and identifying bare land. The VV polarization group typically includes single-band data, and the wavelength can be, but is not limited to, the C-band or X-band. The spatial resolution can be, but is not limited to, 10m. VH The VH polarization group refers to horizontally polarized transmission and vertically polarized reception. Its echoes are more sensitive to vegetation structure. The vegetation layer causes the polarization direction to rotate, making the VH echo intensity positively correlated with vegetation biomass. It is suitable for monitoring forest cover and crop growth status. The VH polarization group usually contains single-band data and comes from the same SAR sensor as the VV polarization group, ensuring spatiotemporal consistency. POL This is a polarimetric synthesis set containing full polarimetric information, a four-polarity combination of HH, HV, VH, and VV. Through polarimetric decomposition, such as Freeman-Durden decomposition, scattering mechanisms can be extracted, including but not limited to surface scattering, volume scattering, and secondary scattering. It can distinguish complex terrain features such as built-up areas and vegetated areas. POL A polarization group typically comprises three characteristic bands synthesized from four-polarization data: entropy, anisotropy, and average scattering angle, preserving the physical information of the polarization dimension. Multispectral image bands are grouped as follows: Where X RGB The RGB (red, green, blue) group is primarily used to capture information within the visible light range, including red (620-670nm), green (500-560nm), and blue (450-500nm) bands, corresponding to the human eye's visible spectrum. It can intuitively reflect differences in ground object color, is suitable for initial ground object identification and visual alignment, and provides visual interpretability for the fusion results. VRE The VRE vegetation red edge group covers the 700-750nm wavelength range and corresponds to the vegetation red edge effect. Its reflectance change rate is directly related to the chlorophyll content and growth vitality of vegetation. It is a key indicator for distinguishing healthy vegetation from withered vegetation and can make up for the shortcomings of SAR in monitoring the vertical structure of vegetation, complementing the VH polarization group of SAR. NIR This is the NIR (Near Infrared) group, used to acquire data in the near-infrared band and X-rays. SWIR The SWIR (Shortwave Infrared) band is used to detect information in the shortwave infrared region. This scientific and systematic band grouping method can effectively improve the accuracy and efficiency of image processing and analysis.

[0073] Specific Implementation Method 5: This embodiment only provides a preferred implementation method. In S3, image features and semantic prompt text features corresponding to each band group are extracted by an image encoder and a text encoder, respectively, to obtain the corresponding image feature vector and text feature vector, specifically including:

[0074] S31. Obtain the image feature vector of the remote sensing image through the image encoder, wherein the network structure of the image encoder is the Inception structure;

[0075] S32. Extract feature vectors of semantic prompt texts constructed for each band group using a text encoder. The network structure of the text encoder is a BERT structure, and a prompt template T is constructed for each group. i Define semantic embedding:

[0076] T i =E t (t i ), i = 1, 2, ..., N g

[0077] In the above formula, N g Indicates the number of band groups.

[0078] Specific Implementation Method Six: This embodiment provides only a preferred implementation method. In S4, calculating the similarity between image features and text features to obtain the predicted image category label includes: calculating image features f(x... i ) and text features g(t) for each category j The similarity between the images is used to calculate the probability that a remote sensing image belongs to a specific category. The category with the highest probability value is taken as the predicted image category label:

[0079]

[0080] In the above formula, <·> represents cosine similarity, f(·) represents image encoder, g(·) represents text encoder, K represents the number of categories in the dataset, and τ represents temperature hyperparameter.

[0081] Specific Implementation Method Seven: This embodiment only provides a preferred implementation method. S5, calculating the classification loss and contrastive loss and training the network, repeating training until the model converges, specifically includes:

[0082] S51, Based on the actual category label y i and the predicted image category labels obtained in step S4 Calculate the classification loss L cls :

[0083]

[0084] In the above formula, B represents the size of the batch;

[0085] S52. Calculate the contrast loss L based on the image features and text features obtained in step S3. con ;

[0086] S53, Constructing the total loss Ltotal =L cls +λL con Where λ represents the balance coefficient, L cls L represents classification loss. con This represents the loss in comparison.

[0087] Specific Implementation Method Eight: This embodiment only provides a preferred implementation method. Step S52, based on the image features and text features obtained in step S3, specifically includes:

[0088] S521, Based on the true category label y of the remote sensing image i Construct a supervision matrix gt, where gt is a B×B matrix, B is the size of the batch, and its elements are:

[0089]

[0090] In the above formula, the range of i and j is [0, B-1];

[0091] S522. Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on image features and text features;

[0092] S523. Calculate the contrast loss L based on the text similarity matrix i2t_logit, the text-image similarity matrix t2i_logit, and the supervision matrix gt. con =(L i2t +L t2i ) / 2, where L i2t and L t2i Both use the BCEWithLogitsLoss loss function:

[0093] BCEWithLogitsLoss={l1,...,l N},l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))]

[0094] In the above formula, N represents the number of batches, n represents the number of labels predicted in each batch, and σ(·) represents the Sigmoid function.

[0095] Specific Implementation Method Nine: This embodiment only provides a preferred implementation method. Calculating the image-text similarity matrix i2t_logit includes:

[0096] i2t_logit=I·TΤ

[0097] In the above formula, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing images, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing images.

[0098] Calculating the text-image similarity matrix t2i_logit includes:

[0099] i2t_logit=T·I Τ

[0100] In the above formula, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing images, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing images.

[0101] Specific Implementation Method 10: This embodiment only provides a preferred implementation method. In S523, the contrast loss L is calculated based on the text similarity matrix i2t_logit, the text-image similarity matrix t2i_logit, and the supervision matrix gt. con include:

[0102] The contrast loss L is calculated based on the supervision matrix gt and the image-text similarity matrix i2t_logit. i2t :

[0103] L i2t =BCEWithLogitsLoss(i2t_logit,gt);

[0104] The contrast loss L is calculated based on the step supervision matrix gt and the text-image similarity matrix t2i_logit. t2i :

[0105] L t2i =BCEWithLogitsLoss(t2i_logit,gt).

[0106] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A SAR and multispectral fusion method based on low-rank adaptive fine-tuning, characterized in that, Includes the following steps: S1. Construct a low-rank adaptive layer and initialize the model. The initialization is performed by pre-training weights for the newly added low-rank matrix parameters through the RemoteCLIP remote sensing visual language model, and then freezing the pre-trained parameters. S2. Using a physical attribute-based band grouping and prompting guidance mechanism, SAR images and multispectral images are grouped by band, wherein the grouping divides SAR images into VV, VH, and PolSAR groups, and multispectral images into RGB, VRE, NIR, and SWIR groups. S3. Extract image features and semantic prompt text features corresponding to each band group through image encoder and text encoder respectively to obtain the corresponding image feature vector and text feature vector; S4. Calculate the similarity between image features and text features to obtain the predicted image category label; S5. Calculate the classification loss and contrastive loss and train the network. Repeat steps S1 to S5 to train the classification model. S6. Use the trained model for inference.

2. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 1, characterized in that, In step S1, specifically, the weights of the low-rank adaptive layer consist of model initialization weights and newly added trainable low-rank matrix parameters. The pre-training of the newly added low-rank matrix parameters through the RemoteCLIP remote sensing visual language model includes initializing the text encoder with pre-trained weights using RemoteCLIP's ViT-B / 32, freezing the initialization weights in the text encoder, and updating the newly added low-rank matrix parameters in the low-rank adaptive layer.

3. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 2, characterized in that, Step S1 also includes: Low-rank adaptive layers are constructed in layers 8, 9, 10, and 11 of the text encoder, and W in each low-rank adaptive layer is adjusted. q W v W k The matrix is ​​adjusted, where the low-rank matrix consists of trainable matrix A and trainable matrix B. Matrix A is initialized using a Gaussian function, and matrix B is initialized to 0. The rank of both matrices is 2.

4. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 3, characterized in that, S2 specifically includes: SAR image band grouping Where X VV For VV polarization group, X VH For VH polarization group, X POL For polarization synthesis group; multispectral image wave Segment grouping Where X RGB For RGB red, green and blue groups, X VRE For VRE vegetation red-edged group, X NIR For NIR and X SWIR SWIR shortwave infrared group.

5. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 4, characterized in that, S3 specifically includes: S31. Obtain the image feature vector of the remote sensing image through the image encoder, wherein the network structure of the image encoder is the Inception structure; S32. Extract feature vectors of semantic prompt texts constructed for each band group using a text encoder. The network structure of the text encoder is a BERT structure, and a prompt template T is constructed for each group. i Define semantic embedding: T i =E t (t i ),i=1,2,...,N g In the above formula, N g Indicates the number of band groups.

6. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 5, characterized in that, In S4, the similarity between image features and text features is calculated to obtain the predicted image category label, including: calculating image features f(x) i ) and text features g(t) for each category j The similarity between the images is used to calculate the probability that a remote sensing image belongs to a specific category. The category with the highest probability value is taken as the predicted image category label: In the above formula, <·> represents cosine similarity, f(·) represents image encoder, g(·) represents text encoder, K represents the number of categories in the dataset, and τ represents temperature hyperparameter.

7. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 6, characterized in that, S5 specifically includes: S51, Based on the actual category label y i and the predicted image category labels obtained in step S4 Calculate the classification loss L cls : In the above formula, B represents the size of the batch; S52. Calculate the contrast loss L based on the image features and text features obtained in step S3. con ; S53, Constructing the total loss L total =L cls +λL con Where λ represents the balance coefficient, L cls L represents classification loss. con This represents the loss in comparison.

8. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 7, characterized in that, Step S52 specifically includes: S521, Based on the true category label y of the remote sensing image i Construct a supervision matrix gt, where gt is a B×B matrix, B is the size of the batch, and its elements are: In the above formula, the range of i and j is [0, B-1]; S522. Calculate the image-text similarity matrix i2t_logit and the text-image similarity matrix t2i_logit based on image features and text features; S523. Calculate the contrast loss L based on the text similarity matrix i2t_logit, the text-image similarity matrix t2i_logit, and the supervision matrix gt. con =(L i2t +L t2i ) / 2, where L i2t and L t2i Both use the BCEWithLogitsLoss loss function: BCEWithLogitsLoss={l1,...,l N },l n =-[y n ·log(σ(x n ))+(1-y n )·log(1-σ(x n ))] In the above formula, N represents the number of batches, n represents the number of labels predicted in each batch, and σ(·) represents the Sigmoid function.

9. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 8, characterized in that, Calculating the image-text similarity matrix i2t_logit includes: i2t_logit=I·T Τ In the above formula, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing images, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing images. Calculating the text-image similarity matrix t2i_logit includes: i2t_logit=T·I Τ In the above formula, · represents matrix multiplication, I is the image feature matrix of a batch of remote sensing images, and T is the text feature matrix of the prompts corresponding to a batch of remote sensing images.

10. The SAR and multispectral fusion method based on low-rank adaptive fine-tuning according to claim 9, characterized in that, In S523, the contrastive loss L is calculated based on the text similarity matrix i2t_logit, the text-image similarity matrix t2i_logit, and the supervision matrix gt. con include: The contrast loss L is calculated based on the supervision matrix gt and the image-text similarity matrix i2t_logit. i2t : L i2t =BCEWithLogitsLoss(i2t_logit,gt); The contrast loss L is calculated based on the step supervision matrix gt and the text-image similarity matrix t2i_logit. t2i : L t2i =BCEWithLogitsLoss(t2i_logit,gt)。

Images