A SAR image classification method based on multi-modal knowledge distillation transmission

Abstract

The application discloses a SAR image classification method based on multi-modal knowledge distillation transmission and relates to the technical field of land cover classification.The SAR image classification method based on multi-modal knowledge distillation transmission constructs a multi-modal knowledge distillation model including a teacher model and a student model, and uses the trained student model to classify and identify the SAR image to be measured.In the training process of the model, the optical-SAR knowledge migration strategy based on the knowledge distillation transmission realizes the knowledge migration between different modal images and between the image and text modalities, can extract multi-level semantic features from the teacher model of the remote sensing optical image to assist the learning of the student model of the SAR image, and aligns the image semantics and the multi-label text semantics by using an optimal transmission method, so that the insufficient recognition ability of the optical and SAR images under different environmental conditions is overcome, and the precision of the SAR image multi-label ground object type discrimination is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of land cover classification technology, specifically relating to a SAR (Synthetic Aperture Radar) image classification method based on multimodal knowledge distillation transmission. Background Technology

[0002] Land cover classification is one of the greatest challenges in remote sensing and a key component of large-scale monitoring of natural and anthropogenic phenomena. With the rapid development of remote sensing technology, researchers can obtain a large amount of highly complex and heterogeneous Earth observation data through various sensors. How to fuse multi-source data for multimodal remote sensing image interpretation is becoming an increasingly important issue. In the field of remote sensing data fusion, two modes have been extensively studied: SAR images and remote sensing optical images.

[0003] Remote sensing optical images describe the same object using hundreds of bands, each with rich spectral characteristics, high resolution, clear color and texture features, and low acquisition cost. They can be used for detailed ground feature classification and environmental monitoring. However, they are susceptible to weather and lighting interference, especially under meteorological conditions such as rain, snow, clouds, and fog, where image quality deteriorates significantly, severely damaging spectral and spatial information. Unlike remote sensing optical images, synthetic aperture radar (SAR) is an active microwave sensor, almost insensitive to these factors. Due to its long wavelength, its radiation can penetrate clouds, smoke, dust, and other climatic conditions except for heavy rainfall. Because of this characteristic, SAR images can be obtained under almost all weather and environmental conditions. While SAR images can provide supplementary knowledge to remote sensing optical images, they are prone to speckle noise, which can severely limit their practical application potential.

[0004] Traditional methods for fusing remote sensing optical images and SAR images include spatial domain and transform domain based fusion techniques. With the development of deep learning technology, deep learning-based fusion methods have gradually become a research hotspot. In recent years, researchers have proposed a variety of deep learning-based optical and SAR image fusion methods. Although these fusion methods have been very successful, all of them are tailored to the contrastive learning competition strategy for SAR optical data fusion, making it difficult to transfer to other multimodal data and frames. This results in relatively low accuracy in identifying multi-label land cover types in SAR images. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a SAR image classification method based on multimodal knowledge distillation transmission. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a SAR image classification method based on multimodal knowledge distillation transmission, comprising:

[0007] Step 1: Construct a training dataset and a multimodal knowledge distillation model that includes teacher and student models;

[0008] Step 2: Use the training dataset to pre-train the teacher model in the multimodal knowledge distillation model to obtain a pre-trained teacher model;

[0009] Step 3: Input the training dataset into the pre-trained teacher model and the student model respectively, and train the student model using the training dataset. During the training process of the student model, knowledge transfer between different modal images and between images and text is realized based on knowledge distillation transfer to obtain the trained student model.

[0010] Step 4: Use the trained student model to classify and identify the SAR image under test, and obtain the SAR image classification result.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0012] This invention presents a SAR image classification method based on multimodal knowledge distillation transfer. It constructs a multimodal knowledge distillation model comprising a teacher model and a student model, and uses the trained student model to classify and identify the SAR image under test. During model training, an optical-SAR knowledge transfer strategy based on knowledge distillation transfer enables knowledge transfer between different modalities of images and between image-text modalities. Multi-level semantic features can be extracted from the teacher model of the remote sensing optical image to assist the learning of the student model of the SAR image. Furthermore, the optimal transfer method is used to align image semantics and multi-label text semantics, overcoming the shortcomings of optical and SAR images in recognizing different environmental conditions and effectively improving the accuracy of multi-label land cover type discrimination in SAR images.

[0013] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a SAR image classification method based on multimodal knowledge distillation transmission provided in an embodiment of the present invention;

[0015] Figure 2 This is an example diagram of land use classification scenario data provided in an embodiment of the present invention;

[0016] Figure 3This is a schematic diagram of knowledge distillation provided in an embodiment of the present invention;

[0017] Figure 4 This is a flowchart of the training process of the multimodal knowledge distillation model provided in this embodiment of the invention;

[0018] Figure 5 This is a schematic diagram of the knowledge transfer architecture provided in an embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of image-text modal knowledge transfer based on optimal transmission provided in an embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail a SAR image classification method based on multimodal knowledge distillation transmission proposed according to the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0021] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0022] This invention provides a SAR image classification method based on multimodal knowledge distillation transmission. The method constructs a multimodal knowledge distillation model including a teacher model and a student model. The multimodal knowledge distillation model is trained based on an optical-SAR knowledge transfer strategy of knowledge distillation transmission. The trained student model is then used to classify and identify the SAR image under test.

[0023] Please see Figure 4 , Figure 4 This is a flowchart of the training process of the multimodal knowledge distillation model provided in this embodiment of the invention, such as... Figure 4As shown, the training process of the multimodal knowledge distillation model in this embodiment includes three stages. The first stage involves pre-training the teacher model using remote sensing optical data from the training dataset. The second stage uses the pre-trained teacher model to guide the learning of the student model, optimizing it through knowledge distillation and attention mechanisms to gradually adapt to the unique patterns of SAR data. During this process, the model's performance on the validation set is closely monitored to ensure its generalization ability. As training progresses, the student model begins to maintain classification accuracy while reducing the gap in feature representation with the teacher model, gradually achieving the expected fusion effect. This process not only promotes the student model's deeper understanding of SAR images but also significantly improves its robustness in different environments. Through continuous iterative learning, the student model can capture more subtle feature differences and reduce the impact of weather and other factors, demonstrating its all-weather, all-day SAR image recognition capability. The third stage evaluates the student model using an independent test dataset.

[0024] Furthermore, it will be combined Figure 1 The SAR image classification method based on multimodal knowledge distillation transmission in this embodiment will be described in detail. Figure 1 This is a schematic diagram of a SAR image classification method based on multimodal knowledge distillation transmission provided by an embodiment of the present invention, as shown below. Figure 1 As shown, the SAR image classification method based on multimodal knowledge distillation transmission in this embodiment includes:

[0025] Step 1: Construct a training dataset and a multimodal knowledge distillation model that includes teacher and student models.

[0026] In this embodiment, the training dataset includes multiple pairs of remote sensing optical images and corresponding SAR images, wherein both the remote sensing optical images and the SAR images are assigned true class labels.

[0027] Optionally, data from the BigEarthNet dataset can be selected as the training dataset. The BigEarthNet dataset is a benchmark dataset containing 590,326 pairs of matched Sentinel-1 and Sentinel-2 image patches for multi-label, multi-class LULC (land cover) scene classification, with spectral resolutions of 10, 20, and 60 meters. Each image patch is annotated with multiple land cover classes (i.e., multi-label) from the 2018 CORINE Land Cover (CLC) database according to its detailed Level 3 class nomenclature. See [link to relevant documentation]. Figure 2 , Figure 2 This is an example diagram of land use classification scenario data provided in an embodiment of the present invention.

[0028] It should be noted that, in this embodiment, the selected BigEarthNet dataset can be preprocessed, such as removing noise and normalizing pixel values, to ensure data consistency and quality, making the data more suitable for subsequent matching model training and feature extraction.

[0029] In an optional embodiment, both the teacher model and the student model are based on the Vision Transformer architecture.

[0030] The Vision Transformer (ViT) is an image model architecture based on the Transformer architecture, which includes cascaded embedding layers, encoding modules, and multilayer perceptrons. The encoding module consists of multiple cascaded encoding units. The embedding layer is used to obtain the image vector distribution of the input image; the encoding module is used to extract multilayer image features of the input image based on the image vector distribution; and the multilayer perceptron is used to perform classification and recognition based on the multilayer image features to obtain the predicted classification label of the input image.

[0031] Step 2: Use the training dataset to pre-train the teacher model in the multimodal knowledge distillation model to obtain the pre-trained teacher model.

[0032] In an optional embodiment, step 2 includes:

[0033] The remote sensing optical images in the training dataset are input into the teacher model. The model parameters of the teacher model are optimized based on the classification loss between the predicted classification labels of the remote sensing optical images output by the teacher model and the true category labels of the remote sensing optical images, thus obtaining the pre-trained teacher model.

[0034] In this embodiment, the classification loss between the predicted classification label of the remote sensing optical image output by the teacher model and the true category label of the remote sensing optical image can be calculated using the cross-entropy loss function. Based on this classification loss, the model parameters of the teacher model can be optimized using the gradient descent optimization algorithm to obtain a pre-trained teacher model.

[0035] Step 3: Input the training dataset into the pre-trained teacher model and student model respectively, and train the student model using the training dataset. During the training process of the student model, knowledge transfer between different modal images and between image and text modalities is achieved based on knowledge distillation transfer, resulting in the trained student model.

[0036] Knowledge distillation is a knowledge transfer technique that allows knowledge from a teacher model to a student model. In this embodiment, knowledge distillation can be used to transfer feature knowledge from remote sensing optical images to SAR image processing, thereby reducing intermodal differences and improving the recognition capability of SAR images. Figure 3The diagram shown is a schematic representation of knowledge distillation provided by an embodiment of the present invention.

[0037] In an optional embodiment, step 3 includes:

[0038] Step 3.1: Input the remote sensing optical images from the training dataset into the pre-trained teacher model to obtain multi-layer image features of the remote sensing optical images.

[0039] In this embodiment, remote sensing optical images from the training dataset are input into the pre-trained teacher model, and the multi-layer image features corresponding to the remote sensing optical images can be extracted using the encoding module of ViT.

[0040] Step 3.2: Input the SAR images from the training dataset into the student model to obtain the image vector distribution, multi-layer image features, and predicted classification labels of the SAR images.

[0041] In this embodiment, SAR images paired with remote sensing optical images from the training dataset are input into the student model. The image vector distribution, multi-layer image features, and predicted classification labels corresponding to the SAR images can be obtained by using the embedding layer, coding module, and multilayer perceptron of ViT.

[0042] Step 3.3: Input the true class labels of the SAR images in the training dataset into the BERT text encoder to obtain the label vector distribution of the SAR images.

[0043] BERT Text Encoder is a pre-trained language model based on a multi-layer Transformer encoder. By combining a tokenizer, multiple embedding layers, and a task-specific output layer, it can capture bidirectional contextual information of text.

[0044] Step 3.4: Based on the multi-layer image features of the remote sensing optical image, the image vector distribution of the SAR image, the multi-layer image features and predicted classification labels, and the label vector distribution of the SAR image, calculate the composite total loss based on knowledge distillation transmission using the constructed composite loss function. Optimize the model parameters of the student model based on the composite total loss to obtain the trained student model.

[0045] In this embodiment, the total composite loss includes: knowledge distillation loss between the multi-layer image features of the remote sensing optical image and the multi-layer image features of the SAR image, classification loss between the predicted classification label of the SAR image and the true category label of the SAR image, and optimal transmission loss between the image vector distribution of the SAR image and the label vector distribution of the SAR image.

[0046] In this embodiment, the knowledge distillation loss includes the alignment distillation loss for each layer of image features in the multi-layer image features of the remote sensing optical image and the multi-layer image features of the SAR image. The alignment distillation loss for each layer ensures that the image features of each layer in the student model are as close as possible to the corresponding layer image features of the pre-trained teacher model, while maintaining good classification performance.

[0047] For example, suppose the ViT encoding module includes L cascaded encoding units. Using these L cascaded encoding units, L layers of image features from the input image can be extracted. The image features output by the l-th encoding unit of the student model are denoted as S1, and the image features output by the l-th encoding unit of the pre-trained teacher model are denoted as T1. Then, the alignment distillation loss of the first layer of image features can be expressed as:

[0048] L distillation (1) = D(S1,T1);

[0049] Where D(S1,T1) represents the distance metric between the l-th layer image features of the student model and the pre-trained teacher model, which can be represented by mean squared error (MSE) or KL divergence.

[0050] So, the knowledge distillation loss L distill That is, L distillation (l), the sum of l = 1, 2, ..., L.

[0051] In this embodiment, the classification loss between the predicted classification label of the SAR image and the true category label of the SAR image can be calculated using the cross-entropy loss function.

[0052] In this embodiment, knowledge distillation loss is used to help with knowledge transfer between different modalities of images, namely remote sensing optical images and SAR images. In addition, optimal transport (OT) loss is added during the final classification to help with knowledge transfer between image and text features, so as to achieve more accurate label classification results.

[0053] It should be noted that a hyperparameter needs to be set when calculating the composite total loss to balance the weights among the knowledge distillation loss, classification loss, and optimal transmission loss. In this embodiment, the constructed composite loss function can be expressed as:

[0054] L = L distill +αL class +βL OT ;

[0055] Among them, L distill For knowledge distillation loss, L class For classification loss, L OTFor optimal transmission loss, α and β are preset hyperparameters.

[0056] Understandably, after obtaining the trained student model, it can be evaluated and validated using SAR image data. This can be done by evaluating the model using an independent test dataset. Evaluation metrics can include accuracy, recall, F1 score, etc., to comprehensively assess the model's performance on different tasks. If the model's performance does not meet the requirements, it can be returned to training for adjustments and optimizations.

[0057] Step 4: Use the trained student model to classify and identify the SAR image under test, and obtain the SAR image classification results.

[0058] In this embodiment, the SAR image to be tested is input into the trained student model to obtain the multi-class prediction label corresponding to the SAR image, thereby realizing the classification and recognition of the SAR image to be tested.

[0059] This invention presents a SAR image classification method based on multimodal knowledge distillation transfer. It constructs a multimodal knowledge distillation model comprising a teacher model and a student model, and uses the trained student model to classify and identify the SAR image under test. During model training, an optical-SAR knowledge transfer strategy based on knowledge distillation transfer enables knowledge transfer between different modalities of images and between image-text modalities. Multi-level semantic features can be extracted from the teacher model of the remote sensing optical image to assist the learning of the student model of the SAR image. Furthermore, the optimal transfer method is used to align image semantics and multi-label text semantics, overcoming the shortcomings of optical and SAR images in recognizing different environmental conditions and effectively improving the accuracy of multi-label land cover type discrimination in SAR images.

[0060] The optical-SAR knowledge transfer strategy based on knowledge distillation proposed in this invention includes an optical-SAR knowledge transfer strategy and an image-text modality knowledge transfer strategy, such as... Figure 5 and Figure 6 As shown, Figure 5 This is a schematic diagram of the knowledge transfer architecture provided in an embodiment of the present invention; Figure 6This is a schematic diagram of image-text modal knowledge transfer based on optimal transport provided in this embodiment of the invention. The optical-SAR knowledge transfer strategy extracts useful semantic knowledge from remote sensing optical images into a network model of a SAR image. It studies the performance gap between the two image modes from the perspective of feature maps, analyzing the multi-layer feature differences between the two image modes at multiple stages of the network model. Specifically, the teacher model in the multimodal knowledge distillation model provides rich semantic information for the remote sensing optical image network, while the student model in the multimodal knowledge distillation model learns the remote sensing optical image features of the teacher model through knowledge distillation, thereby achieving cross-modal knowledge transfer. This effectively integrates the complementary features of remote sensing optical images and SAR images, reducing intermodal differences and improving the model's recognition accuracy and generalization ability by transferring feature knowledge from remote sensing optical images to SAR image processing. The image-text modal knowledge transfer strategy converts the SAR image into a ViT-segmented ensemble distribution and the multi-labeled text of the SAR image into a text-domain feature embedding distribution. The OT (Optimal Transport) algorithm is used to calculate and optimize the distance between the two distributions, performing knowledge transfer between the image and text domains, effectively improving the final classification accuracy.

[0061] Furthermore, simulation experiments are conducted to illustrate the effectiveness of the SAR image classification method based on multimodal knowledge distillation transmission of the present invention.

[0062] The classification accuracy of the knowledge-distilled student model of this invention was compared with that of a network model trained solely on SAR images. The results are shown in Table 1. It can be seen that the knowledge-distilled student model of this invention significantly improves classification accuracy on large-scale benchmark datasets. That is, the knowledge-distilled student model can eliminate optical environmental interference and integrate optical knowledge into the SAR domain, resulting in a significant improvement in the accuracy of SAR image classification.

[0063] Table 1

[0064] Single-modal training Distillation Strategy 1 Distillation Strategy 2 Distillation Strategy 3 Teacher Model 88％ -- -- -- Student Model 77.5％ 78％ 80％ 83.5％

[0065] Table 1 also shows the impact of different distillation strategies on classification accuracy. Distillation strategy 1 is label-level embedding distillation, distillation strategy 2 is deep feature distillation, and distillation strategy 3 is shallow feature distillation, which is the optical-SAR knowledge transfer strategy proposed in this invention. The purpose of label embedding distillation is to improve distillation performance by utilizing the structural relationships between intra-class label embeddings. Generally, the same semantic objects from two different images are often different from each other due to their respective characteristics, and these uniquenesses can confuse the information of the corresponding classification task. Due to its powerful learning ability, the teacher model can capture a highly abstract semantic representation of each class label by ignoring useless individual information. From the perspective of the learned feature representation, the teacher model tends to obtain a more compact intra-class label embedding structure, which often leads to better classification performance. By transferring structural knowledge from the teacher model to the student model, the student model is encouraged to improve the compactness of intra-class label embeddings, thereby improving its classification performance. Shallow features refer to the features extracted using the BLOCK module of the first 50% of the ViT model, and deep features refer to the features extracted using the BLOCK module of the last 50% of the model. As can be seen from Table 1, the optical-SAR knowledge transfer strategy proposed in this invention achieved the best classification accuracy of 83.5% on the student model. This indicates that through the optical-SAR knowledge transfer strategy proposed in this invention, the student model can more effectively learn the high-level feature representation of the remote sensing optical image of the teacher model, thus exhibiting higher performance in the classification task.

[0066] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device comprising said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by terms such as "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0067] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A SAR image classification method based on multimodal knowledge distillation transmission, characterized in that, include: Step 1: Construct a training dataset and a multimodal knowledge distillation model including a teacher model and a student model; the training dataset includes multiple image pairs of remote sensing optical images and corresponding SAR images, wherein the remote sensing optical images and the SAR images are all assigned true class labels; the teacher model and the student model are both based on the Vision Transformer architecture. Step 2: Use the training dataset to pre-train the teacher model in the multimodal knowledge distillation model to obtain a pre-trained teacher model; Step 3: Input the training dataset into the pre-trained teacher model and the student model respectively, and train the student model using the training dataset. During the training process of the student model, knowledge transfer between different modalities of images and between image and text modalities is achieved based on knowledge distillation transfer, resulting in a trained student model; Step 3 includes: Step 3.1: Input the remote sensing optical images in the training dataset into the pre-trained teacher model to obtain the multi-layer image features of the remote sensing optical images; Step 3.2: Input the SAR images from the training dataset into the student model to obtain the image vector distribution, multi-layer image features, and predicted classification labels of the SAR images; Step 3.3: Input the true class labels of the SAR images in the training dataset into the BERT text encoder to obtain the label vector distribution of the SAR images; Step 3.4: Based on the multi-layer image features of the remote sensing optical image, the image vector distribution of the SAR image, the multi-layer image features and predicted classification labels, and the label vector distribution of the SAR image, calculate the composite total loss based on knowledge distillation transmission using the constructed composite loss function, optimize the model parameters of the student model based on the composite total loss, and obtain the trained student model. The composite total loss includes: knowledge distillation loss between the multi-layer image features of the remote sensing optical image and the multi-layer image features of the SAR image; classification loss between the predicted classification label of the SAR image and the true category label of the SAR image; and optimal transmission loss between the image vector distribution of the SAR image and the label vector distribution of the SAR image. The knowledge distillation loss includes alignment distillation loss for each layer of image features in the multi-layer image features of the remote sensing optical image and the multi-layer image features of the SAR image. Step 4: Use the trained student model to classify and identify the SAR image under test, and obtain the SAR image classification result.

2. The SAR image classification method based on multimodal knowledge distillation transmission according to claim 1, characterized in that, The Vision Transformer architecture includes cascaded embedding layers, encoding modules, and multilayer perceptrons. The encoding module comprises multiple cascaded encoding units, wherein... The embedding layer is used to obtain the image vector distribution of the input image; The encoding module is used to extract multi-layer image features of the input image based on the image vector distribution; The multilayer perceptron is used to classify and identify based on the multilayer image features to obtain the predicted classification label of the input image.

3. The SAR image classification method based on multimodal knowledge distillation transmission according to claim 1, characterized in that, Step 2 includes: The remote sensing optical images in the training dataset are input into the teacher model. The model parameters of the teacher model are optimized based on the classification loss between the predicted classification label of the remote sensing optical image output by the teacher model and the true category label of the remote sensing optical image, thus obtaining the pre-trained teacher model.

Images