SAR image classification method based on multi-modal knowledge distillation transmission

Abstract

The invention discloses an SAR image classification method based on multi-modal knowledge distillation transmission, and relates to the technical field of land cover classification. According to the SAR image classification method based on multi-modal knowledge distillation transmission, a multi-modal knowledge distillation model including a teacher model and a student model is constructed, and the trained student model is utilized to perform classification identification on the SAR image to be detected. In the training process of the model, knowledge migration between different modal images and between image text modals is realized based on an optical-SAR knowledge migration strategy of knowledge distillation transmission, and multi-level semantic features can be extracted from a teacher model of a remote sensing optical image to assist learning of a student model of an SAR image. And the image semantics and the multi-label text semantics are aligned by an optimal transmission method, so that the defect of insufficient recognition capability of optical and SAR images under different environment conditions is overcome, and the accuracy of multi-label ground feature type judgment of the SAR image is effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of land cover classification, and in particular relates to a SAR (Synthetic Aperture Radar) image classification method based on multimodal knowledge distillation transmission. Background Art

[0002] Land cover classification is one of the biggest challenges in remote sensing and a key component of large-scale monitoring of natural and man-made phenomena. With the rapid development of remote sensing technology, researchers can obtain a large amount of earth observation data with considerable complex heterogeneity through various sensors. How to fuse multi-source data for multi-modal remote sensing image interpretation is becoming a more interesting issue. In the field of remote sensing data fusion, two modes have been widely studied: SAR images and remote sensing optical images.

[0003] Remote sensing optical images describe the same object using hundreds of bands. Each object has rich spectral characteristics, high resolution, clear color and texture features, and low acquisition cost. They can be used for fine classification perception of ground objects and environmental monitoring, but they are easily affected by weather and light interference. In particular, the imaging quality is seriously degraded under meteorological conditions such as rain, snow, clouds, and fog, which seriously damages the spectral and spatial information. Unlike remote sensing optical images, synthetic aperture radar is an active microwave sensor that is almost insensitive to these factors. Due to its long wavelength, its radiation can penetrate clouds, smoke, dust, and other climatic conditions except heavy rainfall. Due to this feature, SAR images can be obtained under almost all weather and environmental conditions. Although SAR images can provide complementary knowledge for remote sensing optical images, they are prone to speckle noise, which may seriously limit their practical application potential.

[0004] Traditional remote sensing optical image and SAR image fusion methods include fusion techniques based on spatial domain and transform domain. With the development of deep learning technology, fusion methods based on deep learning have gradually become a research hotspot. In recent years, researchers have proposed a variety of optical and SAR image fusion methods based on deep learning. Although these fusion methods are very successful, all methods tailor comparative learning competition strategies for SAR optical data fusion, which are difficult to migrate to other multimodal data and frameworks, resulting in low accuracy in multi-label ground object type discrimination in SAR images. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a SAR image classification method based on multimodal knowledge distillation transmission. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a SAR image classification method based on multimodal knowledge distillation transmission, comprising:

[0007] Step 1: Build a training dataset and a multimodal knowledge distillation model including a teacher model and a student model;

[0008] Step 2: pre-training the teacher model in the multimodal knowledge distillation model using the training data set to obtain a pre-trained teacher model;

[0009] Step 3: input the training data set into the pre-trained teacher model and the student model respectively, and train the student model using the training data set. During the training process of the student model, knowledge transfer between images of different modalities and between image texts is realized based on knowledge distillation transmission to obtain a trained student model;

[0010] Step 4: Use the trained student model to classify and identify the SAR image to be tested to obtain a SAR image classification result.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] The SAR image classification method based on multimodal knowledge distillation and transmission of the present invention constructs a multimodal knowledge distillation model including a teacher model and a student model, and uses the trained student model to classify and identify the SAR image to be tested. During the model training process, the optical-SAR knowledge transfer strategy based on knowledge distillation and transmission realizes the knowledge transfer between different modal images and between image text modalities, and can extract multi-level semantic features from the teacher model of remote sensing optical images to assist the learning of the student model of SAR images, and align the image semantics and multi-label text semantics with the optimal transmission method, thereby overcoming the insufficient recognition ability of optical and SAR images under different environmental conditions, and effectively improving the accuracy of multi-label ground object type discrimination of SAR images.

[0013] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of a SAR image classification method based on multimodal knowledge distillation and transmission provided by an embodiment of the present invention;

[0015] Figure 2 is an example diagram of land use classification scenario data provided by an embodiment of the present invention;

[0016] Figure 3is a schematic diagram of knowledge distillation provided by an embodiment of the present invention;

[0017] Figure 4 is a training flow chart of a multimodal knowledge distillation model provided by an embodiment of the present invention;

[0018] Figure 5 is a schematic diagram of a knowledge transfer architecture provided by an embodiment of the present invention;

[0019] Figure 6 It is a schematic diagram of image-text modal knowledge transfer based on optimal transmission provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, a SAR image classification method based on multimodal knowledge distillation transmission proposed by the present invention is described in detail below in combination with the accompanying drawings and specific implementation methods.

[0021] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0022] The present invention provides a SAR image classification method based on multimodal knowledge distillation and transmission. The SAR image classification method based on multimodal knowledge distillation and transmission of the present invention constructs a multimodal knowledge distillation model including a teacher model and a student model, trains the multimodal knowledge distillation model based on an optical-SAR knowledge transfer strategy of knowledge distillation and transmission, and uses the trained student model to classify and identify the SAR image to be tested.

[0023] See also Figure 4 , Figure 4 is a training flow chart of the multimodal knowledge distillation model provided by an embodiment of the present invention, such as Figure 4As shown, the training process of the multimodal knowledge distillation model of this embodiment includes three stages, wherein the first stage is to pre-train the teacher model using the remote sensing optical data of the training data set, and the second stage uses the pre-trained teacher model to guide the learning of the student model, and gradually adapts to the unique pattern of SAR data through knowledge distillation and attention mechanism optimization. In this process, the generalization ability of the model is ensured by closely monitoring the performance of the model on the validation set. As the training deepens, the student model begins to reduce the gap in feature representation with the teacher model while maintaining the classification accuracy, and gradually achieves the expected fusion effect. This process not only promotes the student model's deep understanding of SAR images, but also significantly improves its robustness in different environments. In the continuous iterative learning, the student model can capture more subtle feature differences and reduce the influence of weather and other factors, reflecting the ability of SAR image recognition all day and all weather. The third stage evaluates the student model by using an independent test data set.

[0024] Furthermore, combining Figure 1 The SAR image classification method based on multimodal knowledge distillation and transmission of this embodiment is described in detail. Figure 1 is a schematic diagram of a SAR image classification method based on multimodal knowledge distillation transmission provided by an embodiment of the present invention, such as Figure 1 As shown, the SAR image classification method based on multimodal knowledge distillation and transmission of this embodiment includes:

[0025] Step 1: Construct a training dataset and a multimodal knowledge distillation model including a teacher model and a student model.

[0026] In this embodiment, the training data set includes a plurality of image pairs of remote sensing optical images and corresponding SAR images, wherein both the remote sensing optical images and the SAR images are assigned with real category labels.

[0027] Optionally, you can select the data from the BigEarthNet dataset as the training dataset. The BigEarthNet dataset is a benchmark dataset consisting of 590,326 pairs of matched Sentinel-1 and Sentinel-2 image patches for multi-label, multi-class LULC (land cover) scene classification, with spectral band resolutions of 10, 20, and 60 meters. Each image patch is annotated with multiple land cover classes (i.e., multi-label) provided by the 2018 CORINE Land Cover (CLC) database according to its detailed Level3 class nomenclature. See Figure 2 , Figure 2 This is an example diagram of land use classification scenario data provided by an embodiment of the present invention.

[0028] It should be noted that, in this embodiment, the data in the selected BigEarthNet dataset can be preprocessed by removing noise, normalizing pixel values, etc. to ensure the consistency and quality of the data and make the data more suitable for subsequent matching model training and feature extraction.

[0029] In an optional embodiment, both the teacher model and the student model are Vision Transformer architectures.

[0030] The Vision Transformer (ViT) is an image model architecture based on the Transformer architecture, including a cascaded embedding layer, an encoding module and a multi-layer perceptron. The encoding module includes multiple cascaded encoding units, wherein the embedding layer is used to obtain the image vector distribution of the input image; the encoding module is used to extract the multi-layer image features of the input image according to the image vector distribution; the multi-layer perceptron is used to perform classification and recognition according to the multi-layer image features to obtain the predicted classification label of the input image.

[0031] Step 2: Use the training dataset to pre-train the teacher model in the multimodal knowledge distillation model to obtain a pre-trained teacher model.

[0032] In an optional embodiment, step 2 includes:

[0033] The remote sensing optical images in the training data set are input into the teacher model, and the model parameters of the teacher model are optimized according to the classification loss between the predicted classification labels of the remote sensing optical images output by the teacher model and the true category labels of the remote sensing optical images to obtain the pre-trained teacher model.

[0034] In this embodiment, the classification loss between the predicted classification label of the remote sensing optical image output by the teacher model and the real category label of the remote sensing optical image can be calculated using a cross entropy loss function. The model parameters of the teacher model can be optimized using a gradient descent optimization algorithm based on the classification loss to obtain a pre-trained teacher model.

[0035] Step 3: Input the training data set into the pre-trained teacher model and student model respectively, and use the training data set to train the student model. During the training process of the student model, knowledge transfer between images of different modalities and between image-text modalities is realized based on knowledge distillation transmission to obtain a trained student model.

[0036] Knowledge distillation is a knowledge transfer technology that allows the knowledge of the teacher model to be transferred to the student model. In this embodiment, knowledge distillation can be used to transfer the feature knowledge of remote sensing optical images to SAR image processing, thereby reducing the difference between modalities and improving the recognition ability of SAR images. Figure 3A schematic diagram of knowledge distillation provided by an embodiment of the present invention is shown.

[0037] In an optional embodiment, step 3 includes:

[0038] Step 3.1: Input the remote sensing optical image in the training data set into the pre-trained teacher model to obtain the multi-layer image features of the remote sensing optical image.

[0039] In this embodiment, the remote sensing optical image in the training data set is input into the pre-trained teacher model, and the encoding module of ViT can be used to extract the multi-layer image features corresponding to the remote sensing optical image.

[0040] Step 3.2: Input the SAR images in the training data set into the student model to obtain the image vector distribution, multi-layer image features and predicted classification labels of the SAR images.

[0041] In this embodiment, the SAR image paired with the remote sensing optical image in the training data set is input into the student model, and the image vector distribution, multi-layer image features and predicted classification labels corresponding to the SAR image can be obtained by using the embedding layer, encoding module and multi-layer perceptron of ViT.

[0042] Step 3.3: Input the true category labels of the SAR images in the training dataset into the Bert text encoder to obtain the label vector distribution of the SAR image.

[0043] The Bert text encoder is a pre-trained language model based on a multi-layer Transformer encoder. It can capture the bidirectional contextual information of the text by combining a word segmenter, multiple embedding layers, and a task-specific output layer.

[0044] Step 3.4: According to the multi-layer image features of the remote sensing optical image, the image vector distribution of the SAR image, the multi-layer image features and the predicted classification labels, and the label vector distribution of the SAR image, the constructed composite loss function is used to calculate the composite total loss based on knowledge distillation transfer, and the model parameters of the student model are optimized according to the composite total loss to obtain the trained student model.

[0045] In this embodiment, the composite total loss includes: knowledge distillation loss between multi-layer image features of the remote sensing optical image and multi-layer image features of the SAR image, classification loss between predicted classification labels of the SAR image and true category labels of the SAR image, and optimal transmission loss between image vector distribution of the SAR image and label vector distribution of the SAR image.

[0046] In this embodiment, the knowledge distillation loss includes the alignment distillation loss of each layer of image features in the multi-layer image features of the remote sensing optical image and the multi-layer image features of the SAR image. Through each layer of alignment distillation loss, each layer of image features of the student model is made as close as possible to the corresponding layer of image features of the pre-trained teacher model while maintaining good classification performance.

[0047] Exemplarily, the encoding module of ViT is assumed to include L cascaded encoding units. The L layers of image features of the input image can be extracted using the L cascaded encoding units. The image features output by the lth encoding unit of the student model are represented as S1, and the image features output by the lth encoding unit of the pre-trained teacher model are represented as T1. Then, the alignment distillation loss of the first layer of image features can be expressed as:

[0048] L distillation (1) = D ( S1 , T1 );

[0049] Among them, D(S1,T1) represents the distance metric between the image features of the student model and the l-th layer of the pre-trained teacher model, which can be expressed by mean square error (MSE) or KL divergence.

[0050] Then, the knowledge distillation loss L distill That is L distillation (l), l = 1, 2,…, the sum of L.

[0051] In this embodiment, the classification loss between the predicted classification label of the SAR image and the true category label of the SAR image can be calculated using a cross entropy loss function.

[0052] In this embodiment, knowledge distillation loss is used to help transfer knowledge between images of different modalities, namely, remote sensing optical images and SAR images. In addition, optimal transport (OT) loss is added in the final classification to help transfer knowledge between image and text features, so as to achieve more accurate label classification results.

[0053] It should be noted that when calculating the composite total loss, a hyperparameter needs to be set to balance the weights among the knowledge distillation loss, the classification loss, and the optimal transmission loss. In this embodiment, the constructed composite loss function can be expressed as:

[0054] L=L distill +αL class +βL OT ;

[0055] Among them, L distill is the knowledge distillation loss, L class is the classification loss, L OTis the optimal transmission loss, and α and β are preset hyperparameters.

[0056] It is understood that after obtaining the trained student model, it can be evaluated and verified using SAR image data, which can be done by evaluating the model using an independent test data set. Evaluation indicators can include accuracy, recall, F1 score, etc. to comprehensively evaluate the performance of the model on different tasks. If the model performance does not meet the requirements, it can be returned to continue training for adjustment and optimization.

[0057] Step 4: Use the trained student model to classify and identify the SAR image to be tested and obtain the SAR image classification result.

[0058] In this embodiment, the SAR image to be tested is input into the trained student model to obtain multi-classification prediction labels corresponding to the SAR image, thereby realizing classification and recognition of the SAR image to be tested.

[0059] The SAR image classification method based on multimodal knowledge distillation and transmission of the embodiment of the present invention constructs a multimodal knowledge distillation model including a teacher model and a student model, and uses the trained student model to classify and identify the SAR image to be tested. During the model training process, the optical-SAR knowledge transfer strategy based on knowledge distillation and transmission realizes the knowledge transfer between images of different modalities and between image text modalities, and can extract multi-level semantic features from the teacher model of remote sensing optical images to assist the learning of the student model of SAR images, and align the image semantics and multi-label text semantics with the optimal transmission method, which overcomes the insufficient recognition ability of optical and SAR images under different environmental conditions and effectively improves the accuracy of multi-label ground object type discrimination of SAR images.

[0060] The optical-SAR knowledge transfer strategy based on knowledge distillation and transfer proposed in the present invention includes an optical-SAR knowledge transfer strategy and an image-text modality knowledge transfer strategy. Figure 5 and Figure 6 As shown, Figure 5 is a schematic diagram of a knowledge transfer architecture provided by an embodiment of the present invention; Figure 6The figure is a schematic diagram of image-text modal knowledge transfer based on optimal transmission provided by an embodiment of the present invention. Among them, the optical-SAR knowledge transfer strategy can extract useful semantic knowledge from remote sensing optical images into a network model of a SAR image, study the performance gap between the two image modes from the perspective of feature graphs, and analyze the multi-layer feature differences between the two image modes in multiple stages of the network model, that is, the teacher model in the multimodal knowledge distillation model provides rich semantic information as a remote sensing optical image network, and the student model in the multimodal knowledge distillation model learns the remote sensing optical image features of the teacher model through knowledge distillation as a SAR image network, thereby realizing cross-modal knowledge transfer, and can effectively integrate the complementary features of remote sensing optical images and SAR images. By migrating the feature knowledge of remote sensing optical images to SAR image processing, the difference between modalities is reduced, and the recognition accuracy and generalization ability of the model are improved. The image-text modal knowledge transfer strategy is to convert the SAR image into a set distribution after ViT slicing, and the multi-label text of the SAR image into a feature embedding distribution of the text domain. The distance between the two distributions is optimized by calculating the OT (Optimal Transport) optimal transmission algorithm, and the knowledge transfer between the image domain and the text domain is performed, which effectively improves the accuracy of the final classification.

[0061] Furthermore, the effect of the SAR image classification method based on multimodal knowledge distillation and transmission of the present invention is illustrated through simulation experiments.

[0062] The classification accuracy of the student model after knowledge distillation of the present invention and the network model trained only with SAR images are compared and tested, and the comparison test results are shown in Table 1. It can be seen that the student model after knowledge distillation of the present invention can significantly improve the classification accuracy on a large-scale benchmark dataset, that is, the student model after knowledge distillation can eliminate the interference of the optical environment, integrate optical knowledge into the SAR field, and greatly improve the accuracy of SAR image classification.

[0063] Table 1

[0064] Single modality training Distillation strategy 1 Distillation Strategy 2 Distillation Strategy 3 Teacher Model 88％ -- -- -- Student Model 77.5％ 78％ 80％ 83.5％

[0065] Table 1 also shows the impact of different distillation strategies on classification accuracy. Distillation strategy 1 is the label-level embedded distillation method, distillation strategy 2 is the deep feature distillation method, and distillation strategy 3 is the shallow feature distillation method, which is the optical-SAR knowledge transfer strategy proposed in this paper. The purpose of label embedding distillation is to improve the distillation performance by exploiting the structural relationship between intra-class label embeddings. In general, the same semantic objects from two different images often differ from each other due to their respective characteristics, and these uniquenesses confuse the information of the corresponding classification task. Due to its strong learning ability, the teacher model is able to capture a highly abstract semantic representation of each class label by ignoring useless individual information. From the perspective of the learned feature representation, the teacher model tends to obtain a more compact intra-class label embedding structure, which often leads to better classification performance. By transferring structural knowledge from the teacher model to the student model, the student model is encouraged to improve the compactness of the intra-class label embedding, thereby improving its classification performance. Shallow features refer to the features extracted by the BLOCK module of the first 50% layer of the ViT model, and deep features refer to the features extracted by the BLOCK module of the last 50% layer. It can also be seen from Table 1 that the optical-SAR knowledge transfer strategy proposed in the present invention achieves the best classification accuracy on the student model, reaching 83.5%. This shows that through the optical-SAR knowledge transfer strategy proposed in the present invention, the student model can more effectively learn the high-level feature representation of the remote sensing optical image of the teacher model, thereby showing higher performance in the classification task.

[0066] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant is intended to cover non-exclusive inclusion, so that the article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the article or device including the elements. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The orientation or position relationship indicated by "up", "down", "left", "right", etc. is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

[0067] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A SAR image classification method based on multimodal knowledge distillation and transmission, characterized in that: include: Step 1: Build a training dataset and a multimodal knowledge distillation model including a teacher model and a student model; Step 2: pre-training the teacher model in the multimodal knowledge distillation model using the training data set to obtain a pre-trained teacher model; Step 3: input the training data set into the pre-trained teacher model and the student model respectively, and train the student model using the training data set. During the training process of the student model, knowledge transfer between images of different modalities and between image-text modalities is realized based on knowledge distillation transmission to obtain a trained student model; Step 4: Use the trained student model to classify and identify the SAR image to be tested to obtain a SAR image classification result.

2. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 1 is characterized in that: The training data set includes a plurality of image pairs of remote sensing optical images and corresponding SAR images, wherein both the remote sensing optical images and the SAR images are assigned real category labels.

3. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 1, characterized in that: Both the teacher model and the student model are Vision Transformer architectures.

4. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 3 is characterized in that: The Vision Transformer architecture includes a cascaded embedding layer, an encoding module and a multi-layer perceptron, wherein the encoding module includes a plurality of cascaded encoding units, wherein: The embedding layer is used to obtain the image vector distribution of the input image; The encoding module is used to extract and obtain multi-layer image features of the input image according to the image vector distribution; The multi-layer perceptron is used to perform classification and recognition according to the multi-layer image features to obtain a predicted classification label of the input image.

5. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 2, characterized in that: The step 2 comprises: The remote sensing optical images in the training data set are input into the teacher model, and the model parameters of the teacher model are optimized according to the classification loss between the predicted classification label of the remote sensing optical image output by the teacher model and the true category label of the remote sensing optical image to obtain the pre-trained teacher model.

6. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 2, characterized in that: The step 3 comprises: Step 3.1: inputting the remote sensing optical image in the training data set into the pre-trained teacher model to obtain multi-layer image features of the remote sensing optical image; Step 3.2: Input the SAR image in the training data set into the student model to obtain the image vector distribution, multi-layer image features and predicted classification labels of the SAR image; Step 3.3: Input the true category labels of the SAR images in the training data set into the Bert text encoder to obtain the label vector distribution of the SAR images; Step 3.4: According to the multi-layer image features of the remote sensing optical image, the image vector distribution of the SAR image, the multi-layer image features and the predicted classification labels, and the label vector distribution of the SAR image, the constructed composite loss function is used to calculate the composite total loss based on knowledge distillation transfer, and the model parameters of the student model are optimized according to the composite total loss to obtain the trained student model.

7. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 6, characterized in that: The composite total loss includes: knowledge distillation loss between multi-layer image features of the remote sensing optical image and multi-layer image features of the SAR image, classification loss between predicted classification labels of the SAR image and true category labels of the SAR image, and optimal transmission loss between image vector distribution of the SAR image and label vector distribution of the SAR image.

8. The SAR image classification method based on multimodal knowledge distillation and transmission according to claim 7, characterized in that: The knowledge distillation loss includes an alignment distillation loss of each layer of image features in the multi-layer image features of the remote sensing optical image and the multi-layer image features of the SAR image.

Images