Multimodal remote sensing image fusion segmentation method based on knowledge distillation
By adopting a multimodal fusion segmentation method based on knowledge distillation in remote sensing image semantic segmentation, the feature extraction and reconstruction module is simplified, and the problem of excessive calculation amount and parameter amount in the prior art is solved, and the segmentation effect with higher accuracy and practicality is achieved.
Patent Information
- Application Number
- CN202411767028.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art in semantic segmentation of remote sensing images has poor accuracy due to the large amount of calculation and parameters, and it is difficult to be practical on satellite platforms or edge devices.
A multimodal remote sensing image fusion segmentation method based on knowledge distillation is used to build a dual-branch encoder and decoder through residual blocks and convolution operations in deep learning. Combined with knowledge distillation technology, the feature extraction and reconstruction modules are simplified, and the calculation complexity and parameter quantity are reduced.
It realizes semantic segmentation of multimodal remote sensing images with less calculation and parameter amount, and is suitable for satellite platforms or edge devices, ensuring practicality and segmentation performance.
Smart Images
Figure CN119723080A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image interpretation, and in particular to a multi-modal remote sensing image fusion and segmentation method. Background Art
[0002] Remote sensing image semantic segmentation refers to classifying each pixel in a remote sensing image according to its semantic category, thereby achieving detailed analysis of the image content. This technology requires not only accurate delineation of the segmented area, but also accurate determination of the category corresponding to the area, providing high-quality data support for subsequent applications. It has important application value in tasks such as urban planning, disaster monitoring, and crop management.
[0003] In the existing technology, on the one hand, the shooting area of remote sensing images is wider and contains complex and diverse ground features, which usually leads to highly heterogeneous and complex categories; on the other hand, due to the particularity of the imaging mechanism, the targets in remote sensing images usually have shadow noise and are easily affected by clouds and lighting, resulting in lower quality or missing data collected. These two factors lead to poor accuracy of semantic segmentation of remote sensing images.
[0004] In order to solve the above problems, multimodal data is usually used to solve the above problems. Specifically, it takes remote sensing optical images as the main data and uses digital surface models (DSM) or synthetic aperture radar (SAR) images as supplementary data to improve the accuracy of semantic segmentation.
[0005] However, the above methods usually involve huge amounts of calculations and parameters, resulting in varying degrees of deficiencies in computational efficiency and accuracy. Summary of the invention
[0006] The purpose of the present invention is to provide a multimodal remote sensing image fusion segmentation method based on knowledge distillation, which can perform semantic segmentation on multimodal remote sensing images with less computational effort and parameter amount and higher accuracy, and has stronger practicality.
[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0008] A multimodal remote sensing image fusion segmentation method based on knowledge distillation, characterized by comprising:
[0009] S1. Obtain an open source data set and divide it into a training set, a validation set, and a test set; the open source data set includes remote sensing optical images and modality supplementary data;
[0010] S2. Design a basic model, use the residual block in deep learning to build a dual-branch encoder, process the remote sensing optical image and modality supplementary data, and extract multi-scale semantic features;
[0011] S3. Design a basic model, use convolution and deconvolution operations in deep learning to build a dual-branch decoder, process the extracted multi-scale semantic features, and generate semantic segmentation results;
[0012] S4, training the basic model and the visual segmentation large model SAM using the training set and the validation set; and adjusting the visual segmentation large model SAM;
[0013] S5. According to the knowledge distillation strategy, the segmentation capability of the large visual segmentation model SAM is transferred to the basic model to obtain the optimal basic model.
[0014] Therefore, the modal supplementary data specifically includes DSM images or SAR images, where DSM images are digital surface model images and SAR images are synthetic aperture radar images. Specifically, DSM images or SAR images are not used at the same time. Usually, remote sensing optical images are used in combination with corresponding DSM images or remote sensing optical images are used in combination with SAR images. Open source datasets such as Potsdam and Vaihingen datasets can be used, which include remote sensing optical images and DSM images; or WHU-OPT-SAR and DFC2023 datasets can be used, which include remote sensing optical images and SAR images. These datasets can be downloaded using scripts or manually.
[0015] Next, we designed a basic model and used the residual block in deep learning to build a dual-branch encoder. Specifically, we can use the Pytorch framework to build a Residual block. Each Residual block contains a convolutional layer, an activation function, and a jump connection. Multiple Residual blocks are combined together to form a dual-branch encoder to process remote sensing optical images and DSM (SAR) images respectively. The dual-branch encoder can effectively extract multi-scale features. The optical branch features are introduced into another branch to assist in extracting the correct multi-scale feature map for subsequent decoders.
[0016] Then, we designed a basic model, and used the convolution and deconvolution operations in deep learning to build a dual-branch decoder. We processed the semantic features of the two modalities separately, restored them to the original image size, and obtained the segmentation results of the two modalities. We then fused these results to obtain the final segmentation result. We used the Pytorch framework to build the Restore block, designed a dual-branch decoder, used convolution and deconvolution to process the multi-scale features output by the encoder, and gradually restored them to the original image size. We designed a fusion strategy to combine the segmentation results generated by each modality to generate the final semantic segmentation result.
[0017] Finally, the training set and the validation set are used to train the base model and the visual segmentation model SAM; and the visual segmentation model SAM is fine-tuned. Knowledge distillation is a machine learning technology that can compress and transfer the knowledge in a complex, large neural network model (usually called a "teacher" model) to a smaller, simpler neural network model (usually called a "student" model).
[0018] In the present invention, SAM is used as a teacher model and the basic model is used as a student model. The feature information and Logit information of the teacher model are used to guide the training of the student model and finally the optimal basic model is obtained. When the optimal basic model is used for semantic segmentation of multimodal remote sensing images, it has less calculation amount and parameter amount and higher accuracy.
[0019] In summary, the method proposed in the present invention achieves minimum parameters by simplifying the feature extraction and reconstruction modules, reduces computational complexity, reduces hardware requirements, and obtains excellent low-latency performance with less computational effort and parameter quantity, which can ensure the practicality of satellite platforms or edge devices; it also utilizes feature complementarity and knowledge distillation technology of multimodal data to ensure accuracy, thereby achieving optimal segmentation performance.
[0020] As a preferred embodiment of the present invention, the modality supplementary data includes DSM images or SAR images.
[0021] As a preferred embodiment of the present invention, S2 further comprises the following sub-steps:
[0022] S21. Use the Pytorch framework to build Residual blocks. Each Residual block contains a convolutional layer, an activation function, and a skip connection.
[0023] S22. Combine multiple Residual blocks together to form a dual-branch encoder.
[0024] As a preferred embodiment of the present invention, S3 further comprises the following sub-steps:
[0025] S31. Use the Pytorch framework to build the Restore block and design a dual-branch decoder;
[0026] S32. Use convolution and deconvolution to process the multi-scale features output by the dual-branch encoder and restore them to the original image size.
[0027] As a preferred embodiment of the present invention, S4 further comprises the following sub-steps:
[0028] S41, defining a loss function and an optimizer, using a training set to train the basic model and adjust parameters;
[0029] S42. Use the validation set to evaluate the performance of the basic model and adjust parameters.
[0030] As a preferred embodiment of the present invention, S5 further comprises the following sub-steps:
[0031] S51, using the visual segmentation large model SAM as a teacher model, the basic model as a student model, and using the feature information and Logit information of the teacher model to guide the training process of the student model;
[0032] S52. Define a distillation loss function to ensure the learning effectiveness of the student model;
[0033] S53. Based on the distillation loss function, retrain the basic model and save the optimal parameters.
[0034] As a preferred embodiment of the present invention, the distillation loss function is specifically:
[0035]
[0036] The pixel value is the smallest evaluation unit, where LossD represents the loss function value; s represents the student model feature; t represents the teacher model feature; w represents the width of the feature map; and h represents the length of the feature map.
[0037] In summary, the present invention has the following beneficial effects:
[0038] The method proposed in the present invention achieves minimum parameters by simplifying the feature extraction and reconstruction modules, reduces computational complexity, reduces hardware requirements, and obtains excellent low-latency performance with less computational effort and parameter quantity, thereby ensuring the practicality of satellite platforms or edge devices; it also utilizes feature complementarity and knowledge distillation technology of multimodal data to ensure accuracy, thereby achieving optimal segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flow chart of the method of the present invention;
[0040] Figure 2 It is a structural diagram of the basic model in the present invention;
[0041] Figure 3 a and b in the figure are the encoder and decoder structure diagrams of the basic model respectively;
[0042] Figure 4 Schematic diagram of the knowledge distillation process in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.
[0044] The terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variation thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0045] like Figure 1 As shown, in this embodiment:
[0046] First, focus on S1 and obtain open source datasets. Open source datasets can be selected, such as the Potsdam and Vaihingen datasets, which contain remote sensing optical images and DSM images; or the WHU-OPT-SAR and DFC2023 datasets, which contain remote sensing optical images and SAR images. These datasets can be downloaded using scripts or manually.
[0047] The open source dataset contains remote sensing optical images and modal supplementary data, wherein the modal supplementary data specifically includes DSM images or SAR images, wherein DSM images are digital surface model images, and SAR images are synthetic aperture radar images. Specifically, DSM images or SAR images are not used at the same time. Usually, remote sensing optical images are used in combination with corresponding DSM images or remote sensing optical images are used in combination with SAR images, that is, in one possible embodiment, the corresponding DSM images are selected as supplementary data for the remote sensing optical images; in another possible embodiment, the corresponding SAR images are selected as supplementary data for the remote sensing optical images.
[0048] Next, in order to facilitate subsequent model training, the remote sensing optical images in the acquired open source dataset were cropped. All remote sensing optical images were cropped to 256×256 size and randomly divided into training set, validation set and test set in a ratio of 7:2:1. The images were preprocessed, including normalization, denoising and image enhancement, to improve the training effect of the subsequent model.
[0049] Then in S2, a basic model is designed, and a dual-branch encoder is constructed using the residual block in deep learning to process optical images and DSM (SAR) images respectively to extract multi-scale semantic features. Figure 2 and Figure 3 As shown in Figure a, the Pytorch framework is used to construct the residual block. Each residual block contains a convolutional layer, an activation function, and a jump connection. Multiple residual blocks are combined together to form a dual-branch encoder. The two branches process the DSM (SAR) image and the remote sensing optical image (RGB) respectively. In this process, Figure 2 As shown in the figure, the optical branch (RGB) features are introduced into another branch to assist in extracting the correct multi-scale feature map for use by the subsequent dual-branch decoder.
[0050] Then in S3, the basic model is designed, and the convolution and deconvolution operations in deep learning are used to build a dual-branch decoder to process the semantic features of the two modalities respectively, restore the original image size to obtain the segmentation results of the two modalities, and fuse these results to obtain the final segmentation result. Figure 2 and Figure 3 As shown in Figure b, the Pytorch framework is used to build the Restore block and design a dual-branch decoder. Convolution and deconvolution are used to process the multi-scale features output by the dual-branch encoder and restore them to the original image size. In this process, each modality will generate its own segmentation result. Then, a fusion strategy is designed to combine the independent segmentation results generated by each modality to generate the final semantic segmentation output, that is, the final segmentation result. The final segmentation result Out can be expressed as:
[0051] Out=argmax(softmax(M,c=-1),c=-1),M∈R h×w×n
[0052] Among them, n represents the number of output channels, that is, the number of categories in the dataset, and c=-1 means performing softmax and argmax operations on the final dimension (i.e., channel).
[0053] Then in S4, the training set and validation set need to be used to train the basic model and the visual segmentation model SAM; and the visual segmentation model SAM needs to be adjusted. Specifically, the loss function and optimizer need to be defined. The loss function here can be the cross entropy loss, that is:
[0054]
[0055] The pixel value is taken as the smallest evaluation unit, where g and p represent the true value and predicted value respectively.
[0056] The training set is then used to train the base model and adjust the parameters; the validation set is then used to evaluate the performance of the base model and adjust the parameters.
[0057] Finally, in S5, according to the knowledge distillation strategy, the segmentation capability of the large visual segmentation model SAM is transferred to the basic model to obtain the optimal basic model. Knowledge distillation technology is a machine learning technology that can compress and transfer the knowledge in a complex, large neural network model (usually called a "teacher" model) to a smaller, simpler neural network model (usually called a "student" model). In the present invention, SAM is used as a teacher model and the basic model is used as a student model. The feature information and Logit information of the teacher model are used to guide the training of the student model and finally obtain the optimal basic model.
[0058] Specifically, in this embodiment, Figure 4 As shown in the figure, the visual segmentation model SAM is used as the teacher model, the basic model is used as the student model, and the feature information (DSM features and RGB features) and Logit information of the teacher model are used to guide the training process of the student model; then, the distillation loss function is defined to ensure the learning effectiveness of the student model; in this embodiment, the distillation loss function is specifically:
[0059]
[0060] The pixel value is taken as the smallest evaluation unit, where LossD represents the loss function value; s represents the student model feature; t represents the teacher model feature; w represents the width of the feature map; and h represents the length of the feature map.
[0061] Finally, based on the distillation loss function, the base model is retrained and the optimal parameters are saved.
[0062] The effect of the present invention can be further illustrated by the following experimental results:
[0063] 1) Experimental indicators:
[0064] In order to evaluate the performance of the algorithm, the present invention uses the common semantic segmentation networks MCANet, ESANet, RedNet, ACNet, and CMGFNet as comparison methods to highlight the effectiveness of the method. The present invention selects overall accuracy (OA), Kappa coefficient and mean intersection over union (mIoU) as evaluation indicators, and the calculation formulas of each indicator are as follows:
[0065]
[0066]
[0067] The pixel value is the smallest evaluation unit. In the above formula: TP (True Positive), the label is foreground, and the segmentation result is also the number of pixels of foreground; TN (True Negative), the label is foreground, and the segmentation result is the number of pixels of background; FP (False Positive), the label is background, and the segmentation result is the number of pixels of foreground; FN (False Negative), the label is background, and the detection result is also the number of pixels of background. m represents the number of categories in the data set; Pre and Gt take the entire image as a unit, representing the predicted value and the true value respectively.
[0068] 2) Comparison of performance with other methods:
[0069] Table 1. Quantitative comparison results with other methods on the Potsdam dataset
[0070] method OA↑ Kappa↑ mIoU↑ MCANet 91.0 87.5 78.1 ESANet 91.9 88.8 81.0 RedNet 91.1 87.6 80.5 ACNet 92.2 89.3 82.6 CMGFNet 92.8 90.0 82.0 Ours 94.1 91.2 87.6
[0071] Table 2. Quantitative comparison results with other methods on the Vaihingen dataset
[0072] method OA↑ Kappa↑ mIoU↑ MCANet 93.6 84.3 56.9 ESANet 94.6 86.7 67.2 RedNet 94.7 87.1 62.3 ACNet 94.8 87.1 64.5 CMGFNet 93.9 84.6 62.9 Ours 94.9 87.4 70.2
[0073] The comparison results on the Potsdam and Vaihingen datasets are shown in Tables 1 and 2. The experimental results show that the proposed method is superior to other methods in all indicators. On the Potsdam dataset, OA, Kappa, and mIoU reached 94.1%, 91.2%, and 87.6% respectively; on the Vaihingen dataset, OA, Kappa, and mIoU reached 94.9%, 87.4%, and 70.2% respectively.
[0074] 3) Comparison with other methods on efficiency:
[0075] Table 3. Comparison results with other methods in terms of computation, parameters and delay
[0076] method Calculation amount↓ Parameter quantity↓ Delay↓ MCANet 88.63 72.03 19.31 ESANet 11.52 54.41 9.35 RedNet 21.21 81.94 12.11 ACNet 26.47 116.6 14.83 CMGFNet 38.83 85.22 9.25 SAM 64.76 90.01 31.58 Ours 14.34 15.26 7.61
[0077] Table 3 lists the calculation, parameter and latency comparison results of all methods, all evaluated on RTX3090Ti. This test has a total of 1958 images and a batch size of 2, which is closer to the real reasoning process. The proposed method achieves minimum parameters by simplifying the feature extraction and reconstruction modules, reduces the computational complexity and reduces the hardware requirements. At the same time, it uses knowledge distillation technology to ensure accuracy, thereby achieving the best segmentation performance. In addition, its low latency ensures real-time effectiveness in practical applications.
[0078] In summary, the method proposed in the present invention is a better solution for solving the task of semantic segmentation of remote sensing images. It utilizes the feature complementarity and knowledge distillation technology of multimodal data to ensure accuracy, and ensures practicality on satellite platforms or edge devices with less computation, parameters and delay.
[0079] The above descriptions are only preferred embodiments disclosed in this application and descriptions of the technical principles used. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this disclosure (but not limited to) to form a technical solution.
[0080] In addition, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.
Claims
1. A multimodal remote sensing image fusion segmentation method based on knowledge distillation, characterized in that: include: S1. Obtain an open source dataset and divide it into training set, validation set and test set; The open source dataset contains remote sensing optical images and modality supplementary data; S2. Design a basic model, use the residual block in deep learning to build a dual-branch encoder, process the remote sensing optical image and modality supplementary data, and extract multi-scale semantic features; S3. Design a basic model, use convolution and deconvolution operations in deep learning to build a dual-branch decoder, process the extracted multi-scale semantic features, and generate semantic segmentation results; S4, training the basic model and the visual segmentation large model SAM using the training set and the validation set; and adjusting the visual segmentation large model SAM; S5. According to the knowledge distillation strategy, the segmentation capability of the large visual segmentation model SAM is transferred to the basic model to obtain the optimal basic model.
2. According to the multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, it is characterized in that: The modality supplementary data includes a DSM image or a SAR image.
3. According to the multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, it is characterized in that: S2 also includes sub-steps: S21. Use the Pytorch framework to build Residual blocks. Each Residual block contains a convolutional layer, an activation function, and a skip connection. S22. Combine multiple Residual blocks together to form a dual-branch encoder.
4. According to the multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, it is characterized in that: S3 also includes sub-steps: S31. Use the Pytorch framework to build the Restore block and design a dual-branch decoder; S32. Use convolution and deconvolution to process the multi-scale features output by the dual-branch encoder and restore them to the original image size.
5. According to the multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, it is characterized in that: S4 also includes sub-steps: S41, defining a loss function and an optimizer, using a training set to train the basic model and adjust parameters; S42. Use the validation set to evaluate the performance of the basic model and adjust parameters.
6. The multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, characterized in that: The loss function defined in S41 is the cross entropy loss.
7. The multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, characterized in that: S5 also includes sub-steps: S51, using the visual segmentation large model SAM as a teacher model, the basic model as a student model, and using the feature information and Logit information of the teacher model to guide the training process of the student model; S52. Define a distillation loss function to ensure the learning effectiveness of the student model; S53. Based on the distillation loss function, retrain the basic model and save the optimal parameters.
8. The multimodal remote sensing image fusion segmentation method based on knowledge distillation according to claim 1, characterized in that: The distillation loss function is specifically: The pixel value is the smallest evaluation unit, where LossD represents the loss function value; s represents the student model feature; t represents the teacher model feature; w represents the width of the feature map; and h represents the length of the feature map.