Remote sensing image segmentation method and system based on modal balance knowledge distillation framework

Abstract

The application provides a remote sensing image segmentation method and system based on a modal balance knowledge distillation framework. For optical and SAR image cross-modal transfer segmentation tasks, the modal balance knowledge distillation framework is innovatively composed of three parts, namely, double-modal supervised learning, mixed-modal virtual learning and double-modal knowledge reasoning, by learning from domain adaptation and knowledge distillation methods. The framework innovatively converts the cross-modal problem into a same-modal knowledge transfer problem by establishing a virtual modal. In the modal balance knowledge distillation, the source domain and the target domain are respectively converted into virtual modal representations similar to the target domain and the source domain. The application aims to eliminate the transfer direction misalignment problem caused by modal imbalance in the cross-modal transfer task, and to realize the learning ability of different modal balance in a more affinitive way.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a remote sensing image segmentation method and system based on a modal equilibrium knowledge distillation framework. Background Technology

[0002] Land cover / land use (LULC) is fundamental geographic information, making significant contributions to climate change research, ecological environment monitoring, and land resource management. Remote sensing imagery is the primary carrier of information about the Earth's surface. In recent years, with the continuous advancement of remote sensing imaging technology, in addition to a large number of optical satellites, synthetic aperture radar (SAR) satellites have also flourished, such as Germany's TerraSAR-X, Japan's ALOS-2, the European Space Agency's Sentinel-1, and China's GaoFen-3 satellite. SAR is an active remote sensing system, possessing the advantage of being unaffected by clouds and fog. Even when optical sensors fail, SAR can continue to record surface information. The comprehensive utilization of optical and SAR satellite imagery for semantic segmentation has become an important approach to achieving all-weather, all-time LULC mapping.

[0003] Currently, the field contains a large number of LULC classification datasets for optical images, such as ISPRS, AID, and GID. Due to the limitations of different imaging principles, the representation of ground features in SAR images differs significantly from that in optical images, requiring a high level of expertise for ground feature labeling. Therefore, there are far fewer LULC classification datasets for SAR images than for optical images, such as WHU-OPT-SAR and SEN12MS, making it difficult to support the growing research and application of SAR image LULC classification. The completely different imaging mechanisms of optical and SAR images lead to significant differences in their ground feature characteristics. When a CNN-based classification model trained using already labeled optical images (source domain) performs significantly worse on unlabeled SAR images (target domain), such as… Figure 1 As shown. To address this problem, the aim is to design a transfer learning method to break down modal barriers and obtain a general semantic segmentation model with similar segmentation capabilities for both optical and SAR images, while training the model using only optical images. This generality means that the model achieves similar and balanced performance for images of different modalities.

[0004] Optical and SAR images have significant differences in features, and current methods primarily rely on transfer learning to achieve cross-modal segmentation capabilities for semantic segmentation models. Transfer learning-based semantic segmentation methods mainly achieve model transfer by reducing the distributional differences between the source and target domains. Classic methods include domain adaptation and knowledge distillation. However, the models obtained using these methods may be biased towards either the source or target domain, often lacking general segmentation capabilities across both domains. Overall, achieving universal semantic segmentation capabilities for both optical and SAR images using transfer learning techniques still faces the following difficulties and challenges:

[0005] (1) Controlling the migration direction: The same type of land cover has a huge difference in appearance between optical and SAR images. How to drive the semantic segmentation model trained by optical images to migrate in the direction that is effective for SAR images is a problem that needs to be overcome.

[0006] (2) Controlling migration balance: In order to obtain general segmentation capabilities for optical and SAR images, it is necessary to accurately control the migration direction of the model to achieve balance. How to ensure that the model does not favor either the source domain or the target domain is a problem that needs to be overcome. Summary of the Invention

[0007] This invention provides a remote sensing image segmentation method and system based on a modal balance knowledge distillation framework, which addresses the shortcomings of existing technologies in transfer learning between optical and SAR images.

[0008] In a first aspect, the present invention provides a remote sensing image segmentation method based on a modal equilibrium knowledge distillation framework, comprising:

[0009] Acquire source-domain optical images and target-domain SAR images;

[0010] Image-level virtual mode generation is performed on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image;

[0011] The source domain optical image and the virtual SAR image are input into dual-modal supervised learning to obtain pseudo-labels for the source domain optical image and virtual SAR image.

[0012] The target domain SAR image and the virtual optical image are input into bimodal knowledge reasoning to obtain pseudo-labels for the target domain SAR image and virtual optical image.

[0013] Based on the source domain real label, the target domain SAR image pseudo label, the source domain optical image and the target domain SAR image, pixel-level virtual modality generation is performed to take into account the proportion of ground cover categories, and the hybrid image and hybrid pseudo label are obtained.

[0014] Based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, pixel-level virtual modal generation is performed to take into account the proportion of ground cover categories, thereby obtaining a hybrid virtual image and a hybrid virtual pseudo-label.

[0015] The blended image and the blended virtual image are input into hybrid modality supervised learning to obtain remote sensing image segmentation results.

[0016] According to a remote sensing image segmentation method based on a modal equilibrium knowledge distillation framework provided by the present invention, image-level virtual modal generation is performed on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image, including:

[0017] The source domain optical image and the target domain SAR image are processed using a pix2pixHD-based style transfer method to generate the virtual SAR image and the virtual optical image.

[0018] According to a remote sensing image segmentation method based on a modal equilibrium knowledge distillation framework provided by the present invention, the source domain optical image and the virtual SAR image are input into a dual-modal supervised learning process to obtain pseudo-labels for the source domain optical image and the virtual SAR image, including:

[0019] Determine the supervisory encoder and supervisory decoder for the bimodal supervised learning;

[0020] The source domain optical image is sequentially passed through the supervisory encoder and the supervisory decoder for multi-scale fusion to obtain the source domain optical image pseudo-label;

[0021] The virtual SAR image is sequentially passed through the encoder and the decoder for multi-scale fusion to obtain the virtual SAR image pseudo-label;

[0022] The loss of the source domain optical image pseudo-label is determined as a first loss function, and the loss of the virtual SAR image pseudo-label is determined as a second loss function. The first loss function and the second loss function constitute a dual-modal supervised learning loss function.

[0023] The first loss function includes the cross-entropy loss between the source domain real label and the virtual SAR image pseudo label, and the softened Dice loss between the source domain real label and the virtual SAR image pseudo label. The second loss function includes the cross-entropy loss between the source domain real label and the virtual SAR image pseudo label, and the softened Dice loss between the source domain real label and the virtual SAR image pseudo label.

[0024] According to a remote sensing image segmentation method based on a modal equilibrium knowledge distillation framework provided by the present invention, the target domain SAR image and the virtual optical image are input into bimodal knowledge inference to obtain pseudo-labels for the target domain SAR image and the virtual optical image, including:

[0025] Determine the reasoning encoder and reasoning decoder for the bimodal knowledge reasoning;

[0026] The knowledge acquired during the dual-modal supervised learning phase is transferred using exponential moving average weights;

[0027] The target domain SAR image is sequentially passed through the inference encoder and the inference decoder for multi-scale fusion to obtain the target domain SAR image pseudo-label;

[0028] The virtual optical image is sequentially passed through the inference encoder and the inference decoder for multi-scale fusion to obtain the virtual optical image pseudo-label.

[0029] According to the present invention, a remote sensing image segmentation method based on a modal balance knowledge distillation framework performs pixel-level virtual modality generation, taking into account the proportion of land cover categories, based on source domain real labels, target domain SAR image pseudo labels, source domain optical images, and target domain SAR images, to obtain a hybrid image and a hybrid pseudo label, including:

[0030] Based on the source domain real labels and the preset land cover category proportions, the background and foreground in the image are determined;

[0031] The background pixel values ​​in the real source domain labels are mapped to 0, and the foreground pixel values ​​are mapped to 1 to generate the first mask;

[0032] The background pixel values ​​in the pseudo-label of the target domain SAR image are mapped to 1, and the foreground pixel values ​​are mapped to 0 to generate a second mask;

[0033] The final mask is obtained by multiplying the first mask and the second mask by a dot product.

[0034] The hybrid image is calculated based on the final mask, the source domain optical image, and the target domain SAR image.

[0035] The hybrid pseudo-label is calculated based on the final mask, the source domain real label, and the target domain SAR image pseudo-label.

[0036] According to the present invention, a remote sensing image segmentation method based on a modal balance knowledge distillation framework performs pixel-level virtual modality generation considering the proportion of ground cover categories based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, to obtain a hybrid virtual image and a hybrid virtual pseudo-label, including:

[0037] The hybrid virtual image is calculated based on the final mask, the virtual SAR image, and the virtual optical image.

[0038] The hybrid virtual pseudo-label is calculated based on the final mask, the virtual SAR image pseudo-label, and the virtual optical image pseudo-label.

[0039] According to the present invention, a remote sensing image segmentation method based on a modal balance knowledge distillation framework is provided, which inputs the hybrid image and the hybrid virtual image into hybrid modality supervised learning to obtain remote sensing image segmentation results, including:

[0040] The hybrid modal supervised learning is determined to employ the supervised encoder and supervised decoder from the dual modal supervised learning.

[0041] The mixed image is sequentially passed through the supervised encoder and the supervised decoder for multi-scale fusion to obtain the mixed pseudo-label segmentation result;

[0042] The hybrid virtual image is sequentially passed through the supervised encoder and the supervised decoder for multi-scale fusion to obtain the hybrid virtual pseudo-label segmentation result;

[0043] The hybrid pseudo-label is used to supervise the hybrid pseudo-label segmentation result, and a third loss function is constructed. The third loss function includes the cross-entropy loss between the hybrid pseudo-label and the hybrid pseudo-label segmentation result, and the softening Dice loss between the hybrid pseudo-label and the hybrid pseudo-label segmentation result.

[0044] The hybrid virtual pseudo-labels are used to supervise the segmentation results of the hybrid virtual pseudo-labels, and a fourth loss function is constructed. The fourth loss function includes the cross-entropy loss between the hybrid virtual pseudo-labels and the segmentation results of the hybrid virtual pseudo-labels, and the softening Dice loss between the hybrid virtual pseudo-labels and the segmentation results of the hybrid virtual pseudo-labels.

[0045] Secondly, the present invention also provides a remote sensing image segmentation system based on a modal equilibrium knowledge distillation framework, comprising:

[0046] The acquisition module is used to acquire source domain optical images and target domain SAR images;

[0047] An image-level generation module is used to perform image-level virtual mode generation on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image;

[0048] A dual-modal supervised learning module is used to input the source domain optical image and the virtual SAR image into dual-modal supervised learning to obtain pseudo-labels for the source domain optical image and the virtual SAR image.

[0049] The dual-modal knowledge reasoning module is used to input the target domain SAR image and the virtual optical image into the dual-modal knowledge reasoning to obtain the target domain SAR image pseudo-label and the virtual optical image pseudo-label;

[0050] The first pixel-level generation module is used to generate a pixel-level virtual modality based on the source domain real label, the target domain SAR image pseudo label, the source domain optical image and the target domain SAR image, taking into account the proportion of land cover categories, to obtain the mixed image and the mixed pseudo label.

[0051] The second pixel-level generation module is used to generate pixel-level virtual modal based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, taking into account the proportion of ground object categories, to obtain the mixed virtual image and the mixed virtual pseudo-label.

[0052] The hybrid modality supervised learning module is used to input the hybrid image and the hybrid virtual image into the hybrid modality supervised learning to obtain the remote sensing image segmentation result.

[0053] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework as described above.

[0054] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework as described above.

[0055] The remote sensing image segmentation method and system based on the modality-balanced knowledge distillation framework provided by this invention acquires modally symmetrical virtual samples of optical and SAR images by employing image-level and pixel-level virtual modality generation strategies, thus supporting modality-unbiased knowledge distillation. Through the modality-balanced optical and SAR image knowledge distillation framework, a cross-modal transfer learning task is transformed into two same-modal transfer learning tasks through bimodal supervised learning, hybrid modality supervised learning, and bimodal knowledge reasoning, thereby achieving modality-unbiased knowledge distillation of optical and SAR images. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a comparison chart showing the prediction results of models trained on optical images using existing technologies on optical images and the prediction results on SAR images.

[0058] Figure 2 This is a comparison between the knowledge extraction in the traditional model provided by this invention and the knowledge extraction in this invention;

[0059] Figure 3 This is a flowchart illustrating the remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework provided by the present invention.

[0060] Figure 4 This is a framework diagram of modal equilibrium knowledge distillation provided by the present invention;

[0061] Figure 5 This is a schematic diagram of the pixel-level virtual modality generation process provided by the present invention;

[0062] Figure 6 This is a pixel-level virtual modality generation result diagram provided by the present invention;

[0063] Figure 7 These are visualization results of different methods provided by this invention on a SAR image test set;

[0064] Figure 8 These are graphs showing the qualitative evaluation results of different methods provided by this invention;

[0065] Figure 9 This invention provides a large-scale visualization map of the entire county in region B. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] To address the problems existing in current technologies, this invention, specifically for cross-modal transfer and segmentation tasks of optical and SAR images, innovatively proposes a modality-balanced knowledge distillation framework consisting of three parts: bimodal supervised learning, hybrid modality virtual learning, and bimodal knowledge reasoning. This framework innovatively transforms the cross-modal problem into a same-modal knowledge transfer problem by establishing a virtual modality. Figure 2 Domain adaptation and knowledge extraction under the traditional model shown in (a) are as follows: Figure 2 The modality balance knowledge extraction framework shown in Figure (b) transforms the source and target domains into virtual modal representations that are similar to the target and source domains, respectively, in modality balance knowledge distillation. This approach aims to eliminate the problem of inaccurate transfer direction caused by modality imbalance in cross-modality transfer tasks and achieve the ability to learn different modality balances in a more affinity way.

[0068] The main methods involved include knowledge distillation and domain adaptation. Knowledge distillation methods utilize supervised information from the source domain to generate pseudo-labels for the target domain, iteratively optimizing the segmentation model to achieve segmentation capability of the target domain. These methods typically employ a student-teacher network structure, including typical structures such as offline distillation, online distillation, and self-distillation.

[0069] (1) Offline distillation: This method uses a pre-trained, fixed-parameter teacher model to guide the student model. During distillation, the student network acquires fixed knowledge at the end of each training round. Examples include models like FSP, SSKD, and SemCKD. This model is simple to implement, but it typically involves a one-way transfer of knowledge from the teacher to the student. When the differences between the student and teacher networks are too large, the student network struggles to learn useful knowledge.

[0070] (2) Online distillation: This type of network trains the student network and the teacher network together, with parameters updated simultaneously, and knowledge is continuously updated during the transfer process. The entire model is trained end-to-end, and classic networks include Rocket-KD, DCM, and ACNs. This is a highly efficient parallel method. However, differences in structure and size can cause differences in model capacity between the teacher network and the student network, which prevents knowledge from being effectively transferred to the student network.

[0071] (3) Self-distillation: Self-distilled student and teacher networks use the same structure for distillation, mainly to improve the ability mismatch problem caused by different structural capacities between student and teacher networks. It can be regarded as a special form of online distillation. The pseudo-labels generated by general distillation models have a lot of noise interference. Therefore, researchers have proposed the Mean Teacher (MT) model to improve the accuracy of the inference stage. Methods such as SePiCo, Dual-Teacher++, DACS and DAFormer have all introduced the MT model. In the field of remote sensing, Luo et al. proposed a two-stage domain adaptive cross-temporal classification method to achieve transfer learning between optical images in different times and spaces. Wang et al. proposed a cross-sensor land cover framework to transfer between aerial images and satellite images to solve the problems of spatial resolution inconsistency and spectral differences. However, these models are all based on optical images for transfer, and the difference between the source domain and the target domain is much smaller than that between optical and SAR images. Therefore, in this scenario, the model is more likely to transfer in a direction that is favorable to the target domain. However, for multimodal images such as optical and SAR, the difficulty of the model controlling the transfer direction is greatly increased.

[0072] Most of the methods mentioned above are geared towards application scenarios where the source and target domains are of the same modality. When these methods are applied to images of different modalities, the model's versatility decreases significantly. It's easy to see that too much or too little guidance from the teacher network to the student network will cause the model to shift towards the target or source domain, easily leading to distillation bias. Therefore, the direction of knowledge distillation is unstable. The method of this invention addresses the significant differences between multimodal images by designing a series of balancing strategies to improve this distillation instability and enhance the model's versatility.

[0073] Domain adaptation addresses the impact of inconsistent data distributions on the performance of learning models. It works by reducing the domain offset between the source and target domains, making them as close as possible to the same feature space. It is typically applied in real-world scenarios where models trained in the source domain directly solve tasks in the target domain. It is mainly divided into two categories: image-level and feature-level.

[0074] (1) Image-level domain adaptation: This method generally uses GANs to minimize appearance differences at the image level to align the style distribution between the source and target domains. This type of method is suitable for mitigating differences in color, texture, and lighting conditions at the image level, and is mainly applicable to differences in the overall appearance of images caused by different imaging conditions. Typical works in the field of computer vision include CycleGAN, AgGAN, and pix2pixHD. Inspired by the field of vision, remote sensing image domain adaptation methods have also been proposed. This type of method translates remote sensing images from different time periods and different sensors into images with the same style to reduce domain shift. The advantage of this type of method is that it can learn complex data distributions to transform the low-level features of the image. For the scenario of optical and SAR image migration, it can shorten the distance between the low-level feature spaces of the two types of images and directly alleviate the feature gap caused by imaging modes. However, its disadvantage is that the training process is sometimes unstable.

[0075] (2) Feature-level domain adaptation: This method applies knowledge from the source domain to the target domain by leveraging the similarity between the source and target domains at the feature level. This approach is suitable for scenarios with differences at the object level, such as object pose and spatial distribution. Classic methods in computer vision include CyCADA, ADVENT, and LITR. Furthermore, researchers have proposed a series of methods for remote sensing images. Examples include ColorMapGANs, TriADA, joint MLP-GNN, SDA, and Ye. These methods typically utilize adversarial approaches with discriminators. When the discriminator cannot distinguish between source and target domain features, the model assumes the features of the two domains are aligned. However, adversarial methods often generate specific, unrepresentative features to deceive the discriminator. For multimodal images like optical and SAR images, which have fundamentally different underlying features, simply forcibly narrowing the distance at higher feature levels has very limited effectiveness.

[0076] The methods described above attempt to learn domain-invariant features between the source and target domains at both the image and feature levels, and then utilize these features to apply knowledge from the source domain to the target domain. However, feature distribution in the global space is discrete and diffuse. Even if the discriminator achieves optimal discrimination at the high-level feature level, the model can still easily bring features from different categories in the source and target domains closer together, leading to negative transfer. Currently, some researchers have proposed combining image-level and feature-level domain adaptation. However, this method only applies to single-channel optical images, not RGB images, limiting its applicability. Therefore, this invention introduces image-level domain adaptation to perform mode transfer between RGB and SAR images at the image level.

[0077] Figure 3 This is a flowchart illustrating the remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework provided in this embodiment of the invention. Figure 3 As shown, it includes:

[0078] Step 100: Acquire source domain optical imagery and target domain SAR imagery;

[0079] Step 200: Perform image-level virtual mode generation on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image;

[0080] Step 300: Input the source domain optical image and the virtual SAR image into dual-modal supervised learning to obtain source domain optical image pseudo-labels and virtual SAR image pseudo-labels;

[0081] Step 400: Input the target domain SAR image and the virtual optical image into bimodal knowledge reasoning to obtain the target domain SAR image pseudo-label and the virtual optical image pseudo-label;

[0082] Step 500: Based on the source domain real label, the target domain SAR image pseudo label, the source domain optical image and the target domain SAR image, perform pixel-level virtual modality generation that takes into account the proportion of land cover categories to obtain the mixed image and the mixed pseudo label;

[0083] Step 600: Based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, perform pixel-level virtual modality generation that takes into account the proportion of ground cover categories to obtain the hybrid virtual image and the hybrid virtual pseudo-label;

[0084] Step 700: Input the blended image and the blended virtual image into the hybrid modality supervised learning to obtain the remote sensing image segmentation result.

[0085] Specifically, the modal balanced knowledge distillation framework proposed in this embodiment of the invention introduces virtual modality generation, enabling models trained solely on optical images to possess general segmentation capabilities for both optical and SAR images. The student network of this framework consists of bimodal supervised learning and hybrid modal supervised learning, while the teacher network consists of bimodal knowledge inference. It is implemented in three stages: Stage 1, using real optical images and virtual SAR images in the source domain for balanced bimodal supervised learning, enabling the model to acquire similar interpretation capabilities for both modalities. Stage 2, transferring the weights from Stage 1 to the bimodal knowledge inference module, and then using it to obtain pseudo-labels for real SAR images and virtual optical images in the target domain. Stage 3, using hybrid images and hybrid labels for hybrid modal supervised learning, this process is supervised and optimized using the pseudo-labels output from Stage 2, ultimately achieving the goal of balanced knowledge distillation. It is worth noting that the input to each stage consists of two parallel modalities. The overall framework structure is as follows: Figure 4 As shown.

[0086] The definitions of the symbols are shown in Table 1:

[0087] Table 1

[0088]

[0089] The first stage is bimodal supervised learning.

[0090] Bimodal supervised learning is the first part of the student network in the balanced knowledge distillation framework. It aims to leverage source domain labels to guide the training of image-level modal balancing samples, thereby establishing the model's initial segmentation capability for balanced optical and SAR images.

[0091] (1) Image-level virtual modality generation

[0092] Image-level modal discrepancies can cause overall distribution shifts, weakening the potential for comprehensive utilization of multimodal data in transfer learning. Inspired by domain adaptation, this invention designs Image-Level Virtual Modality Generation (IVMG) to address the modality imbalance problem in knowledge transfer at the image level. It utilizes style transfer methods... Source domain optical imaging and target domain SAR images The appearance of the images is transferred to each other while retaining their respective content information, resulting in virtual SAR images. and virtual optical imaging As shown in formula (1):

[0093] (1)

[0094] After the above operations are performed... and Belonging to the same mode, and They belong to the same modality. Therefore, IVMG is a prerequisite for transforming a cross-modal transfer task into two intramodal transfer subtasks. The style transfer method used in this embodiment of the invention is pix2pixHD. Compared to the classic CycleGAN method, this model can not only generate high-resolution images more stably, but also capture and reproduce the details and textures of real images more fully.

[0095] (2) Bimodal supervised learning

[0096] like Figure 4 As shown in (a), and The input is fed into Dual-modal supervised learning (DMSL), which extracts dual-modal features through the same encoder to obtain the ability to perceive the balance of different modalities.

[0097] encoder Its structure is a Mix transformer. It learns from each... and The decoder gradually acquires the ability to extract features from both the source and target domains. Multi-scale fusion is performed to obtain the segmentation result. and As shown in formula (2);

[0098] (2)

[0099] and Each by label To conduct oversight. The loss is . The loss is . By cross-entropy loss and soft dice loss The sum is calculated as shown in formula (3):

[0100] (3)

[0101] in:

[0102] (4)

[0103] The second stage is bimodal knowledge reasoning.

[0104] Optimization of the student network requires guidance from the teacher network. Therefore, the model also needs to acquire knowledge of real SAR imagery in the target domain to guide the student network's learning. To achieve this goal, this section uses real SAR imagery to generate virtual optical imagery, and then designs a dual-modal knowledge inference (DMKI) method to predict these two types of imagery in parallel, obtaining pseudo-labels as supervisory information for training the student network.

[0105] like Figure 4 As shown in (c), DMKI has the same network structure as DMSL, but it does not update gradients. It uses exponential moving average (EMA) weights to transfer the knowledge acquired in the DMSL stages. Then it directly predicts the SAR image in the target domain. and virtual optical imaging pseudo-tags and As shown in formula (5):

[0106] (5)

[0107] in, For the encoder in the DMKI stage, This is the decoder for the DMKI stage.

[0108] It should be noted that since DMKI does not update gradients, no additional loss function needs to be designed. The baseline of the entire network framework is the student-teacher structure of DAFormer.

[0109] The third stage is mixed-modal supervised learning.

[0110] Common knowledge distillation methods generate pseudo-labels that contain certain errors. Using these pseudo-labels as supervisory information for student networks may lead to poor convergence and generalization. This embodiment proposes a hybrid modal supervised learning method. This method takes into account the proportion of land cover categories and performs pixel-level mixing of the source domain image and its corresponding label with the target domain image and its corresponding pseudo-label. Then, it uses pseudo-labels containing partial ground truth values ​​for supervised learning to obtain a modally unbiased knowledge distillation model.

[0111] (1) Pixel-level virtual modal generation taking into account the proportion of land cover categories

[0112] This embodiment proposes a pixel-level virtual modality generation (PVMG) method, which blends multimodal images at the pixel level based on the distribution characteristics of remotely sensed ground features. It has two objectives: firstly, to achieve modal equalization within a single image; and secondly, to increase the proportion of minority categories and reduce the influence of long-tailed distributions.

[0113] By statistically analyzing the real labels in the source domain and calculating the proportion of pixels in each category to the total number of pixels, categories with a proportion exceeding 10% are defined as background, and categories with a proportion below 10% are defined as foreground. For example... Figure 5 As shown, since forests, farmland, and water bodies each account for more than 10% of the labels, these categories are defined as background, while cities, rural areas, and roads are defined as foreground. Background pixel values ​​are mapped to 0, and foreground pixel values ​​are mapped to 1, generating a binary mask. . pseudo-tags The mask is generated by mapping in the opposite way (i.e., mapping foreground pixel values ​​to 0 and background pixel values ​​to 1). . Obtain the final mask. .

[0114] use respectively with , Image pairs and , Image pairs are used to calculate the blended image. and corresponding tags As shown in formula (6). By observation It can be seen that the same scene simultaneously possesses the characteristics of both optical and SAR images.

[0115] (6)

[0116] Similarly, , , and Perform the same operation to obtain the mixed result. and .

[0117] like Figure 6 As shown, (a), (e), (h), and (l) are optical images, (b), (d), (i), and (k) are SAR images, and (c) and (g) are images generated by random mixing. We can see that foreground features are severely occluded or damaged by background features, causing the relative relationships between feature distributions to deviate from the real scene. Figure 6 As shown in (j) and (m), PVMG takes into account the remote sensing characteristics and performs reasonable mixing of multimodal images to avoid occlusion and damage to foreground features.

[0118] (2) Virtual Modal Supervised Learning

[0119] After processing, the fake labels were mixed. Some incorrect labels are replaced with truth values ​​from categories with a smaller percentage of errors. Based on this, this embodiment designs a virtual modality supervised learning system, which is the second part of the student network in the balanced knowledge distillation framework. This utilizes pseudo-labels with partial truth values. Supervised information from student networks can significantly improve the accuracy of the training process, especially for improving the accuracy of small-class segmentation, which is more difficult to train.

[0120] Encoders in bimodal supervised learning Extracting pixel-level virtual modal samples and This strengthens the model's ability to extract features balanced between the two modalities. Then, the decoder in dual-modal supervised learning is utilized. The decoding yields the segmentation result, as shown in formula (7):

[0121] (7)

[0122] Segmentation results and Depend on and Supervision is performed separately to improve the accuracy of the model in learning unlabeled target domains. The loss function is shown in formula (8):

[0123] (8)

[0124] Therefore, the loss function of the entire model is shown in equation (9):

[0125] (9)

[0126] To demonstrate the effectiveness and practicality of the modal equilibrium knowledge distillation framework proposed in this invention, this embodiment conducted application experiments on the large public dataset WHU-OPT-SAR and six regions. The results show that the method of this invention is significantly superior to other methods in terms of performance and stability.

[0127] First, let's describe the dataset. Dataset I is the WHU-OPT-SAR dataset, an open-source optical and SAR image segmentation dataset released by a university. The optical images were acquired by the GaoFen-1 satellite and have red, green, and blue bands. The SAR images were acquired by the GaoFen-3 satellite. The WHU-OPT-SAR dataset contains 100 5556×3704 pixel optical images and the same number and size of SAR images. The sampling resolution is 5m for all images. The annotated data categories include farmland, urban, rural, water bodies, forests, roads, and others. These categories account for 35%, 5%, 6%, 14%, 38%, 1%, and 1% of the data, respectively. We use dataset I to verify the effectiveness of the proposed method for semantic segmentation tasks using transfer learning from optical and SAR images in the same region.

[0128] In this embodiment, the images in the WHU-OPT-SAR dataset are cropped to 512×512 pixels, resulting in 7000 patches: 5652 for training and 1348 for testing, as shown in Table 2. The SAR images undergo the same operation. The Dataset I optical image test set and the SAR image test set are used to prepare for testing the model's migration performance in the same region.

[0129] Table 2. Division of Training and Test Sets

[0130]

[0131] The symbol “*” represents the target domain image patches (without their labels).

[0132] Dataset II comprises six distinct regions, as shown in Table 2. Optical imagery was acquired by the GaoFen-2 satellite at a resolution of 1 m. SAR imagery was acquired by the GaoFen-3 refined strip II satellite at a resolution of 10 m. Annotated data is from the Third National Land Cover Survey. To ensure consistency, these images were resampled to a 5 m resolution. The dataset includes five categories: farmland, buildings, water bodies, forests, and roads. The proportions of these categories in order are shown in the last column of Table 3. It can be observed that farmland and water bodies constitute a larger proportion in region A, while forests constitute a larger proportion in region B. The category proportions in region C of the test city are relatively balanced. However, regions D and E have a very large proportion of forests, indicating a significant category imbalance. Region E has a smaller proportion of cities and roads, and the source region, target region, and test city are geographically distant. The source region is located in a large river basin, while the target region is located in another river basin. The test city differs from these two regions and is more widely distributed. These regions have vastly different topography and landforms, and the proportions of different categories vary, which greatly increases the challenge of classification tasks in different regions compared to dataset I, and places higher demands on the transferability, generalization and robustness of the model.

[0133] Table 3. Covered areas of optical and SAR images in different regions

[0134]

[0135] In Dataset II, the training and test sets of optical imagery come from region A. The training and test sets of SAR imagery come from region B and the other four regions listed in Table 3. The optical imagery test set for region A and the SAR imagery test set for region B in Dataset II are used to prepare for testing the transfer performance of the model in different regions. Regions C, D, E, and F in Dataset II are used to prepare for testing the generalization of the model. Following the same cropping method, the dataset division of Dataset II is shown in Table 4.

[0136] Table 4. Division of Training and Test Sets

[0137]

[0138] The symbol “*” represents the target domain image patches (without their labels).

[0139] To evaluate the performance of the BL proposed in this invention, eight common DASS methods were selected, which are mainly divided into two categories:

[0140] (1) Domain Adaptation (DA) Methods. These methods utilize classic image-level techniques such as CycleGAN, AgGAN, and Pix2PixHD to transfer the style of optical images to that of SAR images. This imagery is then used as the training set to train Deeplabv2, directly predicting SAR images. Additionally, feature-level methods such as ADVENT and LTIR are also used. These methods transfer the source and target domains. Unlike the other two methods, LTIR's input also includes the source domain style of the target domain.

[0141] (2) Knowledge distillation methods. These include DACS, DAFormer, and DACST. In cross-domain semantic segmentation tasks, DACS was the first to introduce Cutmix. DAFormer was the first to use a transformer as the network skeleton. DACST is a method from the remote sensing domain that utilizes the source domain style of the target domain to replace the original source domain influence on the input network.

[0142] To ensure fairness, the backbone of the above reference methods uses Deeplabv2. The method of this invention, which uses Deeplabv2 as its backbone, is referred to as BL-D. In other cases, the method of this invention uses MiT-B5 as a pre-trained model.

[0143] The experimental environment was a CentOS 7.5 Linux platform at a university supercomputing center. The model was trained using the Adam optimizer on an Nvidia Tesla V100 supercomputer. The hyperparameters were set as follows: batch size of 4, iter number of 60,000, initial learning rate of 6 × 10⁻⁵, and optimization using a stochastic gradientdescent (SGD) solver with momentum of 0.9 and weight decay of 5 × 10⁻⁴. When the error rate stopped decreasing, the learning rate was divided by 10, and the parameters were updated with the new values.

[0144] The transfer performance of the model on optical and SAR images of the same region was tested using Dataset I, and the results are shown in Table 5. Due to the significant differences in features between optical and SAR images, the model trained using the No adaptation method fails in the target domain. Domain adaptation and knowledge distillation methods show significant improvements over the No adaptation method. For example, the F1 score of Pix2PixHD is 49% higher than that of No adaptation. However, the accuracy of ADVENT, DACS, and DAFormer methods is lower than that of image-level domain adaptation methods. This is because the imaging principles of optical and SAR images are completely different, and implicit learning cannot be used to bridge the feature gap between them. LTIR, DACST, and BL all combine image-level and feature-level modal bridging. Among them, the method of this invention achieved the highest accuracy. The F1 score is 7.40% higher than LTIR, and OA is 7.21% higher than DACST. The LTIR method uses an adversarial approach, which can easily lead to mode collapse. DACST only uses the style-transferred source domain as input, causing the model to completely fail to learn the modality-invariant features inherent in the original source domain. The modality balancing approach of BL allows the model to learn bimodal information in a balanced way, preventing information bias. Furthermore, compared to using the Deeplabv2 structure, the method of this invention, using the MiT_B5 structure as the backbone, improves the F1 score by 7%. This demonstrates that the transformer structure facilitates the learning of contextual information from large-scale remote sensing images, meeting the needs of remote sensing ground feature interpretation.

[0145] Table 5

[0146]

[0147] Figure 7 The visualization results of different methods on a SAR image test set are shown. For example... Figure 7 As shown in (d), the optically trained model is ineffective on SAR imagery. The domain adaptation method results in severe road detection errors and missed detections. Figure 7 (e)-(h)). This is because a lot of detailed information will inevitably be lost during the domain adaptation process, making it difficult for the model to extract small features. Figure 7 (i) and (k) represent knowledge distillation methods. It can be observed that these methods produce a large number of false detections. Due to the significant differences in features between optical and SAR images, these methods cannot achieve good transfer results solely through knowledge transfer at the feature layer. Figure 7(l) and (m) are the methods of this invention. The Transformer structure backbone makes the segmentation results in (m) more detailed and have higher road continuity than those in (l). The model of this invention even predicts small roads that are not marked in the labels. Compared with the reference method, the method of this invention achieves the best visual effect by means of the modal balance knowledge distillation framework and virtual modality generation strategy.

[0148] The model's transfer performance across optical and SAR imagery in different regions was tested using Dataset II. Optical imagery from region A served as the source domain, and SAR imagery from region B served as the target domain. The results are shown in Table 6. Unlike methods for the same region, image-level domain adaptation methods suffer from negative transfer due to the inconsistency between the source and target domains, resulting in unsatisfactory segmentation accuracy. Furthermore, knowledge distillation methods face significant challenges due to the completely different styles and content distributions of the source and target domains, leading to poor performance. Our method utilizes a hierarchical mixing strategy to generate mixed data with consistent content but different modalities, forcing the model to learn multimodal features in a balanced manner, thus achieving satisfactory accuracy. BL's F1 score is 17%, 19%, and 10% higher than LTIR, DACST, and BL-D, respectively, further demonstrating the effectiveness of our proposed method.

[0149] Table 6

[0150]

[0151] Qualitative evaluation, such as Figure 8 As shown, there is no correspondence between optical and SAR image pixels in different regions, leading to severe misclassification by most domain adaptation and knowledge distillation methods in terms of visual perception. The LTIR method improves the visual effect to some extent. However, the method of this invention achieves the best visual effect, with less noise at the boundaries of ground features and smoother edges. In addition, a large-scale visualization of the entire county in region B is shown below. Figure 9 As shown in the figure, compared with the best-performing LTIR method in Table 5, the method of this invention has a more accurate overall classification effect and is closer to the true labels. From the magnified image, the road edges extracted by the method of this invention are more refined. This further demonstrates the effectiveness of pixel-level modal mixing in enhancing features that constitute a small proportion of the terrain. In experiments in different regions, BL showed consistent advantages in both qualitative and quantitative results.

[0152] The above experiments all demonstrate the superiority of the modal equilibrium knowledge distillation framework. This is likely due to the innovative design of this invention in the following three aspects, based on the unique properties of remote sensing imagery:

[0153] (1) We proposed the idea of ​​modal balance symmetry. Multimodal images have completely different features. Currently, most transfer learning networks only operate at the feature level, resulting in obvious bias of the model towards a certain modality. We designed a modal balance strategy that progresses from the image level to the pixel level and from the global to the local level, so that the model can have the ability to segment similar modalities.

[0154] (2) Improved the shortcomings of knowledge distillation in the absence of accurate labels in the target domain. Currently, most knowledge distillation networks rely solely on pseudo-labels in the target domain to determine the accuracy of knowledge transfer during training. However, pseudo-labels are usually inaccurate, affecting the generalization performance of the model. We utilize style conversion between the source and target domains and pixel-level mixing to give the target domain some real labels, which makes the supervision information more accurate during knowledge transfer, promotes the transfer of knowledge from the target domain to the source domain, and thus improves the segmentation accuracy.

[0155] (3) Pixel-level blending rules conforming to remote sensing scenes were formulated. In large-scale real-world scenes, the long-tail distribution of ground features is particularly significant. This is a key characteristic that distinguishes remote sensing images from natural images. Virtual modal blending operations have formulated blending rules conforming to remote sensing images, achieving category enhancement while avoiding disruption of the actual distribution of foreground features.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image segmentation method based on a modal equilibrium knowledge distillation framework, characterized in that, include: Acquire source-domain optical images and target-domain SAR images; Image-level virtual mode generation is performed on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image; The source domain optical image and the virtual SAR image are input into dual-modal supervised learning to obtain pseudo-labels for the source domain optical image and virtual SAR image. The target domain SAR image and the virtual optical image are input into bimodal knowledge reasoning to obtain pseudo-labels for the target domain SAR image and virtual optical image. Based on the source domain real label, the target domain SAR image pseudo label, the source domain optical image and the target domain SAR image, pixel-level virtual modality generation is performed to take into account the proportion of ground cover categories, and the hybrid image and hybrid pseudo label are obtained. Based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, pixel-level virtual modal generation is performed to take into account the proportion of ground cover categories, thereby obtaining a hybrid virtual image and a hybrid virtual pseudo-label. The blended image and the blended virtual image are input into hybrid modality supervised learning to obtain remote sensing image segmentation results; Based on the source domain real label, the target domain SAR image pseudo label, the source domain optical image, and the target domain SAR image, pixel-level virtual modality generation taking into account the proportion of land cover categories is performed to obtain the hybrid image and hybrid pseudo label, including: Based on the source domain real labels and the preset land cover category proportions, the background and foreground in the image are determined; The background pixel values ​​in the real source domain labels are mapped to 0, and the foreground pixel values ​​are mapped to 1 to generate the first mask; The background pixel values ​​in the pseudo-label of the target domain SAR image are mapped to 1, and the foreground pixel values ​​are mapped to 0 to generate a second mask; The final mask is obtained by multiplying the first mask and the second mask by a dot product. The hybrid image is calculated based on the final mask, the source domain optical image, and the target domain SAR image. The hybrid pseudo-label is calculated based on the final mask, the source domain real label, and the target domain SAR image pseudo-label.

2. The remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework according to claim 1, characterized in that, Image-level virtual mode generation is performed on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image, including: The source domain optical image and the target domain SAR image are processed using a pix2pixHD-based style transfer method to generate the virtual SAR image and the virtual optical image.

3. The remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework according to claim 1, characterized in that, The source domain optical image and the virtual SAR image are input into dual-modal supervised learning to obtain pseudo-labels for the source domain optical image and the virtual SAR image, including: Determine the supervisory encoder and supervisory decoder for the bimodal supervised learning; The source domain optical image is sequentially passed through the supervisory encoder and the supervisory decoder for multi-scale fusion to obtain the source domain optical image pseudo-label; The virtual SAR image is sequentially passed through the encoder and the decoder for multi-scale fusion to obtain the virtual SAR image pseudo-label; The loss of the source domain optical image pseudo-label is determined as a first loss function, and the loss of the virtual SAR image pseudo-label is determined as a second loss function. The first loss function and the second loss function constitute a dual-modal supervised learning loss function. The first loss function includes the cross-entropy loss between the source domain real label and the virtual SAR image pseudo label, and the softened Dice loss between the source domain real label and the virtual SAR image pseudo label. The second loss function includes the cross-entropy loss between the source domain real label and the virtual SAR image pseudo label, and the softened Dice loss between the source domain real label and the virtual SAR image pseudo label.

4. The remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework according to claim 1, characterized in that, The target domain SAR image and the virtual optical image are input into bimodal knowledge reasoning to obtain pseudo-labels for the target domain SAR image and the virtual optical image, including: Determine the reasoning encoder and reasoning decoder for the bimodal knowledge reasoning; The knowledge acquired during the dual-modal supervised learning phase is transferred using exponential moving average weights; The target domain SAR image is sequentially passed through the inference encoder and the inference decoder for multi-scale fusion to obtain the target domain SAR image pseudo-label; The virtual optical image is sequentially passed through the inference encoder and the inference decoder for multi-scale fusion to obtain the virtual optical image pseudo-label.

5. The remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework according to claim 1, characterized in that, Based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, pixel-level virtual modality generation taking into account the proportion of ground cover categories is performed to obtain a hybrid virtual image and a hybrid virtual pseudo-label, including: The hybrid virtual image is calculated based on the final mask, the virtual SAR image, and the virtual optical image. The hybrid virtual pseudo-label is calculated based on the final mask, the virtual SAR image pseudo-label, and the virtual optical image pseudo-label.

6. The remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework according to claim 5, characterized in that, The blended image and the blended virtual image are input into hybrid modality supervised learning to obtain remote sensing image segmentation results, including: The hybrid modal supervised learning is determined to employ the supervised encoder and supervised decoder from the dual modal supervised learning. The mixed image is sequentially passed through the supervised encoder and the supervised decoder for multi-scale fusion to obtain the mixed pseudo-label segmentation result; The hybrid virtual image is sequentially passed through the supervised encoder and the supervised decoder for multi-scale fusion to obtain the hybrid virtual pseudo-label segmentation result; The hybrid pseudo-label is used to supervise the hybrid pseudo-label segmentation result, and a third loss function is constructed. The third loss function includes the cross-entropy loss between the hybrid pseudo-label and the hybrid pseudo-label segmentation result, and the softening Dice loss between the hybrid pseudo-label and the hybrid pseudo-label segmentation result. The hybrid virtual pseudo-labels are used to supervise the segmentation results of the hybrid virtual pseudo-labels, and a fourth loss function is constructed. The fourth loss function includes the cross-entropy loss between the hybrid virtual pseudo-labels and the segmentation results of the hybrid virtual pseudo-labels, and the softening Dice loss between the hybrid virtual pseudo-labels and the segmentation results of the hybrid virtual pseudo-labels.

7. A remote sensing image segmentation system based on a modal equilibrium knowledge distillation framework, comprising the remote sensing image segmentation method based on a modal equilibrium knowledge distillation framework as described in any one of claims 1 to 6, characterized in that, include: The acquisition module is used to acquire source domain optical images and target domain SAR images; An image-level generation module is used to perform image-level virtual mode generation on the source domain optical image and the target domain SAR image to obtain virtual SAR image and virtual optical image; A dual-modal supervised learning module is used to input the source domain optical image and the virtual SAR image into dual-modal supervised learning to obtain pseudo-labels for the source domain optical image and the virtual SAR image. The dual-modal knowledge reasoning module is used to input the target domain SAR image and the virtual optical image into the dual-modal knowledge reasoning to obtain the target domain SAR image pseudo-label and the virtual optical image pseudo-label; The first pixel-level generation module is used to generate a pixel-level virtual modality based on the source domain real label, the target domain SAR image pseudo label, the source domain optical image and the target domain SAR image, taking into account the proportion of land cover categories, to obtain the mixed image and the mixed pseudo label. The second pixel-level generation module is used to generate pixel-level virtual modal based on the virtual SAR image pseudo-label, the virtual optical image pseudo-label, the virtual SAR image, and the virtual optical image, taking into account the proportion of ground object categories, to obtain the mixed virtual image and the mixed virtual pseudo-label. The hybrid modality supervised learning module is used to input the hybrid image and the hybrid virtual image into the hybrid modality supervised learning to obtain the remote sensing image segmentation result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the remote sensing image segmentation method based on the modal equilibrium knowledge distillation framework as described in any one of claims 1 to 6.

Images