Cross-modal remote sensing image building change detection system and method

Abstract

The present invention relates to the field of remote sensing image processing technology, and in particular to a cross-modal remote sensing image building change detection system and method. A remote sensing conversion adversarial subnetwork, comprising a generator and a discriminator, is designed based on the adversarial principle. The generator and discriminator work together to achieve efficient cross-modal conversion from SAR images to optical remote sensing images. A teacher-student change detection subnetwork, comprising a student model and a teacher model, is designed based on the teacher-student model and consistency regularization principles. Knowledge distillation is employed to perturb unlabeled samples at the sample level to achieve consistency constraints between the detection results of the teacher and student models before and after the perturbation. This system can effectively restore the spatial details and semantic information of images, combine pseudo-labeling with consistency constraints for semi-supervised change detection, and improve the accuracy and generalization of instance-level building change detection methods when the number of labeled samples is insufficient.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a cross-modal remote sensing image building change detection system and method. Background Art

[0002] With the continuous development of remote sensing technology, high-resolution remote sensing images have been widely used in urban planning, environmental monitoring, disaster assessment and other fields. Among them, building change detection is one of the important tasks in remote sensing image analysis. It can help us to timely understand information such as urban expansion, infrastructure construction, and natural disaster damage.

[0003] In recent years, deep learning technology has made significant progress in remote sensing image analysis, providing new insights and methods for building change detection. Within the deep learning field, convolutional neural networks (CNNs) are widely used in feature extraction and classification tasks for remote sensing images. For example, the U-Net architecture was introduced to remote sensing image processing due to its successful application in medical image segmentation. It uses an encoder-decoder structure to achieve pixel-level classification of images, effectively capturing spatial details in the image. Furthermore, advanced network models based on recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) networks, gated recurrent units (GRUs), as well as Transformers and Mamba, have also been used to process time series data of remote sensing images to explore the patterns of feature changes between time phases.

[0004] However, directly applying these deep learning methods to building change detection in cross-modal remote sensing imagery presents numerous challenges. Firstly, remote sensing images of different modalities, such as optical and Synthetic Aperture Radar (SAR), exhibit significant differences in imaging principles, spectral characteristics, and spatial resolution. This leads to inconsistent feature distributions between images, making it difficult for traditional deep learning models to achieve good detection results directly on cross-modal data. This can lead to significant issues such as missed detections, false detections, and loss of semantic information. Secondly, building change detection requires instance-level accuracy, requiring not only determination of building changes but also the specific location and extent of the changes. This places higher demands on the model's localization accuracy and detail capture capabilities. Furthermore, existing deep learning methods often implement change detection in a fully supervised model, requiring a large number of labeled samples. However, in practice, manual visual interpretation and labeling of changes is labor-intensive and resource-intensive, making it often difficult to obtain sufficient labeled samples. Existing research uses transfer learning methods to improve model detection performance by narrowing the feature distribution of labeled samples and unlabeled samples. However, it still has defects such as poor stability, high complexity and limited applicability, making it difficult to meet application requirements under limited labeled sample conditions in real scenarios. Summary of the Invention

[0005] In view of this, the present invention aims to provide a cross-modal remote sensing image building change detection system and method, which combines generative adversarial networks to achieve cross-modal remote sensing image conversion, combines image segmentation and target detection tasks under consistency regularization constraints, and realizes cross-modal remote sensing image semi-supervised instance-level building change detection, thereby improving the accuracy and intelligence level of the detection method.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0007] A cross-modal remote sensing image building change detection system includes a remote sensing conversion adversarial sub-network and a teacher-student change detection sub-network; wherein;

[0008] The remote sensing conversion adversarial subnetwork consists of a generator and a discriminator. The generator uses multiple parallel residual branches to learn the mapping from the input SAR image to the corresponding optical remote sensing image, and obtains the corresponding converted remote sensing image. The discriminator is used to determine whether the input image is a converted remote sensing image or the real optical remote sensing image corresponding to the SAR image, obtains a quality discrimination score, and feeds the quality discrimination score back to the generator.

[0009] In the teacher-student change detection subnetwork, the converted image and an optical remote sensing image of another time phase imaging and covering the same area are combined to form a remote sensing image pair. The annotations of some remote sensing image pairs are retained to obtain labeled remote sensing image pairs, and the remaining unlabeled remote sensing image pairs are perturbed to obtain perturbed unlabeled remote sensing image pairs. The labeled remote sensing image pairs and the perturbed unlabeled remote sensing image pairs are input into the student model for target detection and semantic segmentation. The output results are compared with the true value of the labeled sample change and the corresponding perturbed pseudo-label, and the student model loss function is calculated; the unperturbed unlabeled remote sensing image pairs are input into the teacher model, the teacher model obtains the weight of the student model, and obtains the corresponding unperturbed pseudo-label; the unperturbed pseudo-label is perturbed in the same way as the corresponding perturbed unlabeled remote sensing image pair to obtain the perturbed pseudo-label, and the perturbed pseudo-label is fed back to the student model.

[0010] Furthermore, in the generator, the SAR image is input into at least two residual branches, and the output of each residual branch is added and convolved with the resampled SAR image to obtain a converted remote sensing image.

[0011] Furthermore, in each residual branch, the SAR image is continuously subjected to residual coding operations for no less than two times; the features after the operation are resampled to obtain output features; in each residual coding operation, no less than two convolution operations of different scales are simultaneously performed on the input features to obtain features of multiple scales; the features of multiple scales are concatenated and fused, and then the corresponding elements are added to the input features to complete the residual connection, and then the features after the residual connection are batch normalized and activated to obtain the output features.

[0012] Furthermore, in the discriminator, the corresponding elements of the input image are added, the added features are subjected to at least two convolution operations, and then at least two pooling operations are performed respectively; the features after the pooling operation are subjected to channel splicing and convolution operations, and then the processed features are subjected to a sigmoid activation operation, and then it is judged whether the input image is a converted remote sensing image or a real optical remote sensing image to obtain a quality discrimination score.

[0013] Furthermore, in the student model, the labeled remote sensing image pairs and the perturbed unlabeled remote sensing image pairs are input into the U-Net network, and the features output by the U-Net network are input into the multi-task output branch for target detection and semantic segmentation. The output results are compared with the true value of the labeled sample changes and the corresponding perturbation pseudo-label to calculate the student model loss function.

[0014] Furthermore, in the teacher model, unperturbed unlabeled remote sensing images are input into the U-Net network, and the features output by the U-Net network are input into the multi-task output branch for target detection and semantic segmentation. The teacher model uses the exponential moving average method to obtain the weights of the student model; the output instance-level change detection results are adaptively thresholded to obtain unperturbed pseudo labels; after perturbing the unperturbed pseudo labels, they are fed back to the student model.

[0015] Furthermore, the exponential moving average method is:

[0016] ;

[0017] in, represents the student model weight corresponding to the step in the training process, and represents the teacher model weight corresponding to the t-1th step training and the tth step training, Represents the adjustment parameter used to control the update amplitude of the teacher model weights.

[0018] A cross-modal remote sensing image building change detection method, comprising:

[0019] S1: Obtain a remote sensing image dataset and preprocess it to obtain a training set;

[0020] S2: Using the training set obtained in step S1, the cross-modal remote sensing image building change detection system provided by the present invention is trained to obtain a change detection model;

[0021] S3: According to the training results of step S2, adjust the hyperparameters in the training and repeat step S2 until the optimal change detection model is obtained;

[0022] S4: Inputting an image pair consisting of a SAR image and an optical remote sensing image that are imaged at different time phases and cover the same area into the optimal change detection model obtained in step S3 to obtain a change detection result.

[0023] Furthermore, in step S2, the generator is trained using an image conversion loss function; wherein:

[0024] The image conversion loss function is:

[0025] ;

[0026] Among them, L Z represents the image conversion loss function, i represents the number of pixels in the converted remote sensing image, M represents the total number of pixels in the converted remote sensing image, represents the input SAR image, G represents the generator, Indicates the pixel value of the i-th pixel in the converted remote sensing image, Represents the input SAR image The pixel value of the i-th pixel in the corresponding real optical remote sensing image;

[0027] The generator and discriminator are trained adversarially by optimizing the objective function as follows:

[0028] ;

[0029] Where D represents the discriminator, represents the discriminator’s expectation of the real optical remote sensing image, represents the discriminator's expectation for the converted remote sensing image.

[0030] Furthermore, in step S2, the teacher-student change detection subnetwork is trained using the following student model loss function:

[0031] ;

[0032] in, represents the student model loss function, represents the loss function of labeled samples, Represents the loss function of labeled samples The weight of represents the loss function of the perturbed unlabeled samples, Represents the loss function of perturbed unlabeled samples The weight of

[0033] Labeled sample loss function It is obtained from the following formula:

[0034] ;

[0035] in, represents the image segmentation loss when the student model performs semantic segmentation, represents the target detection loss when the student model performs target detection;

[0036] Perturbed unlabeled sample loss function It is obtained from the following formula:

[0037] ;

[0038] in, represents the image segmentation loss when the student model performs semantic segmentation, Represents the target detection loss when the student model performs target detection.

[0039] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0040] (1) The present invention creates a cross-modal remote sensing image building change detection system and method, proposes a cross-modal image conversion method, and realizes modal unification and spatial alignment of images of different time phases based on the generative adversarial theory, retains more image spatial details, improves data availability, and thus reduces the phenomenon of pseudo-changes in building change detection results;

[0041] (2) The present invention creates the cross-modal remote sensing image building change detection system and method, proposes a semi-supervised change detection method that uses consistent pseudo-labels of a teacher-student model to assist in model parameter optimization, effectively utilizes a small amount of labeled samples and takes advantage of the data of a large number of unlabeled samples, combines image fine segmentation and target detection tasks, and thereby improves the model's instance-level detection performance for buildings with different changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 A schematic diagram of a cross-modal remote sensing image building change detection system according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a generator according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a residual coding operation according to an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a discriminator according to an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of a teacher-student change detection subnetwork according to an embodiment of the present invention;

[0048] Figure 6 The present invention provides a flow chart of a method for detecting building changes in cross-modal remote sensing images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0051] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second" and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0052] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0053] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0054] like Figure 1 As shown, the cross-modal remote sensing image building change detection system described in the embodiment of the present invention includes a remote sensing conversion adversarial subnetwork and a teacher-student change detection subnetwork. The remote sensing conversion adversarial subnetwork includes a generator and a discriminator. The generator uses multiple parallel residual branches to learn the mapping from the input SAR image to the corresponding optical remote sensing image, thereby obtaining the corresponding converted remote sensing image. The discriminator is used to determine whether the input image is a converted remote sensing image or the true optical remote sensing image corresponding to the SAR image, obtain a quality discrimination score, and feed the quality discrimination score back to the generator. In the teacher-student change detection subnetwork, the converted image and an optical remote sensing image of another time phase imaging and covering the same area are combined to form a remote sensing image pair. The annotations of some remote sensing image pairs are retained to obtain labeled remote sensing image pairs, and the remaining unlabeled remote sensing image pairs are perturbed to obtain perturbed unlabeled remote sensing image pairs. The labeled remote sensing image pairs and the perturbed unlabeled remote sensing image pairs are input into the student model for target detection and semantic segmentation. The output results are compared with the true value of the labeled sample change and the corresponding perturbed pseudo-label, and the student model loss function is calculated; the unperturbed unlabeled remote sensing image pairs are input into the teacher model, the teacher model obtains the weight of the student model, and obtains the corresponding unperturbed pseudo-label; the unperturbed pseudo-label is perturbed in the same way as the corresponding perturbed unlabeled remote sensing image pair to obtain the perturbed pseudo-label, and the perturbed pseudo-label is fed back to the student model.

[0055] In some embodiments, in the generator, the SAR image is input into at least two residual branches, and the output of each residual branch is added and convolved with the SAR image after the resampling operation to obtain a converted remote sensing image. In each residual branch, the SAR image is continuously subjected to at least two residual coding operations; the features after the operation are resampled to obtain output features. In each residual coding operation, at least two different scales of convolution operations are simultaneously performed on the input features to obtain features of multiple scales; the features of multiple scales are spliced ​​and fused, and then the corresponding elements are added to the input features to complete the residual connection, and then the features after the residual connection are batch normalized and activated to obtain output features. In each residual coding operation, the input image or input feature is subjected to multiple convolution operations, batch normalization layer operations and activation function operations, and the input is directly passed to the output through the residual connection, which effectively solves the gradient vanishing problem in deep neural networks, enables the network to be built deeper, and maintains the stability of training.

[0056] The generator provided by the embodiment of the present invention is as follows Figure 2 As shown, the SAR image is input into three residual branches: in the first residual branch, the SAR image is subjected to four consecutive residual coding operations (i.e. Figure 2 Res1-1 to Res1-4 in the residual coding branch), and resample the features after the residual coding operation; in the second residual branch, the SAR image is continuously coded three times (i.e. Figure 2 Res2-1 to Res2-3 in the residual coding branch), and resample the features after the residual coding operation; in the third residual branch, the SAR image is continuously coded twice (i.e. Figure 2 The features after the residual coding operation are resampled. The output of each residual branch is element-wise added to the resampled SAR image and convolved with a 3×3 kernel to obtain the converted remote sensing image. Resampling is performed on each branch to ensure that the output features are of the same size, facilitating subsequent feature addition and fusion.

[0057] The residual coding operation provided by the embodiment of the present invention is as follows Figure 3 As shown, for the input features Simultaneously perform convolution operations of 1×1, 3×3, and 5×5 at three different scales to obtain features at three different scales; after concatenating and fusing the features at three different scales, combine them with the input features. The corresponding elements are added to complete the residual connection, and then the features after the residual connection are batch normalized and ReLU activated to obtain the output features. The splicing fusion in the embodiment of the present invention is specifically to splice the features of three different scales into channels, and then combine the features after the channel splicing into The 1×1 and 3×3 convolution operations are performed in sequence. The residual coding operation provided by the embodiment of the present invention can be expressed as follows:

[0058] ;

[0059] ;

[0060] Among them, concate means channel splicing, Conv 1×1 Represents a 1×1 convolution operation, Conv 3×3 Represents a 3×3 convolution operation, Conv 5×5 represents a 5×5 convolution operation, represents the addition of corresponding elements, and BR(·) represents batch normalization BN and ReLU activation operations.

[0061] In the present invention, the generator serves as the core of the remote sensing conversion adversarial subnetwork, and adopts a multi-branch residual structure to extract the multi-scale features of the SAR image to deeply understand the intrinsic information of the SAR image, and then accurately reconstruct the SAR image into an optical remote sensing image, so as to better preserve the original details of the image and mine semantic features. Among them, the multi-branch residual branch is expanded and optimized on the basis of the residual network (ResNet) widely used in deep learning. By introducing a multi-branch design, the network's ability to extract features of different scales and types is enhanced. Different branches can focus on extracting features of different levels or types. For example, some branches may focus on capturing local detail features, while other branches are responsible for extracting broader global features. The multi-scale feature extraction capability enables the network to understand the input data more comprehensively and improve the cross-modal image conversion results.

[0062] In some embodiments, in the discriminator, after performing at least two convolution operations on the input image, at least two pooling operations are performed respectively; after performing channel splicing and convolution operations on the features after the pooling operation, a sigmoid activation operation is performed on the processed features to determine whether the input image is a converted remote sensing image or a real optical remote sensing image, and obtain a quality discrimination score.

[0063] The discriminator provided by the embodiment of the present invention is as follows Figure 4As shown in the figure, the input image is subjected to three consecutive convolution operations with a convolution kernel of 3×3, and then global maximum pooling and global average pooling operations are performed respectively. The features after the pooling operation are channel-spliced ​​and convolved with a convolution kernel of 1×1. Then, a sigmoid activation operation is performed on the processed features to further distinguish the original optical image from the generated optical image, and obtain quality judgment true (True) and false (False) scores, that is, true (True) indicates that the input image is judged to be a real optical remote sensing image, and false (False) indicates that the input image is judged to be a converted remote sensing image.

[0064] During the training process of the generator and discriminator in the present invention, the discriminator continuously provides feedback to the generator, prompting the generator to gradually improve the quality of the generated image until the generated cross-modal conversion image reaches a high degree of similarity with the optical remote sensing image in appearance and statistical characteristics. Through the collaborative work of the generator and the discriminator, the cross-modal image conversion network model of the present invention can effectively realize the conversion of cross-modal images while retaining spatial details and semantic information. This innovative design provides more reliable and accurate data support for the cross-modal remote sensing image change detection task, promoting technological development in this field.

[0065] In some embodiments, the teacher-student change detection subnetwork is Figure 5 As shown in the figure, in the student model, the labeled remote sensing image pairs and the perturbed unlabeled remote sensing image pairs are input into the U-Net network. The features output by the U-Net network are input into the multi-task output branch for target detection and semantic segmentation. The output results are compared with the true value of the labeled sample changes and the corresponding perturbed pseudo-labels to calculate the student model loss function. In the teacher model, the unperturbed unlabeled remote sensing image pairs are input into the U-Net network. The features output by the U-Net network are input into the multi-task output branch for target detection and semantic segmentation. The teacher model uses the exponential moving average method (EMA) to obtain the weights of the student model, namely:

[0066] ;

[0067] in, represents the student model weight corresponding to the t-th step in the training process, and represents the teacher model weight corresponding to the t-1th step training and the tth step training, Represents the adjustment parameter used to control the update amplitude of the teacher model weights.

[0068] The output instance-level change detection results are threshold-filtered, and the confidence thresholds of the semantic segmentation and target detection results are set to 0.8 to obtain the unperturbed pseudo-labels. After the unperturbed pseudo-labels are perturbed, they are fed back to the student model.

[0069] The U-Net network used in the embodiments of the present invention is disclosed in the paper "U-net: Convolutional networks for biomedical image segmentation" in the "Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference". The multi-task output branch uses the Mask R-CNN output module, which is disclosed in the paper "Mask R-CNN" in the "Proceedings of the IEEE international conference on computer vision".

[0070] The present invention also provides a method for detecting building changes in cross-modal remote sensing images. Figure 6 Shown, including:

[0071] S1: Obtain a remote sensing image dataset and preprocess the remote sensing image dataset to obtain a training set.

[0072] S2: The cross-modal remote sensing image building change detection system provided by the present invention is trained using the training set obtained in step S1 to obtain a change detection model.

[0073] In some embodiments, in step S2, the generator is trained using an image conversion loss function; wherein:

[0074] The image conversion loss function is:

[0075] ;

[0076] Among them, L Z represents the image conversion loss function, i represents the number of pixels in the converted remote sensing image, M represents the total number of pixels in the converted remote sensing image, represents the input SAR image, G represents the generator, Indicates the pixel value of the i-th pixel in the converted remote sensing image, Represents the input SAR image The pixel value of the i-th pixel in the corresponding real optical remote sensing image;

[0077] The generator and discriminator are trained adversarially by optimizing the objective function as follows:

[0078] ;

[0079] Where D represents the discriminator, x represents the sample extracted from the real optical remote sensing image, z represents the sample extracted from the converted remote sensing image, and P data (x) represents the data distribution of real optical remote sensing images, P z (z) represents the data distribution of the converted remote sensing image, represents the discriminator’s expectation of the real optical remote sensing image, represents the discriminator's expectation for the converted remote sensing image.

[0080] The teacher-student change detection subnetwork is trained using the following student model loss function, and the labeled sample loss function is:

[0081] ;

[0082] in, represents the student model loss function, represents the loss function of labeled samples, Represents the loss function of labeled samples The weight of represents the loss function of the perturbed unlabeled samples, Represents the loss function of perturbed unlabeled samples The weight of

[0083] Labeled sample loss function It is obtained from the following formula:

[0084] ;

[0085] in, represents the image segmentation loss when the student model performs semantic segmentation, represents the target detection loss when the student model performs target detection;

[0086] Image segmentation loss is the BCE loss function, specifically:

[0087] L seg =-gtlog(p c )-(1-gt)log(1-p c );

[0088] Among them, gt represents the ground truth value of the binary change of the corresponding pixel, gt=1 means that the ground has changed, gt=0 means that the ground has not changed, p c Indicates the probability that the pixel is predicted to be a change category;

[0089] Object detection loss Use Smooth L1 loss (Smooth L1), specifically:

[0090] ;

[0091] Where a represents the difference between the predicted value and the true value. In the regression box position optimization, a includes the center coordinate difference, width difference, and height difference of the regression box.

[0092] Perturbed unlabeled sample loss function It is obtained from the following formula:

[0093] ;

[0094] in, represents the image segmentation loss when the student model performs semantic segmentation, Represents the target detection loss when the student model performs target detection.

[0095] The training environment in the embodiment of the present invention is as follows: the computer GPU is NVIDIA RTX4090, the operating system is Windows 11, the CPU processor is Intel (R) Core (TM) i9-14900K, the Python version is 3.9.18, the PyTorch version is 2.2.0, the CUDA version is 12.1, and the cuDNN version is 8.8.0.1.

[0096] S3: According to the training results of step S2, adjust the hyperparameters in training and repeat step S2 until the optimal change detection model is obtained.

[0097] In the embodiment of the present invention, the hyper parameters for training the remote sensing conversion adversarial sub-network include: the batch size of the training sample is set to 8; the epoch is set to 60; the stochastic gradient descent (SGD) is used to optimize the model parameters, the weight decay is 0.001, the momentum is 0.8; the initial learning rate is set to 0.1, and the total_iter is 0.1×(1-iter / total_iter) 1.2 The model is decayed in a way to improve the convergence performance of the model, where iter is the training process of one batch and total_iter is the total number of training batches. The hyperparameters for training the teacher-student change detection subnetwork include: the batch size of the training sample is set to 8; the epoch is set to 60; the adaptive moment estimation (Adam) is used to optimize the model parameters, the learning rate is set to 0.01, and the learning rate decays to 80% of the previous learning rate every 10 epochs.

[0098] S4: Inputting an image pair consisting of a SAR image and an optical remote sensing image that are imaged at different time phases and cover the same area into the optimal change detection model obtained in step S3 to obtain a change detection result.

[0099] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0100] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A cross-modal remote sensing image building change detection system, characterized in that: Includes remote sensing conversion adversarial sub-network and teacher-student change detection sub-network; in; The remote sensing conversion adversarial subnetwork includes a generator and a discriminator; the generator uses multiple parallel residual branches to learn the mapping from the input SAR image to the corresponding optical remote sensing image to obtain the corresponding converted remote sensing image; the discriminator is used to determine whether the input image is a converted remote sensing image or a true optical remote sensing image corresponding to the SAR image, obtain a quality discrimination score, and feed the quality discrimination score back to the generator; In the teacher-student change detection subnetwork, the conversion image and another optical remote sensing image that is imaged in the same phase and covers the same area are combined to form a remote sensing image pair, the labels of some remote sensing image pairs are retained to obtain labeled remote sensing image pairs, and the remaining unlabeled remote sensing image pairs are perturbed to obtain perturbed unlabeled remote sensing image pairs. The labeled remote sensing image pairs and the perturbed unlabeled remote sensing image pairs are input into the student model for target detection and semantic segmentation, and the output results are compared with the true value of the labeled sample change and the corresponding perturbed pseudo-label to calculate the student model loss function; the unperturbed unlabeled remote sensing image pair is input into the teacher model, and the teacher model obtains the weight of the student model to obtain the corresponding unperturbed pseudo-label; the unperturbed pseudo-label is perturbed in the same way as the corresponding perturbed unlabeled remote sensing image pair to obtain the perturbed pseudo-label, and the perturbed pseudo-label is fed back into the student model.

2. The cross-modal remote sensing image building change detection system according to claim 1, characterized in that: In the generator, the SAR image is input into at least two residual branches, and the output of each residual branch is added and convolved with the resampled SAR image to obtain the converted remote sensing image.

3. The cross-modal remote sensing image building change detection system according to claim 2, characterized in that: In each residual branch, the residual encoding operation is continuously performed on the SAR image for no less than 2 times; the features after the operation are resampled to obtain output features; In each residual coding operation, at least two convolution operations of different scales are performed on the input features simultaneously to obtain features of multiple scales. After the features of multiple scales are concatenated and fused, the corresponding elements are added to the input features to complete the residual connection. The features after residual connection are then batch normalized and activated to obtain the output features.

4. The cross-modal remote sensing image building change detection system according to claim 1, characterized in that: In the discriminator, after performing at least two convolution operations on the input image, at least two pooling operations are performed respectively; after performing channel splicing and convolution operations on the features after the pooling operation, a sigmoid activation operation is performed on the processed features, and then it is judged whether the input image is a converted remote sensing image or a real optical remote sensing image to obtain the quality discrimination score.

5. The cross-modal remote sensing image building change detection system according to claim 1, characterized in that: In the student model, the labeled remote sensing image pair and the perturbed unlabeled remote sensing image pair are input into the U-Net network, and the features output by the U-Net network are input into the multi-task output branch for target detection and semantic segmentation. The output results are compared with the true value of the labeled sample change and the corresponding perturbation pseudo label to calculate the student model loss function.

6. The cross-modal remote sensing image building change detection system according to claim 5, characterized in that: In the teacher model, unperturbed unlabeled remote sensing images are input into the U-Net network, and the features output by the U-Net network are input into the multi-task output branch for target detection and semantic segmentation. The teacher model obtains the weight of the student model using the exponential moving average method; the output instance-level change detection results are threshold-screened to obtain the unperturbed pseudo-labels; and the unperturbed pseudo-labels are perturbed and fed back to the student model.

7. The cross-modal remote sensing image building change detection system according to claim 6, characterized in that: The exponential moving average method is: ； in, represents the student model weight corresponding to the t-th step in the training process, and represents the teacher model weight corresponding to the t-1th step training and the tth step training, Represents the adjustment parameter used to control the update amplitude of the teacher model weights.

8. A method for detecting building changes in cross-modal remote sensing images, characterized in that: include: S1: Acquire a remote sensing image dataset and preprocess the remote sensing image dataset to obtain a training set; S2: Using the training set obtained in step S1, the cross-modal remote sensing image building change detection system according to any one of claims 1 to 6 is trained to obtain a change detection model; S3: According to the training results of step S2, adjust the hyperparameters in the training and repeat step S2 until the optimal change detection model is obtained; S4: Inputting an image pair consisting of a SAR image and an optical remote sensing image that are imaged at different time phases and cover the same area into the optimal change detection model obtained in step S3 to obtain a change detection result.

9. The cross-modal remote sensing image building change detection method according to claim 8, characterized in that: In step S2, the generator is trained using an image conversion loss function; wherein: The image conversion loss function is: ； Among them, L Z represents the image conversion loss function, i represents the number of pixels in the converted remote sensing image, M represents the total number of pixels in the converted remote sensing image, represents the input SAR image, G represents the generator, represents the pixel value of the i-th pixel in the converted remote sensing image, Represents the input SAR image The pixel value of the i-th pixel in the corresponding real optical remote sensing image; The generator and the discriminator are trained adversarially by optimizing the objective function of the following formula: ； Where D represents the discriminator, represents the expectation of the discriminator for the real optical remote sensing image, represents the discriminator's expectation for converting remote sensing images.

10. The cross-modal remote sensing image building change detection method according to claim 8, characterized in that: In step S2, the teacher-student change detection subnetwork is trained using the following student model loss function: ； in, represents the student model loss function, represents the loss function of labeled samples, Represents the loss function of labeled samples The weight of represents the loss function of the perturbed unlabeled samples, Represents the loss function of perturbed unlabeled samples The weight of The labeled sample loss function It is obtained from the following formula: ； in, represents the image segmentation loss when the student model performs semantic segmentation, represents the target detection loss when the student model performs target detection; The perturbation unlabeled sample loss function It is obtained from the following formula: ； in, represents the image segmentation loss when the student model performs semantic segmentation, Represents the target detection loss when the student model performs target detection.

Images