A remote sensing image change detection method based on image-level labels

Abstract

The application provides a remote sensing image change detection method based on image-level labels, comprising the following steps: step one, inputting a source view and a target view into a twin network to obtain double-view feature output, an initial CAM and a projection embedding; the source view is an original image pair, and the target view is an image pair after spatial change; step two, performing saliency projection based on the initial CAM of the double view to obtain a saliency CAM, forming mutual guidance based on saliency regions, and performing isometry regularization for confusion; step three, calculating a class prototype based on reliable pixels in the CAM, and further comparing the prototype with all pixels in the CAM for contrast learning to optimize a feature space; step four, training a target function, and deriving a pixel-level label based on an image-level label and an original image pair after completion; and step five, training a change detection network based on a full-supervision change detection network and a pseudo label. The application uses image-level labels for model training, and can effectively reduce label acquisition cost.

Description

Technical Field

[0001] This application relates to remote sensing image change detection technology, and more particularly to a method for detecting changes in remote sensing images with image-level labels. Background Technology

[0002] Remote sensing image change detection is a common image interpretation task used to distinguish whether the same location has changed in images from different time phases (two-phase or multi-phase) to generate a binary change detection map showing the change or no change. Change detection is widely used in many fields, such as urban sprawl, disaster detection, and resource management.

[0003] Because manual interpretation of multi-temporal imagery is time-consuming and labor-intensive, automatic change detection algorithms have been continuously developed. These algorithms can be categorized into unsupervised and supervised change detection based on whether labels are required. Early on, only low-to-medium resolution imagery such as MODIS and LandSat was available as data sources. Unsupervised change detection utilizes spectral and various indices for comparison, but due to resolution limitations, it is often coarse-grained.

[0004] With the development of high-resolution Earth imaging technologies, such as WorldView and GaoFen, images now reveal increasingly more details about ground features, leading to their widespread use in semantic segmentation, scene classification, and fine-grained change detection (e.g., single-class change detection of buildings and semantic change detection). However, the rich detail in high-resolution images also presents challenges for change detection. Supervised change detection, especially deep learning methods driven by sample labels, has gradually replaced unsupervised change detection as the mainstream approach. While supervised change detection demonstrates superior performance compared to unsupervised change detection, it requires a large number of labeled samples for automatic feature learning. Furthermore, the change targets in high-resolution images are often small and numerous, making their pixel-level annotations more complex than those in medium- and low-resolution images.

[0005] To leverage supervised learning while reducing label acquisition costs, semi-supervised learning and weakly supervised learning (WSL) can be used. Semi-supervised learning utilizes a limited number of labeled samples to mine the potential of unlabeled samples. Existing techniques propose a progressive semi-supervised change detection framework that learns region-invariant and enhancement-invariant feature representations through feature prediction alignment. WSL refers to using coarse-grained labels to obtain fine-grained prediction results; for example, outputting high-resolution predictions with low-resolution labels and pixel-level predictions with image-level labels. Existing techniques propose a label refinement framework that utilizes low-resolution labels to obtain high-resolution semantic change maps. Existing techniques use seed, expansion, and constraint principles to obtain pixel-level change detection maps and combine them with conditional random fields for optimization. Furthermore, many WSL methods have been proposed for single-time remote sensing tasks. Existing techniques utilize image-level labels for building and green plastic segmentation, respectively. In addition to pixel-level results, image-level labels are often used to obtain box-level predictions. Rafique and Jacobs used box-level labels to obtain pixel-level predictions of buildings. Existing techniques consider point-level labels for semantic segmentation of remote sensing images. Existing techniques use graffiti-level tags to extract roads and buildings. However, WSL has not yet been fully explored in very high-resolution remote sensing general change detection tasks.

[0006] Although weakly supervised image analysis techniques based on class attention maps (CAMs) have been well explored, their application in remote sensing change detection remains challenging. Natural images typically contain a large portion of the image, and the foreground and background are significantly different. Change regions in remote sensing images are often small and fragmented. Furthermore, many areas of change are not salient and can easily be confused with the background. These issues make WSL-based change detection even more difficult.

[0007] Specifically, this invention improves upon equivariant regularization (ER) to make it more suitable for change detection tasks, avoiding activation values ​​tending to zero due to small and fragmented change regions, and provides supervision from reliable regions through mutual learning. Furthermore, the prototype-based contrastive learning (PCL) design enhances the distinction between the foreground (change region) and the background. Summary of the Invention

[0008] The purpose of this invention is to solve the aforementioned problems existing in current remote sensing image change detection technologies.

[0009] To achieve the above objectives, this invention proposes a remote sensing image change detection method based on image-level labels, which includes the following steps:

[0010] Step 1: Input the source view and target view into the Siamese network to obtain the feature output, initial CAM-type attention map and projection embedding of the two views; the source view is the original image pair, and the target view is the image pair after spatial transformation.

[0011] Step 2: Based on the initial CAM from two perspectives, a saliency projection is performed to obtain a saliency CAM. Mutual guidance is formed based on the saliency regions, and isovariant regularization is performed to address confusion.

[0012] Step 3: Calculate category prototypes based on reliable pixels in the CAM, and further compare and learn the prototypes with all pixels in the CAM to optimize the feature space;

[0013] Step 4: Integrate the objective function for training, and then derive pixel-level labels based on image-level labels and original image pairs.

[0014] Step 5: Train the change detection network based on the traditional fully supervised change detection network and pseudo-labels to obtain the change detection network.

[0015] This invention presents a remote sensing image change detection model based on image-level labels. It can derive pixel-level pseudo-labels from image-level labels and further train the change detection model. It has high practical value and can be effectively applied to change detection in high-resolution remote sensing images, reducing the cost of obtaining labels that the deep learning model for change detection relies on. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of remote sensing change detection based on pixel-level labels and image-level labels, where (a) is the previous time-phase image; (b) is the subsequent time-phase image; (c) is the pixel-level label; and (d) is the image-level label.

[0018] Figure 2 A schematic flowchart of a remote sensing image change detection method based on image-level labels provided in an embodiment of the present invention;

[0019] Figure 3 A schematic diagram for Mutual Learning and Equivariant Regularization (MLER);

[0020] Figure 4This is a schematic diagram of Prototype Contrastive Learning (PCL); when pt and v come from the same viewpoint, intra-viewpoint contrast is formed; when they come from different viewpoints, cross-viewpoint contrast is formed.

[0021] Figure 5 This is a schematic diagram of the pseudo-label results. Detailed Implementation

[0022] The present specification will now be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. The described embodiments are only a part of the embodiments described in this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without inventive effort are within the scope of protection of this application.

[0023] The problem definitions and preliminary concepts in this embodiment of the invention are introduced as follows:

[0024] Given a pair of two temporal images [I A ,I B ], and its image-level label Y img ∈{0,1}, the objective of this embodiment of the invention is to obtain the pseudo-label Y based on them. pseudo Furthermore, a change detection model is trained based on the pseudo-labels. Therefore, the weakly supervised change detection task can be divided into two stages: a weakly supervised learning task to generate pseudo-labels; and a fully supervised change detection task.

[0025] For weakly supervised learning tasks, there are

[0026] WSL(I A ,I B ,Y img ) = Y pseudo

[0027] The weakly supervised learning task is denoted as WSL(·), where image pairs [I A ,I B ] and image-level label Y img For input. Similarly, the fully supervised change detection task can be represented as:

[0028] FSCD(I A ,I B ,Y pseudo ) = Y CD

[0029] Where FSCD(·) represents the fully supervised change detection task, Y CD This serves as a mask for detecting predicted changes. Therefore, the key to weakly supervised change detection is generating the most accurate possible Y... pseudoThis has the most significant impact on the final change detection performance. Therefore, this embodiment of the invention focuses on the WSL task and aims to generate high-precision pseudo-tags.

[0030] In this embodiment of the invention, when generating a CAM, a classification model is given, and its output feature map is O∈R. C×H×W Here, H and W represent the height and width of the feature map, respectively, and C is the number of channels. Global average pooling (GAP) is used to aggregate these features. Furthermore, the parameter w∈R 1×C The fully connected layer is used to calculate the probability of belonging to the change category:

[0031]

[0032] Furthermore, A can be calculated as

[0033]

[0034] A is further normalized to [0,1] so that it can be regarded as the probability of belonging to the change category.

[0035] like Figure 1 and Figure 2 As shown, this embodiment of the invention provides a method for detecting changes in remote sensing images based on image-level labels, including the following steps:

[0036] Step 1: Input the original image pair (source viewpoint) and the spatially transformed image pair (target viewpoint) into the Siamese network to obtain the feature output, initial CAM, and projection embedding of the two views.

[0037] Specifically, the framework proposed in this embodiment of the invention is a Siamese network, whose inputs are the original image pairs and the spatially transformed image pairs. When the image pairs are input into the Siamese network, feature maps are obtained through a feature extractor, as follows:

[0038]

[0039]

[0040] Where f(·) is the basic backbone of the CNN, used to extract multi-scale features. For I A Multi-scale features. For I B The multi-scale features are further aggregated as follows:

[0041]

[0042]

[0043] Here, Cat represents channel concatenation. Feature difference is used to fuse multi-scale features across two temporal phases and serves as the final output of the feature extractor, as follows:

[0044] O = |F A -F B |

[0045] The projection features and the initial CAM can be obtained by further using the feature projector and the CAM generator, respectively.

[0046] Step 2: Based on the initial CAM from two perspectives, a saliency projection is performed to obtain a saliency CAM. Mutual guidance is formed based on the saliency regions, and isovariant regularization is performed to address confusion.

[0047] In fully supervised change detection, data augmentation techniques based on spatial transformations, such as scaling and flipping, are often used to improve model performance. That is, after spatial transformation of the input, its prediction should be as similar as possible to the pixel-level label after the same spatial transformation. In weakly supervised learning, due to the lack of pixel-level labels, some studies have attempted to impose consistency constraints on the predictions of the input under different spatial transformations to reduce the gap between weakly supervised and fully supervised learning.

[0048] Furthermore, embodiments of the present invention use salient regions in one CAM to guide corresponding regions in another perspective CAM. The two CAMs mutually guide each other through salient regions to form a mutual learning paradigm, and consistency constraints are applied to both for confusing regions, such as... Figure 3 As shown. First, a saliency CAM is generated from the initial CAM, which consists of three parts: the region of saliency change and the region of confusion. The value of the region of saliency change is set to 1, while the rest remain unchanged. The projection of the initial CAM to the saliency CAM is then performed. Defined as:

[0049]

[0050]

[0051] Where θ is the threshold, and S and A are the significant CAM and the initial CAM, respectively. Therefore, A s and A t By projecting, S can be obtained respectively. s and S t A s and A t Let CAM represent the source and target perspectives, respectively. Mutual learning and isovariant regularization are implemented using the L1 norm; therefore, the source perspective's guidance and regularization of the target can be expressed as:

[0052] L source-target =|T(S s)-A t |1

[0053] Where T(·) represents spatial transformation. Similarly, the guidance and regularization of the target viewpoint on the source viewpoint can be expressed as:

[0054] L target-source =|T(A s )-S t |1

[0055] Note S s and S t There is no gradient. Therefore, mutual learning and isovariant regularization based on the source and target perspectives are defined as:

[0056] L MLER =(L source-target +L target-source ) / 2

[0057] Because the number of changed pixels is much smaller than the number of unchanged pixels, the activation values ​​in the CAM tend to approach zero, which will affect the L... M.ER The influence of this is further aggravated. Therefore, this embodiment of the invention uses a reliable background ignoring strategy, that is, for pixels in the CAM of any view whose activation value is lower than the reliable background threshold φ, when calculating L... M.ER They are often ignored.

[0058] Unlike equivariant regularization, the reliable background ignoring strategy enables the effective use of consistency constraints in WSL for change detection tasks. Furthermore, mutual learning and equivariant regularization achieve consistency constraints in confusing regions, forming a self-supervised paradigm. Mutual learning based on salient regions provides supervision from more reliable regions.

[0059] Step 3: Calculate category prototypes based on reliable pixels in the CAM, and further compare and learn the prototypes with all pixels in the CAM to optimize the feature space.

[0060] Relying solely on mutual learning and isovariant regularization is insufficient to distinguish confused regions. Therefore, this invention constructs a prototype-based contrastive learning mechanism based on the similar representations of pixels of the same category in the feature space. This invention aims to learn more discriminative features by narrowing the distance between pixels and their corresponding positive prototypes and widening the distance between pixels and other negative prototypes. Furthermore, features from different viewpoints should maintain consistency; therefore, this invention constructs intra-viewpoint and cross-viewpoint contrastive mechanisms, such as... Figure 4 As shown.

[0061] In this embodiment of the invention, the prototype is defined as a class-specific discriminative code, which is calculated using salient pixels in the CAM. For all pixels belonging to class c, this embodiment selects the pixels with the highest probability in the CAM. Therefore, the prototype pt is obtained by a weighted average based on pixel-level projective embedding and values ​​in the CAM, as follows:

[0062]

[0063] Among them, A c CAM diagram representing category c, Ω c This represents the Top-K set of category c. K is a hyperparameter used to control the range of pixels used to compute the prototype. Furthermore, to fully utilize reliable pixels across the entire dataset without being limited by batch size, this embodiment of the invention updates the prototype pt using a method similar to exponential moving average (EMA), as follows:

[0064]

[0065] Among them, pt t-1 and pt t These are the prototypes for the (t-1)th and tth iterations during the training process, respectively. To update the weights. Since the classification network only outputs A. change Therefore, A is calculated in the following way. nochange :

[0066] A nochange =1-A change

[0067] The CAM is considered as a pixel-level change probability map. For each pixel, this embodiment of the invention generates a category region mask M by using a threshold:

[0068]

[0069] ρ represents the threshold. In this embodiment of the invention, ρ is set to 0.5, which is similar to the mapping rule from output value to category in sigmoid-based binary classification.

[0070] Furthermore, the projection embedding is divided into v using mask operations. change and v nochange Therefore, embodiments of the present invention should maximize the similarity between the projected embedding and the prototype (v). change and pt change v nochange and pt nochange ), reducing similarity with the negative prototype (v change and pt nochange v nochange and pt change ),like Figure 4 As shown. In this study, the projection embedding and prototype are defined and compared as follows:

[0071]

[0072] Where I represents the entire image, |·| represents the count, and T is the temperature coefficient set to 0.1, sim(·) is the similarity between the pixel-level projective embedding and the prototype, as follows:

[0073] sim(v i ,pt)=exp(v i ·pt)

[0074] Intraview contrast is constructed to perform prototype contrast learning within the same viewpoint, meaning that the projection embedding, category region mask, and prototype all come from the same viewpoint, as follows:

[0075] L intra =(PCL(v s ,pt s M s )+PCL(v t ,pt t M t )) / 2

[0076] Semantic consistency should exist between different perspectives. Therefore, this embodiment of the invention constructs a cross-perspective comparison, using a prototype from another perspective to provide supervision for the current perspective, as follows:

[0077] L inter =9PCL(v s ,pt t M s )+PCL(v t ,pt s M t )) / 2

[0078] Therefore, combining intra-view and cross-view comparisons, prototype-based contrastive learning can be represented as:

[0079] L PCL =(L intra +L cross ) / 2

[0080] Prototype-based contrastive learning obtains supervised information from reliable pixels and extends it for contrast learning. This enhances intra-class compactness and inter-class separability, and addresses the problem of varying regions easily being confused with the background.

[0081] Step 4: Integrate the objective function for training, and then derive pixel-level labels based on image-level labels and original image pairs.

[0082] This invention uses α-weighted binary cross-entropy (BCE) loss to represent the prediction s n and tag y n The differences are as follows:

[0083] L α-BCE (s n ,y n )=-I{α}log(I{s n})

[0084] Where I(·) represents the indicator function. Compared to binary cross-entropy loss, this embodiment of the invention introduces a weighting factor α∈[0,1] to control the importance of the changed and unchanged classes. By increasing the importance of the changed classes and decreasing the importance of the unchanged classes, this embodiment of the invention enables the model to activate a more complete region of change. Combining the source and target perspectives, the classification loss can be expressed as:

[0085]

[0086] In summary, there are three types of loss in the method of this embodiment of the invention. Therefore, the total loss is defined as their sum, as follows:

[0087] L = L cls +L MLER +L PCL

[0088] Training is performed based on the overall dataset, and then pixel-level labels are derived from the image-level labels and the original image pairs.

[0089] Step 5: Train the change detection network based on the traditional fully supervised change detection network and pseudo-labels to obtain the change detection network.

[0090] As described above, this embodiment of the invention divides the weakly supervised change detection task into two steps: generating pseudo-labels using WSL and fully supervised change detection using pseudo-labels. The first part is developed based on the content described in the previous section. The second part simply utilizes U-Net and DeepLab V3+ for fully supervised learning. Specifically, feature extraction employs a Siamese encoder, and the dual-temporal feature fusion method is absolute value difference. Furthermore, a Sigmoid activation function and BCE loss are used for training.

[0091] The remote sensing image change detection model based on image-level labels in this invention can derive pixel-level pseudo-labels from image-level labels and further train the change detection model. It has high practical value and can be effectively applied to change detection in high-resolution remote sensing images, reducing the cost of obtaining labels that the deep learning model for change detection relies on.

[0092] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A method for detecting changes in remote sensing images based on image-level labels, characterized in that, Includes the following steps: Step 1: Input the source view and target view into the Siamese network to obtain the feature output of the two views, and initially based on the attention map (CAM) and projection embedding; the source view is the original image pair, and the target view is the image pair after spatial transformation; Step 2: Based on the initial CAM from two perspectives, a saliency projection is performed to obtain a saliency CAM. Mutual guidance is formed based on the saliency regions, and isovariant regularization is performed to address confusion. Step 3: Calculate category prototypes based on reliable pixels in the CAM, and further compare and learn the prototypes with all pixels in the CAM to optimize the feature space; Step 4: Integrate the objective function for training, and then derive pixel-level labels based on image-level labels and original image pairs. Step 5: Train the change detection network based on the fully supervised change detection network and pseudo-labels to obtain the change detection network.

2. The method according to claim 1, characterized in that, In step one, when the image pair is input into the Siamese network, the feature map obtained by the feature extractor is as follows: Where f(·) is the basic backbone of the CNN, used to extract multi-scale features. For I A Multi-scale features. For I B The multi-scale features are further aggregated as follows: Where Cat represents channel concatenation; feature difference is used to fuse multi-scale features across two time phases and serves as the final output of the feature extractor: O=|F A -F B | The projected features and the initial CAM are obtained through the feature projector and the CAM generator, respectively.

3. The method according to claim 1, characterized in that, In step two, Projection from initial CAM to significant CAM Defined as: Where θ is the threshold, and S and A are the significant CAM and the initial CAM, respectively; therefore, A s and A t S was obtained by projection. s and S t A s and A t Let CAM represent the source and target perspectives respectively; mutual learning and isovariant regularization are implemented using the L1 norm, and the source perspective's guidance and regularization of the target are represented as: L source-target =|T(S s )-A t |1 Where T(·) represents spatial transformation; similarly, the guidance and regularization of the target viewpoint on the source viewpoint can be expressed as: L target-source =|T(A s )-S t |1 S s and S t There is no gradient; therefore, mutual learning and isovariant regularization based on the source perspective and the target are defined as follows: L -LER (L source-target +L target-source ) / 2.

4. The method according to claim 1, characterized in that, In step three, for all pixels belonging to category c, the pixels with the highest probability in the CAM are selected; therefore, the prototype pt is obtained by weighted averaging based on pixel-level projective embedding and values ​​in the CAM, as follows: Among them, A c CAM diagram representing category c, Ω c This represents the Top-K set of category c; the prototype pt is updated using a method similar to exponential moving average, as follows: Among them, pt t-1 and pt t These are the prototypes for the (t-1)th and tth iterations during the training process, respectively. To update the weights; Since the classification network only outputs A change Therefore, A is calculated in the following way. nochange : A nochange =1-A change Among them, A change and A change CAM maps representing the change category and the non-change category respectively; the CAM is treated as a pixel-level change probability map, and a category region mask M is generated for each pixel by applying a threshold: ρ represents the threshold; setting ρ to 0.5 is similar to the mapping rule from output value to category in sigmoid-based binary classification.

5. The method according to claim 4, characterized in that, The projection embedding is divided into v using mask operations. change and v nochange Increase the similarity between the projected embedding and the prototype, v change and pt change v nochange and pt nochange Reduce the similarity with the negative prototype. change and pt nochange v nocchange and pt change v change and v nochange Representing the projective embeddings of the change category and the non-change category, respectively, pt nochange and pt change Let these represent the prototypes of the changed category and the non-changed category, respectively; the projection embedding is defined and compared with the prototype as follows: Where v represents the projected embedding, pt represents the prototype, M represents the category region mask, and T is the temperature coefficient; where I represents the entire image, |·| represents the count, and T is the temperature coefficient set to 0.1, sim(·) is the pixel-level similarity between the projected embedding and the prototype, as follows: sim(v i ,pt)=exp(v i ·pt) Intraview contrast is constructed to perform prototype contrast learning within the same viewpoint, meaning that the projection embedding, category region mask, and prototype all come from the same viewpoint, as follows: L intra =9PCL(v s ,pt s ,M s )+PCL(v t ,pt t ,M t )) / 2 Among them, v s and v t Representing the projection embeddings of the source view and the target view respectively, pt s and pt t M represents the prototypes of the source view and the target view, respectively. s and M t These represent the category region masks for the source and target perspectives, respectively. Semantic consistency should exist between different perspectives. A cross-perspective comparison is constructed, using a prototype from another perspective to provide supervision for the current perspective, as follows: L inter =9PCL(v s ,pt t ,M s )+PCL(v t ,pt s ,M t )) / 2 Combining intra-view and cross-view comparisons, prototype-based contrastive learning can be represented as: L PaL (L intra +L cross ) / 2 Prototype-based contrastive learning obtains supervised information from reliable pixels and extends it for contrast learning.

6. The method according to claim 1, characterized in that, In step four, the α-weighted binary cross-entropy loss is used to represent the prediction s. n and tag y n The differences are as follows: L α-BCE (s n ,y n )=-I{α}log(I{s n }) Where I(·) represents the indicator function; compared to the binary cross-entropy loss, a weight factor α∈[0,1] is introduced to control the importance of the changed class and the unchanged class; by increasing the importance of the changed class and decreasing the importance of the unchanged class, the model activates a more complete region of change; combining the source and target perspectives, the classification loss can be expressed as: The total loss is defined as the sum of them, as follows: L=L cys +L MLER +L PCL Training is performed based on the overall dataset, and then pixel-level labels are derived from the image-level labels and the original image pairs.

7. The method according to claim 1, characterized in that, In step five, the weakly supervised change detection task is divided into two steps: generating pseudo-labels in WSL and fully supervised change detection using pseudo-labels; feature extraction uses a Siamese encoder, and the dual temporal feature fusion method is absolute value difference; the Sigmoid activation function and BCE loss are used for training.

8. The method according to claim 1, characterized in that, When generating a CAM, given a classification model, its output feature map is O∈R. C×H×W , where H and W represent the height and width of the feature map, respectively, and C is the number of channels.

Images