A change detection pre-training method and system based on single-phase semantic segmentation

Abstract

The application discloses a change detection pre-training method and system based on single-phase semantic segmentation, and belongs to the technical field of remote sensing image detection. First, the method uses extensive single-phase semantic segmentation data to construct pseudo-bi-temporal change detection data, then introduces a semantic segmentation detection module into any deep learning change detection network to form a pre-training network, pre-trains on the pseudo-bi-temporal change detection data, and then fine-tunes the obtained pre-training model on a real change detection data set to obtain a final change detection model. The application effectively utilizes rich single-phase semantic segmentation data to construct pseudo-bi-temporal change detection data, thereby relieving the dependence of a deep learning-based remote sensing image change detection method on large-scale bi-temporal labeled data; and the application additionally designs a semantic segmentation detection module in the pre-training stage to strengthen the semantic understanding ability of the model to change targets, thereby improving the detection precision of the model.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image detection technology, specifically relating to a change detection pre-training method and system based on single-temporal semantic segmentation. Background Technology

[0002] Remote sensing image change detection is a technique that uses multi-temporal remote sensing data to analyze surface changes. It is widely used in geological exploration, environmental monitoring, agricultural management, and urban planning, and is of great significance to sustainable social development. In recent years, with the rapid development of deep learning technology, deep learning-based remote sensing image change detection methods have made significant progress, but still face many challenges. Specifically, most existing deep learning methods employ fully supervised training, which largely relies on large-scale bi-temporal change detection annotation data. However, due to the inherent characteristics of remote sensing image change detection tasks, collecting paired and annotated change detection images is extremely difficult. On the one hand, due to the influence of various factors such as weather conditions, shooting time, and differences between different sensors, it is difficult to obtain consistent and high-quality bi-temporal data; on the other hand, annotating the changed areas in bi-temporal remote sensing images requires not only extensive professional background knowledge but is also time-consuming and labor-intensive. Therefore, bi-temporal labeled change detection data is still relatively scarce and valuable. The existing open-source change detection datasets are generally smaller in scale compared to other remote sensing tasks, which significantly limits the application of deep learning-based remote sensing image change detection methods in real-world scenarios. Therefore, it is urgent to reduce their dependence on bi-temporal labeled data. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a change detection pre-training method based on single-temporal semantic segmentation. First, this method constructs "pseudo-dual-temporal" change detection data using a wide range of single-temporal semantic segmentation data. Then, a semantic segmentation detection module is introduced into any deep learning change detection network to form a pre-trained network. This network is pre-trained on the "pseudo-dual-temporal" change detection data, and the resulting pre-trained model is then fine-tuned on a real change detection dataset to obtain the final change detection model. This method uses the constructed "pseudo-dual-temporal" change detection data to pre-train any deep learning change detection network, thereby alleviating the dependence on dual-temporal labeled data. The semantic segmentation detection module introduced during the pre-training stage enhances the model's understanding of the semantics of different land cover types in remote sensing imagery, enabling the model to better capture and understand the features of changing targets, thus improving the model's detection accuracy and generalization performance for changed areas.

[0004] To achieve the objectives of the above invention, the technical solution adopted by this invention is: a change detection pre-training method based on single-temporal semantic segmentation, comprising the following steps:

[0005] Step 1), input single-temporal remote sensing semantic segmentation data, and randomly generate pseudo-dual-temporal change detection data online before the start of each round of pre-training;

[0006] Step 2) Introduce a semantic segmentation detection module into any deep learning change detection network module to form a pre-trained network;

[0007] Step 3) Input the pseudo-dual-temporal change detection data into the pre-trained network, output the pseudo-dual-temporal change detection results and the semantic segmentation detection results of each temporal phase, supervise all results, and finally obtain the pre-trained model.

[0008] Step 4) Initialize the change detection network module in the pre-trained network using the pre-trained model, and then fine-tune it on the real two-phase change detection dataset to obtain the final change detection model;

[0009] Step 5): Based on the final change detection model obtained in Step 4), identify and visualize the change areas in the dual-temporal remote sensing images to be detected.

[0010] Furthermore, the specific implementation method for online random generation of pseudo-dual-phase change detection data in step 1) is as follows:

[0011] Input single-phase remote sensing semantic segmentation dataset Where n is the number of samples in the dataset, Representing remote sensing images and their corresponding semantic segmentation labels, two samples are randomly and non-repeatingly drawn from a single-temporal semantic segmentation dataset. and The pseudo-dual-temporal remote sensing image is constructed, and the corresponding change detection label of the pseudo-dual-temporal image is generated according to formula (3);

[0012] For the binary classification change detection task, the specific construction method is as follows: positive samples of change labels are assigned to the pixel positions where the target object appears only once; if there are two overlapping object instances in the pseudo-dual phase, the pixel positions of the overlapping area are assigned as negative samples, because the change is only related to semantics, not instance, and the remaining pixel positions are assigned as negative samples.

[0013]

[0014] in, Let G represent the loss function, μ represent the model parameters to be optimized, and G represent the loss function. μ (·) indicates a change detection network module. and It is a pseudo-dual-phase image; ⊕ indicates an XOR operation. and semantic tags It provides single-phase supervision.

[0015] Furthermore, the semantic segmentation detection module in step 2) includes two convolutional layers, a batch normalization layer, and a ReLU layer.

[0016] Furthermore, in step 3), the total loss of the pre-trained model includes the change detection loss from the change detection network module and the semantic segmentation loss from the semantic segmentation module.

[0017] Furthermore, the loss calculation method between the output of the change detection network module and the online generated pseudo-bi-temporal supervision label is shown in Equation (4):

[0018]

[0019] Among them, L cd This represents the change detection loss from the change detection network module. Represents the cross-entropy loss function. This represents the output from the change detection network module. This indicates pseudo-bi-temporal supervision labels generated online.

[0020] Furthermore, the loss calculation method between the output of the semantic segmentation module and the single-temporal semantic segmentation supervision label is shown in Equation (5):

[0021]

[0022] Where i represents different time phases, i = 1 represents the previous time phase, and i = 2 represents the subsequent time phase; This represents the loss between the semantic segmentation module's output for time phase i and the semantic segmentation supervision label corresponding to time phase i; Represents the cross-entropy loss function; This represents the output from the semantic segmentation module for phase i; This represents the semantic segmentation supervision label corresponding to phase i.

[0023] Furthermore, the total loss during the pre-training phase is expressed by formula (6):

[0024]

[0025] Here, λ represents the weight coefficient of the semantic segmentation loss, which is used to balance the relative importance of different pre-training objectives.

[0026] Furthermore, in step 4), the specific implementation method for fine-tuning the pre-trained model on a real two-phase change detection dataset is as follows:

[0027] During the fine-tuning phase, the total loss of the model only includes the change detection loss from the change detection network module; the total loss during the fine-tuning phase is expressed by formula (7):

[0028]

[0029] Among them, L fine This represents the change detection loss from the change detection network module. Represents the cross-entropy loss function. Y represents the output from the change detection network module, and Y represents the true bi-temporal supervision label.

[0030] Furthermore, the cross-entropy loss is calculated as shown in Equation (8):

[0031]

[0032] Where N is the number of samples, C is the number of categories, and y i,c Let p be an indicator function representing the true label of the i-th sample under category c; it is 1 if it is the true label of that category, and 0 otherwise. i,c This is the predicted probability of the i-th sample in category c, as predicted by the model.

[0033] The present invention also provides a change detection pre-training system based on single-temporal semantic segmentation, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a change detection pre-training method based on single-temporal semantic segmentation as described in the above technical solution.

[0034] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0035] 1) This invention effectively utilizes abundant single-temporal semantic segmentation data to construct “pseudo-dual-temporal” change detection data, thereby alleviating the dependence of deep learning-based remote sensing image change detection methods on large-scale dual-temporal annotation data.

[0036] 2) Most existing change detection methods only focus on the change features in the preceding and following time-phase remote sensing images, while ignoring the model's understanding of the semantic information of land cover categories in single-time-phase remote sensing images. In the pre-training stage, this invention designs an additional semantic segmentation detection module to enhance the model's ability to understand the semantics of the changed target, thereby improving the model's detection accuracy. Attached Figure Description

[0037] Figure 1 The diagram illustrates the pre-trained network structure for implementing this invention.

[0038] Figure 2 The illustration shows the construction of "pseudo-dual-temporal" change detection data using single-temporal semantic segmentation data for the implementation of this invention.

[0039] Figure 3 A diagram illustrating the overall training strategy for this civilization.

[0040] Figure 4 This illustration shows a comparison of the change detection performance of the model without pre-training and the model with pre-training in accordance with the present invention. Detailed Implementation

[0041] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0042] In change detection tasks, training deep learning-based neural network models typically requires aligned bi-temporal remote sensing images as input and relies on ground truth annotations as supervision signals. The core principle is to learn the ability to detect changed regions in adjacent bi-temporal remote sensing images by using annotated images at the same spatial location but different times. To obtain the "supervision signal," paired semantic information is usually needed to define positive and negative samples. However, the paired semantic information is only related to the semantics of the bi-temporal images and not to their spatial location. The same spatial location is only used to ensure consistency during training and inference. Currently, compared to paired bi-temporal change detection data, single-temporal remote sensing semantic segmentation data is abundant and relatively easy to obtain. When spatial constraints are relaxed, "pseudo-bi-temporal" images can be constructed using any paired single-temporal semantic segmentation images, and the "supervision signal" for the "pseudo-bi-temporal" images can be obtained from the semantic labels of each single-temporal semantic segmentation image. For binary change detection tasks, the specific method for constructing the "supervisory signal" is as follows: positive samples of change labels are assigned to the pixel positions where the target object appears only once; if two object instances overlap in the "pseudo-dual-temporal" data, the pixel positions in the overlapping area are assigned as negative samples, because the change is only semantically related, not instance-related, and the remaining pixel positions are assigned as negative samples. In this way, single-temporal semantic segmentation data can be used to construct "pseudo-dual-temporal" change detection data, thereby effectively alleviating the dependence of deep learning-based remote sensing image change detection methods on large-scale dual-temporal annotation data.

[0043] This invention provides a change detection pre-training method based on single-temporal semantic segmentation, comprising the following steps:

[0044] Step 1), input single-temporal remote sensing semantic segmentation data, and randomly generate "pseudo-dual-temporal" change detection data online before the start of each training round;

[0045] The specific description of generating "pseudo-dual-temporal" change detection data from single-temporal semantic segmentation data in step 1) is as follows:

[0046] 11) In change detection tasks, deep learning model training typically requires aligned bi-temporal remote sensing images as input and relies on real annotations as supervision signals. Its core lies in learning the ability to detect changed regions in bi-temporal remote sensing images by using annotated images at the same spatial location but different times. Therefore, bi-temporal supervised change detection training can be expressed as an optimization problem, as shown in Equation (1).

[0047]

[0048] Where μ represents the model parameters to be optimized. The loss function (also using cross-entropy loss in this embodiment) minimizes the change detection model G. μ (·) in dual-phase and (in Prediction results and change labels on ) (in The loss between ), where the change label represents the region of change from time t1 to time t2. It can be seen from formula (1) that the change label... It is the sole source of monitoring signals. In order to obtain... Typically, paired semantic information is needed to define positive and negative samples. However, paired semantic information is only related to the semantics of the two phases, and not to their spatial location. The same spatial location is only used to ensure consistency during training and inference. If the spatial location constraint is relaxed, the problem expressed by Equation (1) can be simplified to Equation (2):

[0049]

[0050] Where Compare represents the XOR operation, X i and X j It can be any paired image, Y i and Y j These are their semantic labels, and the supervision signal can be obtained from Y. i and Y j The change detection model trained using formula (2) is an extension of the model trained using formula (1). That is, the former not only includes all the knowledge and capabilities of the latter but also possesses additional knowledge and capabilities, enabling change detection in any context, including multi-temporal remote sensing images of the same area. Therefore, "pseudo-dual-temporal" change detection data can be constructed based on formula (2) using single-temporal semantic segmentation data, as shown in formula (3):

[0051]

[0052] in, and It is a "pseudo-dual-phase" image, where ⊕ represents the XOR operation. and semantic tags Single-phase supervision is provided. For binary change detection tasks, the specific construction method is as follows: positive samples of change labels are assigned to the pixel positions where the target object appears only once; if two object instances overlap in the pseudo-dual phase, the pixel positions in the overlapping area are assigned as negative samples, because the change is only related to semantics, not instance, and the remaining pixel positions are assigned as negative samples. Based on this, the change labels in formula (1) It can be replaced with This allows for the use of single-phase data to provide a monitoring signal for change detection tasks.

[0053] 12) Specifically, during the pre-training phase, the input is a single-temporal remote sensing semantic segmentation dataset D. seg ={(X 1 ,Y 1 ),···,(X n ,Y n )}, where n is the number of samples in the dataset, (X i ,Y i () represents the remote sensing image and its corresponding semantic segmentation label. Taking the construction of a single sample as an example, two samples (X) are randomly and non-repeatingly drawn from the single-temporal semantic segmentation dataset. i ,Y i ) and (X j ,Y j The pseudo-dual-temporal remote sensing image is constructed, and the corresponding change detection annotation of the pseudo-dual-temporal image is generated according to formula (3). In this way, the rich single-temporal semantic segmentation data can be effectively used to construct pseudo-dual-temporal change detection data.

[0054] Step 2) Use any deep learning change detection network and introduce a semantic segmentation detection module on top of that network to form a pre-trained network;

[0055] The semantic segmentation and detection module includes two convolutional layers, one batch normalization layer, and one ReLU layer.

[0056] Step 3) Input the "pseudo-dual-temporal" change detection data into the pre-trained network constructed in Step 2), output the "pseudo-dual-temporal" change detection results and the semantic segmentation detection results of each temporal phase, supervise all results, and finally obtain the pre-trained model.

[0057] The specific process of pre-training using the "pseudo-bi-temporal" data constructed in step 1) in step 3) is as follows:

[0058] At this stage, the total loss of the model includes the change detection loss from the change detection network used in step 2) and the semantic segmentation loss from the semantic segmentation module. The loss between the output of the change detection network and the online generated "pseudo-bi-temporal" supervision labels is calculated as shown in Equation (4).

[0059]

[0060] Among them, L cd This represents the change detection loss from the change detection network. Represents the cross-entropy loss function. This represents the output from the change detection network. This refers to the "pseudo-biphase" supervision label generated online.

[0061] The loss calculation method between the output of the semantic segmentation module and the single-phase semantic segmentation supervision label is shown in Equation (5).

[0062]

[0063] Where i represents different time phases, i = 1 represents the previous time phase, and i = 2 represents the subsequent time phase; This represents the loss between the semantic segmentation module's output for time phase i and the semantic segmentation supervision label corresponding to time phase i; Represents the cross-entropy loss function; This represents the output from the semantic segmentation module for phase i; This represents the semantic segmentation supervision label corresponding to phase i.

[0064] In summary, the total loss during the pre-training phase can be expressed by formula (6):

[0065]

[0066] Where λ represents the weight coefficient of semantic segmentation loss, which is used to balance the relative importance of different pre-training objectives, and is set to 1 based on experience.

[0067] Step 4) Input the pre-trained model obtained in Step 3) into the deep learning change detection network used in Step 2) to initialize the weight parameters of the network. Then, input real dual-temporal change detection data to fine-tune the pre-trained model and obtain the final remote sensing image change detection model.

[0068] The specific process of fine-tuning the pre-trained model obtained in step 3) on the real two-phase change detection dataset in step 4) is as follows:

[0069] In the fine-tuning stage, the model obtained in the pre-training stage is applied to the real change detection dataset to fine-tune the change detection network used in step 2). In this stage, the total loss of the model only includes the change detection loss of the change detection network, which can be expressed by formula (7).

[0070]

[0071] Among them, L fine This represents the change detection loss from the change detection network. Represents the cross-entropy loss function. Y represents the output from the change detection network, and Y represents the true bi-temporal supervision label.

[0072] In the overall model training strategy, all loss functions employed are the cross-entropy loss function. This loss function is widely used in classification problems and can effectively measure the difference between the model's predicted distribution and the true label distribution.

[0073] The calculation of cross-entropy loss is shown in Equation (8).

[0074]

[0075] Where N is the number of samples, C is the number of categories, and y i,c Let p be an indicator function representing the true label of the i-th sample in category c (1 if it is the true label, 0 otherwise), and p be an indicator function representing the true label of the i-th sample in category c. i,c This is the predicted probability of the i-th sample in category c, as predicted by the model.

[0076] Step 5): Based on the final change detection model obtained in Step 4), identify and visualize the change areas in the dual-temporal remote sensing images to be detected.

[0077] Figure 1 This is a schematic diagram of the pre-trained network used in this invention. The specific structural details are as follows:

[0078] This network is constructed by adding a semantic segmentation module to an arbitrary change detection network. Semantic segmentation data is used as input, and then "pseudo-tense" change detection data is constructed online and input into the pre-trained network. The output is semantic segmentation results and "pseudo-tense" change detection results, and the change detection network used is pre-trained.

[0079] Figure 2 Here is an example of constructing "pseudo-dual-temporal" change detection data using single-phase semantic segmentation data. The specific construction details are as follows:

[0080] Input single-phase semantic segmentation data and randomly pair them to form "pseudo-dual-phase" data. Then, according to formula (3), use the semantic labels of the single-phase semantic segmentation data to construct the supervision labels of the "pseudo-dual-phase" data. For the binary classification change detection task, the specific construction method is as follows: positive samples of change labels are assigned to the pixel positions where the target object appears only once; if there are two overlapping object instances in the "pseudo-dual-phase", the pixel positions of the overlapping area are assigned as negative samples, because the change is only related to semantics and not to the instance. The remaining pixel positions are assigned as negative samples.

[0081]

[0082] in, and It is a "pseudo-dual-phase" image, where ⊕ represents the XOR operation. and semantic tags It provides single-phase supervision.

[0083] Figure 3 The overall training scheme of this invention is a "pre-training-fine-tuning" strategy. It includes the following steps:

[0084] 1) Input single-temporal remote sensing semantic segmentation data, and randomly generate "pseudo-dual-temporal" change detection data online before the start of each round of pre-training;

[0085] 2) Input the "pseudo-dual-temporal" change detection data into an arbitrary change detection network with a semantic segmentation detection module, output the "pseudo-dual-temporal" change detection results and the segmentation detection results of each single-temporal semantic segmentation data, then supervise all results, and finally obtain the pre-trained model;

[0086] 3) Input the pre-trained model obtained in 2) into the change detection network used in 2) to initialize the weight parameters of the network. Then, input real dual-temporal change detection data to fine-tune the pre-trained model and obtain the final change detection model.

[0087] Figure 4 To compare the change detection performance of untrained and pretrained models, the change detection pretraining strategy based on single-temporal semantic segmentation proposed in this invention can fully mine the rich information contained in the semantic segmentation data, enabling the model to learn features better in the change detection task. In addition, the semantic segmentation detection module enhances the model's ability to understand the semantics of changed targets, enabling the model to learn the deep semantic information of changed ground objects, thereby improving the model's recognition accuracy of changed areas.

[0088] On the other hand, embodiments of the present invention also provide a change detection pre-training system based on single-temporal semantic segmentation, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a change detection pre-training method based on single-temporal semantic segmentation as described in the above technical solution.

[0089] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A single time phase semantic segmentation based change detection pre-training method, characterized in that, The method comprises the following steps: Step 1), input single-phase remote sensing semantic segmentation data, and randomly generate pseudo-bi-temporal change detection data online before starting each round of pre-training; Step 2), introduce a semantic segmentation detection module into any deep learning change detection network module to form a pre-training network; Step 3), input the pseudo-bi-temporal change detection data into the pre-training network, output the pseudo-bi-temporal change detection result and the semantic segmentation detection result of each phase, and supervise all the results, and finally obtain a pre-training model; Step 4), use the pre-training model to initialize the change detection network module in the pre-training network, and then fine-tune on the real bi-temporal change detection data set to obtain a final change detection model; In step 4), the specific implementation of fine-tuning the pre-training model on the real bi-temporal change detection data set is as follows: In the fine-tuning stage, the total loss of the model only includes the change detection loss of the change detection network module; the total loss in the fine-tuning stage is represented by formula (7): （7） wherein, represents a change detection loss from the change detection network module, represents a cross-entropy loss function, represents an output from the change detection network module, represents a real dual-time supervision label; Step 5), according to step 4), obtain the final change detection model, and perform change area recognition and visual output on the bi-temporal remote sensing image to be detected.

2. The single time phase semantic segmentation based change detection pre-training method of claim 1, wherein: The specific implementation of randomly generating pseudo-bi-temporal change detection data online in step 1) is as follows: Input single temporal remote sensing semantic segmentation dataset wherein is the number of samples in the dataset, represents remote sensing images and corresponding semantic segmentation labels, two samples are randomly and non-repeatedly extracted from the single temporal semantic segmentation dataset and constitute a pseudo-bi-temporal remote sensing image, and the corresponding pseudo-bi-temporal image change detection label is generated according to formula (3); For the binary classification change detection task, the specific construction method is: the positive samples of the change label are assigned to the pixel positions where the target object appears only once; if there are two object instances overlapping in the pseudo-bi-temporal phase, the pixel positions in the overlapping area are assigned as negative samples, because the change is only related to semantics, not to instances, and the remaining pixel positions are assigned as negative samples; （3） wherein, denotes a loss function, denotes model parameters to be optimized, denotes a change detection network module, and is a pseudo dual-phase image, denotes an exclusive or operation, and semantic labels provides single-phase supervision.

3. The single time phase semantic segmentation based change detection pre-training method of claim 1, wherein: The semantic segmentation detection module in step 2) comprises two convolutional layers, one batch normalization layer and one ReLU layer.

4. The single time phase semantic segmentation based change detection pre-training method of claim 1, wherein: The total loss of the pre-training model in step 3) includes the change detection loss from the change detection network module and the semantic segmentation loss from the semantic segmentation module.

5. The single time phase semantic segmentation based change detection pre-training method of claim 4, wherein: The loss calculation method between the output of the change detection network module and the pseudo-bi-temporal supervision label generated online is shown in formula (4): （4） wherein, represents a change detection loss from the change detection network module, represents a cross-entropy loss function, represents an output from the change detection network module, represents an online generated pseudo dual-phase supervisory label.

6. The single time phase semantic segmentation based change detection pre-training method of claim 5, wherein: The loss calculation method between the output of the semantic segmentation module and the single-phase semantic segmentation supervision label is shown in formula (5): （5） wherein, represents a preceding time phase, when represents a preceding time phase, when 2 represents a succeeding time phase; represents a loss between an output of the semantic segmentation module for a time phase and a semantic segmentation supervisory label corresponding to the time phase ; represents a cross-entropy loss function; represents an output from the semantic segmentation module for a time phase ; represents a semantic segmentation supervisory label corresponding to the time phase .

7. The single time phase semantic segmentation based change detection pre-training method of claim 6, wherein: The total loss of the pre-training stage is represented by formula (6): （6） wherein, denotes a weight coefficient of the semantic segmentation loss, used to balance the relative importance of different pre-training objectives.

8. The single time phase semantic segmentation based change detection pre-training method of claim 1 or 5 or 6 or 7, wherein: The calculation of the cross-entropy loss is shown in formula (8): （8） wherein, is the number of samples, is the number of classes, denotes the indicator function of the true label of the -th sample in the class , which is 1 if it is the class and 0 otherwise, while is the predicted probability of the -th sample by the model in the class .

9. A single time phase semantic segmentation based change detection pre-training system, characterized in that: The method comprises a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the pre-training method for change detection based on single-phase semantic segmentation according to any one of claims 1-8.

Images