Change detection pre-training method and system based on single temporal semantic segmentation

Abstract

The invention discloses a change detection pre-training method and system based on single temporal semantic segmentation, and belongs to the technical field of remote sensing image detection. The method comprises the following steps: firstly, constructing pseudo dual-temporal change detection data by using wide single-temporal semantic segmentation data, then introducing a semantic segmentation detection module into any deep learning change detection network to form a pre-training network, and performing pre-training on the pseudo dual-temporal change detection data; and then the obtained pre-training model is finely adjusted on a real change detection data set to obtain a final change detection model. According to the method, rich single-temporal semantic segmentation data is effectively utilized to construct pseudo dual-temporal change detection data, so that the dependence of a remote sensing image change detection method based on deep learning on large-scale dual-temporal annotation data is relieved; according to the method, a semantic segmentation detection module is additionally designed in the pre-training stage, so that the understanding ability of the model on changed target semantics is enhanced, and the detection precision of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image detection, and in particular relates to a change detection pre-training method and system based on single-phase semantic segmentation. Background Art

[0002] Remote sensing image change detection is a technology that uses multi-temporal remote sensing data to analyze surface changes. It is widely used in geological exploration, environmental monitoring, agricultural management, and urban planning, and is of great significance to the sustainable development of society. In recent years, with the rapid development of deep learning technology, remote sensing image change detection methods based on deep learning have made significant progress, but they still face many challenges. Specifically, most of the existing deep learning methods use fully supervised training methods, which rely heavily on large-scale dual-temporal change detection labeled data. However, due to the characteristics of the remote sensing image change detection task itself, it is extremely difficult to collect paired and labeled change detection images. On the one hand, due to the influence of multiple factors such as weather conditions, shooting time, and differences between different sensors, it is difficult to obtain consistent and high-quality dual-temporal data; on the other hand, labeling the changed areas in dual-temporal remote sensing images not only requires more professional background knowledge, but also the labeling process is time-consuming and laborious. Therefore, at present, bi-temporal annotated change detection data are still relatively scarce and precious. The scale of existing open source change detection datasets is generally smaller than that of other remote sensing tasks. This significantly limits the application of remote sensing image change detection methods based on deep learning in actual scenarios. Therefore, it is urgent to reduce its dependence on bi-temporal annotated data. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a change detection pre-training method based on single-phase semantic segmentation. First, the method uses a wide range of single-phase semantic segmentation data to construct "pseudo-bi-phase" change detection data, and then introduces a semantic segmentation detection module into any deep learning change detection network to form a pre-training network, and pre-trains on the "pseudo-bi-phase" change detection data. Then, the obtained pre-trained model is fine-tuned on the real change detection data set to obtain the final change detection model. The method uses the constructed "pseudo-bi-phase" change detection data to pre-train any deep learning change detection network, thereby alleviating the dependence on bi-phase labeled data. The semantic segmentation detection module introduced in the pre-training stage can enhance the model's understanding of the semantics of different types of objects in remote sensing images, so that the model can better capture and understand the characteristics of the changing target, thereby improving the model's detection accuracy and generalization performance of the changed area.

[0004] To achieve the purpose of the above invention, the technical solution adopted by the present invention is: a change detection pre-training method based on single-phase semantic segmentation, comprising the following steps:

[0005] Step 1), input single-phase remote sensing semantic segmentation data, and before each round of pre-training, randomly generate pseudo dual-phase change detection data online;

[0006] Step 2), introducing a semantic segmentation detection module into any deep learning change detection network module to form a pre-trained network;

[0007] Step 3), input the pseudo-bi-phase change detection data into the pre-trained network, output the pseudo-bi-phase change detection results and the semantic segmentation detection results of each phase, supervise all the results, and finally obtain the pre-trained model;

[0008] Step 4), using the pre-trained model to initialize the change detection network module in the pre-trained network, and then fine-tuning it on the real dual-temporal change detection dataset to obtain the final change detection model;

[0009] Step 5), according to step 4), the final change detection model is obtained, and the changed area of ​​the dual-temporal remote sensing image to be detected is identified and visualized.

[0010] Further, the specific implementation method of online random generation of pseudo dual-phase change detection data in step 1) is as follows:

[0011] Input single temporal remote sensing semantic segmentation dataset Where n is the number of samples in the data set, Represents remote sensing images and corresponding semantic segmentation labels, and randomly and non-repeatedly extracts two samples from the single-temporal semantic segmentation dataset and Constructing a pseudo dual-temporal remote sensing image, and generating the corresponding change detection annotation of the pseudo dual-temporal image according to formula (3);

[0012] For the binary change detection task, the specific construction method is as follows: the positive samples of the change label are assigned to the pixel positions where the target object appears only once; if there are two object instances overlapping in the pseudo-bi-phase, the pixel positions in the overlapping area are assigned as negative samples, because the change is only related to semantics, not instances, and the remaining pixel positions are assigned as negative samples;

[0013]

[0014] in, represents the loss function, μ represents the model parameters to be optimized, G μ (·) represents the change detection network module, and is a pseudo-bi-temporal image, ⊕ represents an XOR operation, and Semantic tags Provides single-phase supervision.

[0015] Furthermore, the semantic segmentation detection module in step 2) includes two convolutional layers, a batch normalization layer and a ReLU layer.

[0016] Furthermore, the total loss of the pre-trained model in step 3) includes the change detection loss from the change detection network module and the semantic segmentation loss from the semantic segmentation module.

[0017] Furthermore, the loss calculation method between the output of the change detection network module and the pseudo bi-temporal supervision label generated online is shown in formula (4):

[0018]

[0019] Among them, L cd represents the change detection loss from the change detection network module, represents the cross entropy loss function, represents the output from the change detection network module, represents the pseudo bi-temporal supervision labels generated online.

[0020] Furthermore, the loss calculation method between the output of the semantic segmentation module and the single-phase semantic segmentation supervision label is shown in formula (5):

[0021]

[0022] Where i represents different phases, when i = 1 it indicates the anterior phase, and when i = 2 it indicates the posterior phase; represents the loss between the output of the semantic segmentation module for phase i and the semantic segmentation supervision label corresponding to phase i; represents the cross entropy loss function; represents the output from the semantic segmentation module for phase i; Represents the semantic segmentation supervision label corresponding to phase i.

[0023] Furthermore, the total loss in the pre-training phase is expressed by formula (6):

[0024]

[0025] Among them, λ represents the weight coefficient of semantic segmentation loss, which is used to balance the relative importance of different pre-training objectives.

[0026] Furthermore, in step 4), the specific implementation method of fine-tuning the pre-trained model on the real dual-phase change detection dataset is as follows:

[0027] In the fine-tuning stage, the total loss of the model only includes the change detection loss of the change detection network module; the total loss in the fine-tuning stage is expressed by formula (7):

[0028]

[0029] Among them, L fine represents the change detection loss from the change detection network module, represents the cross entropy loss function, represents the output from the change detection network module, and Y represents the true bi-temporal supervision label.

[0030] Furthermore, the calculation of the cross entropy loss is shown in formula (8):

[0031]

[0032] Among them, N is the number of samples, C is the number of categories, and y i,c represents the indicator function of the true label of the i-th sample under category c. If it is in this category, it is 1, otherwise it is 0, and p i,c It is the predicted probability of the i-th sample under category c predicted by the model.

[0033] The present invention also provides a change detection pre-training system based on single-phase semantic segmentation, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a change detection pre-training method based on single-phase semantic segmentation as described in the above technical solution.

[0034] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0035] 1) The present invention effectively utilizes abundant single-phase semantic segmentation data to construct “pseudo-bi-phase” change detection data, thereby alleviating the dependence of deep learning-based remote sensing image change detection methods on large-scale bi-phase annotated data.

[0036] 2) Most of the existing change detection methods only focus on the change features in the previous and next phases of remote sensing images, but ignore the model's understanding of the semantic information of the object category in the single-phase remote sensing image. The present invention additionally designs a semantic segmentation detection module in the pre-training stage to enhance the model's ability to understand the semantics of the changed target, thereby improving the model's detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A diagram illustrating the structure of a pre-trained network for implementing the present invention.

[0038] Figure 2 A "pseudo-bi-phase" change detection data diagram is constructed for the implementation of the present invention using single-phase semantic segmentation data.

[0039] Figure 3 Diagram of the overall training strategy implemented for this civilization.

[0040] Figure 4 The figure is a comparative diagram of the change detection effects of the model without pre-training and with pre-training implemented in the present invention. DETAILED DESCRIPTION

[0041] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0042] In the change detection task, the training of neural network models based on deep learning usually requires the input of aligned dual-temporal remote sensing images and relies on real annotations as supervisory signals. The core of deep learning is to learn the ability to detect the changed areas in the remote sensing images of the previous and next phases through annotated images of the same spatial position but different time. In order to obtain the "supervisory signal", paired semantic information is usually required to define positive and negative samples. However, paired semantic information is only related to the semantics of the dual-temporal phase, but not to its spatial position. The same spatial position is only used to ensure consistency during training and inference. At present, compared with paired dual-temporal change detection data, single-temporal remote sensing semantic segmentation data is abundant and relatively easy to obtain. When the spatial position constraints are relaxed, any paired single-temporal semantic segmentation images can be used to construct "pseudo-bi-temporal", and the "supervisory signal" of "pseudo-bi-temporal" can be obtained from the semantic label of each single-temporal semantic segmentation image. For the binary change detection task, the specific "supervisory signal" construction method is as follows: the positive sample of the change label is assigned to the pixel position where the target object appears only once; if there are two object instances overlapping in the "pseudo-bi-phase", the pixel position of the overlapping area is assigned as a negative sample, because the change is only related to semantics, not instances, and the remaining pixel positions are assigned as negative samples. In this way, the single-phase semantic segmentation data can be used to construct the "pseudo-bi-phase" change detection data, thereby effectively alleviating the dependence of the remote sensing image change detection method based on deep learning on large-scale bi-phase annotated data.

[0043] The embodiment of the present invention provides a change detection pre-training method based on single-phase semantic segmentation, comprising the following steps:

[0044] Step 1), input single-phase remote sensing semantic segmentation data, and before each round of training, randomly generate "pseudo-bi-phase" change detection data online;

[0045] The specific expression of generating "pseudo-bi-phase" change detection data from single-phase semantic segmentation data in step 1) is as follows:

[0046] 11) In the change detection task, deep learning model training usually requires input of aligned dual-temporal remote sensing images and relies on real annotations as supervisory signals. The core of deep learning model training is to learn the ability to detect the changed areas in the previous and next phase remote sensing images through annotated images at the same spatial location but at different times. Therefore, dual-temporal supervised change detection training can be expressed as an optimization problem, as shown in formula (1).

[0047]

[0048] Among them, μ represents the model parameters to be optimized, represents the loss function (cross entropy loss is also used in this embodiment), which minimizes the change detection model G μ (·) In bi-phase and (in ) on the prediction results and change labels (in ), where the change label represents the change region from time t1 to time t2. From formula (1), it can be seen that the change label is the only source of supervision signal. Paired semantic information is usually required to define positive and negative samples. However, paired semantic information is only related to the semantics of the two phases, but not to their spatial positions. The same spatial position is only used to ensure consistency during training and inference. If the spatial position constraint is relaxed, the problem expressed in formula (1) can be simplified to formula (2):

[0049]

[0050] Where Compare represents an XOR operation, X i and X j Can be any paired image, Y i and Y j are their semantic labels, and the supervision signal can be obtained from Y i and Y j The change detection model trained by formula (2) will be an extension of the model trained by formula (1), that is, the former not only contains all the knowledge and capabilities of the latter, but also has additional knowledge and capabilities, and can perform change detection in any context, including multi-temporal remote sensing images of the same area. Therefore, based on formula (2), the "pseudo-bi-temporal" change detection data can be constructed using single-temporal semantic segmentation data, as shown in formula (3):

[0051]

[0052] in, and is a "pseudo-bi-phase" image, ⊕ represents an XOR operation, and Semantic tags Single-phase supervision is provided. For the binary change detection task, the specific construction method is as follows: the positive sample of the change label is assigned to the pixel position where the target object appears only once; if there are two object instances overlapping in the pseudo-bi-phase, the pixel position in the overlapping area is assigned as a negative sample, because the change is only related to semantics, not to instances, and the remaining pixel positions are assigned as negative samples. Based on this, the change label in formula (1) Can be replaced by Thus, single-phase data is used to provide supervisory signals for change detection tasks.

[0053] 12) Specifically, in the pre-training stage, the single-temporal remote sensing semantic segmentation dataset D is input seg ={(X 1 ,Y 1 ),···,(X n ,Y n )}, where n is the number of samples in the data set, (X i ,Y i ) represents the remote sensing image and the corresponding semantic segmentation label. Taking the construction of a single sample as an example, two samples (X i ,Y i ) and (X j ,Y j ) to form a "pseudo-bi-temporal" remote sensing image, and generate the corresponding pseudo-bi-temporal image change detection annotation according to formula (3). Through this method, the rich single-phase semantic segmentation data can be effectively used to construct the "pseudo-bi-temporal" change detection data.

[0054] Step 2) using any deep learning change detection network, and introducing a semantic segmentation detection module based on the network to form a pre-trained network;

[0055] Among them, the semantic segmentation detection module consists of two convolutional layers, a batch normalization layer and a ReLU layer.

[0056] Step 3), input the "pseudo-bi-phase" change detection data into the pre-trained network constructed in step 2), output the "pseudo-bi-phase" change detection results and the semantic segmentation detection results of each phase, supervise all the results, and finally obtain the pre-trained model;

[0057] The specific process of pre-training using the "pseudo-bi-phase" data constructed in step 1) in step 3) is as follows:

[0058] At this stage, the total loss of the model includes the change detection loss from the change detection network used in step 2) and the semantic segmentation loss from the semantic segmentation module. The loss calculation method between the output of the change detection network and the online generated "pseudo bi-temporal" supervision label is shown in formula (4).

[0059]

[0060] Among them, L cd represents the change detection loss from the change detection network, represents the cross entropy loss function, represents the output from the change detection network, Represents the online generated “pseudo bi-temporal” supervision labels.

[0061] The loss calculation method between the output of the semantic segmentation module and the single-temporal semantic segmentation supervision label is shown in formula (5).

[0062]

[0063] Where i represents different phases, when i = 1 it indicates the anterior phase, and when i = 2 it indicates the posterior phase; represents the loss between the output of the semantic segmentation module for phase i and the semantic segmentation supervision label corresponding to phase i; represents the cross entropy loss function; represents the output from the semantic segmentation module for phase i; Represents the semantic segmentation supervision label corresponding to phase i.

[0064] To sum up, the total loss in the pre-training stage can be expressed by formula (6):

[0065]

[0066] Among them, λ represents the weight coefficient of semantic segmentation loss, which is used to balance the relative importance of different pre-training objectives and is empirically set to 1.

[0067] Step 4), input the pre-trained model obtained in step 3) into the deep learning change detection network used in step 2) to initialize the weight parameters of the network, then input the real dual-phase change detection data, fine-tune the pre-trained model, and obtain the final remote sensing image change detection model.

[0068] The specific process of fine-tuning the pre-trained model obtained in step 3) on the real dual-phase change detection dataset in step 4) is as follows:

[0069] In the fine-tuning stage, the model obtained in the pre-training stage is applied to the real change detection dataset to fine-tune the change detection network used in step 2). In this stage, the total loss of the model only includes the change detection loss of the change detection network, which can be expressed by formula (7).

[0070]

[0071] Among them, L fine represents the change detection loss from the change detection network, represents the cross entropy loss function, represents the output from the change detection network and Y represents the true bi-temporal supervision label.

[0072] In the overall model training strategy, all loss functions use the cross entropy loss function. This loss function is widely used in classification problems and can effectively measure the difference between the model prediction distribution and the true label distribution.

[0073] The calculation of the cross entropy loss is shown in formula (8).

[0074]

[0075] Among them, N is the number of samples, C is the number of categories, and y i,c represents the indicator function of the true label of the i-th sample under category c (1 if it is this category, otherwise 0), and p i,c It is the predicted probability of the i-th sample under category c predicted by the model.

[0076] Step 5), according to step 4), the final change detection model is obtained, and the changed area of ​​the dual-temporal remote sensing image to be detected is identified and visualized.

[0077] Figure 1 This is a schematic diagram of the pre-trained network results used in the present invention. The specific structural details are as follows:

[0078] This network is composed of adding a semantic segmentation module to any change detection network. The semantic segmentation data is used as input, and then the "pseudo-bi-phase" change detection data is constructed online and input into the pre-trained network. The semantic segmentation results and "pseudo-bi-phase" change detection results are output to pre-train the change detection network used.

[0079] Figure 2 An example of constructing "pseudo-bi-temporal" change detection data using single-phase semantic segmentation data. The specific construction details are as follows:

[0080] The single-phase semantic segmentation data is input and randomly paired to form "pseudo-bi-phase" data. Then, the semantic labels of the single-phase semantic segmentation data are used to construct the supervision labels of the "pseudo-bi-phase" data according to formula (3). For the binary change detection task, the specific construction method is as follows: the positive samples of the change label are assigned to the pixel positions where the target object appears only once; if there are two object instances overlapping in the "pseudo-bi-phase", the pixel positions in the overlapping area are assigned as negative samples, because the change is only related to semantics but not to instances, and the remaining pixel positions are assigned as negative samples.

[0081]

[0082] in, and is a "pseudo-bi-phase" image, ⊕ represents an XOR operation, and Semantic tags Provides single-phase supervision.

[0083] Figure 3 The overall training scheme of the present invention is the "pre-training-fine-tuning" strategy. It includes the following steps:

[0084] 1) Input single-phase remote sensing semantic segmentation data, and randomly generate "pseudo-bi-phase" change detection data online before each round of pre-training;

[0085] 2) Input the "pseudo-bi-phase" change detection data into any change detection network with a semantic segmentation detection module, output the "pseudo-bi-phase" change detection results and the segmentation detection results of each single-phase semantic segmentation data, then supervise all the results and finally obtain the pre-trained model;

[0086] 3) Input the pre-trained model obtained in 2) into the change detection network used in 2) to initialize the weight parameters of the network, then input the real dual-phase change detection data, fine-tune the pre-trained model, and obtain the final change detection model.

[0087] Figure 4 To compare the change detection effects of models that have not been pre-trained and those that have been pre-trained, the change detection pre-training strategy based on single-phase semantic segmentation proposed in the present invention can fully mine the rich information contained in the semantic segmentation data, so that the model can better learn features in the change detection task. In addition, the semantic segmentation detection module enhances the model's ability to understand the semantics of the changing target, so that the model can learn the deep semantic information of the changing ground objects, thereby improving the model's recognition accuracy of the changed area.

[0088] On the other hand, an embodiment of the present invention also provides a change detection pre-training system based on single-phase semantic segmentation, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a change detection pre-training method based on single-phase semantic segmentation as described in the above technical solution.

[0089] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A change detection pre-training method based on single-phase semantic segmentation, characterized in that: The steps include: Step 1), input single-phase remote sensing semantic segmentation data, and before each round of pre-training, randomly generate pseudo dual-phase change detection data online; Step 2), introducing a semantic segmentation detection module into any deep learning change detection network module to form a pre-trained network; Step 3), input the pseudo-bi-phase change detection data into the pre-trained network, output the pseudo-bi-phase change detection results and the semantic segmentation detection results of each phase, supervise all the results, and finally obtain the pre-trained model; Step 4), using the pre-trained model to initialize the change detection network module in the pre-trained network, and then fine-tuning it on the real dual-temporal change detection dataset to obtain the final change detection model; Step 5), according to step 4), the final change detection model is obtained, and the changed area of ​​the dual-temporal remote sensing image to be detected is identified and visualized.

2. The change detection pre-training method based on single-phase semantic segmentation according to claim 1, characterized in that: The specific implementation method of online random generation of pseudo dual-phase change detection data in step 1) is as follows: Input single temporal remote sensing semantic segmentation dataset Where n is the number of samples in the data set, Represents remote sensing images and corresponding semantic segmentation labels, and randomly and non-repeatedly extracts two samples from the single-temporal semantic segmentation dataset and Constructing a pseudo dual-temporal remote sensing image, and generating the corresponding change detection annotation of the pseudo dual-temporal image according to formula (3); For the binary change detection task, the specific construction method is as follows: the positive samples of the change label are assigned to the pixel positions where the target object appears only once; if there are two object instances overlapping in the pseudo-bi-phase, the pixel positions in the overlapping area are assigned as negative samples, because the change is only related to semantics, not instances, and the remaining pixel positions are assigned as negative samples; in, represents the loss function, μ represents the model parameters to be optimized, G μ (·) represents the change detection network module, and It is a pseudo-dual-phase image. Represents the exclusive OR operation, and Semantic tags Provides single-phase supervision.

3. The change detection pre-training method based on single-phase semantic segmentation according to claim 1, characterized in that: The semantic segmentation detection module in step 2) includes two convolutional layers, a batch normalization layer and a ReLU layer.

4. The change detection pre-training method based on single-phase semantic segmentation according to claim 1, characterized in that: The total loss of the pre-trained model in step 3) includes the change detection loss from the change detection network module and the semantic segmentation loss from the semantic segmentation module.

5. The change detection pre-training method based on single-phase semantic segmentation according to claim 4, characterized in that: The loss calculation method between the output of the change detection network module and the pseudo bi-temporal supervision label generated online is shown in formula (4): Among them, L cd represents the change detection loss from the change detection network module, represents the cross entropy loss function, represents the output from the change detection network module, represents the pseudo bi-temporal supervision labels generated online.

6. The change detection pre-training method based on single-phase semantic segmentation according to claim 5, characterized in that: The loss calculation method between the output of the semantic segmentation module and the single-phase semantic segmentation supervision label is shown in formula (5): Where i represents different phases, when i = 1 it indicates the anterior phase, and when i = 2 it indicates the posterior phase; represents the loss between the output of the semantic segmentation module for phase i and the semantic segmentation supervision label corresponding to phase i; represents the cross entropy loss function; represents the output from the semantic segmentation module for phase i; Represents the semantic segmentation supervision label corresponding to phase i.

7. The change detection pre-training method based on single-phase semantic segmentation according to claim 6, characterized in that: The total loss in the pre-training phase is expressed by formula (6): Among them, λ represents the weight coefficient of semantic segmentation loss, which is used to balance the relative importance of different pre-training objectives.

8. The change detection pre-training method based on single-phase semantic segmentation according to claim 1, characterized in that: In step 4), the specific implementation method of fine-tuning the pre-trained model on the real dual-phase change detection dataset is as follows: In the fine-tuning stage, the total loss of the model only includes the change detection loss of the change detection network module; the total loss in the fine-tuning stage is expressed by formula (7): Among them, L fine represents the change detection loss from the change detection network module, represents the cross entropy loss function, represents the output from the change detection network module, and Y represents the true bi-temporal supervision label.

9. A change detection pre-training method based on single-phase semantic segmentation as claimed in claim 4, 5, 6 or 7, characterized in that: The calculation of cross entropy loss is shown in formula (8): Among them, N is the number of samples, C is the number of categories, and y i,c represents the indicator function of the true label of the i-th sample under category c. If it is in this category, it is 1, otherwise it is 0, and p i,c It is the predicted probability of the i-th sample under category c predicted by the model.

10. A change detection pre-training system based on single-phase semantic segmentation, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a change detection pre-training method based on single-phase semantic segmentation as described in any one of claims 1 to 9.

Images