A domain knowledge integrated heterogeneous image domain migration network and change detection method

Abstract

The application discloses a domain knowledge integrated heterogeneous image domain migration network and a change detection method, and aims at the change detection problem of the ground cover information of the heterogeneous image. The domain knowledge of the change detection is integrated into the heterogeneous image domain migration network, a domain migration network CK_CycleGAN suitable for the change detection of the heterogeneous image is obtained, the CK_CycleGAN network is used, and the heterogeneous images with large feature differences obtained by different sensors can be well converted into pseudo-homogeneous images with good consistent features in the unchanged area. Thus, the change detection problem of the heterogeneous image is converted into the change detection problem of the pseudo-homogeneous image, and then the change map is obtained by using the change detection method of the homogeneous image, so that the precision of detecting the change area of the ground cover information from two heterogeneous images is improved. Therefore, the application provides a new way for the change detection of the ground cover information of the heterogeneous image.

Description

Technical Field

[0001] This invention relates to an image change detection method, specifically to a heterogeneous image domain migration network incorporating domain knowledge and a change detection method thereon. Background Technology

[0002] Heterogeneous image change detection, belonging to the field of image processing, refers to extracting information on land cover changes using multi-temporal heterogeneous images covering the same geographical area. Heterogeneous images are those acquired by two different sensors. This technology has been widely applied in agricultural surveys, urban expansion, disaster assessment, and geographic national condition monitoring. The heterogeneous image change detection process typically consists of four steps: image preprocessing, heterogeneous image homogenization, generating difference images by comparing homogenized multi-temporal images, and difference image analysis. Among these, heterogeneous image homogenization is the key step in heterogeneous image change detection.

[0003] Current research on change detection using imagery from the same source often utilizes multi-temporal imagery covering the same geographical area acquired by the same satellite. This is because heterogeneous imagery is acquired by different sensors, resulting in significant feature differences between them, which poses a major challenge to extracting change information from multi-temporal imagery. However, different satellite sensors have their own characteristics. For example, optical remote sensing satellites can acquire ground images with rich spectral features, while synthetic aperture radar has the ability to acquire ground information around the clock and in all weather conditions. With the advancement of image processing technology, how to utilize multi-temporal heterogeneous imagery acquired by different satellites for change detection research has gradually become a hot topic in change detection research.

[0004] Image homogenization is a crucial step in heterogeneous remote sensing image change detection. Traditional heterogeneous image change detection methods aim to map heterogeneous images into the same feature domain before detection. Examples include methods based on structural features and image self-similarity. However, these methods suffer from poor detection of edges and zonal changes in changed areas due to patch selection issues. Machine learning-based methods address feature differences by constructing mapping relationships between pixel pairs at the same location in heterogeneous images. Representative methods include those based on affinity matrix similarity and homogeneous pixel transformation. However, the reliability of these pixel pair mapping relationships is significantly affected by the training set construction method. In recent years, deep learning technology has been widely applied to change detection. Deep learning methods for heterogeneous image change detection also seek image feature mapping and obtain homogenized images within the same feature space, then perform change detection based on these homogenized images. Currently, the heterogeneous image change detection method based on GAN network is the most favored technique among researchers. This method uses a network to construct a generator to transfer an image into a feature space similar to another image, and uses pseudo-homogeneous image pairs to perform change detection. However, existing GAN networks are all proposed by computer scientists and do not make full use of domain knowledge for change detection, resulting in room for improvement in the quality of the acquired pseudo-homogeneous remote sensing images, thus limiting the accuracy of change detection results.

[0005] Therefore, developing a heterogeneous image domain migration network and change detection method that incorporates domain knowledge has significant practical implications. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a heterogeneous image domain migration network and change detection method incorporating domain knowledge. This invention features a heterogeneous image domain migration network and change detection method incorporating domain knowledge with certain interpretability, high domain migration quality, and good detection effect.

[0007] To achieve the aforementioned inventive objectives, the present invention employs the following technical solution:

[0008] A heterogeneous image domain migration network incorporating domain knowledge and a change detection method, the method specifically includes the following steps:

[0009] Step 1: Data Collection

[0010] Two different satellite sensors were used to acquire two images of different time phases covering the same geographical area. Both images were registered and geometrically corrected, and were identified as image X at time T1 and image Y at time T2.

[0011] Step 2: Obtain a rough change graph:

[0012] A rough binary change map Cmap is obtained from images X and Y using traditional heterogeneous image change detection methods. In the binary change map Cmap, 0 indicates no change and 1 indicates a change.

[0013] Step 3: Creating training samples:

[0014] Image X and image Y are cropped into an image block x of size N×N at position i using a sliding window of size N×N. i and image block y i Using a sliding window, the entire image X and image Y are cropped separately, and the cropped image patches are aggregated to obtain the sample dataset Sxy{x i y i Based on this, a subset of sample data Sun{x is created by combining the transformation graph Cmap. i y i}, paired sample data subset Sun{x i y i The sample position i in} satisfies the following conditions: the cell value at position i in Cmap is 0, and the number of cells with a value of 0 in a neighborhood window of size N×N centered at position i in Cmap is greater than th×N×N, where th represents the ratio of the number of cells with a value of 0 to the total number of cells in the neighborhood, and the value of th is between 0 and 1.

[0015] Step 4: Model Design

[0016] We designed and built a heterogeneous image domain transfer network CK_CycleGAN that incorporates domain knowledge. CK_CycleGAN includes a generator G′ y→x (y) and generator G′ x→y (x), and contains a discriminator D Y Discriminator D X Generator G′ y→x (y) and discriminator D X Connection, Generator G′ x→y (x) and discriminator D Y Connection; where discriminator D Y Discriminator D X Similar to the discriminator structure used in the classic CycleGAN network, the generator G′ y→x (y) Input image patch y obtained by cropping from image Y. i Generate an image patch y with the same feature space as image X. Xi Generator G′ x→y (x) Input image patch x obtained by cropping from image X i Generate an image patch x with the same feature space as image Y.Yi Discriminator D Y It is to determine the image block x Yi With y i Similarity, discriminator D X Determine y Xi With x i The similarity is used to calculate the loss function L for training the CK_CycleGAN network, which incorporates knowledge from the change detection domain. CK_CycleGAN ;

[0017] Step 5: Model Training

[0018] Training the heterogeneous image domain transfer network CK_CycleGAN: using the created sample dataset Sxy{x i y i} and sample data subset Sun{x i y i The heterogeneous image domain transfer network CK_CycleGAN was trained, and the model loss during training was achieved using the function L. CK_CycleGAN calculate;

[0019] Step 6: Detection of changes in pseudo-homogeneous images:

[0020] The trained CK_CycleGAN domain transfer network is used to transform image X into image X with a similar feature space to image Y. Y Transform image Y into an image Y with a similar feature space to dataset X. X Image Y and image X Y It is a set of pseudo-homogeneous images, image X and image Y X This is another set of pseudo-homogeneous images; subsequently, a homogeneous image change detection method was used to distinguish between pseudo-homogeneous image Y and image X. Y Or pseudo-homogeneous images X and Y X Obtain the change graph from the middle.

[0021] The heterogeneous image domain migration network and change detection method incorporating domain knowledge, in the second step of obtaining a coarse binary change map Cmap, utilizes a traditional change detection method that does not rely on sample data to obtain the coarse change map Cmap, specifically including the method based on iterative robust graph mapping and Markov co-occurrence matrix.

[0022] The heterogeneous image domain migration network and change detection method incorporating domain knowledge, in the fourth step, involves designing the generator G′ of the CK_CycleGAN network. y→x (y) and generator G′ x→y(x) incorporates domain knowledge that the same land feature can be described at different scales in remote sensing images. Based on the generator structure used by CycleGAN, a multi-scale convolution module and a residual module are added: a multi-scale convolution module is added after the convolutional layer of the network encoder in the two generators of CycleGAN to utilize the fused multi-scale features for land feature representation, and a 3-layer residual module is added at the end of the encoder, which makes the generator easier to optimize and avoids the problem of performance degradation caused by simply increasing the model depth.

[0023] The heterogeneous image domain migration network and change detection method incorporating domain knowledge includes a multi-scale convolutional module comprising four sets of convolutional modules in parallel, a normalization function, and a ReLU activation function. The first set of modules includes a 1×1 convolutional kernel; the second set includes a cascaded combination of 1×1 and 3×3 convolutional kernels; the third set includes a cascaded combination of 1×1 and 5×5 convolutional kernels; and the fourth set includes a cascaded combination of a 3×3 max-pooling kernel and a 1×1 convolutional kernel. The outputs of the four sets of parallel modules are first spliced ​​and fused along the channel dimension. The fused result is then input to the normalization function and the ReLU activation function, and the output of the ReLU activation function serves as the output of the multi-scale convolutional module.

[0024] The heterogeneous image domain migration network and change detection method incorporating domain knowledge, wherein the residual module is composed of convolutional layers connected in series, after the feature map is input, it is expanded by 3×3, then passed through 2 times of 3×3 convolution kernels, input to the normalization function, and finally output after operation by the ReLU activation function.

[0025] The heterogeneous image domain migration network and change detection method incorporating domain knowledge, and the generator G′ of the CK_CycleGAN network in the fourth step. y→x (y) and generator G′ x→y The structure of (x) is U-Net, which includes an encoder and a decoder. The input is an image patch of size N×N×C, where N is the length and width, and C is the number of channels in the image X or Y. The output of the model is a domain-transferred image patch of size N×N×C.

[0026] The heterogeneous image domain migration network and change detection method incorporating domain knowledge includes an encoder comprising three convolutional layers of different depths and a multi-scale module. The first layer (Down_I) of the encoder includes a convolutional layer and a multi-scale module. After receiving input, it undergoes 3×3 edge expansion, and after one convolution, the image block scale changes from N×N×C to N×N×64. Then, after passing through the multi-scale convolutional module, the scale changes to N×N×64 to N×N×256. The second layer (Down_II) of the encoder includes a convolutional layer and a multi-scale module. The scaling module, after one convolution, changes the image block scale from N×N×256 to N / 2×N / 2×128, and then after a multi-scale convolution module, the scale changes to N / 2×N / 2×128 to N / 2×N / 2×512; the encoder's third layer, Down_III, is a convolutional layer, after one convolution, changing the image block scale from N / 2×N / 2×512 to N / 4×N / 4×256; the encoder output then passes through three sets of residual modules with the same structure, and the residual modules do not change the feature map scale.

[0027] The heterogeneous image domain migration network and change detection method incorporating domain knowledge includes a decoder comprising three transposed convolutional layers of different depths and a multi-scale module. The first layer, Up_III, is a transposed convolutional layer that receives the output of the residual module. After one transposed convolution, the image block scale changes from N / 4×N / 4×256 to N / 2×N / 2×128. The second layer, Up_II, includes two convolutions and a transposed convolution. The input of the second layer, Up_II, is output from the third layer, Up_III. The image block size changes after two convolutions: N / 2×N / 2×640→N / 2×N / 2×128→N / 2×N / 2×64. After transposed convolution, the image block size changes to N / 2×N / 2×64→N×N×64. The first layer, Up_I, is a convolutional layer. After 3×3 edge expansion, it undergoes one more convolution, changing the image block size to N×N×64→N×N×C, where N×N×C is the output size of the generator.

[0028] The heterogeneous image domain migration network and change detection method incorporating domain knowledge, in the fourth step, the loss function L CK_CycleGAN The calculation method is as follows:

[0029] L CK_CycleGAN (G′ x→y , G′ y→x D X D Y ) = L CycleGan (G′ x→y , G′ y→x D X D Y )+b L CD_Knowledge (G′x→y , G′ y→x )

[0030] Where L CycleGAN The loss consists of cycle consistency loss and discriminator loss, based on the sample dataset Sxy{x i y i The loss calculation is performed using the classic CycleGAN function, with the knowledge weight b used to control the loss L during domain transfer. CD_Knowledge L's contribution to the total loss CD_Knowledge The primary metric is the quality of the generated images on unchanged paired data, based on the sample dataset Sun{x. i y i The samples in} are obtained using the following formula:

[0031] L CD_Knowledge (G′ x→y , G′ y→x )=E||x Yi -y i ||1+E||y Xi -x i ||1+E||x Yi -y′ i ||1+E||y Xi -x′ i ||1

[0032] In the formula, x Yi Represents sample image block y i The pseudo-homogeneous image patch is composed of sample image patch x i Input generator G′ x→y (x) is then obtained; y Xi It is a sample image block x i The pseudo-homogeneous images are derived from sample y. i Input generator G′ y→x (y) is then obtained; y′ i Indicates y Xi Input G′ x→y The generated image obtained after (x), x′ i It is x Yi Input G′ y→x The generated image obtained after (y); E represents the expectation in mathematics, and || ||1 is the L1 norm in mathematics.

[0033] The heterogeneous image domain migration network and change detection method incorporating domain knowledge, wherein the method for obtaining the change map from the pseudo-homogeneous image in the sixth step is any one of the traditional homogeneous image change detection methods, such as the difference method, ratio method, change vector analysis method, or spectral angle mapping method.

[0034] By employing the technical solution described above, the present invention has the following advantages:

[0035] This invention addresses the problem of detecting changes in land cover information from heterogeneous images. By integrating domain knowledge of change detection into a heterogeneous image domain transfer network (CK_CycleGAN), a domain transfer network suitable for heterogeneous image change detection is obtained. Using the CK_CycleGAN network of this invention, heterogeneous images acquired from different sensors with significant feature differences can be effectively converted into pseudo-homogeneous images where unchanged areas exhibit good consistency. This transforms the heterogeneous image change detection problem into a pseudo-homogeneous image change detection problem, allowing the use of homogeneous image change detection methods to obtain change maps. This improves the accuracy of detecting changes in land cover information from two heterogeneous images. Therefore, this invention provides a new approach for detecting changes in land cover information from heterogeneous images. Attached Figure Description

[0036] Figure 1 This is a flowchart of the present invention;

[0037] Figure 2 In this embodiment of the invention, the generator is a structural diagram;

[0038] Figure 3 Image X in this embodiment of the invention;

[0039] Figure 4 Image Y in this embodiment of the invention;

[0040] Figure 5 Image X in this embodiment of the invention Y ;

[0041] Figure 6 Image Y in the embodiments of the present invention X . Detailed Implementation

[0042] The present invention can be explained in more detail through the following embodiments, but the present invention is not limited to the following embodiments;

[0043] Combined with appendix Figures 1-6 The present invention discloses a heterogeneous image domain migration network and change detection method incorporating domain knowledge, the method specifically including the following steps:

[0044] Step 1: Data Collection

[0045] Two different satellite sensors were used to acquire two images of different time phases covering the same geographical area. Both images were registered and geometrically corrected, and were identified as image X at time T1 and image Y at time T2.

[0046] Step 2: Obtain a rough change graph:

[0047] A rough binary change map Cmap is obtained from images X and Y using traditional heterogeneous image change detection methods. In the binary change map Cmap, 0 indicates no change and 1 indicates a change.

[0048] Furthermore, in practice, when obtaining the coarse binary change map Cmap, a traditional change detection method that does not rely on sample data is used to obtain the coarse change map Cmap, specifically including the method based on iterative robust graph mapping and Markov co-occurrence matrix.

[0049] Step 3: Creating training samples:

[0050] Image X and image Y are cropped into an image block x of size N×N at position i using a sliding window of size N×N. i and image block y i Using a sliding window, the entire image X and image Y are cropped separately, and the cropped image patches are aggregated to obtain the sample dataset Sxy{x i y i Based on this, a subset of sample data Sun{x is created by combining the transformation graph Cmap. i y i}, paired sample data subset Sun{x i y i The sample position i in} satisfies the following conditions: the cell value at position i in Cmap is 0, and the number of cells with a value of 0 in a neighborhood window of size N×N centered at position i in Cmap is greater than th×N×N, where th represents the ratio of the number of cells with a value of 0 to the total number of cells in the neighborhood, and the value of th is between 0 and 1.

[0051] Step 4: Model Design

[0052] We designed and built a heterogeneous image domain transfer network CK_CycleGAN that incorporates domain knowledge. CK_CycleGAN includes a generator G′ y→x (y) and generator G′ x→y (x), and contains a discriminator D Y Discriminator D X Generator G′ y→x (y) and discriminator D X Connection, Generator G′ x→y (x) and discriminator D Y Connection; where discriminator D Y Discriminator D X Similar to the discriminator structure used in the classic CycleGAN network, the generator G′ y→x(y) Input image patch y obtained by cropping from image Y. i Generate an image patch y with the same feature space as image X. Xi Generator G′ x→y (x) Input image patch x obtained by cropping from image X i Generate an image patch x with the same feature space as image Y. Yi Discriminator D Y It is to determine the image block x Yi With y i Similarity, discriminator D X Determine y Xi With x i The similarity is used to calculate the loss function L for training the CK_CycleGAN network, which incorporates knowledge from the change detection domain. CK_CycleGAN ;

[0053] Furthermore, in specific implementation, the generator G′ in the design of the CK_CycleGAN network... y→x (y) and generator G′ x→y (x) incorporates domain knowledge that the same land feature can be described at different scales in remote sensing images. Based on the generator structure used by CycleGAN, a multi-scale convolution module and a residual module are added: a multi-scale convolution module is added after the convolutional layer of the network encoder in the two generators of CycleGAN to utilize the fused multi-scale features for land feature representation, and a 3-layer residual module is added at the end of the encoder, which makes the generator easier to optimize and avoids the problem of performance degradation caused by simply increasing the model depth;

[0054] Furthermore, the generator G′ of the CK_CycleGAN network y→x (y) and generator G′ x→y (x) has a U-Net structure, which includes an encoder and a decoder. The input is an image patch of size N×N×C, where N is the length and width, and C is the number of channels in image X or Y. The output of the model is a domain-transferred image patch of size N×N×C.

[0055] Furthermore, the loss function L CK_CycleGAN The calculation method is as follows:

[0056] L CK_CycleGAN (G′ x→y , G′ y→x D X D Y ) = L CycleGan (G′ x→y , G′ y→x D X D Y )+b LCD_Knowledge (G′ x→y , G′ y→x )

[0057] Where L CycleGAN The loss consists of cycle consistency loss and discriminator loss, based on the sample dataset Sxy{x i y i The loss calculation is performed using the classic CycleGAN function, with the knowledge weight b used to control the loss L during domain transfer. CD_Knowledge L's contribution to the total loss CD_Knowledge The primary metric is the quality of the generated images on unchanged paired data, based on the sample dataset Sun{x. i y i The samples in} are obtained using the following formula:

[0058] L CD_Knowledge (G′ x→y , G′ y→x )=E||x Yi -y i ||1+E||y Xi -x i ||1+E||x Yi -y′ i ||1+E||y Xi -x′ i ||1

[0059] In the formula, x Yi Represents sample image block y i The pseudo-homogeneous image patch is composed of sample image patch x i Input generator G′ x→y (x) is then obtained; y Xi It is a sample image block x i The pseudo-homogeneous images are derived from sample y. i Input generator G′ y→x (y) is then obtained; y′ i Indicates y Xi Input G′ x→y The generated image obtained after (x), x′ i It is x Yi Input G′ y→x The generated image obtained after (y); E represents the expectation in mathematics, and || ||1 is the L1 norm in mathematics;

[0060] Step 5: Model Training

[0061] Training the heterogeneous image domain transfer network CK_CycleGAN: using the created sample dataset Sxy{x i y i} and sample data subset Sun{x i y i The heterogeneous image domain transfer network CK_CycleGAN was trained, and the model loss during training was achieved using the function L. CK_CycleGAN calculate;

[0062] Step 6: Detection of changes in pseudo-homogeneous images:

[0063] The trained CK_CycleGAN domain transfer network is used to transform image X into image X with a similar feature space to image Y. Y Transform image Y into an image Y with a similar feature space to dataset X. X Image Y and image X Y It is a set of pseudo-homogeneous images, image X and image Y X This is another set of pseudo-homogeneous images; subsequently, a homogeneous image change detection method was used to distinguish between pseudo-homogeneous image Y and image X. Y Or pseudo-homogeneous images X and Y X Obtain the change graph from the middle;

[0064] In practice, the method for obtaining the change map from the pseudo-homogeneous image is any one of the traditional homogeneous image change detection methods, such as the difference method, ratio method, change vector analysis method, and spectral angle mapping method.

[0065] In a specific implementation of this invention, the multi-scale convolution module includes four sets of convolution modules, a normalization function, and a ReLU activation function, wherein the first set of modules includes a 1×1 convolution kernel; the second set of modules includes a cascaded 1×1 and 3×3 convolution kernels; the third set of convolution kernels includes a cascaded 1×1 and 5×5 convolution kernels; and the fourth set of convolution kernels includes a cascaded 3×3 max pooling kernel and a cascaded 1×1 convolution kernel. The outputs of the four sets of parallel modules are first spliced ​​and fused along the channel dimension, and the fused result is input into the normalization function and the ReLU activation function. The output of the ReLU activation function is used as the output of the multi-scale convolution module.

[0066] Furthermore, the residual module is composed of convolutional layers connected in series. After the feature map is input, it undergoes 3×3 edge expansion, then passes through two 3×3 convolutional kernels, is input to a normalization function, and is finally output after being processed by a ReLU activation function.

[0067] Furthermore, the encoder includes three convolutional layers of different depths and a multi-scale module. The first layer, Down_I, includes a convolutional layer and a multi-scale module. After receiving the input, it undergoes 3×3 edge expansion. After one convolution, the image block scale changes from N×N×C to N×N×64, and then after passing through the multi-scale convolution module, the scale changes to N×N×64 to N×N×256. The second layer, Down_II, includes a convolutional layer and a multi-scale module. After one convolution, the image block scale changes from N×N×256 to N / 2×N / 2×128, and then after passing through the multi-scale convolution module, the scale changes to N / 2×N / 2×128 to N / 2×N / 2×512. The third layer, Down_III, consists of a convolutional layer. After one convolution, the image block scale changes from N / 2×N / 2×512 to N / 4×N / 4×256. The encoder output then passes through three sets of residual modules with the same structure. The residual modules do not change the feature map scale.

[0068] Furthermore, the decoder includes three transposed convolutional layers of different depths and a multi-scale module. The first layer, Up_III, is a transposed convolutional layer that receives the output of the residual module. After one transposed convolution, the image block scale changes from N / 4×N / 4×256 to N / 2×N / 2×128. The second layer, Up_II, includes two convolutions and a transposed convolution. The input to the second layer, Up_II, is the output of the third layer, Up_III, and the second layer, Down_II. The output jump connection is constructed. After two convolutions, the image block scale changes to N / 2×N / 2×640→N / 2×N / 2×128→N / 2×N / 2×64. Then, after transposed convolution, the image block scale changes to N / 2×N / 2×64→N×N×64. The first layer Up_I is a convolutional layer. After 3×3 edge expansion, it undergoes one more convolution, and the image block scale changes to N×N×64→N×N×C, where N×N×C is the output scale of the generator.

[0069] In specific implementations of this invention, such as Figure 1 As shown, the present invention discloses a heterogeneous image domain migration network and change detection method incorporating domain knowledge. The method specifically includes the following steps:

[0070] Step 1: Data Acquisition

[0071] like Figure 3 and Figure 4 As shown, two images from different time periods and origins, which have been registered and geometrically corrected and cover the same geographic area, are obtained: image X at time T1 and image Y at time T2.

[0072] Step 2: Obtain a rough change graph:

[0073] A coarse binary change map Cmap is obtained from images X and Y using a traditional heterogeneous image change detection method, where 0 indicates no change and 1 indicates a change. The method for obtaining the coarse change map Cmap is a traditional change detection method that does not rely on sample data. There is no particular limitation on whether the method based on iterative robust graph mapping and Markov co-occurrence matrix or other methods is used. This embodiment uses the traditional heterogeneous image change detection method based on iterative robust graph mapping and Markov co-occurrence matrix.

[0074] Step 3: Create training samples:

[0075] Image X and image Y are cropped into an image block x of size N×N at position i using a sliding window of size N×N. i and image block y i Using a sliding window, the entire image X and image Y are cropped separately, and the cropped image patches are aggregated to obtain the sample dataset Sxy{x i y i Based on this, a subset of sample data Sun{x is created by combining the transformation graph Cmap. i y i}, paired sample data subset Sun{x i y i The sample position i in} satisfies the following conditions: the cell value at position i in Cmap is 0, and the number of cells with a value of 0 in a neighborhood window of size N×N centered at position i in Cmap is greater than th×N×N, where th represents the ratio of the number of cells with a value of 0 to the total number of cells in the neighborhood, and th takes a value between 0 and 1; where N = 32 and th = 0.7;

[0076] Step 4: Model Design

[0077] We designed and built a heterogeneous image domain transfer network, CK_CycleGAN, which incorporates domain knowledge. CK_CycleGAN contains two generators G′. y→x (y) and generator G′ x→y (x), and contains a discriminator D Y Discriminator D X Generator G′ y→x (y) and discriminator D X Connection, Generator G′ x→y (x) and discriminator D Y Connection; where discriminator D Y Discriminator D X Similar to the discriminator structure used in the classic CycleGAN network, the generator G′ y→x (y) Input image patch y obtained by cropping from image Y. iGenerate an image patch y with the same feature space as image X. Xi Generator G′ x→y (x) Input image patch x obtained by cropping from image X i Generate an image patch x with the same feature space as image Y. Yi Discriminator D Y It is to determine the image block x Yi With y i Similarity, discriminator D X Determine y Xi With x i The similarity is used to calculate the loss function L for training the CK_CycleGAN network, which incorporates knowledge from the change detection domain. CK_CycleGAN ;

[0078] Among them, such as Figure 2 As shown, in designing the generator G′ of the CK_CycleGAN network y→x (y) and generator G′ x→y (x) incorporates domain knowledge that the same land cover can be described at different scales in remote sensing imagery. Based on the generator structure used in CycleGAN, multi-scale convolutional modules and residual modules are added: a multi-scale convolutional module is added after the convolutional layer of the network encoder in both generators of CycleGAN to utilize fused multi-scale features for land cover representation. A three-layer residual module is added at the end of the encoder, making the generator easier to optimize and avoiding the performance degradation caused by simply increasing model depth. The specific structures of the multi-scale convolutional module and residual module are described below.

[0079] The multi-scale convolution module specifically includes four sets of convolutional modules set in parallel, a normalization function, and a ReLU activation function. The first set of modules includes a 1×1 convolutional kernel; the second set of modules includes a cascaded 1×1 and 3×3 convolutional kernels; the third set of convolutional kernels includes a cascaded 1×1 and 5×5 convolutional kernels; and the fourth set of convolutional kernels includes a cascaded 3×3 max pooling kernel and a cascaded 1×1 convolutional kernel. The outputs of the four sets of parallel modules are first concatenated and fused along the channel dimension. The fused result is then input into the normalization function and the ReLU activation function. The output of the ReLU activation function serves as the output of the multi-scale convolution module.

[0080] The residual module is composed of convolutional layers connected in series. After the feature map is input, it undergoes 3×3 edge expansion, then passes through two 3×3 convolutional kernels, is input to the normalization function, and finally outputs after being processed by the ReLU activation function.

[0081] The generator G′ of the CK_CycleGAN network y→x (y) and G′ x→y(x) has a U-Net-like structure, consisting of an encoder and a decoder. The input is an image patch of size N×N×C, where C is the number of channels in image X or Y. The output of the model is a domain-transferred image patch of size N×N×C; N=32. The specific details of the encoder and decoder structure are as follows:

[0082] The encoder consists of three convolutional layers of different depths and a multi-scale module, with the following structure: The Down_I layer consists of a convolutional layer and a multi-scale module. After receiving the input, it undergoes 3×3 edge expansion, followed by one convolution to change the image patch scale from 32×32×C to 32×32×64, and then through the multi-scale convolution module to change the scale to 32×32×64 to 32×32×256. The Down_II layer consists of a convolutional layer and a multi-scale module. After one convolution, the image patch scale changes from 32×32×256 to 16×16×128, and then through the multi-scale convolution module to change the scale to 16×16×128 to 16×16×512. The Down_III layer consists of a convolutional layer, which changes the image patch scale from 16×16×512 to 8×8×256 after one convolution. The encoder output then passes through three sets of residual modules with the same structure; these residual modules do not change the feature map scale.

[0083] The decoder consists of three transposed convolutional layers of different depths and a multi-scale module, with the following structure: The Up_III layer of the decoder is composed of a transposed convolutional layer, receiving the output of the residual module. After one transposed convolution, the image block scale changes from 8×8×256 to 16×16×128. The input of the Up_II layer is composed of the outputs of the Up_III layer and the Down_II layer, which are connected in a skip connection. After two convolutions, the image block scale changes from 16×16×640 to 16×16×128 to 16×16×64. Then, after transposed convolution, the image block scale changes from 16×16×64 to 32×32×64. The Up_I layer is composed of a convolutional layer. After 3×3 edge expansion, it undergoes one convolution, and the image block scale changes from 32×32×64 to 32×32×C, where 32×32×C is the output scale of the generator.

[0084] Where the loss function L CK_CycleGAN The calculation method is as follows:

[0085] L CK_CycleGAN (G′ x→y , G′ y→x D X D Y ) = L CycleGan (G′ x→y , G′ y→x D X D Y )+b LCD_Knowledge (G′ x→y , G′ y→x )

[0086] Where L CycleGAN The loss consists of cycle consistency loss and discriminator loss, and can be based on the sample dataset Sxy{x i y i The loss calculation is performed using the classic CycleGAN function, with the knowledge weight b used to control the loss L during domain transfer. CD_Knowledge L's contribution to the total loss CD_Knowledge The primary metric is the quality of the generated images on unchanged paired data, based on the sample dataset Sun{x. i y i The samples in} are calculated using the following formula:

[0087] L CD_Knowledge (G′ x→y , G′ y→x )=E||x Yi -y i ||1+E||y Xi -x i ||1+E||x Yi -y′ i ||1+E||y Xi -x′ i ||1

[0088] In the formula, x Yi Represents sample image block y i The pseudo-homogeneous image patch is composed of sample image patch x i Input generator G′ x→y (x) is then obtained; y Xi It is a sample image block x i The pseudo-homogeneous images are derived from sample y. i Input generator G′ y→x (y) is then obtained; y′ i Indicates y Xi Input G′ x→y The generated image obtained after (x), x′ i It is x Yi Input G′ y→x The generated image obtained after (y); E represents the expectation in mathematics, and || ||1 is the L1 norm in mathematics.

[0089] Step 5: Model Training

[0090] The CK_CycleGAN domain transfer model was trained using the created sample dataset Sxy and the sample data subset Sun. The model loss during training was calculated using the function L.CK_CycleGAN calculate;

[0091] Step 6: Detection of changes in pseudo-homogeneous images:

[0092] The trained CK_CycleGAN domain transfer network is used to transform image X into... Figure 5 Image X, which has a similar feature space to image Y, is shown. Y Convert the image Y into... Figure 6 The image Y shown has a similar feature space to dataset X. X Image Y and image X Y It is a set of pseudo-homogeneous images, image X and image Y X This is another set of pseudo-homogeneous images; based on this, the traditional homogeneous image change detection method is used to distinguish between pseudo-homogeneous image Y and image X. Y Or pseudo-homogeneous images X and Y X The method for obtaining the change map from the pseudo-homogeneous image is one of the traditional homogeneous image change detection methods, such as the difference method, ratio method, change vector analysis method, and spectral angle mapping method. There is no particular limitation on which method is used. In this embodiment, the change vector analysis method is used.

[0093] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0094] It should be understood that the above description of the implementation examples is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A heterogeneous image domain migration network and change detection method incorporating domain knowledge, characterized by: The method specifically includes the following steps: Step 1: Data Collection Two different satellite sensors were used to acquire two images of different time phases covering the same geographical area. Both images were registered and geometrically corrected, and were identified as image X at time T1 and image Y at time T2. Step 2: Obtain a rough change graph: A rough binary change map Cmap is obtained from images X and Y using traditional heterogeneous image change detection methods. In the binary change map Cmap, 0 indicates no change and 1 indicates a change. Step 3: Creating training samples: Image X and image Y are cropped into an image block of size N×N at position i using a sliding window of size N×N. and image blocks Using a sliding window, the entire image X and image Y are cropped separately, and the cropped image patches are aggregated to obtain the sample dataset Sxy{ , Based on this, and combined with the transformation graph Cmap, a subset of sample data Sun{ is created. , }, paired sample data subset Sun{ , The sample position i in} satisfies the following conditions: the cell value at position i in Cmap is 0, and the number of cells with a value of 0 in a neighborhood window of size N×N centered at position i in Cmap is greater than th×N×N, where th represents the ratio of the number of cells with a value of 0 to the total number of cells in the neighborhood, and the value of th is between 0 and 1. Step 4: Model Design Design and build a heterogeneous image domain transfer network CK_CycleGAN that incorporates domain knowledge. CK_CycleGAN includes a generator. and generator And contains a discriminator and discriminator generator With discriminator Connect, Generator With discriminator connect; Among them, the discriminator and discriminator Similar to the discriminator structure used in the classic CycleGAN network, the generator... Input an image patch cropped from image Y Generate image patches with the same feature space as image X. generator Input an image patch cropped from image X. Generate image patches with the same feature space as image Y. Discriminator It is to determine image blocks and Similarity, discriminator judge and The similarity is used to calculate the loss function for training the CK_CycleGAN network, which incorporates knowledge from the change detection domain. ; The loss function The calculation method is as follows: ； in The loss consists of cycle consistency loss and discriminator loss, based on the sample dataset Sxy{ , The loss function from the classic CycleGAN is used for calculation, and the knowledge weights are calculated. Loss during control domain migration Contribution to the total loss, The primary metric is the quality of the generated images on unchanged paired data, based on the sample dataset Sun{ , The samples in} are obtained using the following formula: ； In the formula, Represents sample image blocks Pseudo-homogeneous image blocks, composed of sample image blocks Input generator Later obtained; It is a sample image block Pseudo-homogeneous images, from samples Input generator Later obtained; express enter The generated image obtained later, yes enter The resulting image; In mathematics, |||1 represents the expectation, and it is the L1 norm. Step 5: Model Training Training the heterogeneous image domain transfer network CK_CycleGAN: using the created sample dataset Sxy{ , } and sample data subset Sun{ , The heterogeneous image domain transfer network CK_CycleGAN was trained, and the model loss during training was achieved using a function. calculate; Step 6: Detection of changes in pseudo-homogeneous images: The trained CK_CycleGAN domain transfer network is used to transform image X into an image with a similar feature space to image Y. Transform image Y into an image with a similar feature space to dataset X. Image Y and image It is a set of pseudo-homogeneous images, image X and image... This is another set of pseudo-homogeneous images; subsequently, a homogeneous image change detection method was used to analyze the pseudo-homogeneous image Y and the image. Or pseudo-homogeneous image X and image Obtain the change graph from the middle.

2. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 1, characterized in that: In the second step, when obtaining the coarse binary change map Cmap, traditional change detection methods that do not rely on sample data are used to obtain the coarse change map Cmap, specifically including the method based on iterative robust graph mapping and Markov co-occurrence matrix.

3. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 1, characterized in that: The fourth step involves designing the generator for the CK_CycleGAN network. and generator It incorporates domain knowledge that the same ground feature can be described at different scales in remote sensing images. Based on the generator structure used by CycleGAN, it adds multi-scale convolutional modules and residual modules: multi-scale convolutional modules are added after the convolutional layers of the network encoder in the two generators of CycleGAN to utilize the fused multi-scale features for ground feature representation, and three residual modules are added at the end of the encoder, which makes the generator easier to optimize and avoids the performance degradation caused by simply increasing the model depth.

4. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 3, characterized in that: The multi-scale convolution module includes four sets of convolution modules, a normalization function, and a ReLU activation function, wherein the first set of modules includes a 1×1 convolution kernel; the second set of modules includes a cascaded 1×1 and 3×3 convolution kernels; the third set of convolution kernels includes a cascaded 1×1 and 5×5 convolution kernels; and the fourth set of convolution kernels includes a cascaded 3×3 max pooling kernel and a cascaded 1×1 convolution kernel. The outputs of the four sets of parallel modules are first spliced ​​and fused along the channel dimension. The fused result is input into the normalization function and the ReLU activation function, and the output of the ReLU activation function is used as the output of the multi-scale convolution module.

5. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 3, characterized in that: The residual module is composed of convolutional layers connected in series. After the feature map is input, it undergoes 3×3 edge expansion, then passes through two 3×3 convolutional kernels, is input to a normalization function, and is finally output after being processed by the ReLU activation function.

6. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 1, characterized in that: The generator of the CK_CycleGAN network in the fourth step and generator The structure is U-Net, which includes an encoder and a decoder. The input is an image patch of size N×N×C, where N is the length and width, and C is the number of channels in the image X or Y. The output of the model is a domain-transferred image patch of size N×N×C.

7. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 6, characterized in that: The encoder comprises three convolutional layers of different depths and a multi-scale module. The first layer, Down_Ⅰ, includes a convolutional layer and a multi-scale module. After receiving the input, it undergoes 3×3 edge expansion, and after one convolution, the image block scale changes from N×N×C to N×N×64. Then, after passing through the multi-scale convolution module, the scale changes to N×N×64 to N×N×256. The second layer, Down_Ⅱ, also includes a convolutional layer and a multi-scale module. After one convolution, the image block scale changes from N×N×256 to N / 2×N / 2×128. Then, after passing through the multi-scale convolution module, the scale changes to N / 2×N / 2×128 to N / 2×N / 2×512. The third layer, Down_Ⅲ, consists of a convolutional layer. After one convolution, the image block scale changes from N / 2×N / 2×512 to N / 4×N / 4×256. The encoder output then passes through three sets of residual modules with the same structure. The residual modules do not change the feature map scale.

8. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 6, characterized in that: The decoder includes three transposed convolutional layers of different depths and a multi-scale module. The first layer, Up_Ⅲ, is a transposed convolutional layer used to receive the output of the residual module. After one transposed convolution, the image block scale changes from N / 4×N / 4×256 to N / 2×N / 2×128. The second layer, Up_Ⅱ, includes two convolutions and a transposed convolution. The input of the second layer, Up_Ⅱ, is formed by a jump connection between the output of the third layer, Up_Ⅲ, and the output of the second layer, Down_Ⅱ. After two convolutions, the image block scale changes from N / 2×N / 2×640 to N / 2×N / 2×128 to N / 2×N / 2×64. Then, after transposed convolution, the image block scale changes from N / 2×N / 2×64 to N×N×64. The first layer, Up_Ⅰ, is a convolutional layer. After being expanded by 3×3 edges, it undergoes one more convolution to change the image block scale to N×N×64→N×N×C, where N×N×C is the output scale of the generator.

9. The heterogeneous image domain migration network and change detection method incorporating domain knowledge as described in claim 1, characterized in that: The method for obtaining the change map from the pseudo-homogeneous image in the sixth step is any one of the traditional homogeneous image change detection methods, such as the difference method, ratio method, change vector analysis method, and spectral angle mapping method.

Images