Multi-source high-resolution remote sensing image change detection method and system

Abstract

The invention belongs to the technical field of image data processing, and discloses a multi-source high-resolution remote sensing image change detection method and system, and the method comprises the steps: obtaining a remote sensing image; converting the remote sensing image into a composite image; extracting global features of the first image, the first composite image, the second image and the second composite image; performing weighted fusion on the extracted global features to form a fusion feature at a first moment and a fusion feature at a second moment; carrying out correlation modeling on the fusion feature at the first moment and the fusion feature at the second moment, extracting a space-time relationship between images of the ground features in different time phases, and generating a final fusion feature at the first moment and a final fusion feature at the second moment; and detecting image changes at the first moment and the second moment according to the final fusion feature at the first moment and the final fusion feature at the second moment. The problems of low efficiency and low accuracy in existing remote sensing image change detection are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a multi-source high-resolution remote sensing image change detection method and system. Background Art

[0002] Remote sensing image change detection is an important task in remote sensing image processing research. It aims to identify and locate changes in ground objects by comparing multi-temporal images. Currently, the sources of remote sensing images provided on the Internet are very wide, covering a variety of remote sensing platforms, such as optical satellites, synthetic aperture radar satellites, drones, Internet map platforms Google Earth and OpenStreetMap. The data information types of remote sensing images obtained by these platforms vary, including optical images, radar images, thermal infrared images and other modalities. The diversity of remote sensing image data information provides a richer and more comprehensive data source for change detection, making it possible to observe the target area from different perspectives, thereby reflecting the changes in ground objects more comprehensively and accurately.

[0003] The existing change detection technology for multi-source high-resolution remote sensing images is mainly based on the recurrent generative adversarial network model to perform modal conversion on data information of different modalities, convert different modalities into the same modality, and perform feature extraction and change detection under the same modality. However, the existing detection methods have problems such as information loss in modal conversion, insufficient feature fusion, and high consumption of computing resources, resulting in low detection efficiency and accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-source high-resolution remote sensing image change detection method and system to solve the problems of low efficiency and low accuracy in existing remote sensing image change detection.

[0005] The present invention is achieved through the following technical solutions: A multi-source high-resolution remote sensing image change detection method comprises the following steps: S1. Acquire a remote sensing image, where the remote sensing image includes a first image at a first moment and a second image at a second moment, where the first image is an image of a first modality, and the second image is an image of a second modality, and the first modality and the second modality are different modalities; S2, converting the remote sensing image into a synthetic image, including: converting the first image into a first synthetic image having a second modality style, and converting the second image into a second synthetic image having the first modality style; S3, extracting global features of the first image, the first synthesized image, the second image, and the second synthesized image; S4, weighted fusion is performed on the extracted global features to form fusion features at the first moment and fusion features at the second moment; and the fusion features at the first moment and the fusion features at the second moment are associated with each other for modeling, the spatiotemporal relationship between the images of the ground objects at different time phases is extracted, and the final fusion features at the first moment and the final fusion features at the second moment are generated; S5. Detect image changes between the first moment and the second moment according to the final fusion features at the first moment and the final fusion features at the second moment.

[0006] In some embodiments, the step of generating the final fusion feature at the first moment in step S4 includes: S401, subjecting the fused features at the first moment and the fused features at the second moment to different linear transformations, respectively, to obtain Q1 corresponding to the fused features at the first moment, and K1 and V1 corresponding to the fused features at the second moment. The linear transformation formula is: Q1=W Q F combined,t1 , K1=W K F combined,t2 , V1=W V F combined,t2 , Among them, W Q , W K , W V Both represent learnable linear transformation matrices, which are used to adjust the dimension of features; F combined,t1 represents the fusion feature at the first moment, F combined,t2 represents the fused features at the second moment; Q1 represents the query vector of each feature position at the first moment; K1 represents the key vector of each feature position at the second moment; V1 represents the weight value of each feature position at the second moment; S402, calculating the similarity of each feature position according to the query vector Q1 of each feature position at the first moment and the key vector K1 of each feature position at the second moment to obtain a first attention weight C1, and the calculation formula of the first attention weight C1 is: , in, d k Represents the dimension of the feature, used to scale the similarity score; K1 T represents the transpose of K1, which represents the transposed form of the key vector of each feature position at the second moment, and is used to calculate the similarity between Q1 and K1; C1 represents the correlation between each feature position at the first moment and the second moment; S403: Perform weighted summation on the weight value V1 of each feature position at the second moment according to the first attention weight C1 to generate a weighted feature representation F1 at the first moment. attended ; S404, the fusion feature F at the first moment combined,t1 And the weighted feature representation F1 at the first moment attended Perform fusion processing to generate the final fusion feature F1 at the first moment final ; The fusion formula is: F1 final =F combined,t1 +F1 attended .

[0007] In some embodiments, the step of generating the final fusion feature at the second moment in step S4 includes: S411, subjecting the fused features at the first moment and the fused features at the second moment to different linear transformations, respectively, to obtain Q2 corresponding to the fused features at the second moment, and K2 and V2 corresponding to the fused features at the first moment. The linear transformation formula is: Q2=W Q F combined,t2 , K2=W K F combined,t1 , V2=W V F combined,t1 , Wherein, Q2 represents the query vector of each feature position at the second moment; K2 represents the key vector of each feature position at the first moment; V2 represents the weight value of each feature position at the first moment; S412: Calculate the similarity of each feature position according to the query vector Q2 of each feature position at the second moment and the key vector K2 of each feature position at the first moment to obtain a second attention weight C2. The calculation formula of the second attention weight C2 is: , Among them, K2 T represents the transpose of K2, which represents the transposed form of the key vector of each feature position at the first moment, and is used to calculate the similarity between Q2 and K2; C2 represents the correlation between each feature position at the second moment and the first moment; S413: Perform weighted summation on the weight value V2 of each feature position at the first moment according to the second attention weight C2 to generate a weighted feature representation F2 at the second moment. attended ; S414, the fusion feature F at the second moment combined,t2 And the weighted feature representation F2 at the second moment attended Perform fusion processing to generate the final fusion feature F2 at the second moment final ; The fusion formula is: F2 final =F combined,t2 +F2 attended .

[0008] In some embodiments, step S2 includes: Inputting the first image and the second image into the CycleGAN model, converting the first image into a first synthetic image having the style of the second modality through the generator G1 of the CycleGAN model, and converting the second image into a second synthetic image having the style of the first modality through the generator G2 of the CycleGAN model; The discriminator D1 of the CycleGAN model determines the difference in modal features between the first synthetic image and the second image, and feeds back to the generator G1. The generator G1 is improved according to the difference information fed back by the discriminator D1, so that the generator G1 can output the first synthetic image that is closer to the second modal style. At the same time, the discriminator D2 of the CycleGAN model determines the difference in modal features between the second synthetic image and the first image, and feeds back to the generator G2. The generator G2 is improved according to the difference information fed back by the discriminator D2, so that the generator G2 can output the second synthetic image that is closer to the first modal style.

[0009] In some embodiments, step S2 further includes the step of performing edge preservation loss processing on the first synthetic image and the second synthetic image, including: The edge differences between the first image, the second image and the first synthetic image and the second synthetic image are calculated by the Sobel operator, and fed back to the generator G1 and the generator G2 of the CycleGAN model. The generator G1 and the generator G2 are improved according to the feedback edge difference information, so that the generator G1 can output the first synthetic image retaining the edge features of the first image, and the generator G2 can output the second synthetic image retaining the edge features of the second image. The improved generator G1 outputs the first synthetic image retaining the edge features of the first image according to the first image, and the improved generator G2 outputs the second synthetic image retaining the edge features of the second image according to the second image. The Sobel operator is: L edge =E A [||Edge(A)-Edge(G1(A))||1]+E B [||Edge(B)-Edge(G2(B))||1], Among them, L edge represents the edge difference information between the first image and the second image and the first synthesized image and the second synthesized image, E A represents the edge loss value of the first image, E B represents the edge loss value of the second image, A represents the first image, B represents the second image, G1(A) represents the first synthesized image, and G2(B) represents the second synthesized image.

[0010] In some embodiments, step S2 further includes the step of performing spatial consistency loss processing on the remote sensing image and the corresponding synthetic image, including: The spatial difference between the remote sensing image and the corresponding synthetic image is calculated through the spatial consistency loss formula and fed back to the corresponding generator in the CycleGAN model. The generator is improved according to the feedback spatial difference information so that the generator can output a synthetic image that retains the spatial structure of the remote sensing image. The improved generator is used to output a synthetic image that retains the spatial structure of the remote sensing image based on the remote sensing image; the spatial consistency loss formula is: L spatial =E x [||A spatial -Gi(X) spatial ||2], Among them, L spatial Represents the spatial difference information between the remote sensing image and the corresponding synthetic image, E x Represents the edge loss value of the remote sensing image, A spatial Represents the spatial feature representation of remote sensing images, Gi(X) spatial Representation of spatial features of synthetic images corresponding to remote sensing images.

[0011] In some embodiments, step S3 includes: Convolution kernels of different sizes of a multi-scale convolutional neural network are used to extract global features of the first image, the first synthesized image, the second image, and the second synthesized image.

[0012] In some embodiments, step S3 further includes performing a channel attention mechanism on the global feature map generated when extracting the global feature, and using a multi-scale convolutional neural network to perform feature extraction on the processed global feature map to generate a global feature; The steps of processing the channel attention mechanism on the global feature map include: When extracting the global features of the first image, the first synthetic image, the second image, and the second synthetic image, global feature maps of different sizes are generated, and global average pooling is performed on the global feature maps to obtain the global statistical information of each channel in the global feature maps. A fully connected network is used to calculate the weight of the importance of each channel according to the global statistical information of each channel, and the weight is applied to each channel of the global feature map.

[0013] In some embodiments, step S5 includes: According to the final fusion feature F1 at the first moment final And the final fusion feature F2 at the second moment final After performing element-by-element subtraction, the absolute value is taken to obtain the differential feature representation D. The calculation formula of the differential feature representation D is: D=|F1final - F2 final |; The differential feature representation D is input into the classifier to detect the image changes between the first moment and the second moment and generate a change map.

[0014] The present invention also relates to a multi-source high-resolution remote sensing image change detection system based on the multi-source high-resolution remote sensing image change detection method, comprising: a modality conversion module, configured to convert a first image into a first synthetic image having a second modality style, and convert a second image into a second synthetic image having a first modality style, wherein the first image is a first modality image corresponding to a first moment, and the second image is a second modality image corresponding to a second moment, and the first modality and the second modality are different modalities; A feature extraction module, used for extracting global features of the first image, the first synthetic image, the second image, and the second synthetic image; The feature fusion module is used to perform weighted fusion on the extracted global features to form fusion features at the first moment and fusion features at the second moment; and to perform correlation modeling on the fusion features at the first moment and the fusion features at the second moment, extract the spatiotemporal relationship between the images of the ground objects at different time phases, and generate the final fusion features at the first moment and the final fusion features at the second moment; The change detection module is used to detect the image change between the first moment and the second moment based on the final fusion feature of the first moment and the final fusion feature of the second moment.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The existing fusion method cannot fully consider the information difference between different modalities, ignores the importance of weights between different modal features, and cannot make full use of complementary information from different data sources, resulting in the lack of sufficient discrimination of the fused features in change detection, making some modalities have too much influence on the final decision, thereby reducing the accuracy of change detection. The present invention adopts weighted fusion, which can dynamically adjust the fusion weight according to the feature importance of each modality, thereby improving the effectiveness and accuracy of multimodal feature fusion; for remote sensing images of different phases and different modalities, the existing fusion method fails to fully consider the spatiotemporal correlation between different phases, ignores the time dimension, and cannot fully consider the spatial and temporal correlation between different phases. The invention takes the remote sensing images at different time points as independent inputs, thereby reducing the accuracy of change detection. The present invention associates the fusion features of the first moment with the fusion features of the second moment for modeling, extracts the spatiotemporal relationship between the images of the ground objects at different time phases, comprehensively considers the change information in the time dimension, enhances the spatiotemporal modeling capability of change detection, better understands the relationship between the first moment and the second moment in change detection, effectively captures the spatiotemporal relationship between different time phases, more comprehensively captures the change information, and enables the multimodal features to be fully integrated, thereby improving the efficiency and accuracy of change detection and reducing the cost of computing resources.

[0016] 2. The edge preservation loss method of the CycleGAN model is used to improve the generator through the Sobel operator, which can effectively retain the edge information and detail information of the image during the modal conversion process, so that the output synthetic image retains the edge features of the remote sensing image, ensuring that the edge features of the remote sensing image and the synthetic image are as consistent as possible, preventing the details from being blurred after the high-resolution image is converted, thereby improving the accuracy of change detection.

[0017] 3. The spatial consistency loss method of the CycleGAN model is used to improve the generator, so that the output synthetic image retains the spatial structure of the remote sensing image corresponding to the synthetic image, which solves the problem of geometric distortion of remote sensing images during the modal conversion of high-resolution remote sensing images, ensures that the position and geometric shape of objects in the remote sensing image remain unchanged before and after the conversion, ensures the consistency of the spatial structure of the converted synthetic image and the remote sensing image, reduces the distortion of the geometric shape, ensures that the position and structure of objects in change detection remain accurate, and thus improves the accuracy of change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 The figure is a flow chart of the existing remote sensing image change detection method in the implementation of the present invention.

[0020] Figure 2 The present invention is a flowchart of a remote sensing image change detection method in the implementation of the present invention.

[0021] Figure 3 It is a block diagram of the remote sensing image change detection system in the implementation of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0023] In the current change detection task for multi-source high-resolution remote sensing images, the change detection method is based on the cyclic generative adversarial network model CycleGAN. CycleGAN can perform modal conversion on data information of different modalities so that they are converted into the same modality. For example, if change detection is required between optical images obtained from a multi-source Internet platform and SAR images from Synthetic Aperture Radar, it is very difficult to directly perform change detection on optical images and SAR images from different remote sensing platforms because the modal characteristics of these images are quite different. However, CycleGAN can convert SAR images into optical-style images, or convert optical images into SAR-style images, so that feature extraction and change detection can be performed in the same modality. This modal conversion greatly reduces the impact of modal differences on the change detection task, making feature extraction and detection results more consistent and reliable.

[0024] Taking the optical image and SAR image in the high-resolution image data information of the multi-source Internet platform as an example, refer to Figure 1 ,The process of existing change detection methods includes: 1) Multimodal image preprocessing: First, the optical image and SAR image are geo-referenced and geometrically corrected, and then the optical image and SAR image are cropped into a preset size, such as 256×256 image blocks.

[0025] 2) Modal conversion: The preprocessed optical images and SAR images are input into the modal conversion module CycleGAN, which uses CycleGAN to convert the data information of different modalities to convert the optical images and SAR images into the same modality. Specifically, CycleGAN contains two generators and two discriminators. The generator is responsible for converting images of one modality into images of another modality, that is, converting optical images into images in the style of SAR images, or converting SAR images into images in the style of optical images; the discriminator is used to determine whether the converted image is consistent with the target modality, thereby continuously improving the conversion effect of the generator to reduce the difference between the modalities.

[0026] 3) Feature extraction: The modality-converted image is input into the convolutional neural network to extract representative spatial features. The convolutional neural network can capture features such as edges, textures, and local structures in the extracted image through multiple layers of convolution, providing a basis for subsequent change detection.

[0027] 4) Feature fusion: The features converted from different modalities are fused by simple concatenation or superposition to form a joint feature representation. This fusion method aims to combine information from different modalities so that subsequent change detection can utilize these features.

[0028] 5) Change detection: Use feature difference method to generate change map, and identify the changed area of ​​the object by comparing the image features at different times.

[0029] 6) Change detection map generation: Calculate the difference between the joint feature representations of two time phases, such as the feature difference between time points t1 and t2. Usually, the differential features are obtained by element-by-element subtraction and then taking the absolute value. Then, based on the feature difference results, a threshold method or classifier is used to determine which areas have changed significantly, and finally a change map is generated.

[0030] Existing technical solutions use the CycleGAN model to perform modal conversion to eliminate the differences between different modalities, such as the difference between optical images and SAR images. However, since the conversion process of CycleGAN relies on adversarial generation, the converted images often have information loss problems, especially in high-resolution images, where the detailed features after conversion cannot be fully preserved. This information loss will directly affect the accuracy of subsequent change detection and reduce the accuracy of distinguishing changed areas from unchanged areas.

[0031] In the prior art, the fusion of multimodal features is mainly achieved through simple splicing or superposition. This simple fusion method ignores the importance of weights between different modal features and fails to fully utilize complementary information from different data sources, resulting in the lack of sufficient discriminative power of the fused features in change detection tasks. This simple fusion method also fails to fully consider the spatiotemporal correlation between different phases, limiting the performance of multimodal features in change detection.

[0032] The above method also has the problem of high consumption of computing resources.

[0033] Example 1 A multi-source high-resolution remote sensing image change detection method, referring to Figure 2 , including the following steps: S1. Acquire a remote sensing image, where the remote sensing image includes a first image at a first moment and a second image at a second moment, where the first image is an image of a first modality, and the second image is an image of a second modality, and the first modality and the second modality are different modalities; High-resolution remote sensing images are obtained from a multi-source Internet platform, and the obtained remote sensing images are preprocessed, including: georeferencing and geometric correction of the first image and the second image, and then the first image and the second image are cropped into images of 256×256 size.

[0034] S2, converting the preprocessed remote sensing image into a synthetic image, including: converting the first image into a first synthetic image having a second modality style, and converting the second image into a second synthetic image having the first modality style; S21. Input the first image and the second image into the CycleGAN model, convert the first image into a first synthetic image having a second modal style through the generator G1 of the CycleGAN model, and convert the second image into a second synthetic image having the first modal style through the generator G2 of the CycleGAN model.

[0035] S22. The discriminator D1 of the CycleGAN model determines the difference in modal features between the first synthetic image and the second image, and feeds back the difference to the generator G1. The generator G1 is improved according to the difference information fed back by the discriminator D1, so that the generator G1 can output the first synthetic image that is closer to the second modal style. At the same time, the discriminator D2 of the CycleGAN model determines the difference in modal features between the second synthetic image and the first image, and feeds back the difference to the generator G2. The generator G2 is improved according to the difference information fed back by the discriminator D2, so that the generator G2 can output the second synthetic image that is closer to the first modal style.

[0036] S23, performing edge preservation loss processing on the first synthetic image and the second synthetic image.

[0037] The method comprises: calculating the edge differences between the first image, the second image and the first synthetic image and the second synthetic image respectively by using the Sobel operator, and feeding back the differences to the generator G1 and the generator G2 of the CycleGAN model; improving the generator G1 and the generator G2 according to the feedback edge difference information, so that the generator G1 can output the first synthetic image retaining the edge features of the first image, and the generator G2 can output the second synthetic image retaining the edge features of the second image; using the improved generator G1 to output the first synthetic image retaining the edge features of the first image according to the first image, and using the improved generator G2 to output the second synthetic image retaining the edge features of the second image according to the second image; the Sobel operator is: L edge =E A [||Edge(A)-Edge(G1(A))||1]+E B [||Edge(B)-Edge(G2(B))||1], Among them, L edge represents the edge difference information between the first image and the second image and the first synthesized image and the second synthesized image, E A represents the edge loss value of the first image, E B represents the edge loss value of the second image, A represents the first image, B represents the second image, G1(A) represents the first synthesized image, and G2(B) represents the second synthesized image.

[0038] The edge preservation loss method of the CycleGAN model is used to improve the generator through the Sobel operator, which can effectively retain the edge information and detail information of the image during the modal conversion process, so that the output synthetic image retains the edge features of the remote sensing image, ensuring that the edge features of the remote sensing image and the synthetic image are as consistent as possible, preventing the details from being blurred after the high-resolution image is converted, thereby improving the accuracy of change detection.

[0039] S24. Perform spatial consistency loss processing on the remote sensing image and the corresponding synthetic image.

[0040] The method includes: calculating the spatial difference between the remote sensing image and the corresponding synthetic image through the spatial consistency loss formula, and feeding it back to the corresponding generator in the CycleGAN model; the generator is improved according to the fed-back spatial difference information so that the generator can output a synthetic image that retains the spatial structure of the remote sensing image; the improved generator is used to output a synthetic image that retains the spatial structure of the remote sensing image according to the remote sensing image; the spatial consistency loss formula is: L spatial =E x [||A spatial -Gi(X) spatial ||2], Among them, L spatial Represents the spatial difference information between the remote sensing image and the corresponding synthetic image, E x Represents the edge loss value of the remote sensing image, A spatial Represents the spatial feature representation of remote sensing images, Gi(X) spatial Representation of spatial features of synthetic images corresponding to remote sensing images.

[0041] The spatial consistency loss method of the CycleGAN model is used to improve the generator, so that the output synthetic image retains the spatial structure of the remote sensing image corresponding to the synthetic image, which solves the problem of geometric distortion of remote sensing images during the modal conversion of high-resolution remote sensing images, ensures that the position and geometric shape of objects in the remote sensing image remain unchanged before and after the conversion, ensures the consistency of the converted synthetic image with the remote sensing image in spatial structure, reduces the distortion of geometric shapes, ensures that the position and structure of objects in change detection remain accurate, and thus improves the accuracy of change detection.

[0042] S3, extracting global features of the first image, the first synthesized image, the second image, and the second synthesized image; S31. Use convolution kernels of different sizes of a multi-scale convolutional neural network to extract global features of the first image, the first synthesized image, the second image, and the second synthesized image.

[0043] Convolution kernels include 3x3, 5x5, 7x7, etc.

[0044] S32, performing a channel attention mechanism process on the global feature map generated when extracting the global feature, and using a multi-scale convolutional neural network to perform feature extraction on the processed global feature map to generate a global feature; The steps of processing the channel attention mechanism on the global feature map include: When extracting the global features of the first image, the first synthetic image, the second image, and the second synthetic image, convolution processing is performed using convolution kernels of different sizes of a multi-scale convolutional neural network to capture information of different scales in the image, generate global feature maps of different sizes, perform global average pooling processing on the global feature map to obtain the global statistical information of each channel in the global feature map, use a fully connected network to calculate the weight of the importance of each channel according to the global statistical information of each channel, and apply the weights to each channel of the global feature map, so that important features in the global feature map can be amplified and unimportant features in the global feature map can be suppressed.

[0045] Through the channel attention mechanism, the multi-scale convolutional neural network can learn that certain global features are most important for change detection, thereby enhancing these global features, improving the expressiveness of global features and the distinguishability of change areas.

[0046] A multi-scale convolutional neural network is used with convolution kernels of different sizes to extract global features of different scales, and a channel attention mechanism is introduced to improve the expressiveness of global features and the distinguishability of change areas, so that the multi-scale convolutional neural network improves the accuracy and robustness of change detection when facing multi-source images.

[0047] S4, weighted fusion is performed on the extracted global features to form fusion features at the first moment and fusion features at the second moment; and the fusion features at the first moment and the fusion features at the second moment are associated with each other for modeling, the spatiotemporal relationship between the images of the ground objects at different time phases is extracted, and the final fusion features at the first moment and the final fusion features at the second moment are generated; S41. Assign weights according to the global features of the first image, the first synthetic image, the second image, and the second synthetic image, and perform weighted summation according to the assigned weights to generate fusion features at the first moment and fusion features at the second moment, thereby realizing multimodal feature fusion.

[0048] In view of the problem that the existing fusion methods cannot fully consider the information differences between different modalities, ignore the importance of weights between different modal features, and cannot make full use of complementary information from different data sources, resulting in the fused features lacking sufficient discriminative power in change detection, making certain modalities have too much influence on the final decision, thereby reducing the accuracy of change detection, the present invention adopts weighted fusion, which can dynamically adjust the fusion weights according to the feature importance of each modality, thereby improving the effectiveness and accuracy of multimodal feature fusion.

[0049] S42. Use the spatiotemporal attention mechanism to correlate the fusion features of the first moment and the fusion features of the second moment to model, extract the spatiotemporal relationship between the images of the objects in different time phases, and generate the final fusion features of the first moment and the final fusion features of the second moment.

[0050] The step of generating the final fusion feature at the first moment includes: S401, subjecting the fused features at the first moment and the fused features at the second moment to different linear transformations, respectively, to obtain Q1 corresponding to the fused features at the first moment, and K1 and V1 corresponding to the fused features at the second moment. The linear transformation formula is: Q1=W Q F combined,t1 , K1=W K F combined,t2 , V1=W V F combined,t2 , Among them, W Q , W K , W VBoth represent learnable linear transformation matrices, which are used to adjust the dimension of features; F combined,t1 represents the fusion feature at the first moment, F combined,t2 represents the fused features at the second moment; Q1 represents the query vector of each feature position at the first moment; K1 represents the key vector of each feature position at the second moment; V1 represents the weight value of each feature position at the second moment; S402, calculating the similarity of each feature position according to the query vector Q1 of each feature position at the first moment and the key vector K1 of each feature position at the second moment to obtain a first attention weight C1, and the calculation formula of the first attention weight C1 is: , in, d k Represents the dimension of the feature, which is used to scale the similarity score to ensure the stability of the value; K1 T represents the transpose of K1, which represents the transposed form of the key vector of each feature position at the second moment, and is used to calculate the similarity between Q1 and K1; C1 represents the correlation between each feature position at the first moment and the second moment; S403: Perform weighted summation on the weight value V1 of each feature position at the second moment according to the first attention weight C1 to generate a weighted feature representation F1 at the first moment. attended ; S404, the fusion feature F at the first moment combined,t1 And the weighted feature representation F1 at the first moment attended Perform fusion processing to generate the final fusion feature F1 at the first moment final ; The fusion formula is: F1 final =F combined,t1 +F1 attended .

[0051] The steps to generate the final fusion features at the second moment include: S411, subjecting the fused features at the first moment and the fused features at the second moment to different linear transformations, respectively, to obtain Q2 corresponding to the fused features at the second moment, and K2 and V2 corresponding to the fused features at the first moment. The linear transformation formula is: Q2=W Q F combined,t2 , K2=W K F combined,t1 , V2=W V F combined,t1 , Wherein, Q2 represents the query vector of each feature position at the second moment; K2 represents the key vector of each feature position at the first moment; V2 represents the weight value of each feature position at the first moment; S412: Calculate the similarity of each feature position according to the query vector Q2 of each feature position at the second moment and the key vector K2 of each feature position at the first moment to obtain a second attention weight C2. The calculation formula of the second attention weight C2 is: , Among them, K2 T represents the transpose of K2, which represents the transposed form of the key vector of each feature position at the first moment, and is used to calculate the similarity between Q2 and K2; C2 represents the correlation between each feature position at the second moment and the first moment; S413: Perform weighted summation on the weight value V2 of each feature position at the first moment according to the second attention weight C2 to generate a weighted feature representation F2 at the second moment. attended ; S414, the fusion feature F at the second moment combined,t2 And the weighted feature representation F2 at the second moment attended Perform fusion processing to generate the final fusion feature F2 at the second moment final ; The fusion formula is: F2 final =F combined,t2 +F2 attended .

[0052] For remote sensing images of different phases and different modalities, the existing fusion methods fail to fully consider the spatiotemporal correlation between different phases, ignore the influence of the time dimension, namely the phase change, and regard the remote sensing images at different time points as independent inputs, thereby reducing the accuracy of change detection. The present invention adopts the spatiotemporal attention mechanism to associate the fusion features of the first moment with the fusion features of the second moment to model, extract the spatiotemporal relationship between the images of the ground objects in different phases, comprehensively consider the change information in the time dimension, enhance the spatiotemporal modeling ability of change detection, better understand the relationship between the first moment and the second moment in change detection, effectively capture the spatiotemporal relationship between different phases, and more comprehensively capture the change information, so that the multimodal features can be fully integrated, thereby improving the efficiency and accuracy of change detection and reducing the cost of computing resources.

[0053] S5. Detect image changes between the first moment and the second moment according to the final fusion features at the first moment and the final fusion features at the second moment.

[0054] According to the final fusion feature F1 at the first moment final And the final fusion feature F2 at the second momentfinal After performing element-by-element subtraction, the absolute value is taken to obtain the differential feature representation D. The calculation formula of the differential feature representation D is: D=|F1 final - F2 final |; The differential feature representation D is input into the classifier, and the classifier detects the image changes between the first moment and the second moment, and generates a change map by marking the changed area obtained by the change judgment.

[0055] The present invention also relates to a multi-source high-resolution remote sensing image change detection system based on the multi-source high-resolution remote sensing image change detection method. Figure 3 ,include: a modality conversion module, configured to convert a first image into a first synthetic image having a second modality style, and convert a second image into a second synthetic image having a first modality style, wherein the first image is a first modality image corresponding to a first moment, and the second image is a second modality image corresponding to a second moment, and the first modality and the second modality are different modalities; A feature extraction module, used for extracting global features of the first image, the first synthetic image, the second image, and the second synthetic image; The feature fusion module is used to perform weighted fusion on the extracted global features to form fusion features at the first moment and fusion features at the second moment; and to perform correlation modeling on the fusion features at the first moment and the fusion features at the second moment, extract the spatiotemporal relationship between the images of the ground objects at different time phases, and generate the final fusion features at the first moment and the final fusion features at the second moment; The change detection module is used to detect the image change between the first moment and the second moment based on the final fusion feature of the first moment and the final fusion feature of the second moment.

[0056] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-source high-resolution remote sensing image change detection method, characterized in that: The following steps are involved: S1. Acquire a remote sensing image, wherein the remote sensing image includes a first image at a first moment and a second image at a second moment, wherein the first image is an image of a first modality, and the second image is an image of a second modality, and the first modality and the second modality are different modalities; S2, converting the remote sensing image into a synthetic image, including: converting the first image into a first synthetic image having a second modality style, and converting the second image into a second synthetic image having the first modality style; S3, extracting global features of the first image, the first synthesized image, the second image, and the second synthesized image; S4, weighted fusion is performed on the extracted global features to form fusion features at the first moment and fusion features at the second moment; and the fusion features at the first moment and the fusion features at the second moment are associated with each other for modeling, the spatiotemporal relationship between the images of the ground objects at different time phases is extracted, and the final fusion features at the first moment and the final fusion features at the second moment are generated; S5. Detect image changes between the first moment and the second moment according to the final fusion features at the first moment and the final fusion features at the second moment.

2. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 1, characterized in that: The step of generating the final fusion feature at the first moment in step S4 includes: S401, subjecting the fused features at the first moment and the fused features at the second moment to different linear transformations, respectively, to obtain Q1 corresponding to the fused features at the first moment, and K1 and V1 corresponding to the fused features at the second moment. The linear transformation formula is: Q1=W Q F combined,t1 ,K1=W K F combined,t2 ,V1=W V F combined,t2 , Among them, W Q , W K , W V Both represent learnable linear transformation matrices, which are used to adjust the dimension of features; F combined,t1 represents the fusion feature at the first moment, F combined,t2 represents the fused features at the second moment; Q1 represents the query vector of each feature position at the first moment; K1 represents the key vector of each feature position at the second moment; V1 represents the weight value of each feature position at the second moment; S402, calculating the similarity of each feature position according to the query vector Q1 of each feature position at the first moment and the key vector K1 of each feature position at the second moment to obtain a first attention weight C1, and the calculation formula of the first attention weight C1 is: ， in, d k Represents the dimension of the feature, used to scale the similarity score; K1 T represents the transpose of K1, which represents the transposed form of the key vector of each feature position at the second moment, and is used to calculate the similarity between Q1 and K1; C1 represents the correlation between each feature position at the first moment and the second moment; S403: Perform weighted summation on the weight value V1 of each feature position at the second moment according to the first attention weight C1 to generate a weighted feature representation F1 at the first moment. attended ; S404, the fusion feature F at the first moment combined,t1 And the weighted feature representation F1 at the first moment attended Perform fusion processing to generate the final fusion feature F1 at the first moment final ; The fusion formula is: F1 final =F combined,t1 +F1 attended 。 3. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 2, characterized in that: The step of generating the final fusion feature at the second moment in step S4 includes: S411, subjecting the fused features at the first moment and the fused features at the second moment to different linear transformations, respectively, to obtain Q2 corresponding to the fused features at the second moment, and K2 and V2 corresponding to the fused features at the first moment. The linear transformation formula is: <h2 style=";text-align:left;direction:ltr">Q2=W<h2 style=";text-align:left;direction:ltr"> Q <h2 style=";text-align:left;direction:ltr"> F<h2 style=";text-align:left;direction:ltr"> combined,t2 <h2 style=";text-align:left;direction:ltr"> K2=W<h2 style=";text-align:left;direction:ltr"> K <h2 style=";text-align:left;direction:ltr"> F<h2 style=";text-align:left;direction:ltr"> combined,t1 <h2 style=";text-align:left;direction:ltr"> V2=W<h2 style=";text-align:left;direction:ltr"> V <h2 style=";text-align:left;direction:ltr"> F<h2 style=";text-align:left;direction:ltr"> combined,t1 <h2 style=";text-align:left;direction:ltr"> , Wherein, Q2 represents the query vector of each feature position at the second moment; K2 represents the key vector of each feature position at the first moment; V2 represents the weight value of each feature position at the first moment; S412: Calculate the similarity of each feature position according to the query vector Q2 of each feature position at the second moment and the key vector K2 of each feature position at the first moment to obtain a second attention weight C2. The calculation formula of the second attention weight C2 is: ， Among them, K2 T represents the transpose of K2, which represents the transposed form of the key vector of each feature position at the first moment, and is used to calculate the similarity between Q2 and K2; C2 represents the correlation between each feature position at the second moment and the first moment; S413: Perform weighted summation on the weight value V2 of each feature position at the first moment according to the second attention weight C2 to generate a weighted feature representation F2 at the second moment. attended ; S414, the fusion feature F at the second moment combined,t2 And the weighted feature representation F2 at the second moment attended Perform fusion processing to generate the final fusion feature F2 at the second moment final ; The fusion formula is: <h2 style=";text-align:left;direction:ltr">F2<h2 style=";text-align:left;direction:ltr"> final <h2 style=";text-align:left;direction:ltr"> =F<h2 style=";text-align:left;direction:ltr"> combined,t2 <h2 style=";text-align:left;direction:ltr"> +F2<h2 style=";text-align:left;direction:ltr"> attended <h2 style=";text-align:left;direction:ltr"> 。 4. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 1, characterized in that: Step S2 includes: Inputting the first image and the second image into the CycleGAN model, converting the first image into a first synthetic image having the style of the second modality through the generator G1 of the CycleGAN model, and converting the second image into a second synthetic image having the style of the first modality through the generator G2 of the CycleGAN model; The discriminator D1 of the CycleGAN model determines the difference in modal features between the first synthetic image and the second image, and feeds back to the generator G1. The generator G1 is improved according to the difference information fed back by the discriminator D1, so that the generator G1 can output the first synthetic image that is closer to the second modal style. At the same time, the discriminator D2 of the CycleGAN model determines the difference in modal features between the second synthetic image and the first image, and feeds back to the generator G2. The generator G2 is improved according to the difference information fed back by the discriminator D2, so that the generator G2 can output the second synthetic image that is closer to the first modal style.

5. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 4, characterized in that: Step S2 also includes the step of performing edge preservation loss processing on the first synthetic image and the second synthetic image, including: The edge differences between the first image, the second image and the first synthetic image and the second synthetic image are calculated respectively by the Sobel operator, and fed back to the generator G1 and the generator G2 of the CycleGAN model. The generator G1 and the generator G2 are improved according to the feedback edge difference information, so that the generator G1 can output the first synthetic image retaining the edge features of the first image, and the generator G2 can output the second synthetic image retaining the edge features of the second image. The improved generator G1 outputs the first synthetic image retaining the edge features of the first image according to the first image, and the improved generator G2 outputs the second synthetic image retaining the edge features of the second image according to the second image. The Sobel operator is: L edge =E A [||Edge(A)-Edge(G1(A))||1]+E B [||Edge(B)-Edge(G2(B))||1], Among them, L edge represents the edge difference information between the first image and the second image and the first synthesized image and the second synthesized image, E A represents the edge loss value of the first image, E B represents the edge loss value of the second image, A represents the first image, B represents the second image, G1(A) represents the first synthesized image, and G2(B) represents the second synthesized image.

6. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 4, characterized in that: Step S2 also includes the step of performing spatial consistency loss processing on the remote sensing image and the corresponding synthetic image, including: The spatial difference between the remote sensing image and the corresponding synthetic image is calculated by the spatial consistency loss formula and fed back to the corresponding generator in the CycleGAN model. The generator is improved according to the fed-back spatial difference information so that the generator can output a synthetic image that retains the spatial structure of the remote sensing image. The improved generator is used to output a synthetic image that retains the spatial structure of the remote sensing image based on the remote sensing image. The spatial consistency loss formula is: L spatial =E x [||A spatial -Gi(X) spatial ||2], Among them, L spatial Represents the spatial difference information between the remote sensing image and the corresponding synthetic image, E x Represents the edge loss value of the remote sensing image, A spatial Represents the spatial feature representation of remote sensing images, Gi(X) spatial Representation of spatial features of synthetic images corresponding to remote sensing images.

7. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 1, characterized in that: Step S3 includes: Convolution kernels of different sizes of a multi-scale convolutional neural network are used to extract global features of the first image, the first synthesized image, the second image, and the second synthesized image.

8. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 7, characterized in that: Step S3 also includes performing a channel attention mechanism on the global feature map generated when extracting the global feature, and using a multi-scale convolutional neural network to extract features from the processed global feature map to generate global features; The steps of processing the channel attention mechanism on the global feature map include: When extracting the global features of the first image, the first synthetic image, the second image, and the second synthetic image, global feature maps of different sizes are generated, and global average pooling is performed on the global feature maps to obtain the global statistical information of each channel in the global feature maps. A fully connected network is used to calculate the weight of the importance of each channel according to the global statistical information of each channel, and the weight is applied to each channel of the global feature map.

9. The method for detecting changes in multi-source high-resolution remote sensing images according to claim 1, characterized in that: Step S5 includes: According to the final fusion feature F1 at the first moment final And the final fusion feature F2 at the second moment final After performing element-by-element subtraction, the absolute value is taken to obtain the differential feature representation D. The calculation formula of the differential feature representation D is: D=|F1 final - F2 final |; The differential feature representation D is input into the classifier to detect the image changes between the first moment and the second moment and generate a change map.

10. A multi-source high-resolution remote sensing image change detection system based on the multi-source high-resolution remote sensing image change detection method according to any one of claims 1 to 9, characterized in that: include: a modality conversion module, configured to convert a first image into a first synthetic image having a second modality style, and convert a second image into a second synthetic image having a first modality style, wherein the first image is a first modality image corresponding to a first moment, the second image is a second modality image corresponding to a second moment, and the first modality and the second modality are different modalities; A feature extraction module, used for extracting global features of the first image, the first synthetic image, the second image, and the second synthetic image; The feature fusion module is used to perform weighted fusion on the extracted global features to form fusion features at the first moment and fusion features at the second moment; and to perform correlation modeling on the fusion features at the first moment and the fusion features at the second moment, extract the spatiotemporal relationship between the images of the ground objects at different time phases, and generate the final fusion features at the first moment and the final fusion features at the second moment; The change detection module is used to detect the image change between the first moment and the second moment based on the final fusion feature of the first moment and the final fusion feature of the second moment.

Images