Twin change detection method based on information interaction and fusion

Abstract

The invention relates to a twin change detection method based on information interaction and fusion, and belongs to the field of remote sensing and computer vision. A pair of remote sensing images at different moments is read into the model, the model comprises a symmetric backbone network, a feature information interaction module, a feature pair fusion model and a multi-scale contrast learning module, the model is optimized through a loss function, a change graph is obtained for the remote sensing images at different moments through the optimized model, and a change detection task is achieved. The method has the advantages that the feature information interaction module models context information of local features and enhances interdependence between the features, the feature pair fusion model models semantic relationships between feature pairs at different moments, edge information of images is enhanced, and model performance is improved; and obtaining a fine change chart of the monitored area by adopting a supervision mode based on comparative learning. The method can be widely applied to remote sensing change detection tasks of various scenes, and subsequent disaster analysis, emergency response and other tasks are facilitated.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing and computer vision, and in particular refers to a twin change detection method based on information interaction and fusion. Background Art

[0002] In recent years, with the continuous development of remote sensing technology, the use of satellite remote sensing to observe the earth can realize the rational development and utilization of natural resources according to the state and change information of the earth's surface, so as to protect the earth from a more macro perspective and promote the harmonious coexistence of man and nature. At the same time, many researchers have also begun to use remote sensing images to monitor the surface. Among them, change detection technology based on remote sensing images has been widely used in many fields, especially for monitoring tasks of major natural disasters. This technology can monitor, evaluate and assist in decision-making of natural disasters in disaster prevention, mitigation and relief work. At the same time, it can use the changes in the monitoring area to conduct disaster assessment and provide certain support for the prevention and control of subsequent disasters. Therefore, the study of change detection technology based on remote sensing images is of great importance and is an indispensable technical means in disaster analysis and rescue plan formulation.

[0003] In the embryonic stage of change detection, change detection of remote sensing images is mostly based on traditional methods, which can be roughly divided into: algebra-based methods, feature-based methods, and post-classification comparison methods. The algebra-based method is simple and easy to operate, and can respond quickly to the changed area. The feature-based change detection method first needs to select a variety of image features, then transform and fuse the selected image features, and finally realize the change detection task. The post-classification comparison method is different from the algebra-based method and the feature-based method. The advantage of this method is that it can not only detect changes (or no changes) in a certain area, but also identify the category of changes.

[0004] Although the above change detection algorithms based on traditional methods have promoted the development of change detection tasks to a certain extent and achieved good performance. However, with the rapid development of artificial intelligence and deep learning, using deep learning technology to detect changes in images has become a research hotspot. The method of extracting advanced features using deep learning technology has shown broad application prospects. This powerful feature extraction capability of deep neural networks can greatly improve the performance of change detection.

[0005] Therefore, in order to extract more detailed deep features and make the network more robust and generalizable, many researchers have proposed a large number of change detection networks based on deep learning for different tasks, and have achieved good results. Among these methods, whether based on full convolutional networks, twin networks or GAN change detection networks, there are some limitations: (1) Full convolutional networks based on symmetric structures aim to extract more information by designing different feature fusion methods. Among them, Daudt et al. use a simple skip connection mechanism to operate on intermediate features, and Peng et al. use different attention mechanisms to fuse deep feature maps with feature difference maps. Although these methods have achieved certain results, simple fusion methods cannot obtain detailed change features and will introduce certain noise. (2) Twin change detection networks based on UNet aim to improve network performance by designing different supervision methods. Among them, Zhan et al. use contrastive learning ideas to supervise the final output of the network. Although satisfactory results are achieved, they ignore the effective supervision of intermediate features, which causes the network to lose some information of intermediate features. (3) Gong et al. achieved good network performance by combining GAN networks with change detection tasks, but these networks lack the consideration and modeling of the relationship between feature pairs at different times, and modeling the relationship between feature pairs is one of the important factors in obtaining a more refined change map. Therefore, how to effectively extract and fuse features at different times and better constrain these features is a problem that we need to think about seriously, and it is also a challenge faced by the change detection task. Summary of the invention

[0006] The present invention provides a twin change detection method based on information interaction and fusion to solve the problem that most current networks only process change features through a simple fusion method, lack modeling of the relationship between feature pairs at different times, and ignore effective supervision of intermediate features, resulting in poor network performance, aiming to obtain a more detailed regional change map.

[0007] The technical solution adopted by the present invention comprises the following steps:

[0008] (1) Read a pair of remote sensing images T at different times into the model 1 and T 2 ;

[0009] (2) Use 3×3 convolution to obtain the input image T 1 and T 2 The shallow features x 1 and x 2 ;

[0010] (3) In the twin encoding stage of the network, the twin dual-branch network uses four downsampling layers to implement the shallow feature x 1 and x2 Downsampling operations are performed, and output encoding features with scales of 1 / 2, 1 / 4, and 1 / 8 are obtained respectively, where each downsampling layer includes convolution, batch normalization, and ReLU operations;

[0011] (4) In the twin decoding stage of the network, the twin dual-branch network uses three upsampling layers to upsample the output encoding features with a scale of 1 / 8, and obtains output decoding features with scales of 1 / 4, 1 / 2 and 1 respectively. In order to avoid the loss of important information, a feature information interaction module is used for each output decoding feature, which uses the spatial attention mechanism to model the contextual information of local features, while enhancing the mutual dependence between features, giving greater weight to certain feature representations with specific semantics, and better distinguishing the change areas; then, the feature pair fusion module is used to obtain the delicate change information of the twin branches; finally, a multi-scale contrast learning module is added to measure the distance between feature pairs of different scales, so that the unchanged negative samples are as close as possible and the changed positive samples are kept away from each other, so that different change pairs can be distinguished more easily;

[0012] (5) Network total loss function

[0013] In order to utilize more image pair feature information to obtain more accurate change detection results, the following loss constraint features are designed:

[0014]

[0015] in, represents the supervised approach for multi-scale contrastive learning, It represents the constraint on the final change characteristics, expressed by formula (8) and (9):

[0016]

[0017] in, Represents the change characteristics at the pixel position (h, w), Y hw is the true label, is the cross entropy loss function; for the constraints on the changing features, a multi-scale training strategy is adopted. This constraint on the features of different scales in the decoding stage can improve the network performance and ultimately obtain more refined change information;

[0018] (6) For the input image pair T 1 and T 2 ,Through step (5), the model is optimized to obtain a more refined change graph and achieve the change detection task.

[0019] The convolution kernel size used in the convolution in step (3) of the present invention is 3×3.

[0020] In step (4) of the present invention, the feature information interaction module first performs a transpose operation on the input feature f, wherein the input feature C is the number of feature channels, H and W are the height and width of the feature respectively. After that, the module performs matrix multiplication on the transposed feature with a scale of C×H×W and the reshaped feature C×H×W after the transformation operation, and then obtains the spatial attention relationship map through softmax operation. Afterwards, the module uses the dot multiplication method to apply the obtained spatial attention relationship map to the features after the transformation operation, obtains features with spatial correlation information, and processes them into features of size C×H×W. In order to compensate for the problem of feature information loss that may be caused in this process, it also uses the residual connection method to fuse the features with spatial information with the original input information to achieve better feature extraction and obtain the final output feature f s , the process is expressed by the following formula:

[0021]

[0022] Among them, f i and f j They represent feature vectors at different positions, N is the number of features, and λ is a parameter automatically optimized by the network. Through this calculation process, the degree of association between vectors at different positions can be obtained, that is, the stronger the association between two features, the stronger the correlation between them. s The larger the value.

[0023] In step (4) of the present invention, the feature pair fusion module is divided into two branches, the upper branch I is a difference branch, and the lower branch II is a sum branch. The difference branch has two inputs They are the remote sensing image features extracted at different times, and then three parallel convolution and Relu operations are used to further process the input features. The size of the convolution kernel is 3×3 and the step size is 2. At the same time, in order to make the output feature information richer, this module uses multiple residual connection operations to fuse the features in the feature extraction stage. In addition, in order to generate fine change areas, the two features are processed by difference. Finally, after a convolution, batch normalization and Relu operation, the output feature f of the upper branch can be obtained. I The summation branch parallel to the upper branch is to enhance the edge information of the feature and reduce the pseudo-change phenomenon caused by feature mismatch. Similar to the difference branch, this module only sums the two features after the convolution layer to obtain the output feature f of the summation branch. II At the end of this module, the output features of the two branches are fused by vector-by-vector summation to obtain the final output f of the module. c .

[0024] In the step (4) of the present invention, the multi-scale contrast learning module allows unchanged pixel pairs to be close to each other and changed pixel pairs to be far away from each other in the feature space of multiple scales. In order to assign different distance measurement values ​​to different categories, thereby making it easier to distinguish different changed pairs, the changed area at the same position is defined as a changed pair, i.e., a positive sample, and the unchanged area at the same position of different image pairs is defined as an unchanged pair, i.e., a negative sample, so that the unchanged negative samples can be as close as possible and the changed positive samples can be far away from each other. First, the input of this module is the feature pair processed by the feature information interaction module. In order to make the network better distinguish the information of the changed area and speed up the convergence speed, for positive samples, the distance value between them should be greater than a boundary value θ (θ>0), and only when the distance between positive samples is within the range of this θ value, the changed pixel pairs will affect the loss function. For negative samples, the distance value between them should be less than a boundary value ε (ε>0), and only when the distance between negative samples is within the range of this ε value, the unchanged pixel pairs will affect the loss function. The complete process is expressed by formulas (3)-(5). The multi-scale contrastive learning loss function in the decoding stage can be defined as formula (6);

[0025] D(f s1 (i,j) m ,f s2 (i,j) m )=||f s1 (i,j) m -f s2 (i,j) m || 2 ,m=1,2,(3)

[0026]

[0027]

[0028] Among them, f s1 (i,j) and f s2 (i, j) are the vectors at position (i, j) in the image, m represents the mth layer feature in the decoding stage, and its feature scale is 1 / 2 4-m , D(·) is the distance function that the network needs to learn, and D is used in formulas (4)-(5) i,j express, Represents the feature of the real label of the mth layer in the decoding stage mapped to the input image pair, whose size is H×W, and when y i,j = 0, indicating that the corresponding pixel pair has not changed. i,j =1, it means that the corresponding pixel pair has changed, λ1 With λ 2 Different weights are assigned to different losses to solve the problem of imbalance between positive and negative samples. Through the above calculations, this method can use the distance between features of different scales in the feature space to obtain a more detailed change map.

[0029] The present invention constructs a twin change detection network based on information interaction and fusion. It designs a feature information interaction module based on the spatial attention mechanism to increase the semantic information of the features, and a feature pair fusion module models the semantic relationship between feature pairs at different times, so that it can make full use of the spatial information of the input image pairs to obtain richer detail information, enhance the edge information of the image, and improve the model performance. Through the design of these two modules, the network can obtain more accurate change features. In addition, this network also designs a supervision method based on contrastive learning to constrain the feature pairs obtained in multiple decoding stages, and finally obtains a detailed change map of the monitored area. A large number of experimental results show that this method can obtain a detailed change map and is superior to the existing representative change detection methods, which is of great significance to urban planning, forestry, agriculture, environmental protection and other fields.

[0030] The present invention conducts experimental comparison and analysis on two public datasets CDD and LEVIR-CD, and evaluates the effectiveness and superiority of the proposed method from both qualitative and quantitative perspectives. A large number of comparative experimental results show that the method proposed in the present invention can obtain more detailed change maps and achieve better change detection tasks.

[0031] The beneficial effects of the present invention are as follows: (1) The present invention is an accurate change detection method for remote sensing images; (2) The present invention designs a feature information interaction module to improve the feature extraction capability of the model; the feature pair fusion module models the relationship between features and increases the semantic information of the features; the self-supervision method based on contrastive learning constrains the features of multiple decoding stages to improve the change detection performance. (3) The present invention can be widely used in remote sensing image change detection tasks in various scenarios, which is helpful for subsequent disaster analysis and emergency response tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a pair of remote sensing images at different times on the LEVIR-CD dataset;

[0033] Figure 2 It is the overall framework diagram of the network of the present invention;

[0034] Figure 3 It is a structural diagram of the feature information interaction module;

[0035] Figure 4 It is the structural diagram of the feature pair fusion module;

[0036] Figure 5 This is a visual comparison of the change graphs obtained with other methods on the CDD dataset;

[0037] Figure 6 This is a visual comparison of the change maps obtained with other methods on the LEVIR-CD dataset;

[0038] Figure 7 This is a comparison chart of the visualization effects of the feature information interaction module on the CDD dataset.

[0039] Figure 8 This is a comparison of the visualization effects of the feature information interaction module on the LEVIR-CD dataset.

[0040] Fig. 9 This is a comparison of the visualization effects of the feature pair fusion module on the CDD dataset.

[0041] Fig.10 This is a comparison of the visualization effects of the feature pair fusion module on the LEVIR-CD dataset.

[0042] Fig.11 This is a comparison of the visualization effects of the contrast learning module on the CDD dataset.

[0043] Fig.12 This is a comparison of the visualization effects of the contrastive learning module on the LEVIR-CD dataset. DETAILED DESCRIPTION

[0044] The following steps are involved:

[0045] (1) Read a pair of remote sensing images T at different times into the model 1 and T 2 , see Figure 1 ;

[0046] (2) First, use 3×3 convolution to obtain the input image T 1 and T 2 The shallow features x 1 and x 2 ;

[0047] (3) In the twin encoding stage of the network, the twin dual-branch network uses four downsampling layers to implement the shallow feature x 1 and x 2 Downsampling operations are performed, and output encoding features with scales of 1 / 2, 1 / 4, and 1 / 8 are obtained respectively, where each downsampling layer includes convolution, batch normalization, and ReLU operations;

[0048] (4) In the twin decoding stage of the network, the twin dual-branch network uses three upsampling layers to upsample the output encoding features with a scale of 1 / 8, and obtains output decoding features with scales of 1 / 4, 1 / 2 and 1 respectively. In order to avoid the loss of important information, a feature information interaction module is used for each output decoding feature. It uses the spatial attention mechanism to model the contextual information of local features, while enhancing the mutual dependence between features and giving greater weight to certain feature representations with specific semantics, which can better distinguish the change area. Subsequently, the feature pair fusion module is used to obtain the delicate change information of the twin branches. The designed dual-branch parallel structure can improve the model performance, and in the process of feature extraction, multiple residual connections are used to process the features, which to a certain extent compensates for the possible information loss problem and makes the obtained change features more accurate. Finally, a multi-scale contrast learning module is added to measure the distance between feature pairs of different scales, so that the unchanged negative samples are as close as possible and the changed positive samples are kept away from each other, so that different change pairs can be distinguished more easily.

[0049] (5) Network total loss function

[0050] In order to utilize more image pair feature information to obtain more accurate change detection results, the following loss constraint features are designed:

[0051]

[0052] in, represents the supervised approach for multi-scale contrastive learning, It represents the constraint on the final change characteristics, expressed by formula (2) and (3):

[0053]

[0054] in, Represents the change characteristics at the pixel position (h, w), Y hw is the true label, is the cross entropy loss function; for the constraint of the changing features, the invention adopts a multi-scale training strategy. This constraint on features of different scales during the decoding stage can improve the network performance and ultimately obtain more refined change information;

[0055] (6) For the input image pair T 1 and T 2 ,Through step (5), the model is optimized to obtain a more refined change graph and achieve the change detection task.

[0056] The convolution kernel size used in the convolution in step (3) of the present invention is 3×3.

[0057] In step (4) of the present invention, the feature information interaction module first performs a transpose operation on the input feature f, wherein the input feature C is the number of feature channels, H and W are the height and width of the feature respectively. After that, the module performs matrix multiplication on the transposed feature with a scale of C×H×W and the reshaped feature C×H×W after the transformation operation, and then obtains the spatial attention relationship map through softmax operation. Afterwards, the module uses the dot multiplication method to apply the obtained spatial attention relationship map to the features after the transformation operation, obtains features with spatial correlation information, and processes them into features of size C×H×W. In order to compensate for the problem of feature information loss that may be caused in this process, it also uses the residual connection method to fuse the features with spatial information with the original input information to achieve better feature extraction and obtain the final output feature f s , see Figure 3 , the process can be expressed by the following formula:

[0058]

[0059] Among them, f i and f j They represent feature vectors at different positions, N is the number of features, and λ is a parameter automatically optimized by the network. Through this calculation process, the degree of association between vectors at different positions can be obtained, that is, the stronger the association between two features, the stronger the correlation between them. s The larger the value of , at the same time, it can be seen from the above formula that the output feature f of the feature information interaction module s is the weighted sum of the feature information at all positions and the original input features, so f s It has rich global context information. In addition, due to this output feature f s It is obtained based on the calculation method of the spatial attention mechanism. Therefore, it can adaptively select and aggregate contextual information, and can model long-term semantic dependencies between features, improve semantic consistency, and enable similar semantic features to promote each other, ultimately enabling the network to better distinguish between changed features and unchanged features.

[0060] In step (4) of the present invention, the feature pair fusion module is divided into two branches, the upper branch I is a difference branch, and the lower branch II is a sum branch. The difference branch has two inputs They are the remote sensing image features extracted at different times, and then three parallel convolution and Relu operations are used to further process the input features. The size of the convolution kernel is 3×3 and the step size is 2. At the same time, in order to make the output feature information richer, this module uses multiple residual connection operations to fuse the features in the feature extraction stage. In addition, in order to generate fine change areas, the two features are processed by difference. Finally, after a convolution, batch normalization and Relu operation, the output feature f of the upper branch can be obtained. I The summation branch parallel to the upper branch is to enhance the edge information of the feature and reduce the pseudo-change phenomenon caused by feature mismatch. Similar to the difference branch, this module only sums the two features after the convolution layer to obtain the output feature f of the summation branch. II At the end of this module, the output features of the two branches are fused by vector-by-vector summation to obtain the final output f of the module. c , see Figure 4 .

[0061] In the step (4) of the present invention, the multi-scale contrast learning module allows unchanged pixel pairs to be close to each other and changed pixel pairs to be far away from each other in the feature space of multiple scales. In order to assign different distance measurement values ​​to different categories, thereby making it easier to distinguish different changed pairs, the changed area at the same position is defined as a changed pair, i.e., a positive sample, and the unchanged area at the same position of different image pairs is defined as an unchanged pair, i.e., a negative sample, so that the unchanged negative samples can be as close as possible and the changed positive samples can be far away from each other. First, the input of this module is the feature pair processed by the feature information interaction module. In order to make the network better distinguish the information of the changed area and speed up the convergence speed, for positive samples, the distance value between them should be greater than a boundary value θ (θ>0), and only when the distance between positive samples is within the range of this θ value, the changed pixel pairs will affect the loss function. For negative samples, the distance value between them should be less than a boundary value ε (ε>0), and only when the distance between negative samples is within the range of this ε value, the unchanged pixel pairs will affect the loss function. The complete process is expressed by formulas (3)-(5). The multi-scale contrastive learning loss function in the decoding stage can be defined as formula (6);

[0062] D(f s1 (i,j) m ,f s2 (i,j) m )=||f s1 (i,j) m -f s2 (i,j) m || 2,m=1,2,(3)

[0063]

[0064] Among them, f s1 (i,j) and f s2 (i, j) are the vectors at position (i, j) in the image, m represents the mth layer feature in the decoding stage, and its feature scale is 1 / 2 4-m , D(·) is the distance function that the network needs to learn, and D is used in formulas (4)-(5) i,j express, Represents the feature of the real label of the mth layer in the decoding stage mapped to the input image pair, whose size is H×W, and when y i,j = 0, indicating that the corresponding pixel pair has not changed. i,j =1, it means that the corresponding pixel pair has changed, λ 1 With λ 2 Different weights are assigned to different losses to solve the problem of imbalance between positive and negative samples. Through the above calculations, this method can use the distance between features of different scales in the feature space to obtain a more detailed change map.

[0065] The beneficial effects of the present invention are further illustrated below through analysis and comparison of specific experimental results.

[0066] Experimental example

[0067] In order to effectively and systematically evaluate the proposed method, experimental comparisons and analyses were conducted on two public datasets, CDD and LEVIR-CD. The CDD dataset is an open seasonal remote sensing change detection dataset. The dataset consists of multi-source remote sensing images and 11 pairs of original images, including 7 pairs of 4725×2200 pixel change images and 4 pairs of 1900×1000 pixel change images, with a spatial resolution of 3cm-100cm. After processing, the CDD dataset is divided into a training set consisting of 10,000 samples, a validation set consisting of 3,000 samples, and a test set consisting of 3,000 samples, with a pixel size of 256×256. The LEVIR-CD dataset contains a pair of image scenes captured at the same location in 2012 and 2016, as well as semantic labels and change detection labels of buildings. The location is the area where the 6.3 magnitude earthquake occurred in Christchurch, New Zealand in February 2011. The LEVIR-CD dataset contains 637 pairs (445 pairs for training) of high-resolution images with a spatial resolution of 50 cm and a pixel size of 1024×1024. Figure 1 Given a pair of images from the LEVIR-CD dataset.

[0068] In the specific implementation process, θ is set to 1, ε is set to 0.1, weight decay is set to 5e-5, initial learning rate is set to 1e-4, epoch is set to 2000, Batch size is set to 2, and Adam optimizer is used to optimize the network of the present invention.

[0069] In order to evaluate the performance of the method, precision, recall and F1 score are used to quantitatively evaluate the method.

[0070] Table 1 Performance comparison with other advanced methods on the CDD dataset

[0071]

[0072] As can be seen from Table 1, the performance of the present invention is better than other advanced methods. The reasons are as follows: (1) The methods based on UNet and twin networks often only use simple network structures and skip connection mechanisms to process features. This method cannot extract rich semantic features of the input image. In addition, since the design of these two network structures will lose a lot of detail information during the downsampling process, and the upsampling process does not effectively supervise the features, this method cannot obtain good results; (2) The UNet++_MSOF network is a change detection network based on deep supervision. It first uses a fully convolutional network with a symmetrical structure to extract features from the input image, and then uses a difference discrimination network to detect changes in the output feature pairs. This network also uses an attention mechanism to fuse deep feature maps and image difference feature maps at different scales. Although the performance of UNet++_MSOF networks with different numbers of channels (16 and 32 represent the number of output channels, respectively) is better than FC-EF, FC-Siam-conc and FC-Siam-diff, it still uses a simple skip connection mechanism to fuse intermediate features, resulting in the inability to capture the deep features of the image, and thus the inability to effectively reconstruct the image. Therefore, the F1 score of the UNet++_MSOF network is not very high, which is 0.062 (16 channels) and 0.020 (32 channels) lower than that of SNIIF-Net respectively; (3) The DASNet network is a change detection network based on contrastive learning and dual-headed attention mechanism. Although it improves the network performance by introducing contrastive learning method, DASNet only uses contrastive learning to supervise the final output features and lacks constraints on intermediate features. In addition, the use of a dual-headed attention mechanism also increases the burden on the network to a certain extent, so the performance is not optimal, and the obtained F1 score is 0.044 lower than that of SNIIF-Net; (4) The SNUNet-CD network is a twin network based on UNet, which uses an integrated channel attention fusion method to process intermediate features. It is better than the DASNet network in performance, but the simple focal loss cannot supervise the network well. Therefore, the performance of the DASNet network is not the best; (5) Some recent methods have achieved higher F1 scores by using some new fusion methods or network structures. However, whether it is the Transformer data block method, the feature fusion method of different attention mechanisms, or the different design methods based on the UNet network, these methods ignore the effective constraints on the intermediate features, so the network performance is not optimal.The present invention designs a feature information interaction module that uses the attention mechanism to model the contextual information of local features, enhances the mutual dependence between features, and can better distinguish the change area; the feature pair fusion module makes full use of the spatial and relational information of the input image pair to obtain more refined detail information, enhance the edge information of the image, and improve the model performance; the supervision method based on contrastive learning can effectively constrain the intermediate features, so that the network can obtain a more refined change map, thereby improving the accuracy of the network.

[0073] pass Figure 5 It can also be seen from the visual comparison results that the performance of the present invention is better than other methods. For example, the DASNet network lacks effective supervision of the intermediate features, which makes it unable to extract rich semantic features. Therefore, the obtained change map is the roughest and there is a phenomenon of missed detection of regions. The SNUNet-CD network uses a simple feature fusion method to generate change features. Since this method introduces noise, it inevitably affects the fineness of the final change map. Since the present invention uses a supervision method based on contrastive learning to supervise the intermediate features, and uses an information interaction module and a feature pair fusion module to learn the spatial and semantic relationship of the image, the edge information of the image is enhanced. Therefore, compared with the change maps obtained by the DASNet and SNUNet-CD networks, the obtained change map is closer to the real label, more refined in contour, and can better reflect the changes in the region.

[0074] Table 2 Performance comparison with other advanced methods on the LEVIR-CD dataset

[0075]

[0076]

[0077] From the experimental results in Table 2 and Figure 6 The advantages of the present invention can be clearly seen in the visual comparison results. That is, the present invention can obtain richer change features and extract more refined change result graphs by introducing the feature information interaction module, feature pair relationship module and comparative learning supervision method for output features of different scales.

[0078] In order to verify the effectiveness of each module in the model proposed by the present invention, the feature information interaction module in the network is first replaced by the traditional convolution operation. The experimental results on the CDD and LEVIR-CD datasets are shown in Table 3. The experimental results on the two datasets show that in the absence of the feature information interaction module, the F1 scores obtained by the present invention are reduced by 0.025 and 0.020 respectively. This means that replacing this module with the traditional convolution will cause the extracted intermediate features to lose some important detail information, while extracting feature information using an attention-based method can enrich the feature information and help improve network performance. Figure 7 and Figure 8 The visualization effects on CDD and LEVIR-CD datasets are compared. It can be seen from the comparison chart that the contour of the change map obtained by the feature information interaction module is more refined.

[0079] Table 3. Influence of feature information interaction module on experimental performance

[0080]

[0081] Secondly, the feature pair fusion module is removed from the network, and only a simple difference operation is used to process the intermediate features. The experimental results on the two data sets are shown in Table 4. The experimental results show that using a simple difference operation to replace this module will cause the network to shift in spatial position and color during feature extraction, and will also introduce considerable noise. The use of the feature pair fusion module can make up for the shortcomings of the simple difference method and enrich the feature information. Therefore, the feature fusion method using this module can improve the accuracy of change detection. Fig. 9 and Fig.10 The following is a comparison of the visualization effects on the CDD and LEVIR-CD datasets. From the comparison chart, we can see that the change graph obtained by this module is closer to the true label.

[0082] Table 4. The impact of features on the fusion module on experimental performance

[0083]

[0084] Then, the present invention removes the contrastive learning module from the network, which means that the network does not use the idea of ​​contrastive learning to supervise the multi-scale output features in the intermediate layer of the decoding stage. The experimental results on the two data sets are shown in Table 5. From the experimental results, it can be seen that the lack of the contrastive learning module leads to the most obvious reduction in network performance. This is because the contrastive learning module effectively supervises multiple intermediate features and improves the overall performance of the network. This also shows that the contrastive learning module is very important for generating fine change maps. Fig.11 and Fig.12The comparison of visualization effects on the two datasets is given in detail. From the comparison chart, we can also see that the change chart obtained by the contrast learning module is better.

[0085] Table 5. Effect of comparative learning modules on experimental performance

[0086]

[0087] Finally, the present invention also tests the impact of the use of the contrastive learning module on the performance of the network, and the experimental results are shown in Tables 6 and 7, where 1 / 8, 1 / 4, 1 / 2, and 1 / 1 represent features of different scales, respectively. It can be seen from Tables 6 and 7 that if only the features of a certain scale are supervised by contrastive learning, the performance of the network is poor. As contrastive learning is used to supervise multi-layer features, the network performance gradually improves. When the network performs contrastive learning supervision on the features of the four scales in the decoder, the highest F1 score is achieved, which shows that supervision of the features of each scale can enable the network to obtain more refined information and improve network performance.

[0088] Table 6 Comparison of the results of using learning modules on the CDD dataset

[0089]

[0090] Table 7 Comparison of the results of using learning modules on the LEVIR-CD dataset

[0091]

[0092] In view of this, the present invention proposes a twin change detection network based on information interaction and fusion. It aims at the problems existing in previous change detection methods: lack of constraints on intermediate features, insufficient modeling of the relationship between features at different times, and simple fusion methods between features. The present invention effectively increases the semantic information of features, enhances the edge information of images, and improves model performance by designing a feature information interaction module and a feature pair fusion modeling module based on the spatial attention mechanism. At the same time, the supervision method based on contrastive learning effectively constrains the intermediate features, so that the network can obtain a more refined change result map. Comparative experiments on two data sets show that the network proposed in the present invention has more advantages than other state-of-the-art methods.

[0093] The above is only a preferred embodiment of the present invention. The protection scope of the present invention is not limited to the above embodiment. All technical solutions belonging to the principle of the present invention belong to the protection scope of this aspect. For those skilled in the art, several improvements and modifications made without departing from the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A twin change detection method based on information interaction and fusion, characterized in that: The following steps are involved: (1) Read a pair of remote sensing images T1 and T2 at different times into the model; (2) Use 3×3 convolution to obtain the shallow features x1 and x2 of the input images T1 and T2 respectively; (3) In the twin encoding stage of the network, the twin dual-branch network uses four downsampling layers to implement the downsampling operation of the shallow features x1 and x2, and obtains output encoding features with scales of 1 / 2, 1 / 4, and 1 / 8 respectively. Each downsampling layer contains convolution, batch normalization, and ReLU operations; (4) In the twin decoding stage of the network, the twin dual-branch network uses three upsampling layers to upsample the output encoding features with a scale of 1 / 8, and obtains output decoding features with scales of 1 / 4, 1 / 2 and 1 respectively. In order to avoid the loss of important information, a feature information interaction module is used for each output decoding feature, which uses the spatial attention mechanism to model the contextual information of local features, while enhancing the mutual dependence between features, giving greater weight to certain feature representations with specific semantics, and better distinguishing the change areas; then, the feature pair fusion module is used to obtain the delicate change information of the twin branches; finally, a multi-scale contrast learning module is added to measure the distance between feature pairs of different scales, so that the unchanged negative samples are as close as possible and the changed positive samples are kept away from each other, so that different change pairs can be distinguished more easily; (5) Network total loss function In order to utilize more image pair feature information to obtain more accurate change detection results, the following loss constraint features are designed: l loss ＝l change +l MCLM , (7) Among them, l MCLM represents the supervision method of multi-scale contrastive learning, l change It represents the constraint on the final change characteristics, expressed by formula (8) and (9): in, Represents the change characteristics at the pixel position (h, w), Y hw is the true label, l(X change ,Y) is the cross entropy loss function; for the constraint of the change feature, a multi-scale training strategy is adopted. This constraint on the features of different scales in the decoding stage can improve the network performance and finally obtain more refined change information; (6) For the input image pair T1 and T2, the model is optimized through step (5) to obtain a more refined change map to achieve the change detection task.

2. According to claim 1, a twin change detection method based on information interaction and fusion is characterized in that: The convolution kernel size used in the convolution in step (3) is 3×3.

3. According to claim 1, a twin change detection method based on information interaction and fusion is characterized in that: In step (4), the feature information interaction module first performs a transpose operation on the input feature f, where the input feature C is the number of feature channels, H and W are the height and width of the feature respectively. After that, the module performs matrix multiplication on the transposed feature with a scale of C×H×W and the reshaped feature C×H×W after the transformation operation, and then obtains the spatial attention relationship map through softmax operation. Afterwards, the module uses the dot multiplication method to apply the obtained spatial attention relationship map to the features after the transformation operation, obtains features with spatial correlation information, and processes them into features of size C×H×W. In order to compensate for the problem of feature information loss that may be caused in this process, it also uses the residual connection method to fuse the features with spatial information with the original input information to achieve better feature extraction and obtain the final output feature f s , the process is expressed by the following formula: Among them, f i and f j They represent feature vectors at different positions, N is the number of features, and λ is a parameter automatically optimized by the network. Through this calculation process, the degree of association between vectors at different positions can be obtained, that is, the stronger the association between two features, the stronger the correlation between them. s The larger the value.

4. According to claim 1, a twin change detection method based on information interaction and fusion is characterized in that: In step (4), the feature pair fusion module is divided into two branches, the upper branch I is a difference branch, and the lower branch II is a summation branch. The difference branch has two inputs They are the remote sensing image features extracted at different times, and then three parallel convolution and Relu operations are used to further process the input features. The size of the convolution kernel is 3×3 and the step size is 2. At the same time, in order to make the output feature information richer, this module uses multiple residual connection operations to fuse the features in the feature extraction stage. In addition, in order to generate fine change areas, the two features are processed by difference. Finally, after a convolution, batch normalization and Relu operation, the output feature f of the upper branch can be obtained. I The summation branch parallel to the upper branch is to enhance the edge information of the feature and reduce the pseudo-change phenomenon caused by feature mismatch. Similar to the difference branch, this module only sums the two features after the convolution layer to obtain the output feature f of the summation branch. II At the end of this module, the output features of the two branches are fused by vector-by-vector summation to obtain the final output f of the module. c .

5. The twin change detection method based on information interaction and fusion according to claim 1 is characterized in that: In the step (4), the multi-scale contrast learning module makes unchanged pixel pairs close to each other and changed pixel pairs far away from each other in the feature space of multiple scales. In order to assign different distance measurement values ​​to different categories and make it easier to distinguish different changed pairs, the changed area at the same position is defined as a changed pair, i.e., a positive sample, and the unchanged area at the same position of different image pairs is defined as an unchanged pair, i.e., a negative sample, so that the unchanged negative samples can be as close as possible and the changed positive samples can be far away from each other. First, the input of this module is the feature pair processed by the feature information interaction module. In order to make the network better distinguish the information of the changed area and speed up the convergence speed, for positive samples, the distance value between them should be greater than a boundary value θ (θ>0), and only when the distance between positive samples is within the range of this θ value, the changed pixel pairs will affect the loss function. For negative samples, the distance value between them should be less than a boundary value ε (ε>0), and only when the distance between negative samples is within the range of this ε value, the unchanged pixel pairs will affect the loss function. The complete process is expressed by formulas (3)-(5). The multi-scale contrastive learning loss function in the decoding stage can be defined as formula (6); D(f s1 (i,j) m ,f s2 (i,j) m )=||f s1 (i,j) m -f s2 (i,j) m ||2,m=1,2, (3) Among them, f s1 (i,j) and f s2 (i, j) are the vectors at position (i, j) in the image, m represents the mth layer feature in the decoding stage, and its feature scale is 1 / 2 4-m , D(·) is the distance function that the network needs to learn, and D is used in formulas (4)-(5) i,j express, Represents the feature of the real label of the mth layer in the decoding stage mapped to the input image pair, whose size is H×W, and when y i,j = 0, indicating that the corresponding pixel pair has not changed. i,j =1, it means that the corresponding pixel pair has changed. λ1 and λ2 assign different weights to different losses to solve the problem of imbalance between positive and negative samples. Through the above calculation, this method can use the distance between features of different scales in the feature space to obtain a more detailed change map.

Images