Remote sensing image change detection method based on semantic interaction and differential representation

Abstract

The invention discloses a remote sensing image change detection method based on semantic interaction and differential representation, and the method comprises the following steps: S1, inputting a to-be-detected dual-temporal remote sensing image into an improved ResNet34 network, and obtaining the multi-scale features of the to-be-detected dual-temporal remote sensing image; s2, inputting the multi-scale features of the to-be-detected dual-temporal remote sensing image into a semantic interaction module to obtain multi-scale dual-temporal semantic features; s3, inputting the multi-scale dual-temporal semantic features into a difference representation module to obtain multi-scale difference features; s4, inputting the multi-scale difference features into a multi-scale feature aggregation module to obtain difference aggregation features; inputting the multi-scale features of the to-be-detected dual-time-phase remote sensing image into a multi-scale feature aggregation module to obtain dual-time-phase aggregation features; s5, inputting the difference aggregation features after up-sampling convolution into a binary change detection module to obtain a binary change graph; and S6, performing mask multiplication on the dual-temporal clustering features after up-sampling convolution and the binary change graph to obtain a dual-temporal semantic change graph. According to the invention, the accuracy of detecting the change position of the dual-temporal remote sensing image and identifying the change element type can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image change detection, and more particularly to a remote sensing image change detection method based on semantic interaction and difference representation. Background Art

[0002] With the rapid development of aerospace and satellite technology, remote sensing images have become a powerful tool for analyzing changes in the earth's surface. As a key technology, remote sensing image semantic change detection aims to monitor changes in surface features in the same area at different times and has been widely used in environmental monitoring, urban expansion and reconstruction, and disaster assessment.

[0003] However, existing remote sensing image change detection methods have low accuracy in detecting the changed positions and identifying the types of changed elements in dual-temporal remote sensing images.

[0004] Therefore, how to provide a remote sensing image change detection method that can improve the accuracy of detecting the changed position of dual-temporal remote sensing images and identifying the type of changed elements is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a remote sensing image change detection method based on semantic interaction and difference representation.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A remote sensing image change detection method based on semantic interaction and difference representation includes the following steps:

[0008] S1: Input the dual-temporal remote sensing image to be detected into the improved ResNet34 network to obtain the multi-scale features of the dual-temporal remote sensing image to be detected;

[0009] Among them, the improved ResNet34 network includes the improved Layer0, Layer1, Layer2, Layer3 and improved Layer4 connected in sequence; the improved Layer0 includes the convolution layer, batch normalization layer, Relu activation layer and maximum pooling layer connected in sequence; Layer1, Layer2 and Layer3 are the original ResNet34 network structure; the improved Layer4 is an identity mapping;

[0010] S2: Input the multi-scale features of the dual-temporal remote sensing image to be detected into the semantic interaction module to obtain multi-scale dual-temporal semantic features;

[0011] S3: Input the multi-scale bi-temporal semantic features into the difference representation module to obtain multi-scale difference features;

[0012] S4: input the multi-scale difference features into the multi-scale feature aggregation module to obtain difference aggregation features;

[0013] Input the multi-scale features of the dual-temporal remote sensing image to be detected into the multi-scale feature aggregation module to obtain the dual-temporal aggregation features;

[0014] S5: Input the difference aggregation features after upsampling convolution into the binary change detection module to obtain a binary change map;

[0015] S6: Mask-multiply the bi-temporal aggregate features after upsampling convolution with the binary change map to obtain a bi-temporal semantic change map.

[0016] Preferably, S1 specifically includes:

[0017] The remote sensing image of the previous phase to be detected is input into the improved Layer0 and Layer1 in sequence to obtain the first scale feature F1 of the previous phase pre ;

[0018] The first scale feature F1 of the previous phase pre Input to Layer2 to obtain the second scale feature of the previous phase

[0019] The second scale feature of the previous phase Input to Layer3 and Layer4 in turn to obtain the third scale feature F3 of the previous phase pre ;

[0020] The remote sensing image of the later phase to be detected is input into the improved Layer0 and Layer1 in sequence to obtain the first scale feature F1 of the later phase post ;

[0021] The first scale feature F1 of the later phase post Input to Layer2 to obtain the second scale feature of the later phase

[0022] The second scale feature of the later phase Input to Layer3 and Layer4 in turn to obtain the third scale feature F3 of the later phase post .

[0023] Preferably, S2 specifically includes:

[0024] S21: The i-th scale feature F of the previous phase i pre and the i-th scale feature F of the later phase i post Add the corresponding elements to obtain the i-th scale addition feature F i pre +F ipost ; Where i = {1, 2, 3};

[0025] Add the feature F at the i-th scale i pre +F i post Perform global average pooling to obtain the i-th scale global feature p i ;

[0026] The i-th scale global feature p i Input into the convolution layer and Relu activation layer in sequence to obtain the global simplified features of the i-th scale

[0027] The i-th scale global reduction feature Input to the first 1*1 convolutional layer to obtain the channel weight of the i-th scale in the previous phase

[0028] The i-th scale global reduction feature Input to the second 1*1 convolutional layer to obtain the channel weight of the i-th scale in the later phase

[0029] The weight of the channel at the i-th scale in the previous phase is Input to the softmax layer to obtain the normalized channel weight of the i-th scale in the previous phase

[0030] The weight of the channel at the i-th scale in the later phase is Input to the softmax layer to obtain the normalized channel weight of the i-th scale in the later phase

[0031] Normalize the channel weight of the i-th scale of the previous phase and the i-th scale feature F of the previous phase i pre Multiply them together to obtain the semantic interaction features of the i-th scale in the previous phase

[0032] Normalize the channel weight of the i-th scale in the later phase and the i-th scale feature F of the later phase i post Multiply them together to obtain the semantic interaction features of the i-th scale in the later phase

[0033] Preferably, S3 specifically includes:

[0034] The semantic interaction feature of the i-th scale in the previous phase and the semantic interaction features of the i-th scale in the later phase Perform cascade operation to obtain the i-th scale cascade semantic interaction feature F i concat ;

[0035] The i-th scale cascade semantic interaction feature F i concat Input to the channel attention module to obtain the i-th scale channel attention distribution feature W i CAM ;

[0036] The i-th scale channel attention distribution feature W i CAM Cascade semantic interaction feature F with the i-th scale i concat Multiply them together to obtain the i-th scale channel fusion feature F i CAM ;

[0037] Get the semantic interaction features of the i-th scale in the previous phase and the semantic interaction features of the i-th scale in the later phase The absolute value of the difference F i avod ;

[0038] The absolute value of the difference F i avod Input it to the convolution layer and Relu activation layer in turn to obtain the nonlinear difference absolute value F i avod + ;

[0039] The absolute value of the nonlinear difference F i avod+ Input to the spatial attention module to obtain the spatial attention distribution feature W of the i-th scale i SAM ;

[0040] The i-th scale channel attention distribution feature W i CAM and the i-th scale spatial attention distribution feature W i SAM After multiplication, it is input into the convolution layer, batch normalization layer and Relu activation layer in turn to obtain the i-th scale difference feature F i d .

[0041] Preferably, the i-th scale channel attention distribution feature W i CAM Based on the following steps:

[0042] The i-th scale cascade semantic interaction feature F i concatPerform maximum pooling and average pooling respectively;

[0043] The maximum pooling result and the average pooling result are respectively input into the multi-layer perceptron and added, and then input into the Sigmoid function to obtain the attention distribution feature W of the i-th scale channel i CAM .

[0044] Preferably, the i-th scale spatial attention distribution feature W i SAM Based on the following steps:

[0045] The absolute value of the nonlinear difference F i avod+ Perform maximum pooling and average pooling respectively;

[0046] The maximum pooling result and the average pooling result are concatenated in the channel dimension, and then input into the convolution layer and the Sigmoid function in turn to obtain the spatial attention distribution feature W at the i-th scale i SAM .

[0047] Preferably, S4 specifically includes:

[0048] The first scale difference feature F1 d Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1) ;in,

[0049] The first scale difference feature F1 d Input to the upsampling layer to obtain the second intermediate feature F (2) ;

[0050] The second scale difference feature Input to the convolution layer to obtain the third intermediate feature F (3) ;

[0051] The first intermediate feature F (1) and the second scale difference characteristics Multiply them together to obtain the fourth intermediate feature F (4) ;

[0052] The second intermediate feature F (2) With the third intermediate feature F (3) Multiply them together to obtain the fifth intermediate feature F (5) ;

[0053] The fourth intermediate feature F (4) and the fifth intermediate feature F (5) Cascade to obtain the sixth intermediate feature F (6) ;

[0054] The sixth intermediate feature F (6) Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7) ;

[0055] The seventh intermediate feature F (7) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8) ;

[0056] The seventh intermediate feature F (7) Input to the upsampling layer to obtain the ninth intermediate feature F (9) ;

[0057] The third scale difference feature F3 d Input to the convolution layer to obtain the tenth intermediate feature F (10) ;

[0058] The eighth intermediate feature F (8) and the third scale difference feature F3 d Multiply them together to obtain the eleventh intermediate feature F (11) ;

[0059] The ninth intermediate feature F (9) and the tenth intermediate feature F (10) Multiply them together to get the twelfth intermediate feature F (12) ;

[0060] The eleventh intermediate feature F (11) and the twelfth intermediate feature F (12) Cascade to obtain the thirteenth intermediate feature F (13) ;

[0061] The thirteenth intermediate feature F (13) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the difference aggregation feature D agg ;

[0062] The first scale feature F1 of the previous phase pre Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1 ' ) ;

[0063] The first scale feature F1 of the previous phase pre Input to the upsampling layer to obtain the second intermediate feature F (2 ' ) ;

[0064] The second scale feature of the previous phase Input to the convolution layer to obtain the third intermediate feature F (3 ' ) ;

[0065] The first intermediate feature F (1 ' ) and the second scale characteristics of the previous phase Multiply them together to obtain the fourth intermediate feature F (4 ' ) ;

[0066] The second intermediate feature F (2 ' ) With the third intermediate feature F (3 ' ) Multiply them together to obtain the fifth intermediate feature F (5 ' ) ;

[0067] The fourth intermediate feature F (4 ' ) and the fifth intermediate feature F (5 ' ) Cascade to obtain the sixth intermediate feature F (6 ' ) ;

[0068] The sixth intermediate feature F (6 ' ) Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7 ' ) ;

[0069] The seventh intermediate feature F (7 ' ) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8 ' ) ;

[0070] The seventh intermediate feature F (7 ' ) Input to the upsampling layer to obtain the ninth intermediate feature F (9 ' ) ;

[0071] The third scale feature F3 of the previous phase pre Input to the convolution layer to obtain the tenth intermediate feature F (10') ;

[0072] The eighth intermediate feature F (8') and the third scale feature F3 of the previous phase pre Multiply them together to obtain the eleventh intermediate feature F (11 ' ) ;

[0073] The ninth intermediate feature F (9 ' ) and the tenth intermediate feature F (10 ') Multiply them together to get the twelfth intermediate feature F (12 ' ) ;

[0074] The eleventh intermediate feature F (11 ' ) and the twelfth intermediate feature F (12 ' ) Cascade to obtain the thirteenth intermediate feature F (13 ' ) ;

[0075] The thirteenth intermediate feature F (13 ' ) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the front phase aggregation features

[0076] The first scale feature F1 of the later phase post Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1") ;

[0077] The first scale feature F1 of the later phase post Input to the upsampling layer to obtain the second intermediate feature F (2") ;

[0078] The second scale feature of the later phase Input to the convolution layer to obtain the third intermediate feature F (3") ;

[0079] The first intermediate feature F (1") and the second scale characteristics of the later phase Multiply them together to obtain the fourth intermediate feature F (4 " ) ;

[0080] The second intermediate feature F (2 " ) With the third intermediate feature F (3 " ) Multiply them together to obtain the fifth intermediate feature F (5") ;

[0081] The fourth intermediate feature F (4") and the fifth intermediate feature F (5") Cascade to obtain the sixth intermediate feature F (6 " ) ;

[0082] The sixth intermediate feature F (6") Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7") ;

[0083] The seventh intermediate feature F (7") Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8") ;

[0084] The seventh intermediate feature F (7") Input to the upsampling layer to obtain the ninth intermediate feature F (9") ;

[0085] The third scale feature F3 of the later phase post Input to the convolution layer to obtain the tenth intermediate feature F (10") ;

[0086] The eighth intermediate feature F (8") And the third scale feature F3 of the later phase post Multiply them together to obtain the eleventh intermediate feature F (11") ;

[0087] The ninth intermediate feature F (9") and the tenth intermediate feature F (10") Multiply them together to get the twelfth intermediate feature F (12") ;

[0088] The eleventh intermediate feature F (11") and the twelfth intermediate feature F (12") Cascade to obtain the thirteenth intermediate feature F (13") ;

[0089] The thirteenth intermediate feature F (13") Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the post-phase aggregation features

[0090] Preferably, the binary change detection module is formed by stacking five residual blocks in series.

[0091] Preferably, S6 specifically includes:

[0092] Aggregate the features of the previous phase after upsampling convolution Multiply the binary change map by mask to obtain the semantic change map of the previous phase;

[0093] The post-temporal aggregation features after upsampling convolution The mask is multiplied with the binary change map to obtain the post-temporal semantic change map.

[0094] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a remote sensing image change detection method based on semantic interaction and difference representation, which can improve the accuracy of detecting the change position of dual-phase remote sensing images and identifying the type of changed elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0096] Figure 1 A flow chart of a remote sensing image change detection method based on semantic interaction and difference representation provided by the present invention;

[0097] Figure 2 A schematic diagram of the structure of the improved ResNet34 network provided by the present invention;

[0098] Figure 3 A schematic diagram of the structure of the semantic interaction module provided by the present invention;

[0099] Figure 4 A schematic diagram of the structure of the difference representation module provided by the present invention;

[0100] Figure 5 A schematic diagram of the structure of the multi-scale feature aggregation module provided by the present invention;

[0101] Figure 6 This is a structural schematic diagram of the binary change detection module provided by the present invention. DETAILED DESCRIPTION

[0102] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0103] like Figure 1 As shown, the embodiment of the present invention discloses a remote sensing image change detection method based on semantic interaction and difference representation, comprising the following steps:

[0104] S1: Input the dual-temporal remote sensing image to be detected into the improved ResNet34 network to obtain the multi-scale features of the dual-temporal remote sensing image to be detected;

[0105] Among them, Figure 2As shown, the improved ResNet34 network includes improved Layer0, Layer1, Layer2, Layer3 and improved Layer4 connected in sequence; the improved Layer0 includes a convolutional layer, a batch normalization layer, a Relu activation layer and a maximum pooling layer connected in sequence; Layer1, Layer2 and Layer3 are the original ResNet34 network structure; the improved Layer4 is an identity mapping;

[0106] Furthermore, the dual-phase remote sensing image to be detected is normalized before being input into the improved ResNet34 network, and the uint type remote sensing image with a pixel value in the interval [0, 255] is normalized to a double type remote sensing image with a pixel value in the interval [0, 1].

[0107] It can be understood that the identity mapping means that no transformation or operation is performed on the input, and the input is simply returned as the output.

[0108] Figure 2 The two improved ResNet34 networks in share weight parameters.

[0109] S2: Input the multi-scale features of the dual-temporal remote sensing image to be detected into the semantic interaction module to obtain multi-scale dual-temporal semantic features;

[0110] S3: Input the multi-scale bi-temporal semantic features into the difference representation module to obtain multi-scale difference features;

[0111] S4: input the multi-scale difference features into the multi-scale feature aggregation module to obtain difference aggregation features;

[0112] Input the multi-scale features of the dual-temporal remote sensing image to be detected into the multi-scale feature aggregation module to obtain the dual-temporal aggregation features;

[0113] S5: Input the difference aggregation features after upsampling convolution into the binary change detection module to obtain a binary change map;

[0114] S6: Mask-multiply the bi-temporal aggregate features after upsampling convolution with the binary change map to obtain a bi-temporal semantic change map.

[0115] In one embodiment, S1 specifically includes:

[0116] The remote sensing image of the previous phase to be detected is input into the improved Layer0 and Layer1 in sequence to obtain the first scale feature F1 of the previous phase pre ;

[0117] The first scale feature F1 of the previous phase pre Input to Layer2 to obtain the second scale feature of the previous phase

[0118] The second scale feature of the previous phase Input to Layer3 and Layer4 in turn to obtain the third scale feature F3 of the previous phase pre ;

[0119] The remote sensing image of the later phase to be detected is input into the improved Layer0 and Layer1 in sequence to obtain the first scale feature F1 of the later phase post ;

[0120] The first scale feature F1 of the later phase post Input to Layer2 to obtain the second scale feature of the later phase

[0121] The second scale feature of the later phase Input to Layer3 and Layer4 in turn to obtain the third scale feature F3 of the later phase post .

[0122] Specifically: The above process can be expressed by the following formula:

[0123]

[0124] Wherein, x1 and x2 represent the remote sensing image before and after the detection, respectively; Conv represents convolution; BN represents batch normalization; Relu represents Relu activation function; max pool represents maximum pooling.

[0125] In one embodiment, if Figure 3 As shown, S2 specifically includes:

[0126] S21: The i-th scale feature F of the previous phase i pre and the i-th scale feature F of the later phase i post Add the corresponding elements to obtain the i-th scale addition feature F i pre +F i post ; Where i = {1, 2, 3};

[0127] Add the feature F at the i-th scale i pre +F i post Perform global average pooling to obtain the i-th scale global feature p i ;

[0128] The i-th scale global feature pi Input into the convolution layer and Relu activation layer in sequence to obtain the global simplified features of the i-th scale

[0129] The i-th scale global reduction feature Input to the first 1*1 convolutional layer to obtain the channel weight of the i-th scale in the previous phase

[0130] The i-th scale global reduction feature Input to the second 1*1 convolutional layer to obtain the channel weight of the i-th scale in the later phase

[0131] The weight of the channel at the i-th scale in the previous phase is Input to the softmax layer to obtain the normalized channel weight of the i-th scale in the previous phase

[0132] The weight of the channel at the i-th scale in the later phase is Input to the softmax layer to obtain the normalized channel weight of the i-th scale in the later phase

[0133] Normalize the channel weight of the i-th scale of the previous phase and the i-th scale feature F of the previous phase i pre Multiply them together to obtain the semantic interaction features of the i-th scale in the previous phase

[0134] Normalize the channel weight of the i-th scale in the later phase and the i-th scale feature F of the later phase i post Multiply them together to obtain the semantic interaction features of the i-th scale in the later phase

[0135] Specifically, the above process can be expressed by the following formula:

[0136] p i =GAP(F i pre +F i post );

[0137]

[0138] In the formula, GAP represents global average pooling; represents the Relu activation function; w1, w2, and w3 represent the convolution weights of the convolution layer, the first 1*1 convolution layer, and the second 1*1 convolution layer respectively; j represents the channel index, and exp represents the exponential function; C i Represents the number of channels of the i-th scale bi-temporal semantic feature map;

[0139] In one embodiment, if Figure 4 As shown, S3 specifically includes:

[0140] The semantic interaction feature of the i-th scale in the previous phase and the semantic interaction features of the i-th scale in the later phase Perform cascade operation to obtain the i-th scale cascade semantic interaction feature F i concat ;

[0141] The i-th scale cascade semantic interaction feature F i concat Input to the channel attention module to obtain the i-th scale channel attention distribution feature W i CAM ;

[0142] The i-th scale channel attention distribution feature W i CAM Cascade semantic interaction feature F with the i-th scale i concat Multiply them together to obtain the i-th scale channel fusion feature F i CAM ;

[0143] Get the semantic interaction features of the i-th scale in the previous phase and the semantic interaction features of the i-th scale in the later phase The absolute value of the difference F i avod ;

[0144] The absolute value of the difference F i avod Input it to the convolution layer (1*1 convolution layer) and Relu activation layer in sequence to obtain the nonlinear difference absolute value F i avod+ ;

[0145] The absolute value of the nonlinear difference F i avod+ Input to the spatial attention module to obtain the spatial attention distribution feature W at the i-th scale i SAM ;

[0146] The i-th scale channel attention distribution feature W i CAM and the i-th scale spatial attention distribution feature W i SAM After multiplication, the data is input into the convolution layer (1*1 convolution layer), batch normalization layer and Relu activation layer in turn to obtain the i-th scale difference feature F id ;

[0147] Specifically, the above process can be expressed by the following formula:

[0148]

[0149] W i CAM =CAM(F i Concat );

[0150] F i CAM =F i Concat ·W i CAM ;

[0151]

[0152] F i avod+ =Relu(Conv 1*1 (F i avod ));

[0153] W i SAM =SAM(F i avod+ );

[0154] F i d =Relu(BN(Conv 1*1 (F i CAM ·W i SAM )));

[0155] In the formula, Conv 1*1 Represents 1*1 convolution; CAM represents channel attention; SAM represents spatial attention; Concat represents cascade operation; BN represents batch normalization; Relu represents Relu activation function.

[0156] In one embodiment, the i-th scale channel attention distribution feature W i CAM Based on the following steps:

[0157] The i-th scale cascade semantic interaction feature F i concat Perform maximum pooling and average pooling respectively;

[0158] The maximum pooling result and the average pooling result are respectively input into the multi-layer perceptron and added, and then input into the Sigmoid function to obtain the attention distribution feature W of the i-th scale channel i CAM .

[0159] Specifically, the above process can be expressed by the following formula:

[0160] W i CAM =σ(MLP(Avg pool(F i Concat ))+MLP(Max pool(F i Concat )));

[0161] In the formula, Avg pool represents the average pooling operation; Maxpool represents the maximum pooling operation; MLP represents the multi-layer perceptron; σ represents the Sigmoid function.

[0162] In one embodiment, the i-th scale spatial attention distribution feature W i SAM Based on the following steps:

[0163] The absolute value of the nonlinear difference F i avod+ Perform maximum pooling and average pooling respectively;

[0164] The maximum pooling result and the average pooling result are concatenated in the channel dimension, and then input into the convolution layer (7*7 convolution layer) and the Sigmoid function in turn to obtain the spatial attention distribution feature W at the i-th scale i SAM ;

[0165] Specifically, the above process can be expressed by the following formula:

[0166] W i SAM =σ(Conv 7*7 (Cat[Avgpool(F i avod+ );Maxpool(F i avod+ )]));

[0167] Where, σ represents the Sigmoid function; Conv 7*7 Represents a 7*7 convolution operation; Cat represents channel dimension concatenation; Avgpool represents an average pooling operation; Maxpool represents a maximum pooling operation.

[0168] In one embodiment, if Figure 5As shown, S4 specifically includes:

[0169] The first scale difference feature F1 d Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1) ;in,

[0170] The first scale difference feature F1 d Input to the upsampling layer to obtain the second intermediate feature F (2) ;

[0171] The second scale difference feature Input to the convolution layer to obtain the third intermediate feature F (3) ;

[0172] The first intermediate feature F (1) and the second scale difference characteristics Multiply them together to obtain the fourth intermediate feature F (4) ;

[0173] The second intermediate feature F (2) With the third intermediate feature F (3) Multiply them together to obtain the fifth intermediate feature F (5) ;

[0174] The fourth intermediate feature F (4) and the fifth intermediate feature F (5) Cascade to obtain the sixth intermediate feature F (6) ;

[0175] The sixth intermediate feature F (6) Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7) ;

[0176] The seventh intermediate feature F (7) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8) ;

[0177] The seventh intermediate feature F (7) Input to the upsampling layer to obtain the ninth intermediate feature F (9) ;

[0178] The third scale difference feature F3 d Input to the convolution layer to obtain the tenth intermediate feature F (10) ;

[0179] The eighth intermediate feature F (8) and the third scale difference feature F3 d Multiply them together to obtain the eleventh intermediate feature F (11) ;

[0180] The ninth intermediate feature F (9) and the tenth intermediate feature F (10) Multiply them together to get the twelfth intermediate feature F (12) ;

[0181] The eleventh intermediate feature F (11) and the twelfth intermediate feature F (12) Cascade to obtain the thirteenth intermediate feature F (13) ;

[0182] The thirteenth intermediate feature F (13) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the difference aggregation feature D agg ;

[0183] The first scale feature F1 of the previous phase pre Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1 ' ) ;

[0184] The first scale feature F1 of the previous phase pre Input to the upsampling layer to obtain the second intermediate feature F (2 ' ) ;

[0185] The second scale feature of the previous phase Input to the convolution layer to obtain the third intermediate feature F (3 ' ) ;

[0186] The first intermediate feature F (1 ' ) and the second scale characteristics of the previous phase Multiply them together to obtain the fourth intermediate feature F (4 ' ) ;

[0187] The second intermediate feature F (2') With the third intermediate feature F (3') Multiply them together to obtain the fifth intermediate feature F (5') ;

[0188] The fourth intermediate feature F (4') and the fifth intermediate feature F (5') Cascade to obtain the sixth intermediate feature F (6') ;

[0189] The sixth intermediate feature F (6') Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7') ;

[0190] The seventh intermediate feature F(7 ' ) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8 ' ) ;

[0191] The seventh intermediate feature F (7 ' ) Input to the upsampling layer to obtain the ninth intermediate feature F (9 ' ) ;

[0192] The third scale feature F3 of the previous phase pre Input to the convolution layer to obtain the tenth intermediate feature F (10 ' ) ;

[0193] The eighth intermediate feature F (8 ' ) and the third scale feature F3 of the previous phase pre Multiply them together to obtain the eleventh intermediate feature F (11 ' ) ;

[0194] The ninth intermediate feature F (9 ' ) and the tenth intermediate feature F (10 ' ) Multiply them together to get the twelfth intermediate feature F (12 ' ) ;

[0195] The eleventh intermediate feature F (11 ' ) and the twelfth intermediate feature F (12 ' ) Cascade to obtain the thirteenth intermediate feature F (13 ' ) ;

[0196] The thirteenth intermediate feature F (13 ' ) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the front phase aggregation features

[0197] The first scale feature F1 of the later phase post Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1") ;

[0198] The first scale feature F1 of the later phase post Input to the upsampling layer to obtain the second intermediate feature F (2 " ) ;

[0199] The second scale feature of the later phase Input to the convolution layer to obtain the third intermediate feature F (3") ;

[0200] The first intermediate feature F (1") and the second scale characteristics of the later phase Multiply them together to obtain the fourth intermediate feature F (4") ;

[0201] The second intermediate feature F (2") With the third intermediate feature F (3") Multiply them together to obtain the fifth intermediate feature F (5") ;

[0202] The fourth intermediate feature F (4") and the fifth intermediate feature F (5") Cascade to obtain the sixth intermediate feature F (6") ;

[0203] The sixth intermediate feature F (6") Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7") ;

[0204] The seventh intermediate feature F (7") Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8") ;

[0205] The seventh intermediate feature F (7") Input to the upsampling layer to obtain the ninth intermediate feature F (9 " ) ;

[0206] The third scale feature F3 of the later phase post Input to the convolution layer to obtain the tenth intermediate feature F (10") ;

[0207] The eighth intermediate feature F (8") And the third scale feature F3 of the later phase post Multiply them together to obtain the eleventh intermediate feature F (11") ;

[0208] The ninth intermediate feature F (9") and the tenth intermediate feature F (10") Multiply them together to get the twelfth intermediate feature F (12") ;

[0209] The eleventh intermediate feature F (11") and the twelfth intermediate feature F (12") Cascade to obtain the thirteenth intermediate feature F (13") ;

[0210] The thirteenth intermediate feature F (13") Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the post-phase aggregation features

[0211] Specifically, the above process can be expressed by the following formula:

[0212]

[0213] D agg =Conv(Concat(Up(Conv(F (7) ))·F3 d ,Conv(F3 d )·Up(F (7) )));

[0214]

[0215] In the formula, Concat represents a cascade operation; Up represents upsampling; · represents the Hadamard product; and Conv represents a convolution operation.

[0216] In one embodiment, the binary change detection module is formed by stacking five residual blocks in series.

[0217] In one embodiment, if Figure 6 As shown, S6 specifically includes:

[0218] Aggregate the features of the previous phase after upsampling convolution Multiply the binary change map by mask to obtain the semantic change map of the previous phase;

[0219] The post-temporal aggregation features after upsampling convolution The mask is multiplied with the binary change map to obtain the post-temporal semantic change map.

[0220] Finally, it should be noted that:

[0221] The present invention selects a public remote sensing image semantic change detection dataset (SECOND dataset) as the original remote sensing image data, which contains a total of g groups of high-resolution optical remote sensing images, each group has 2 p-pixel × p-pixel high-resolution optical images and their respective corresponding semantic change labels; the original remote sensing image data is divided into training samples and test samples.

[0222] Then the training samples and test samples are normalized;

[0223] Specifically: normalize the uint type training samples and test samples with pixel values ​​in the range [0,255] to double type training samples and test samples with pixel values ​​in the range [0,1].

[0224] Perform data augmentation on the normalized training and test samples;

[0225] Specifically: data enhancement is performed by means of rotation, translation, scaling, cropping, flipping, etc.

[0226] The improved ResNet34 network, semantic interaction module, difference representation module, multi-scale feature aggregation module, and binary change detection module are trained and tested using the enhanced training samples and test samples to obtain the improved ResNet34 network, semantic interaction module, difference representation module, multi-scale feature aggregation module, and binary change detection module in S1.

[0227] Finally, Table 1 shows the evaluation indicators of this method and other semantic change detection methods on the SECOND dataset. The bold data represent the optimal values ​​of this evaluation indicator in different methods. It can be seen that the indicators of this method basically exceed those of other methods, including mIoU, Sek, F scd The indicators reached 73.03%, 22.16% and 61.70% respectively. This shows that this method can well model the semantic information of the distant context and detect the change category of interest by extracting and identifying complex features. Compared with other methods, the remote sensing image change detection method based on semantic interaction and difference representation proposed in this invention has more advantages for semantic change detection of remote sensing images.

[0228] Table 1

[0229]

[0230] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0231] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing image change detection method based on semantic interaction and difference representation, characterized in that: The following steps are involved: S1: Input the dual-temporal remote sensing image to be detected into the improved ResNet34 network to obtain the multi-scale features of the dual-temporal remote sensing image to be detected; Among them, the improved ResNet34 network includes the improved Layer0, Layer1, Layer2, Layer3 and improved Layer4 connected in sequence; the improved Layer0 includes the convolution layer, batch normalization layer, Relu activation layer and maximum pooling layer connected in sequence; Layer1, Layer2 and Layer3 are the original ResNet34 network structure; the improved Layer4 is an identity mapping; S2: Input the multi-scale features of the dual-temporal remote sensing image to be detected into the semantic interaction module to obtain multi-scale dual-temporal semantic features; S3: Input the multi-scale bi-temporal semantic features into the difference representation module to obtain multi-scale difference features; S4: input the multi-scale difference features into the multi-scale feature aggregation module to obtain difference aggregation features; Input the multi-scale features of the dual-temporal remote sensing image to be detected into the multi-scale feature aggregation module to obtain the dual-temporal aggregation features; S5: Input the difference aggregation features after upsampling convolution into the binary change detection module to obtain a binary change map; S6: Mask-multiply the bi-temporal aggregate features after upsampling convolution with the binary change map to obtain a bi-temporal semantic change map.

2. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 1, characterized in that: S1 specifically includes: The remote sensing image of the previous phase to be detected is input into the improved Layer0 and Layer1 in sequence to obtain the first scale feature F1 of the previous phase pre ; The first scale feature F1 of the previous phase pre Input to Layer2 to obtain the second scale feature of the previous phase The second scale feature of the previous phase Input to Layer3 and Layer4 in turn to obtain the third scale feature F3 of the previous phase pre ; The remote sensing image of the later phase to be detected is input into the improved Layer0 and Layer1 in sequence to obtain the first scale feature F1 of the later phase post ; The first scale feature F1 of the later phase post Input to Layer2 to obtain the second scale feature of the later phase The second scale feature F2 of the later phase post Input to Layer3 and Layer4 in turn to obtain the third scale feature F3 of the later phase post .

3. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 2 is characterized in that: S2 specifically includes: S21: The i-th scale feature F of the previous phase i pre and the i-th scale feature F of the later phase i post Add the corresponding elements to obtain the i-th scale addition feature F i pre +F i post ; where i = {1, 2, 3}; F i pre ={F1 pre ,F2 pre ,F3 pre }; F i post ={F1 post ,F2 post ,F3 post }; Add the feature F at the i-th scale i pre +F i post Perform global average pooling to obtain the i-th scale global feature p i ; The i-th scale global feature p i Input into the convolution layer and Relu activation layer in sequence to obtain the global simplified features of the i-th scale The i-th scale global reduction feature Input to the first 1*1 convolutional layer to obtain the channel weight of the i-th scale in the previous phase The i-th scale global reduction feature Input to the second 1*1 convolutional layer to obtain the channel weight of the i-th scale in the later phase The weight of the channel at the i-th scale in the previous phase is Input to the softmax layer to obtain the normalized channel weight of the i-th scale in the previous phase The weight of the channel at the i-th scale in the later phase is Input to the softmax layer to obtain the normalized channel weight of the i-th scale in the later phase Normalize the channel weight of the i-th scale of the previous phase and the i-th scale feature F of the previous phase i pre Multiply them together to obtain the semantic interaction features of the i-th scale in the previous phase Normalize the channel weight of the i-th scale in the later phase and the i-th scale feature F of the later phase i post Multiply them together to obtain the semantic interaction features of the i-th scale in the later phase 4. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 3 is characterized in that: S3 specifically includes: The semantic interaction feature of the i-th scale in the previous phase and the semantic interaction features of the i-th scale in the later phase Perform cascade operation to obtain the i-th scale cascade semantic interaction feature F i concat ; The i-th scale cascade semantic interaction feature F i concat Input to the channel attention module to obtain the i-th scale channel attention distribution feature W i CAM ; The i-th scale channel attention distribution feature W i CAM Cascade semantic interaction feature F with the i-th scale i concat Multiply them together to obtain the i-th scale channel fusion feature F i CAM ; Get the semantic interaction features of the i-th scale in the previous phase and the semantic interaction features of the i-th scale in the later phase The absolute value of the difference F i avod ; The absolute value of the difference F i avod Input it to the convolution layer and Relu activation layer in turn to obtain the nonlinear difference absolute value F i avod+ ; The absolute value of the nonlinear difference F i avod+ Input to the spatial attention module to obtain the spatial attention distribution feature W of the i-th scale i SAM ; The i-th scale channel attention distribution feature W i CAM and the i-th scale spatial attention distribution feature W i SAM After multiplication, it is input into the convolution layer, batch normalization layer and Relu activation layer in turn to obtain the i-th scale difference feature F i d .

5. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 4, characterized in that: The i-th scale channel attention distribution feature W i CAM Based on the following steps: The i-th scale cascade semantic interaction feature F i concat Perform maximum pooling and average pooling respectively; The maximum pooling result and the average pooling result are respectively input into the multi-layer perceptron and added, and then input into the Sigmoid function to obtain the attention distribution feature W of the i-th scale channel i CAM .

6. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 4 is characterized in that: The spatial attention distribution feature W at the i-th scale i SAM Based on the following steps: The absolute value of the nonlinear difference F i avod+ Perform maximum pooling and average pooling respectively; The maximum pooling result and the average pooling result are concatenated in the channel dimension, and then input into the convolution layer and the Sigmoid function in turn to obtain the spatial attention distribution feature W at the i-th scale i SAM .

7. A remote sensing image change detection method based on semantic interaction and difference representation according to any one of claims 4 to 6, characterized in that: S4 specifically includes: The first scale difference feature F1 d Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1) ; Among them, F i d ={F1 d ,F2 d ,F3 d }; The first scale difference feature F1 d Input to the upsampling layer to obtain the second intermediate feature F (2) ; The second scale difference feature Input to the convolution layer to obtain the third intermediate feature F (3) ; The first intermediate feature F (1) and the second scale difference characteristics Multiply them together to obtain the fourth intermediate feature F (4) ; The second intermediate feature F (2) With the third intermediate feature F (3) Multiply them together to obtain the fifth intermediate feature F (5) ; The fourth intermediate feature F (4) and the fifth intermediate feature F (5) Cascade to obtain the sixth intermediate feature F (6) ; The sixth intermediate feature F (6) Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7) ; The seventh intermediate feature F (7) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8) ; The seventh intermediate feature F (7) Input to the upsampling layer to obtain the ninth intermediate feature F (9) ; The third scale difference feature F3 d Input to the convolution layer to obtain the tenth intermediate feature F (10) ; The eighth intermediate feature F (8) and the third scale difference feature F3 d Multiply them together to obtain the eleventh intermediate feature F (11) ; The ninth intermediate feature F (9) and the tenth intermediate feature F (10) Multiply them together to get the twelfth intermediate feature F (12) ; The eleventh intermediate feature F (11) and the twelfth intermediate feature F (12) Cascade to obtain the thirteenth intermediate feature F (13) ; The thirteenth intermediate feature F (13) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the difference aggregation feature D agg ; The first scale feature F1 of the previous phase pre Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1 ' ) ; The first scale feature F1 of the previous phase pre Input to the upsampling layer to obtain the second intermediate feature F (2 ' ) ; The second scale feature F2 of the previous phase pre Input to the convolution layer to obtain the third intermediate feature F (3 ' ) ; The first intermediate feature F (1 ' ) and the second scale feature F2 of the previous phase pre Multiply them together to obtain the fourth intermediate feature F (4 ' ) ; The second intermediate feature F (2 ' ) With the third intermediate feature F (3 ' ) Multiply them together to obtain the fifth intermediate feature F (5 ' ) ; The fourth intermediate feature F (4 ' ) and the fifth intermediate feature F (5 ' ) Cascade to obtain the sixth intermediate feature F (6 ' ) ; The sixth intermediate feature F (6 ' ) Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7 ' ) ; The seventh intermediate feature F (7 ' ) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8 ' ) ; The seventh intermediate feature F (7 ' ) Input to the upsampling layer to obtain the ninth intermediate feature F (9 ' ) ; The third scale feature F3 of the previous phase pre Input to the convolution layer to obtain the tenth intermediate feature F (10 ' ) ; The eighth intermediate feature F (8 ' ) and the third scale feature F3 of the previous phase pre Multiply them together to obtain the eleventh intermediate feature F (11 ' ) ; The ninth intermediate feature F (9 ' ) and the tenth intermediate feature F (10 ' ) Multiply them together to get the twelfth intermediate feature F (12 ' ) ; The eleventh intermediate feature F (11 ' ) and the twelfth intermediate feature F (12 ' ) Cascade to obtain the thirteenth intermediate feature F (13 ' ) ; The thirteenth intermediate feature F (13 ' ) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the front phase aggregation features The first scale feature F1 of the later phase post Input into the convolution layer and upsampling layer in sequence to obtain the first intermediate feature F (1 " ) ; The first scale feature F1 of the later phase post Input to the upsampling layer to obtain the second intermediate feature F (2 " ) ; The second scale feature F2 of the later phase post Input to the convolution layer to obtain the third intermediate feature F (3 " ) ; The first intermediate feature F (1") and the second scale feature F2 of the later phase post Multiply them together to obtain the fourth intermediate feature F (4") ; The second intermediate feature F (2") With the third intermediate feature F (3 " ) Multiply them together to obtain the fifth intermediate feature F (5") ; The fourth intermediate feature F (4 " ) and the fifth intermediate feature F (5 " ) Cascade to obtain the sixth intermediate feature F (6 " ) ; The sixth intermediate feature F (6 " ) Input it to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the seventh intermediate feature F (7 " ) ; The seventh intermediate feature F (7 " ) Input into the convolution layer and upsampling layer in turn to obtain the eighth intermediate feature F (8 " ) ; The seventh intermediate feature F (7 " ) Input to the upsampling layer to obtain the ninth intermediate feature F (9 " ) ; The third scale feature F3 of the later phase post Input to the convolution layer to obtain the tenth intermediate feature F (10 " ) ; The eighth intermediate feature F (8 " ) And the third scale feature F3 of the later phase post Multiply them together to obtain the eleventh intermediate feature F (11 " ) ; The ninth intermediate feature F (9 " ) and the tenth intermediate feature F (10 " ) Multiply them together to get the twelfth intermediate feature F (12 " ) ; The eleventh intermediate feature F (11 " ) and the twelfth intermediate feature F (12 " ) Cascade to obtain the thirteenth intermediate feature F (13 " ) ; The thirteenth intermediate feature F (13 " ) Input to the convolution layer, batch normalization layer and Relu activation layer in sequence to obtain the post-phase aggregation features 8. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 7, characterized in that: The binary change detection module is composed of five residual blocks stacked in series.

9. The remote sensing image change detection method based on semantic interaction and difference representation according to claim 8, characterized in that: S6 specifically includes: Aggregate the features of the previous phase after upsampling convolution Multiply the binary change map by mask to obtain the semantic change map of the previous phase; The post-temporal aggregation features after upsampling convolution The mask is multiplied with the binary change map to obtain the post-temporal semantic change map.

Images