Remote Sensing Image Semantic Change Detection Method and System Incorporating Triple Attention Mechanism

Abstract

The present invention discloses a remote sensing image semantic change detection method and system integrating a triple attention mechanism, including: dividing remote sensing image data of a research area into image patches, and generating training, validation, and test data sets after preprocessing; constructing a remote sensing image semantic change detection model, including an image change detection network and an image semantic segmentation network introducing a triple attention mechanism module; training the model, calculating the losses between the output of the model and the change detection pixel-level labels and the semantic segmentation pixel-level labels, and using an adaptive learning weight to set a joint loss function for the change detection task and the semantic segmentation task; inputting the test data into the model to obtain a binary change map and two semantic mapping maps, and obtaining a corresponding semantic change map after image masking. The present invention can obtain specific change information of the image, and the introduced triple attention can more effectively obtain the significant feature information of the target, thereby improving the performance of image semantic change detection.

Description

Technical Field

[0001] The present invention relates to a remote sensing image semantic change detection method and system integrating a triple attention mechanism, belonging to the fields of spatial information science and computer vision. Background Art

[0002] Remote sensing change detection is a method of obtaining dynamic changes in various surface information through comparison and various processing means using the information reflected by multiple static remote sensing images. The object of remote sensing change detection is images of the same area at different times. Remote sensing semantic change detection studies how the image pairs have changed based on the changes in the image pairs. Most of the existing deep learning-based change detection methods rely on a large amount of labeled data. The subjectivity of annotating change detection data sets is too large. There must be many unreliable labels in the change detection data sets with large-scale annotation, and the area where changes occur in the image pairs is small, resulting in low utilization rate of remote sensing image data. And in the image, pixel blocks that are spatially adjacent are likely to have a greater impact on each other, but pixel blocks that are far apart in the image space but close in the feature space will also have a great impact on each other. Therefore, it is necessary to further explore the image pairs from the feature perspective. Summary of the Invention

[0003] Object of the Invention: Aiming at the deficiencies of the prior art, the object of the present invention is to provide a remote sensing image semantic change detection method and system integrating a triple attention mechanism to explore the semantic change information of image pairs at the same location at different times with higher efficiency.

[0004] Technical Solution: To achieve the above object of the invention, a remote sensing image semantic change detection method integrating a triple attention mechanism provided by the present invention includes the following steps:

[0005] Step 1: After preprocessing the remote sensing image data, obtain dual-temporal change detection task data and corresponding change detection pixel-level labels, as well as semantic segmentation task data and corresponding semantic segmentation pixel-level labels, and divide the two task data into training, validation, and test data sets;

[0006] Step 2: Construct a remote sensing image semantic change detection model based on a fused triple attention mechanism: The model includes an image change detection network with a triple attention mechanism module introduced and an image semantic segmentation network; the triple attention mechanism module fuses image channel attention, neighboring pixel block attention, and similar pixel block attention; the image change detection network includes two encoders and a decoder. The difference between the feature maps output by the corresponding convolutional modules in the two encoders is input into the decoder after passing through an attention module, and a binary change map is output through the decoder; the difference between the feature maps passes through a channel attention module, a neighboring pixel block attention module, and a similar pixel block attention module in sequence; the channel attention module performs channel attention learning and then restores to the original scale. The neighboring pixel block attention module takes pixel blocks as the research object, learns the correlation between spatially neighboring pixel blocks, and then restores to the original scale. The similar pixel block attention module takes pixel blocks as the research object, clusters the pixel blocks, and performs attention learning on similar pixel blocks and then restores to the original scale;

[0007] Step 3: Train the remote sensing image semantic change detection model based on the fused triple attention mechanism, calculate the loss between the output of the model and the change detection pixel-level label and the semantic segmentation pixel-level label, and use an adaptive learning weight to set the joint loss function for the change detection task and the semantic segmentation task;

[0008] Step 4: Test the remote sensing image semantic change detection model based on the fused triple attention mechanism. After inputting the sample data into the model, a binary change map and two semantic mapping maps are obtained. The binary change map and the two semantic mapping maps are subjected to image masking respectively to obtain the corresponding semantic change maps.

[0009] Preferably, the step 1 includes the following steps:

[0010] Step 11: Divide the original dual-temporal remote sensing image data into image blocks of a set pixel size to obtain change detection task data and corresponding change detection labels; the change detection task data is multiple pairs of original image pairs at the same location but different times, and the label is 0 / 1, that is, the label for the image pair without change is 0, and the label for the changed image pair is 1; the ratio of the training set, validation set, and test data set is 8:1:1. The test data set stores the remote sensing image data to be detected, that is, multiple pairs of original image pairs at the same location but different times;

[0011] Step 12: Divide the dual-temporal remote sensing image data into image blocks of a set pixel size to obtain semantic segmentation task data and corresponding pixel-level semantic segmentation labels; in the semantic segmentation task, no separate test data set is set, and the ratio of the training set to the validation data set is 8:2.

[0012] Preferably, in step 2, the image change detection network consists of two encoders and one decoder. Encoder 1 contains three convolutional modules and three pooling layers, and encoder 2 contains three convolutional modules and two pooling layers; the two encoders respectively reduce the dimensions of the dual-temporal images; the decoder contains three convolutional modules and three deconvolutional layers; the difference between the feature maps output by each convolutional module in the two encoders is introduced into a triple attention mechanism module and then input into the decoder, and the decoder raises the dimension of the feature map and outputs a binary change map; the image semantic segmentation network contains one encoder and one decoder. The input of the encoder is the dual-temporal remote sensing image data with merged channels, and the output is the image feature map; the input of the decoder is the image feature map. After the last convolutional layer in the decoder, the output feature map is divided into two halves to obtain two corresponding semantic map; the binary change map is respectively subjected to image masking with the two semantic maps, that is, the binary change map is respectively projected onto the two semantic maps to obtain two semantic change label maps.

[0013] Preferably, in step 2, the triple attention mechanism (CASA) module consists of one channel attention (CA) module, one adjacent pixel block attention (APBA) module, one similar pixel block attention (SPBA) module, and three layer normalization (LN) modules. The specific calculation steps are as follows:

[0014] The difference between the feature maps output by each convolutional module in the two encoders is used as the input feature map in of the triple attention mechanism module. The feature map is subjected to average pooling to obtain a 1×1×C feature vector, and then the relationships between multiple adjacent channels are calculated, and then channel attention learning is performed through the Sigmoid function and upsampled to the original scale; where C is the number of channels; the input feature map in passes through an LN module and a CA module, and the overall calculation process is expressed as: F CA = in + CA(LN(in));

[0015] The feature map F CA after passing through the channel attention module is sliced into N pixel blocks of the same size. Taking the pixel block as the smallest research object, the correlation between adjacent pixel blocks in space is studied and finally upsampled to the original scale; use to represent the set of input pixel blocks, where The calculation process is as follows: <000>

[0016]

[0017] Among them, y ei represents the adjacent attention feature encoding result of pixel block e i , represents the central position coordinate of pixel block e i , represents the central position coordinate of the surrounding pixel block, and m represents pixel block ei The set of pixel blocks within the neighborhood is the Euclidean distance of the pixel block represents the average eigenvalue of the input of surrounding pixel blocks, g αm (·) is the Gaussian function; after each pixel block obtains the neighboring attention feature encoding value, this encoding value is assigned to each pixel point in the pixel block to obtain the neighboring encoding map; in the calculation of neighboring pixel block attention, the output F of the CA module CA is input into another LN module and an APBA module, and the overall expression is:

[0018] W APBA = F CA + APBA(LN(F CA ))

[0019] The feature map F that has passed through the neighboring pixel block attention module CA is sliced into N pixel blocks of the same size, and the pixel blocks with similar feature contents are clustered using a clustering function. A feature similarity value between 0 and 1 is assigned to each pixel block according to the feature category, and then the feature similarity value is assigned to each pixel point in the pixel block to obtain the feature encoding map. Finally, the input feature map and the feature encoding map are multiplied element-wise, and the feature encoding map is upsampled to the original scale; in the similar pixel block attention module, the output W of the APBA module APBA is input into an LN module and an SPBA module, and the overall expression is:

[0020] W SPBA = W APBA + SPBA(LN(W APBA ))

[0021] W SPBA is the feature map output by the similar pixel block attention module

[0022] Preferably, in step 2, the K-means++ algorithm is used to cluster the pixel blocks with similar feature contents, and the clustering process includes:

[0023] 1) Initialize K clustering centers: ① represents the set of input pixel blocks, and a pixel block is randomly selected from the set as the first clustering center; ② for each pixel block in the dataset, calculate its distance D(e i ) from the nearest clustering center, and then calculate the probability p(e i ) that each sample is selected as the next clustering center:

[0024]

[0025] Then, select the next clustering center by the roulette method; ③ Repeat the previous step until K clustering centers are selected;

[0026] 2) Calculate the distances between each pixel block and each clustering center;

[0027] 3) Assign the pixel blocks to the nearest clustering center to form K clusters;

[0028] 4) Calculate the average value of all points in each cluster and use it as the clustering center;

[0029] 5) Repeat steps 2), 3), and 4) until the cluster centers remain unchanged or reach the set number of iterations or the set error tolerance range.

[0030] Preferably, in step 3, the change detection network and the semantic segmentation network are trained respectively, and the training process is as follows: sample the training tasks of the change detection network, and select the same number of changed image pairs and unchanged image pairs according to the change detection labels in the training set; perform data augmentation on the training sample images and their labels, and perform horizontal transformation, vertical flipping transformation, and counterclockwise rotation on the images respectively; after normalizing the samples, input them into the semantic change detection model and train the model.

[0031] Preferably, three loss functions are set in step 3: the change detection loss L cd , the semantic segmentation loss L ss and the joint loss L sc . Merge the semantic mapping maps P1 and P2, mark the same places as 1 and the different places as 0, thus generating a binary change label map L; the change loss L cd is the binary cross-entropy loss between the binary change map C predicted by the change prediction network and the binary change label map L; the semantic loss L ss is the multi-class cross-entropy loss between the semantic mapping maps P1 and P2 and the semantic change maps S1 and S2; during the model training process, adaptively adjust the loss function weights of the two tasks to obtain the joint loss function, and the core formula of L sc is:

[0032]

[0033] where σ1 and σ2 are parameters that the model needs to learn, and L ss1 , L ss2 are the multi-class cross-entropy losses between the semantic mapping map P1 and the semantic change map S1, and between the semantic mapping map P2 and the semantic change map S2 respectively.

[0034] Based on the same inventive concept, a remote sensing image semantic change detection system incorporating a triple attention mechanism provided by the present invention includes:

[0035] A preprocessing module, which is used to preprocess remote sensing image data to obtain dual-temporal change detection task data and corresponding change detection pixel-level labels, as well as semantic segmentation task data and corresponding semantic segmentation pixel-level labels, and divide the two task data into training, validation, and test data sets;

[0036] A model construction module, which is used to construct a remote sensing image semantic change detection model based on a fused triple attention mechanism: the model includes an image change detection network introducing a triple attention mechanism module and an image semantic segmentation network; the triple attention mechanism module fuses image channel attention, neighboring pixel block attention, and similar pixel block attention; the image change detection network includes two encoders and a decoder, and the difference between the feature maps output by the corresponding convolutional modules in the two encoders is input into the decoder after introducing an attention module, and a binary change map is output through the decoder; the difference of the feature maps sequentially passes through a channel attention module, a neighboring pixel block attention module, and a similar pixel block attention module; the channel attention module performs channel attention learning and then restores to the original scale, the neighboring pixel block attention module takes pixel blocks as the research object, learns the correlation between spatially neighboring pixel blocks and then restores to the original scale, and the similar pixel block attention module takes pixel blocks as the research object, clusters the pixel blocks, and performs attention learning on similar pixel blocks and then restores to the original scale;

[0037] A model training module, which is used to train the remote sensing image semantic change detection model based on the fused triple attention mechanism, calculate the losses between the output of the model and the change detection pixel-level labels and the semantic segmentation pixel-level labels, and use an adaptive learning weight to set the joint loss function for the change detection task and the semantic segmentation task;

[0038] And a testing module, which is used to test the remote sensing image semantic change detection model based on the fused triple attention mechanism. After inputting sample data into the model, a binary change map and two semantic mapping maps are obtained. After the binary change map is masked with the two semantic mapping maps respectively, corresponding semantic change maps are obtained.

[0039] Based on the same inventive concept, a computer system provided by the present invention includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the remote sensing image semantic change detection method based on the fused triple attention mechanism are implemented.

[0040] Based on the same inventive concept, a computer-readable storage medium provided by the present invention stores a computer program, and when the computer program is executed by a processor, the steps of the remote sensing image semantic change detection method based on the fused triple attention mechanism are implemented.

[0041] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. The present invention combines the two tasks of change detection and semantic segmentation, and studies how the image pair has changed on the basis of the change of the image pair. 2. Most of the existing deep learning-based change detection methods rely on a large amount of labeled data. The subjectivity of annotating the change detection data set is too large, and there must be many unreliable labels. Moreover, the changed area in the image pair is small, resulting in low utilization rate of remote sensing image data. To solve this problem, the present invention uses a weakly supervised method, that is, the change label of the image pair is 0 / 1, and only the information of change or no change is used. 3. For the two tasks of change detection and semantic segmentation, the present invention adopts two separate networks for training, reducing the computational amount in the training process. 4. In the image, pixel blocks that are spatially adjacent are likely to have a greater impact on each other. However, pixel blocks that are far apart in the image space but close in the feature space will also have a great impact on each other. Therefore, the present invention further explores the image pair from the feature perspective. A triple attention mechanism that combines image channel attention, adjacent pixel block attention, and similar pixel block attention is proposed, and a clustering function is used to cluster pixel pairs with similar features. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of an embodiment of the present invention.

[0043] Figure 2 It is a structural diagram of a semantic change detection model in an embodiment of the present invention.

[0044] Figure 3 It is a structural diagram of an encoder and a decoder of a change detection network in an embodiment of the present invention.

[0045] Figure 4 It is a structural diagram of three attention mechanisms in an embodiment of the present invention.

[0046] Figure 5 It is a diagram of a channel attention module in an embodiment of the present invention.

[0047] Figure 6 It is a diagram of an adjacent pixel block attention module in an embodiment of the present invention.

[0048] Figure 7 It is a diagram of a similar pixel block attention module in an embodiment of the present invention.

[0049] Figure 8 It is a schematic diagram of loss function calculation in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0050] The present invention will be further clarified below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, those skilled in the art's various equivalent modifications of the present invention fall within the scope defined by the appended claims of this application.

[0051] As Figure 1 shown, a remote sensing image semantic change detection method integrating a triple attention mechanism disclosed in an embodiment of the present invention mainly includes the following steps

[0052] Step 1: Divide the original multi-temporal data of remote sensing images into image patches according to a certain pixel size to obtain change detection task data and corresponding change detection labels. Divide the bi-temporal remote sensing image data into image patches according to a certain pixel size to obtain semantic segmentation task data and corresponding pixel-level semantic segmentation labels. After preprocessing, divide the two task data into training, validation, and test data sets according to a certain ratio. The test data set stores the remote sensing image data to be detected, that is, multiple pairs of original image pairs at the same location but different times.

[0053] Step 1 includes the following steps:

[0054] Step 11: Divide the original bi-temporal data of remote sensing images into image patches of a set pixel size (such as 256 pixel × 256 pixel) to obtain change detection task data and corresponding change detection labels. The change detection task data is multiple pairs of original image pairs at the same location but different times, and the label is 0 / 1, that is, the label for the image pair without change is 0, and the label for the changed image pair is 1. In this embodiment, the ratio of the training set, validation set, and test data set is 8:1:1, and the test data set stores the remote sensing image data to be detected, that is, multiple pairs of original image pairs at the same location but different times.

[0055] Step 12: Divide the bi-temporal remote sensing image data into image patches of the same size of 256 pixel × 256 pixel to obtain semantic segmentation task data and corresponding pixel-level semantic segmentation labels. In model testing, the change detection task and the semantic segmentation task are tested simultaneously. Therefore, in the semantic segmentation task, no separate test data set is set, and the ratio of the training set to the validation data set is 8:2.

[0056] Step 2: Construct a remote sensing image semantic change detection model based on the integrated triple attention mechanism: The constructed model structure consists of an image change detection network with a triple attention module introduced and an image semantic segmentation network; embed a triple attention mechanism module that integrates image channel attention, neighboring pixel block attention, and similar pixel block attention into the change detection network.

[0057] Specifically, as Figure 2 、 Figure 3As shown in the figure, both networks constructed in Step 2 have a decoder-encoder as the basic structure. The change detection network consists of two encoders and one decoder. Encoder 1 contains three convolutional modules and three pooling layers, and Encoder 2 contains three convolutional modules and two pooling layers. The two encoders respectively reduce the dimension of the dual-temporal images. The input of the encoder is the normalized remote sensing image data, and the output is the remote sensing image feature map. The decoder contains three convolutional modules and three deconvolution layers. The difference between the feature maps output by each convolutional module in the two encoders is introduced into the attention module and then input into the decoder. The decoder raises the dimension of the feature map and outputs a binary change map. The semantic segmentation network is constructed based on U-net and contains one encoder and one decoder. The input of the encoder is the dual-temporal remote sensing image data with merged channels, and the output is the image feature map. The input of the decoder is the image feature map. After the last convolutional layer in the decoder, the output feature map is divided into two halves to obtain two corresponding semantic map images. Finally, the binary change map is respectively subjected to image masking with the two semantic map images, that is, the binary change map is respectively projected onto the two semantic map images to obtain two semantic change label maps.

[0058] As Figure 4 shown, the triple attention mechanism (CASA) module consists of one channel attention (CA) module, one adjacent pixel block attention (APBA) module, one similar pixel block attention (SPBA) module, and three layer normalization (LN) modules.

[0059] The embedding of the triple attention mechanism module includes the following steps:

[0060] Step 21: As Figure 5 shown, the difference between the feature maps output by each convolutional module in the two encoders is subjected to average pooling to obtain a feature vector of 1×1×C. Then, the relationship between k adjacent channels is calculated, and channel attention learning is performed through the Sigmoid function. Finally, the feature vector is raised to the original scale through element-wise multiplication. Where C is the number of channels, k is taken as 4, and in is the input feature map. The input feature map passes through one LN module and one CA module, and the calculation process is expressed by a simple overall function:

[0061] F CA = in + CA(LN(in))

[0062] Step 22: In order to obtain useful information in the image space of the data and at the same time reduce the amount of calculation, as Figure 6 in the present invention, the feature map passing through the channel attention module is cut into N pixel blocks of the same size. Taking the pixel block as the smallest research object, the correlation between adjacent pixel blocks in space is studied, and finally it is restored to the original scale. Let represent the set of input pixel blocks, where C is the number of channels. The calculation process is as follows:

[0063]

[0064] Among them, y ei represents the neighboring attention feature encoding result of the current pixel block e i represents the central position coordinates of the pixel block e i represents the central position coordinates of the surrounding pixel blocks, and m represents the set of pixel blocks within the 7×7 range neighboring the pixel block e i is the Euclidean distance of the pixel block represents the average eigenvalue of the input of the surrounding pixel blocks. g αm (·) is a Gaussian function, and its expression is:

[0065]

[0066] Among them, d represents the Euclidean distance between two pixel points, and αm is a parameter set to 1.5

[0067] After the calculation, after each pixel block obtains the neighboring attention feature encoding value, this encoding value is assigned to each pixel point in the pixel block to obtain the neighboring encoding map. Finally, the element-wise product of the feature map of the input module and the neighboring encoding map is performed, and the neighboring encoding map is upsampled to the original scale

[0068] In the calculation of neighboring pixel block attention, the output CA of the CA module is input into another LN module and an APBA module, which is expressed by a simple overall function:

[0069] W APBA = F CA + APBA(LN(F CA ))

[0070] Step 23: In order to obtain the information of the lost features (specific content) in the image space, the present invention adds a similar pixel block attention (SPBA) module to obtain the correlation between pixel blocks that are far apart in distance but highly similar in pixel block features. As Figure 7 shown, as described in Step 22, the feature map passing through the neighboring pixel block attention module is sliced into N pixel blocks of the same size, and the pixel blocks with similar feature contents are clustered using a clustering function. Each pixel block is assigned a feature similarity value between 0 and 1 according to the feature category (1 for the same category and 0 for those with completely dissimilar types). Then, the feature similarity value is assigned to each pixel point in the pixel block to obtain the feature encoding map. Finally, the element-wise product of the feature map of the input module and the feature encoding map is performed, and the feature encoding map is upsampled to the original scale. The output W of the APBA module APBA ​​​Input to an LN module and an SPBA module, expressed by a simple overall function:

[0071] W SPBA = W APBA + SPBA(LN(W APBA ))

[0072] In the pixel block clustering operation in step 23, the K-means++ algorithm is used to cluster pixel blocks with similar feature contents. The basic principle of the K-Means++ algorithm in the initialization process of the clustering centers is to make the mutual distances between the initial clustering centers as far as possible. The process of the K-Means++ algorithm is as follows: 1) Initialize K clustering centers: ① denotes the set of input pixel blocks, and a pixel block is randomly selected from the set as the first clustering center. ② For each pixel block in the dataset, calculate its distance D(w i ) from the nearest clustering center. Then calculate the probability p(w i ) that each sample is selected as the next clustering center:

[0073]

[0074] Then, the next clustering center is selected by the roulette method. ③ Repeat the previous step until K clustering centers are selected. 2) Calculate the distance between each pixel block and each clustering center. 3) Assign the pixel blocks to the nearest clustering center to form K clusters. 4) Calculate the average value of all points in each cluster and use it as the clustering center. 5) Repeat steps 2), 3), and 4) until the cluster centers remain unchanged or reach the set number of iterations or the set error tolerance range.

[0075] Step 3: Train the remote sensing image semantic change detection model based on the fusion triple attention mechanism, and train the change detection network and the semantic segmentation network respectively. Set the loss function, and calculate the loss between the output of the remote sensing image semantic change detection model based on the fusion triple attention mechanism and the change detection pixel-level label and the semantic segmentation pixel-level label. At the same time, use the adaptive learning weight to set the joint loss function of the change detection task and the semantic segmentation task.

[0076] The model training includes the following steps:

[0077] Step 31: Sample the training tasks of the change detection network. Since the number of change pairs and non-change pairs in the image pairs is highly unbalanced, the present invention uses a sampler to select an equal number of change image pairs and non-change image pairs from the training set according to the change detection label, so that the network will not be overly biased towards the non-change decision.

[0078] Step 32: Sufficient sample data is required during the training of the neural network. The more the number of samples, the better the training effect of the network, the stronger the generalization ability, and thus the occurrence of overfitting can be effectively prevented. Therefore, data augmentation is performed on the small-sized training sample images and their labels obtained by the previous cutting. The images are respectively subjected to horizontal transformation, vertical flipping transformation, and counterclockwise rotation of 90°, 180°, and 270°.

[0079] Step 33: After normalizing the samples, input them into the semantic change model and train the model. Normalize each pixel value x (x ∈ [0, 255]) to (x - 180) / 160.

[0080] As Figure 8 shown, the three loss functions include the change detection loss L cd , the semantic segmentation loss L ss and the joint loss L sc , which are specifically as follows:

[0081] Merge the semantic mapping graphs P1 and P2, mark the same places as 1 and the different places as 0, thus generating a binary change label graph L. The change loss L cd is the binary cross-entropy loss between the binary change graph C predicted by the change prediction network and the binary change label graph L:

[0082]

[0083] where x, t(x), and p C (x) respectively represent the pixel coordinates of the output binary change graph, the true change label, and the predicted label. t(x) and p C (x) respectively use per-pixel softmax to calculate each output feature map.

[0084] The semantic loss L ss is the multi-class cross-entropy loss between the semantic mapping results P1 and P2 and the semantic change graphs S1 and S2. The calculation of L ss for each pixel is

[0085]

[0086] where k is the index of the class (1 ≤ k ≤ K, K: the number of classes), t(x, k) represents the ground truth, and p(x, k) represents the prediction calculated from each output feature map through per-pixel softmax.

[0087] During the model training process, adaptively adjust the weights of the loss functions of the two tasks to obtain the joint loss function, L sc The core formula is:

[0088]

[0089] where σ1 and σ2 are parameters that the model needs to learn, and L ss1 and L ss2 are the multi-class cross-entropy losses between the semantic mapping results P1 and the semantic change map S1, and between the semantic mapping results P2 and the semantic change map S2, respectively.

[0090] Step 4: Test the remote sensing image semantic change detection model based on the fusion triple attention mechanism. After inputting the sample data into the remote sensing image semantic change detection model based on the fusion triple attention mechanism, a binary change map and two semantic mapping maps are obtained. The binary change map is subjected to image masking with the two semantic mapping maps respectively to obtain the corresponding semantic change label maps.

[0091] Based on the same inventive concept, a remote sensing image semantic change detection system disclosed in an embodiment of the present invention includes: a preprocessing module for preprocessing remote sensing image data to obtain dual-temporal change detection task data and corresponding change detection pixel-level labels, as well as semantic segmentation task data and corresponding semantic segmentation pixel-level labels, and dividing the two task data into training, validation, and test data sets; a model construction module for constructing a remote sensing image semantic change detection model based on the fusion triple attention mechanism: the model includes an image change detection network introducing a triple attention mechanism module and an image semantic segmentation network; the triple attention mechanism module fuses image channel attention, adjacent pixel block attention, and similar pixel block attention; the image change detection network includes two encoders and a decoder. The difference between the feature maps output by the corresponding convolutional modules in the two encoders is input into the decoder after introducing an attention module, and a binary change map is output through the decoder; the difference of the feature maps passes through a channel attention module, an adjacent pixel block attention module, and a similar pixel block attention module in sequence; the channel attention module performs channel attention learning and then restores to the original scale, the adjacent pixel block attention module takes pixel blocks as the research object, learns the correlation between adjacent pixel blocks in space and then restores to the original scale, and the similar pixel block attention module takes pixel blocks as the research object, clusters the pixel blocks, and performs attention learning on similar pixel blocks and then restores to the original scale; a model training module for training the remote sensing image semantic change detection model based on the fusion triple attention mechanism, calculating the losses between the output of the model and the change detection pixel-level labels and the semantic segmentation pixel-level labels, and setting a joint loss function for the change detection task and the semantic segmentation task using an adaptive learning weight; and a test module for testing the remote sensing image semantic change detection model based on the fusion triple attention mechanism. After inputting the sample data into the model, a binary change map and two semantic mapping maps are obtained, and the binary change map is subjected to image masking with the two semantic mapping maps respectively to obtain the corresponding semantic change maps.

[0092] Based on the same inventive concept, a computer system disclosed in an embodiment of the present invention includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the remote sensing image semantic change detection method integrating a triple attention mechanism are implemented.

[0093] Based on the same inventive concept, a computer-readable storage medium disclosed in an embodiment of the present invention stores a computer program. When the computer program is executed by a processor, the steps of the remote sensing image semantic change detection method integrating a triple attention mechanism are implemented.

Claims

1. A method for semantic change detection of remote sensing images integrating triple attention mechanism, characterized in that, It includes the following steps: Step 1: After preprocessing the remote sensing image data, obtain the dual-temporal change detection task data and the corresponding change detection pixel-level labels, as well as the semantic segmentation task data and the corresponding semantic segmentation pixel-level labels, and divide the two task data into training, validation, and test data sets; Step 2: Construct a remote sensing image semantic change detection model based on a fused triple attention mechanism: The model includes an image change detection network with a triple attention mechanism module and an image semantic segmentation network; the triple attention mechanism module consists of a channel attention (CA) module, an adjacent pixel block attention (APBA) module, a similar pixel block attention (SPBA) module, and three layer normalization (LN) modules; the image change detection network includes two encoders and a decoder. The difference between the feature maps output by the corresponding convolutional modules in the two encoders is input into the decoder after introducing the attention module, and a binary change map is output through the decoder; the input feature map in passes through an LN module and a CA module, and the overall calculation process is expressed as: F CA = in + CA(LN(in)); The feature map F CA after passing through the channel attention module is sliced into N pixel blocks of the same size. Taking the pixel block as the smallest research object, the correlation between adjacent pixel blocks in space is studied, and finally it is restored to the original scale; use to represent the set of input pixel blocks, where The calculation process is as follows: where y ei represents the adjacent attention feature encoding result of pixel block e i , represents the central position coordinate of pixel block e i , represents the central position coordinate of the surrounding pixel block, m represents the set of pixel blocks within the neighborhood of pixel block e i , is the Euclidean distance of the pixel block, represents the average eigenvalue of the input of the surrounding pixel block, g αm (·) is the Gaussian function; after each pixel block obtains the adjacent attention feature encoding value, this encoding value is assigned to each pixel point in the pixel block to obtain the adjacent encoding map; in the adjacent pixel block attention calculation, the output F CA of the CA module is input into another LN module and an APBA module, and the overall expression is: W APBA = F CA + APBA(LN(F CA )); The feature map F CA after passing through the adjacent pixel block attention module is sliced into N pixel blocks of the same size. The pixel blocks with similar feature contents are clustered using the clustering function, and a feature similarity value between 0 and 1 is assigned to each pixel block according to the feature category. Then, the feature similarity value is assigned to each pixel point in the pixel block to obtain the feature encoding map. Finally, the input feature map and the feature encoding map are element-wise multiplied, and the feature encoding map is upsampled to the original scale; in the similar pixel block attention module, the output W APBA is input into an LN module and an SPBA module, and the overall expression is: W SPBA = W APBA + SPBA(LN(W APBA )); W SPBA is the feature map output by the similar pixel block attention module; Step 3: Train the remote sensing image semantic change detection model based on the fusion triple attention mechanism, calculate the loss between the output of the model and the change detection pixel-level labels and the semantic segmentation pixel-level labels, and use the adaptive learning weight to set the joint loss function for the change detection task and the semantic segmentation task; Step 4: Test the remote sensing image semantic change detection model based on the fusion triple attention mechanism. After inputting the sample data into the model, obtain a binary change map and two semantic mapping maps. After performing image masking on the binary change map with the two semantic mapping maps respectively, obtain the corresponding semantic change maps.

2. A semantic change detection method for remote sensing images integrating a triple attention mechanism according to claim 1, characterized in that The following steps are included in Step 1: Step 11: Divide the original dual-temporal data of the remote sensing image into image patches of a set pixel size to obtain the change detection task data and the corresponding change detection labels; the change detection task data is multiple pairs of original image pairs at the same location but different times, and the label is 0 / 1, that is, the label for the image pair without change is 0, and the label for the changed image pair is 1; the ratio of the training set, validation set, and test data set is 8:1:1, and the test data set stores the remote sensing image data to be detected, that is, multiple pairs of original image pairs at the same location but different times; Step 12: Divide the dual-temporal remote sensing image data into image patches of a set pixel size to obtain the semantic segmentation task data and the corresponding pixel-level semantic segmentation labels; in the semantic segmentation task, no separate test data set is set, and the ratio of the training set to the validation data set is 8:

2.

3. A semantic change detection method for remote sensing images integrating a triple attention mechanism according to claim 1, characterized in that In Step 2, the image change detection network consists of two encoders and one decoder. Encoder 1 contains three convolutional modules and three pooling layers, and Encoder 2 contains three convolutional modules and two pooling layers; the two encoders respectively reduce the dimension of the dual-temporal images; the decoder contains three convolutional modules and three deconvolutional layers; the difference between the feature maps output by each convolutional module in the two encoders is introduced into the triple attention mechanism module and then input into the decoder. The decoder raises the dimension of the feature maps and outputs a binary change map; the image semantic segmentation network contains one encoder and one decoder. The input of the encoder is the dual-temporal remote sensing image data with merged channels, and the output is the image feature map; the input of the decoder is the image feature map. After the last convolutional layer in the decoder, the output feature map is divided into two halves to obtain two corresponding semantic mapping maps; perform image masking on the binary change map with the two semantic mapping maps respectively, that is, project the binary change map onto the two semantic mapping maps respectively to obtain two semantic change label maps.

4. A semantic change detection method for remote sensing images integrating a triple attention mechanism according to claim 1, characterized in that In Step 2, the K-means++ algorithm is used to cluster the pixel blocks with similar feature contents. The clustering process includes: 1) Initialize K clustering centers: ① Denote the set of input pixel blocks, and randomly select a pixel block from the set as the first clustering center; ② For each pixel block in the dataset, calculate its distance D(e i ), and then calculate the probability p(e i ) that each sample is selected as the next clustering center: Then, select the next clustering center by the roulette method; ③ Repeat the previous step until K clustering centers are selected; 2) Calculate the distance between each pixel block and each clustering center; 3) Assign the pixel blocks to the nearest cluster centers to form K clusters; 4) Calculate the average value of all points in each cluster and use it as the cluster center; 5) Repeat steps 2), 3), and 4) until the cluster centers remain unchanged or reach the set number of iterations or the set error tolerance range.

5. A semantic change detection method for remote sensing images integrating triple attention mechanisms according to claim 1, characterized in that In step 3), the change detection network and the semantic segmentation network are trained respectively, and the training process is as follows: sample the training tasks of the change detection network, and select the same number of changed image pairs and unchanged image pairs according to the change detection labels in the training set; perform data augmentation on the training sample images and their labels, and perform horizontal transformation, vertical flipping transformation, and counterclockwise rotation on the images respectively; after normalizing the samples, input them into the semantic change detection model to train the model.

6. A method for semantic change detection of remote sensing images integrating a triple attention mechanism, characterized in that, In step 3, three loss functions are set: the change detection loss L cd , the semantic segmentation loss L ss and the joint loss L sc . The semantic mapping graphs P1 and P2 are merged, with the same parts marked as 1 and the different parts marked as 0, thus generating a binary change label graph L. The change loss L cd is the binary cross-entropy loss between the binary change graph C predicted by the change prediction network and the binary change label graph L. The semantic loss L ss is the multi-class cross-entropy loss between the semantic mapping graphs P1 and P2 and the semantic change graphs S1 and S2. During the model training process, the weights of the loss functions of the two tasks are adaptively adjusted to obtain the joint loss function, and the core formula of L sc is: where σ1 and σ2 are parameters that the model needs to learn, L ss1 and L ss2 are the multi-class cross-entropy losses between the semantic mapping graph P1 and the semantic change graph S1, and between the semantic mapping graph P2 and the semantic change graph S2, respectively.

7. A remote sensing image semantic change detection system integrating a triple attention mechanism, characterized in that, It includes: A preprocessing module, which is used to preprocess the remote sensing image data to obtain dual-temporal change detection task data and corresponding change detection pixel-level labels, as well as semantic segmentation task data and corresponding semantic segmentation pixel-level labels, and divide the two task data into training, validation, and test data sets; The model construction module is used to construct a remote sensing image semantic change detection model based on a fusion triple attention mechanism: the model includes an image change detection network introducing a triple attention mechanism module and an image semantic segmentation network; the triple attention mechanism module consists of a channel attention (CA) module, a neighboring pixel block attention (APBA) module, a similar pixel block attention (SPBA) module, and three layer normalization (LN) modules; the image change detection network includes two encoders and a decoder, and the difference between the feature maps output by the corresponding convolutional modules in the two encoders is input into the decoder after introducing the attention module, and a binary change map is output through the decoder; the input feature map in passes through an LN module and a CA module, and the overall calculation process is expressed as: F CA = in + CA(LN(in)); the feature map F CA after passing through the channel attention module is sliced into N pixel blocks of the same size. Taking the pixel block as the smallest research object, the correlation between neighboring pixel blocks in space is studied, and finally it is restored to the original scale; using to represent the set of input pixel blocks, where the calculation process is as follows: where y ei represents the neighboring attention feature encoding result of pixel block e i , represents the central position coordinate of pixel block e i , represents the central position coordinate of the surrounding pixel block, m represents the set of pixel blocks within the neighborhood of pixel block e i , is the Euclidean distance of the pixel block, represents the average eigenvalue of the input of the surrounding pixel block, g αm (·) is the Gaussian function; after each pixel block obtains the neighboring attention feature encoding value, this encoding value is assigned to each pixel point in the pixel block to obtain the neighboring encoding map; in the calculation of neighboring pixel block attention, the output F CA of the CA module is input into another LN module and an APBA module, and the overall expression is: W APBA = F CA + APBA(LN(F CA )); the feature map F CA after passing through the neighboring pixel block attention module is sliced into N pixel blocks of the same size, and the pixel blocks with similar feature contents are clustered using a clustering function. A feature similarity value between 0 and 1 is assigned to each pixel block according to the feature category, and then the feature similarity value is assigned to each pixel point in the pixel block to obtain the feature encoding map. Finally, the input feature map and the feature encoding map are element-wise multiplied, and the feature encoding map is upsampled to the original scale; in the similar pixel block attention module, the output W APBA is input into an LN module and an SPBA module, and the overall expression is: W SPBA = W APBA + SPBA(LN(W APBA )); W SPBA is the feature map output by the similar pixel block attention module; A model training module, which is used to train the remote sensing image semantic change detection model based on the fusion triple attention mechanism, calculate the loss between the output of the model and the change detection pixel-level label and the semantic segmentation pixel-level label, and use the adaptive learning weight to set the joint loss function of the change detection task and the semantic segmentation task; And a testing module, which is used to test the remote sensing image semantic change detection model based on the fusion triple attention mechanism. After inputting the sample data into the model, a binary change map and two semantic mapping maps are obtained. After masking the binary change map with the two semantic mapping maps respectively, the corresponding semantic change maps are obtained.

8. A computer system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of the remote sensing image semantic change detection method based on the fusion triple attention mechanism according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the remote sensing image semantic change detection method based on the fusion triple attention mechanism according to any one of claims 1-6.

Images