Building change detection method and system based on ultra-high resolution remote sensing image

Abstract

The invention provides a building change detection method and system based on an ultrahigh-resolution remote sensing image, and the method comprises the steps: firstly obtaining a dual-tense remote sensing image data set, and constructing a multi-scale discrete cosine transform network model which comprises a spatial DCT attention path and a channel DCT attention path; the two-dimensional DCT attention network comprises a spatial DCT attention path and a channel DCT attention path, the spatial DCT attention path comprises a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure which are connected in sequence, and the two-dimensional DCT attention network simultaneously acquires frequency information of a spatial dimension and a channel dimension by using a discrete cosine transform network; spatial information and channel information in a feature map are effectively enhanced to perfect the feature map in the network, the multi-scale DCT pyramid network obtains multi-scale frequency information through the multi-scale discrete cosine transform network, objects of different sizes can be better recognized, and detection of building changes of remote sensing images is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for detecting building changes based on ultra-high resolution remote sensing images. Background Art

[0002] As the spatial resolution of remote sensing images continues to increase, the requirements for various applications of earth observation are also increasing, especially for the observation of finer ground objects such as buildings. This is because buildings are one of the important products of human beings' continuous transformation of the earth's surface for production and life. Studying the changes of buildings in relevant areas over time has gradually become a research hotspot for scholars. Change detection is a technology that has been widely used in the field of remote sensing. The purpose of change detection is to obtain land cover changes by comparing and analyzing multi-temporal remote sensing images. In this context, building change detection has received increasing attention in the past decade. It can achieve effective change monitoring of a single target building (including new buildings and disappearing buildings). Therefore, change detection has potential value in many practical applications, such as urban development planning, urbanization process assessment, and urban disaster prevention and mitigation.

[0003] In the early days, scholars designed hand-crafted features to achieve building extraction and change detection, but the representation effect was not ideal. The deep network structure has a strong multi-level feature extraction capability and can automatically learn complex features in remote sensing images, including geometric information, spectral information and semantic information of ground objects, avoiding the reliance on prior knowledge and the tedious process of manually constructing features, and has better representation. However, most scholars only use high-frequency information in the frequency domain when using deep learning models, without considering the influence of low-frequency information and other frequency information on the representation of building features. High-frequency information corresponds to the rich edge details of ground objects in remote sensing images, while low-frequency information contains more backbone features of ground objects. Introducing high-frequency information such as edges or contours can help improve the accuracy of building change detection, but this method obviously does not make enough use of frequency. Therefore, there is an urgent need for a building change detection method based on ultra-high resolution remote sensing images. Summary of the invention

[0004] In order to solve the problem of insufficient utilization of frequency when using high-frequency information in building change detection, the present invention provides a building change detection method based on ultra-high resolution remote sensing images. Based on a dual-temporal remote sensing image data set, a multi-scale discrete cosine transform network model is established through a dual-dimensional discrete cosine transform attention network and a multi-scale discrete cosine transform pyramid network. By using different frequencies in the frequency domain to refine features and enhance the feature representation ability of the network, the frequency domain information can be better mined and utilized, and the performance of building change detection can be improved. The present invention also relates to a building change detection system based on ultra-high resolution remote sensing images.

[0005] The technical solution of the present invention is as follows:

[0006] A method for detecting building changes based on ultra-high resolution remote sensing images, characterized in that it comprises the following steps:

[0007] Image acquisition step: obtain a bi-temporal remote sensing image dataset;

[0008] Model establishment steps: a multi-scale discrete cosine transform network model is established using a U-shaped network structure and a jump connection as a backbone architecture. The multi-scale discrete cosine transform network model is a novel multi-scale DCT network structure improved on the basis of discrete cosine transform, including a two-dimensional DCT attention network and a multi-scale DCT pyramid network; the two-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure connected in sequence; feature extraction is performed on a dual-temporal remote sensing image through the backbone architecture in the multi-scale discrete cosine transform network model to obtain feature maps of different scales, and the feature maps are respectively input into the spatial DCT attention path and the channel DCT attention path;

[0009] Steps for obtaining a spatial attention score: the spatial feature extraction structure in the spatial DCT attention path performs maximum and average operations on the tensor of the input feature map in the spatial dimension to obtain a maximum feature tensor and an average feature tensor, and connects the maximum feature tensor and the average feature tensor in the channel dimension to form a new feature tensor; then a hybrid convolution method is used to perform convolution processing on the new feature tensor, and a self-attention mechanism is used to weight the new feature tensor after the convolution processing to obtain an enhanced feature tensor, and the spatial frequency extraction structure uses DCT convolution to process the enhanced feature tensor to obtain fine-grained spatial frequency information; the spatial attention mask generation structure uses multiple convolution layers and sigmoid functions to process the fine-grained spatial frequency information to obtain a spatial attention mask, and then obtain a spatial attention score;

[0010] Channel attention score acquisition step: perform pooling operations of multiple scales on the input feature map, and splice the pooling operation results of different scales in the channel dimension to form a multi-scale feature map, the channel DCT attention path uses feature DCT to extract global frequency information from the formed multi-scale feature map to obtain a multi-scale feature map with global frequency information; then use global maximum pooling and global average pooling to process the multi-scale feature map with global frequency information to obtain a new feature map at each scale, and connect the new feature maps at all scales in the channel dimension to form multi-scale channel frequency information, and then process the multi-scale channel frequency information through multiple convolutional layers to obtain a channel attention score;

[0011] Spatial and channel information enhancement step: Based on the obtained spatial attention score and channel attention score, the Hadamard product is used to calculate the two-dimensional attention score, and the two-dimensional attention score is multiplied by the input feature map to obtain the feature map with spatial and channel information enhancement;

[0012] Multi-scale frequency information extraction and information fusion steps: The multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation, and uses multi-scale DCT convolution to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature maps enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of building changes in remote sensing images.

[0013] Preferably, in the model building step, the multi-scale discrete cosine transform network model also includes a CBR structure, and the CBR structure is composed of a Conv convolution layer, a BN layer and a ReLU activation function. The Conv convolution layer performs a convolution operation on the input feature map, and the BN layer normalizes the feature map after the convolution operation to obtain a linear result of the feature map. The ReLU activation function performs feature extraction and feature fusion calculation on the linear result of the feature map to obtain a nonlinear result of the feature map.

[0014] Preferably, in the spatial attention score acquisition step, the hybrid convolution method includes depthwise separable convolution and ordinary convolution, and the depthwise separable convolution includes depthwise convolution and pointwise convolution.

[0015] Preferably, in the multi-scale frequency information extraction and information fusion steps, the pre-operation includes a convolution operation, an extended convolution operation and a global average pooling operation.

[0016] Preferably, the convolution operation performs a convolution operation on input feature maps of different scales through a convolution layer of a specific size; the extended convolution operation performs a convolution operation on input feature maps of different scales through an extended convolution of a specific rate and a specific size; and the global average pooling operation performs global average pooling on input feature maps of different scales.

[0017] A building change detection system based on ultra-high resolution remote sensing images, characterized by comprising an image acquisition module, a model building module, a spatial attention score acquisition module, a channel attention score acquisition module, a spatial and channel information enhancement module and a multi-scale frequency information extraction and information fusion module connected in sequence,

[0018] The image acquisition module acquires a dual-temporal remote sensing image dataset;

[0019] The model building module adopts a U-shaped network structure and a jump connection as a backbone architecture to establish a multi-scale discrete cosine transform network model. The multi-scale discrete cosine transform network model is a novel multi-scale DCT network structure improved on the basis of discrete cosine transform, including a two-dimensional DCT attention network and a multi-scale DCT pyramid network; the two-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure connected in sequence; the dual-temporal remote sensing image is feature extracted through the backbone architecture in the multi-scale discrete cosine transform network model to obtain feature maps of different scales, and the feature maps are respectively input into the spatial DCT attention path and the channel DCT attention path;

[0020] The spatial attention score acquisition module, the spatial feature extraction structure in the spatial DCT attention path performs maximum and average operations on the tensor of the input feature map in the spatial dimension to obtain a maximum feature tensor and an average feature tensor, and connects the maximum feature tensor and the average feature tensor in the channel dimension to form a new feature tensor; then a hybrid convolution method is used to convolve the new feature tensor, and a self-attention mechanism is used to weight the new feature tensor after the convolution process to obtain an enhanced feature tensor, and the spatial frequency extraction structure uses DCT convolution to process the enhanced feature tensor to obtain fine-grained spatial frequency information; the spatial attention mask generation structure uses multiple convolution layers and sigmoid functions to process the fine-grained spatial frequency information to obtain a spatial attention mask, and then obtain a spatial attention score;

[0021] The channel attention score acquisition module performs pooling operations of multiple scales on the input feature map, and splices the pooling operation results of different scales in the channel dimension to form a multi-scale feature map. The channel DCT attention path uses feature DCT to extract global frequency information from the formed multi-scale feature map to obtain a multi-scale feature map with global frequency information; then global maximum pooling and global average pooling are used to process the multi-scale feature map with global frequency information to obtain a new feature map at each scale, and the new feature maps at all scales are connected in the channel dimension to form multi-scale channel frequency information, and then the multi-scale channel frequency information is processed through multiple convolutional layers to obtain a channel attention score;

[0022] The spatial and channel information enhancement module obtains a two-dimensional attention score based on the obtained spatial attention score and channel attention score and uses the Hadamard product to calculate the two-dimensional attention score, and multiplies the two-dimensional attention score by the input feature map to obtain a feature map with enhanced spatial and channel information;

[0023] The multi-scale frequency information extraction and information fusion module, the multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation, and uses multi-scale DCT convolution to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature map enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of building changes in remote sensing images.

[0024] Preferably, in the model building module, the multi-scale discrete cosine transform network model also includes a CBR structure, which is composed of a Conv convolution layer, a BN layer and a ReLU activation function. The Conv convolution layer performs a convolution operation on the input feature map, and the BN layer normalizes the feature map after the convolution operation to obtain a linear result of the feature map. The ReLU activation function performs feature extraction and feature fusion calculation on the linear result of the feature map to obtain a nonlinear result of the feature map.

[0025] Preferably, in the spatial attention score acquisition module, the hybrid convolution method includes depthwise separable convolution and ordinary convolution, and the depthwise separable convolution includes depthwise convolution and pointwise convolution.

[0026] Preferably, in the multi-scale frequency information extraction and information fusion module, the pre-operation includes a convolution operation, an extended convolution operation and a global average pooling operation.

[0027] Preferably, the convolution operation performs a convolution operation on input feature maps of different scales through a convolution layer of a specific size; the extended convolution operation performs a convolution operation on input feature maps of different scales through an extended convolution of a specific rate and a specific size; and the global average pooling operation performs global average pooling on input feature maps of different scales.

[0028] The beneficial effects of the present invention are:

[0029] The present invention provides a method for detecting building changes based on ultra-high resolution remote sensing images. The method first obtains a dual-temporal remote sensing image data set, and uses a U-shaped network structure and a jump connection as a backbone architecture to establish a multi-scale discrete cosine transform network model. The backbone architecture is used to extract features from the dual-temporal remote sensing images to obtain feature maps of different scales, which can effectively capture the features of land cover targets with different scales and complexities, and help to better obtain the cognition of multi-scale land cover targets; the multi-scale discrete cosine transform network model includes a dual-dimensional DCT attention network and a multi-scale DCT pyramid network, the dual-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure, and a spatial frequency extraction structure connected in sequence. The structure and spatial attention mask generation structure are as follows; the spatial DCT attention path processes the input feature map and uses a hybrid convolution and self-attention mechanism to enhance the spatial features, thereby obtaining a spatial attention score, which effectively enhances the spatial information in the feature map; and multi-scale pooling is introduced to capture information obtained from different scales, making the model more comprehensive when processing features, and then the channel DCT attention path processes the input multi-scale feature map to obtain a channel attention score, which is used to extract and enhance the channel information in the feature map; then, based on the obtained spatial attention score and channel attention score, the Hadamard product is used to calculate the two-dimensional attention score, which is multiplied by the two-dimensional attention score and the input feature map to obtain a feature map with enhanced spatial and channel information. Finally, the multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation to extract and organize feature information to obtain multi-scale feature mapping; and uses multi-scale DCT to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature maps enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of architectural changes in remote sensing images, greatly improving the model's understanding and characterization capabilities of the input feature map. The dual-dimensional DCT attention network of the present invention uses a discrete cosine transform network to simultaneously obtain frequency information of the spatial dimension and the channel dimension, effectively enhancing the spatial information and channel information in the feature map to improve the feature map in the network, and the multi-scale DCT pyramid network obtains multi-scale frequency information through a multi-scale discrete cosine transform network, which can better identify objects of different sizes.

[0030] The present invention also relates to a building change detection system based on ultra-high resolution remote sensing images. The system corresponds to the above-mentioned building change detection method based on ultra-high resolution remote sensing images, and can be understood as a system for realizing the above-mentioned building change detection method based on ultra-high resolution remote sensing images, including an image acquisition module, a model building module, a spatial attention score acquisition module, a channel attention score acquisition module, a spatial and channel information enhancement module, and a multi-scale frequency information extraction and information fusion module connected in sequence. The modules work together to establish a multi-scale discrete cosine transform network model based on a dual-temporal remote sensing image data set through a dual-dimensional discrete cosine transform attention network and a multi-scale discrete cosine transform pyramid network. By using different frequencies in the frequency domain to refine features and enhance the feature representation capability of the network, the frequency domain information can be better mined and utilized, thereby improving the performance of building change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the building change detection method based on ultra-high resolution remote sensing images of the present invention.

[0032] Figure 2 It is a structural schematic diagram of the multi-scale discrete cosine transform network model of the present invention.

[0033] Figure 3 It is a structural diagram of the dual-dimensional DCT attention network of the present invention.

[0034] Figure 4 It is a schematic diagram of the structure of the multi-scale DCT pyramid network of the present invention.

[0035] Figure 5 It is a schematic diagram of the heat map visualization output by the present invention on three benchmark building change detection datasets. DETAILED DESCRIPTION

[0036] The present invention will be described below in conjunction with the accompanying drawings.

[0037] The present invention relates to a building change detection method based on ultra-high resolution remote sensing images, which is used to solve the problem of insufficient frequency utilization when using high-frequency information in building change detection. A new multi-scale discrete cosine transform network model (MDNet) is established using a U-shaped network structure and jump connections as the backbone architecture, and different frequencies are used in the frequency domain to refine features, further enhancing the feature representation ability of the network. Different from traditional deep neural networks, the multi-scale discrete cosine transform network has two new modules, namely the dual-dimensional DCT attention network (D 3 AM) and multi-scale DCT pyramid network (MDP) to better mine and utilize frequency domain information and improve the performance of building change detection; through the dual-dimensional DCT attention network (D 3AM) uses a discrete cosine transform network to simultaneously obtain frequency information in the spatial dimension and the channel dimension, effectively enhancing the spatial information and channel information in the feature map to improve the feature map in the network, and obtains multi-scale frequency information through a multi-scale discrete cosine transform network through a multi-scale DCT pyramid network (MDP), which can better identify objects of different sizes. The flowchart of this method is shown in the figure. Figure 1 As shown, the following steps are included in sequence:

[0038] 1. Image acquisition step: Obtain the benchmark remote sensing change detection dataset (also known as a dual-temporal remote sensing image dataset, or a dual-phase remote sensing image dataset). The benchmark remote sensing change detection dataset includes the WHU-CD dataset, the LEVIR-CD dataset, and the Google dataset. The WHU-CD dataset was released by Wuhan University in 2019. The dataset includes aerial images taken at the same location in Christchurch, New Zealand in 2012 and 2016, respectively. These aerial images depict the reconstruction of the area after the 6.3-magnitude earthquake in February 2011. The LEVIR-CD dataset covers various types of buildings, including villa residences, high-rise apartments, small garages, and large warehouses. The dataset consists of 637 Google Earth image fragments with a resolution of 0.5 meters per pixel and a size of 1024×1024 pixels per fragment. The Google dataset contains a variety of complex buildings, partly due to large-scale terrain changes brought about by urbanization.

[0039] 2. Model establishment steps: A multi-scale discrete cosine transform network model is established using a U-shaped network structure and jump connections as the backbone architecture. The multi-scale discrete cosine transform network model is a new multi-scale DCT network structure improved on the basis of discrete cosine transform, including a two-dimensional DCT attention network and a multi-scale DCT pyramid network; the two-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure connected in sequence; the backbone architecture in the multi-scale discrete cosine transform network model is used to extract features of the dual-temporal remote sensing image to obtain feature maps of different scales, and the feature maps are respectively input into the spatial DCT attention path and the channel DCT attention path.

[0040] Specifically, Figure 2As shown in the figure, a u-shaped network is used, and a multi-scale discrete cosine transform network model (MDNet) is established using jump connections as the backbone architecture. The u-shaped network structure has an encoder-decoder architecture, in which the encoder part can extract high-level feature representations of the input data, and the decoder part can map these features back to the original input space. Jump connections can help transfer information between the encoder and the decoder, and can better retain detail information, which helps to better obtain the cognition of multi-scale land cover targets. Through this backbone architecture, the multi-scale feature representation of dual-temporal remote sensing images can be automatically obtained, and hierarchical feature maps of different scales can be obtained, which helps to better obtain the cognition of multi-scale land cover targets.

[0041] The multi-scale discrete cosine transform network model is a deep neural network structure improved on the basis of discrete cosine transform, including a dual-dimensional DCT attention network (D 3 AM) and a multi-scale DCT pyramid network (MDP), where the dual-dimensional DCT attention network D 3 AM obtains more complete frequency information from both spatial and channel dimensions, and uses the obtained dual-dimensional frequency information to generate spatial attention scores and channel attention scores to enhance the useful information in the feature map, thereby improving the performance of building change detection. 3 AM includes a spatial DCT attention path and a channel DCT attention path. The spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure which are connected in sequence.

[0042] Furthermore, the multi-scale discrete cosine transform network model can also include a CBR structure, which is composed of a Conv convolution layer, a BN layer and a ReLU activation function, wherein the Conv convolution layer performs a convolution operation on the input feature map; the BN layer normalizes the feature map after the convolution operation to obtain a linear result of the feature map; the ReLU activation function performs feature extraction and feature fusion calculation on the linear result of the feature map to obtain a nonlinear result of the feature map.

[0043] 3. Steps for obtaining spatial attention scores: The spatial feature extraction structure in the spatial DCT attention path performs maximum and average operations on the tensor of the input feature map in the spatial dimension to obtain a maximum feature tensor and an average feature tensor, and connects the maximum feature tensor and the average feature tensor in the channel dimension to form a new feature tensor; then a hybrid convolution method is used to convolve the new feature tensor, and a self-attention mechanism is used to weight the new feature tensor after convolution to obtain an enhanced feature tensor; the spatial frequency extraction structure uses DCT convolution to process the enhanced feature tensor to obtain fine-grained spatial frequency information; the spatial attention mask generation structure uses multiple convolution layers and sigmoid functions to process the fine-grained spatial frequency information to obtain a spatial attention mask, and then obtains a spatial attention score.

[0044] like Figure 3 As shown, assuming that the input feature maps of different scales are a tensor of size C×H×W, the feature map is represented by F in ∈R C×H×W , where C×H×W represents the number of channels, height, and width respectively. In order to obtain information on both the spatial dimension and the channel dimension, the feature map F in They are input into the spatial DCT attention path and the channel DCT attention path respectively. Specifically, the spatial feature extraction structure in the spatial DCT attention path extracts the most representative spatial features through the maximum and average values ​​of the spatial dimension, and then concatenates them to form a new feature tensor. First, by taking the maximum value operation on the spatial dimension, a maximum value feature tensor of size 1×H×W is obtained, and by taking the average value operation on the spatial dimension, an average value feature tensor of size 1×H×W is obtained. Then, the maximum value feature tensor and the average value feature tensor are concatenated on the channel dimension to form a new feature tensor. It is expressed as follows:

[0045]

[0046] Among them, Max S (·) and Avg S (·) respectively represent the process of taking the maximum and average in the spatial dimension. Concat{·} represents the connection in the channel dimension.

[0047] Then, a hybrid convolution method (i.e., a combination of depthwise separable convolution and ordinary convolution) is used to convolve the new feature tensor. The depthwise separable convolution decomposes the convolution process into depthwise convolution and pointwise convolution. First, the depthwise convolution is used to extract the local features of the new feature tensor, and then the pointwise convolution is used to integrate these local features, which can effectively reduce the complexity of calculation and improve the feature expression ability. The self-attention mechanism (such as the multi-head self-attention in Transformer) is then used to weight the new feature tensor after convolution to obtain an enhanced feature tensor, thereby capturing long-range dependencies, allowing the model to focus on more important feature positions, thereby enhancing the overall feature representation. In addition, the spatial frequency extraction structure in the spatial DCT attention path uses a DCT convolution with a convolution kernel size of 3×3 (i.e., a convolution operation using DCT as the kernel function, where a 3×3 DCT kernel is applied to the new feature tensor) to enhance the feature tensor. Processing to obtain fine-grained spatial frequency information It is expressed as follows:

[0048]

[0049] in, is a discrete cosine transform convolution with a kernel size of 3×3, and P represents the kernel size used for discrete cosine transform (DCT), where the kernel size is 3×3.

[0050] Finally, at the end of the spatial DCT attention path, several convolutional layers with kernel size 1×1 and a sigmoid function are used to obtain fine-grained spatial frequency information. Processing is performed to obtain the spatial attention mask To enhance the spatial information in the feature map, It is expressed as follows:

[0051]

[0052] Among them, Conv s (·) represents several 1×1 convolutional layers with BN layers and ReLU activation functions, which are used to evaluate features in the spatial dimension through supervised learning. In addition, the δ(·) function is used to obtain the spatial attention score.

[0053] 4. Channel attention score acquisition steps: perform pooling operations of multiple scales on the input feature map, and splice the pooling operation results of different scales on the channel dimension to form a multi-scale feature map. The channel DCT attention path uses feature DCT to extract global frequency information from the multi-scale feature map to obtain a multi-scale feature map with global frequency information; then use global maximum pooling and global average pooling to process the multi-scale feature map with global frequency information to obtain a new feature map at each scale, and connect the new feature maps at all scales on the channel dimension to form multi-scale channel frequency information, and then process the multi-scale channel frequency information through multiple convolutional layers to obtain a channel attention score.

[0054] Specifically, firstly, the input feature map is pooled at multiple scales (for example, using pooling kernels of different sizes), that is, using pooling layers of different sizes (such as 2x2, 4x4, 8x8, etc.) to extract local features from the input feature map; then the results of the pooling operations of different scales are spliced ​​in the channel dimension to form a multi-scale feature map to capture information obtained from different scales, so as to obtain more comprehensive channel features and make the subsequent model more comprehensive when processing features. Then, feature DCT is used, that is, discrete cosine transform (DCT) based on height and width features is used to extract channel-based global frequency information from the input multi-scale feature map. It is expressed as: A multi-scale feature map with global frequency information is obtained. Among them, f represents the DCT of the global frequency mapping of the feature, which is the overall DCT transform of the input feature map. Then, the multi-scale feature map with global frequency information is processed by global maximum pooling and global average pooling to obtain a new feature map at each scale, and the new feature maps at all scales are connected in the channel dimension to form multi-scale channel frequency information. Among them, the global maximum pooling and global average pooling operation process is expressed as:

[0055]

[0056] Where Max C (·) and Avg C (·) represent global maximum pooling and global average pooling, respectively.

[0057] Finally, the multi-scale channel frequency information is processed through several 1×1 convolutional layers to generate channel attention scores It is expressed as follows:

[0058]

[0059] Among them, Conv c(·) represents BN, ReLU, and several convolutional layers with kernel size of 1×1.

[0060] 5. Spatial and channel information enhancement step: According to the obtained spatial attention score A s and channel attention score A c , and use the Hadamard product to calculate the two-dimensional attention score, multiply the two-dimensional attention score by the input feature map to get D 3 The feature map with enhanced spatial and channel information in the final output of AM It is expressed as: in, is the Hadamard product. In this stage, the residual link is used to keep the output stable, which is convenient for supervised learning.

[0061] 6. Multi-scale frequency information extraction and information fusion steps: The multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation, and uses multi-scale DCT convolution to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature maps enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of architectural changes in remote sensing images. Furthermore, the pre-operation may include convolution operation, dilated convolution operation and global average pooling operation, wherein the convolution operation performs convolution operation on the input feature maps of different scales through a convolution layer of a specific size, the dilated convolution operation performs convolution operation on the input feature maps of different scales through a dilated convolution of a specific rate and a specific size, and the global average pooling operation performs global average pooling on the input feature maps of different scales.

[0062] Specifically, a multi-scale DCT pyramid network (MDP) is used to further mine multi-scale frequency information. Figure 4 As shown, first assume that the input feature maps of different scales are a tensor of size C×H×W, and the feature map is represented by P in ∈R C ×H×W , where C×H×W represents the number of channels, height, and width respectively. Then, a series of convolution operations, dilated convolution operations, and global average pooling operations are performed on the input feature maps P of different scales. in Processing is used to enhance and integrate the feature map to obtain a richer and more useful feature representation, and a multi-scale feature map P can be obtained. x ∈R C ×H×W {x=1,2,3,4,5}; where

[0063]

[0064] In the above formula, Represents a convolutional layer with a kernel size of 1×1, BN, and ReLU. represents a 3×3 dilated convolution with a dilation rate of n. GAP(·) represents global average pooling.

[0065] Then, multi-scale discrete cosine transform DCT convolution (using DCT kernel as convolution kernel to perform convolution operation) is used to process the feature map of each scale (i.e., multi-scale feature map) respectively. That is, in the multi-scale DCT convolution process, the feature map (P x ) applies multiple DCT kernels of different scales (N×N), where (N=3,5,7,9,11). For feature maps of each scale, multiple DCT kernels of different sizes are used for processing to extract multi-scale frequency information, which can help the model better understand the structure and characteristics of the input features in the frequency domain, thereby improving the understanding and representation of the input features. Among them, the refinement of these features is represented as: Among them, MS-DCT N×N (·) denotes multi-scale DCT with kernel size of N×N {N=3,5,7,9,11}.

[0066] Example:

[0067] In order to verify the effectiveness of the method, four common indicators, namely precision, recall, F1-score and intersection over union (IOU), are used to report the quantitative results. TP, TN, FP and FN represent the number of true positives, true negatives, false positives and false negatives, respectively. Eight excellent peers are selected as comparison algorithms, namely FC-EF, FC-Siam-Diff, FC-Siam-Conc, STANet, SNUNet, BIT and LGPNet to verify the effectiveness of the method. The quantitative analysis of the accuracy, recall, F1-score and IOU indicators of the WHU-CD dataset is shown in Table 1.

[0068] Table 1

[0069] method Accuracy Recall F1-Score IOU FC-Siam-Diff 67.116 88.850 76.468 61.902 FC-EF 90.134 79.473 84.468 73.112 FC-Siam-Conc 50.710 88.402 64.490 47.546 STANet 82.268 88.552 85.294 74.360 SNUNet 88.436 62.294 73.098 57.602 BIT 83.094 86.350 84.691 73.446 LGPNet 84.705 88.544 86.582 76.340 MDNet 94.678 88.445 91.456 84.257

[0070] From the experimental results, we can see that MDNet achieves the best results in three of the four indicators, except that the recall indicator is slightly lower than the best performing algorithm (FC-Siam-Diff).

[0071] Table 2 lists the qualitative analysis results of MDNet and the comparison algorithms for building change detection in four scenarios of the LEVIR-CD dataset. From the results in Table 2, it can be seen that except that the accuracy (precision) is not optimal, MDNet continues to maintain its excellent performance.

[0072] Table 2

[0073]

[0074]

[0075] Table 3 lists the results of four indicators of MDNet and peer comparators on the Google dataset. MDNet ranks first in accuracy (precision), F1 score and IOU, and ranks second in recall.

[0076] Table 3

[0077] method Accuracy Recall F1-Score IOU FC-Siam-Diff 80.525 41.560 54.825 37.764 FC-EF 84.087 55.141 66.605 49.931 FC-Siam-Conc 81.876 54.643 65.543 48.747 STANet 80.231 52.434 63.420 46.434 SNUNet 55.669 47.705 51.381 34.572 BIT 89.623 68.641 77.741 63.587 LGPNet 83.917 82.467 83.186 71.212 MDNet 88.356 80.359 84.168 72.664

[0078] Table 3 further demonstrates the effectiveness of the proposed MDP. Figure 5 The paper provides a method to use D on three benchmark datasets (WHU-CD dataset, LEVIR-CD dataset and Google dataset). 3 Heatmap of the spatial attention masks obtained by AM. As can be seen from the figure, some of them can enhance the boundaries of the changing land cover objects, while others can globally delineate the entire building. This shows that D 3 AM can use DCT to capture global frequency information from low-frequency to high-frequency regions, thereby enhancing the feature map in the network and helping to improve the recognition ability of multi-scale land cover targets. 3 With the help of AM, MDNet is also very capable of extracting the contour features of key changes in the scene. From the visualization results, it can be seen that the edges of the building changes are clearly visible in the extracted overall features.

[0079] The present invention also relates to a building change detection system based on ultra-high resolution remote sensing images. The system corresponds to the above-mentioned building change detection method based on ultra-high resolution remote sensing images, and can be understood as a system for implementing the above-mentioned method. The system includes an image acquisition module, a model building module, a spatial attention score acquisition module, a channel attention score acquisition module, a spatial and channel information enhancement module, and a multi-scale frequency information extraction and information fusion module connected in sequence. Specifically,

[0080] The image acquisition module acquires a dual-temporal remote sensing image dataset;

[0081] The model building module adopts a U-shaped network structure and a jump connection as a backbone architecture to establish a multi-scale discrete cosine transform network model. The multi-scale discrete cosine transform network model is a novel multi-scale DCT network structure improved on the basis of discrete cosine transform, including a two-dimensional DCT attention network and a multi-scale DCT pyramid network; the two-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure connected in sequence; the dual-temporal remote sensing image is feature extracted through the backbone architecture in the multi-scale discrete cosine transform network model to obtain feature maps of different scales, and the feature maps are respectively input into the spatial DCT attention path and the channel DCT attention path;

[0082] The spatial attention score acquisition module, the spatial feature extraction structure in the spatial DCT attention path performs maximum and average operations on the tensor of the input feature map in the spatial dimension to obtain a maximum feature tensor and an average feature tensor, and connects the maximum feature tensor and the average feature tensor in the channel dimension to form a new feature tensor; then a hybrid convolution method is used to convolve the new feature tensor, and a self-attention mechanism is used to weight the new feature tensor after the convolution process to obtain an enhanced feature tensor, and the spatial frequency extraction structure uses DCT convolution to process the enhanced feature tensor to obtain fine-grained spatial frequency information; the spatial attention mask generation structure uses multiple convolution layers and sigmoid functions to process the fine-grained spatial frequency information to obtain a spatial attention mask, and then obtain a spatial attention score;

[0083] The channel attention score acquisition module performs pooling operations of multiple scales on the input feature map, and splices the pooling operation results of different scales in the channel dimension to form a multi-scale feature map. The channel DCT attention path uses feature DCT to extract global frequency information from the multi-scale feature map to obtain a multi-scale feature map with global frequency information; then global maximum pooling and global average pooling are used to process the multi-scale feature map with global frequency information to obtain a new feature map at each scale, and the new feature maps at all scales are connected in the channel dimension to form multi-scale channel frequency information, and then the multi-scale channel frequency information is processed through multiple convolutional layers to obtain a channel attention score;

[0084] The spatial and channel information enhancement module obtains a two-dimensional attention score based on the obtained spatial attention score and channel attention score and uses the Hadamard product to calculate the two-dimensional attention score, and multiplies the two-dimensional attention score by the input feature map to obtain a feature map with enhanced spatial and channel information;

[0085] The multi-scale frequency information extraction and information fusion module, the multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation, and uses multi-scale DCT convolution to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature map enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of building changes in remote sensing images.

[0086] Preferably, in the model building module, the multi-scale discrete cosine transform network model also includes a CBR structure, which is composed of a Conv convolution layer, a BN layer and a ReLU activation function. The Conv convolution layer performs a convolution operation on the input feature map, and the BN layer normalizes the feature map after the convolution operation to obtain a linear result of the feature map. The ReLU activation function performs feature extraction and feature fusion calculation on the linear result of the feature map to obtain a nonlinear result of the feature map.

[0087] Preferably, in the spatial attention score acquisition module, the hybrid convolution method includes depthwise separable convolution and ordinary convolution, and the depthwise separable convolution includes depthwise convolution and pointwise convolution.

[0088] Preferably, the pre-operation includes a convolution operation, a dilated convolution operation and a global average pooling operation.

[0089] Preferably, the convolution operation performs a convolution operation on input feature maps of different scales through a convolution layer of a specific size; the extended convolution operation performs a convolution operation on input feature maps of different scales through an extended convolution of a specific rate and a specific size; and the global average pooling operation performs global average pooling on input feature maps of different scales.

[0090] The present invention provides an objective and scientific building change detection method and system based on ultra-high resolution remote sensing images. Based on a dual-temporal remote sensing image dataset, a multi-scale discrete cosine transform network model is established through a dual-dimensional discrete cosine transform attention network (D3AM) and a multi-scale discrete cosine transform pyramid network (MDP). By using different frequencies in the frequency domain to refine features and enhance the feature representation ability of the network, the frequency domain information can be better mined and utilized, thereby improving the performance of building change detection.

[0091] It should be noted that the above-described specific implementations can enable those skilled in the art to more fully understand the invention, but do not limit the invention in any way. Therefore, although this specification has described the invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents. In short, all technical solutions and improvements that do not deviate from the spirit and scope of the invention should be included in the protection scope of the patent for the invention.

Claims

1. A building change detection method based on ultra-high resolution remote sensing images, characterized in that: The following steps are involved: Image acquisition step: obtain a bi-temporal remote sensing image dataset; Model establishment steps: a multi-scale discrete cosine transform network model is established using a U-shaped network structure and a jump connection as a backbone architecture. The multi-scale discrete cosine transform network model is a novel multi-scale DCT network structure improved on the basis of discrete cosine transform, including a two-dimensional DCT attention network and a multi-scale DCT pyramid network; the two-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure connected in sequence; feature extraction is performed on a dual-temporal remote sensing image through the backbone architecture in the multi-scale discrete cosine transform network model to obtain feature maps of different scales, and the feature maps are respectively input into the spatial DCT attention path and the channel DCT attention path; Steps for obtaining a spatial attention score: the spatial feature extraction structure in the spatial DCT attention path performs maximum and average operations on the tensor of the input feature map in the spatial dimension to obtain a maximum feature tensor and an average feature tensor, and connects the maximum feature tensor and the average feature tensor in the channel dimension to form a new feature tensor; then a hybrid convolution method is used to perform convolution processing on the new feature tensor, and a self-attention mechanism is used to weight the new feature tensor after the convolution processing to obtain an enhanced feature tensor, and the spatial frequency extraction structure uses DCT convolution to process the enhanced feature tensor to obtain fine-grained spatial frequency information; the spatial attention mask generation structure uses multiple convolution layers and sigmoid functions to process the fine-grained spatial frequency information to obtain a spatial attention mask, and then obtain a spatial attention score; Channel attention score acquisition step: perform pooling operations of multiple scales on the input feature map, and splice the pooling operation results of different scales in the channel dimension to form a multi-scale feature map, the channel DCT attention path uses feature DCT to extract global frequency information from the formed multi-scale feature map to obtain a multi-scale feature map with global frequency information; then use global maximum pooling and global average pooling to process the multi-scale feature map with global frequency information to obtain a new feature map at each scale, and connect the new feature maps at all scales in the channel dimension to form multi-scale channel frequency information, and then process the multi-scale channel frequency information through multiple convolutional layers to obtain a channel attention score; Spatial and channel information enhancement step: Based on the obtained spatial attention score and channel attention score, the Hadamard product is used to calculate the two-dimensional attention score, and the two-dimensional attention score is multiplied by the input feature map to obtain the feature map with spatial and channel information enhancement; Multi-scale frequency information extraction and information fusion steps: The multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation, and uses multi-scale DCT convolution to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature maps enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of building changes in remote sensing images.

2. The building change detection method based on ultra-high resolution remote sensing images according to claim 1 is characterized in that: In the model building step, the multi-scale discrete cosine transform network model also includes a CBR structure, which is composed of a Conv convolution layer, a BN layer and a ReLU activation function. The Conv convolution layer performs a convolution operation on the input feature map, and the BN layer normalizes the feature map after the convolution operation to obtain a linear result of the feature map. The ReLU activation function performs feature extraction and feature fusion calculation on the linear result of the feature map to obtain a nonlinear result of the feature map.

3. The building change detection method based on ultra-high resolution remote sensing images according to claim 1 is characterized in that: In the spatial attention score acquisition step, the hybrid convolution method includes depthwise separable convolution and ordinary convolution, and the depthwise separable convolution includes depthwise convolution and pointwise convolution.

4. The building change detection method based on ultra-high resolution remote sensing images according to claim 1 is characterized in that: In the multi-scale frequency information extraction and information fusion steps, the pre-operation includes a convolution operation, an extended convolution operation and a global average pooling operation.

5. The building change detection method based on ultra-high resolution remote sensing images according to claim 4 is characterized in that: The convolution operation performs a convolution operation on input feature maps of different scales through a convolution layer of a specific size; the extended convolution operation performs a convolution operation on input feature maps of different scales through an extended convolution of a specific rate and a specific size; the global average pooling operation performs global average pooling on input feature maps of different scales.

6. A building change detection system based on ultra-high resolution remote sensing images, characterized in that: It includes an image acquisition module, a model building module, a spatial attention score acquisition module, a channel attention score acquisition module, a spatial and channel information enhancement module, and a multi-scale frequency information extraction and information fusion module, which are connected in sequence. The image acquisition module acquires a dual-temporal remote sensing image dataset; The model building module adopts a U-shaped network structure and a jump connection as a backbone architecture to establish a multi-scale discrete cosine transform network model. The multi-scale discrete cosine transform network model is a novel multi-scale DCT network structure improved on the basis of discrete cosine transform, including a two-dimensional DCT attention network and a multi-scale DCT pyramid network; the two-dimensional DCT attention network includes a spatial DCT attention path and a channel DCT attention path, and the spatial DCT attention path includes a spatial feature extraction structure, a spatial frequency extraction structure and a spatial attention mask generation structure connected in sequence; the dual-temporal remote sensing image is feature extracted through the backbone architecture in the multi-scale discrete cosine transform network model to obtain feature maps of different scales, and the feature maps are respectively input into the spatial DCT attention path and the channel DCT attention path; The spatial attention score acquisition module, the spatial feature extraction structure in the spatial DCT attention path performs maximum and average operations on the tensor of the input feature map in the spatial dimension to obtain a maximum feature tensor and an average feature tensor, and connects the maximum feature tensor and the average feature tensor in the channel dimension to form a new feature tensor; then a hybrid convolution method is used to convolve the new feature tensor, and a self-attention mechanism is used to weight the new feature tensor after the convolution process to obtain an enhanced feature tensor, and the spatial frequency extraction structure uses DCT convolution to process the enhanced feature tensor to obtain fine-grained spatial frequency information; the spatial attention mask generation structure uses multiple convolution layers and sigmoid functions to process the fine-grained spatial frequency information to obtain a spatial attention mask, and then obtain a spatial attention score; The channel attention score acquisition module performs pooling operations of multiple scales on the input feature map, and splices the pooling operation results of different scales in the channel dimension to form a multi-scale feature map. The channel DCT attention path uses feature DCT to extract global frequency information from the formed multi-scale feature map to obtain a multi-scale feature map with global frequency information; then global maximum pooling and global average pooling are used to process the multi-scale feature map with global frequency information to obtain a new feature map at each scale, and the new feature maps at all scales are connected in the channel dimension to form multi-scale channel frequency information, and then the multi-scale channel frequency information is processed through multiple convolutional layers to obtain a channel attention score; The spatial and channel information enhancement module obtains a two-dimensional attention score based on the obtained spatial attention score and channel attention score and uses the Hadamard product to calculate the two-dimensional attention score, and multiplies the two-dimensional attention score by the input feature map to obtain a feature map with enhanced spatial and channel information; The multi-scale frequency information extraction and information fusion module, the multi-scale DCT pyramid network processes the input feature maps of different scales through pre-operation, and uses multi-scale DCT convolution to process the feature maps of each scale after processing, extracts multi-scale frequency information, and fuses the multi-scale frequency information with the feature map enhanced by space and channel information to obtain the final output feature map, thereby realizing the detection of building changes in remote sensing images.

7. The building change detection system based on ultra-high resolution remote sensing images according to claim 6 is characterized in that: In the model building module, the multi-scale discrete cosine transform network model also includes a CBR structure, which is composed of a Conv convolution layer, a BN layer and a ReLU activation function. The Conv convolution layer performs a convolution operation on the input feature map, and the BN layer normalizes the feature map after the convolution operation to obtain a linear result of the feature map. The ReLU activation function performs feature extraction and feature fusion calculation on the linear result of the feature map to obtain a nonlinear result of the feature map.

8. The building change detection system based on ultra-high resolution remote sensing images according to claim 6 is characterized in that: In the spatial attention score acquisition module, the hybrid convolution method includes depthwise separable convolution and ordinary convolution, and the depthwise separable convolution includes depthwise convolution and pointwise convolution.

9. The building change detection system based on ultra-high resolution remote sensing images according to claim 6 is characterized in that: In the multi-scale frequency information extraction and information fusion module, the pre-operation includes a convolution operation, an extended convolution operation and a global average pooling operation.

10. The building change detection system based on ultra-high resolution remote sensing images according to claim 9 is characterized in that: The convolution operation performs a convolution operation on input feature maps of different scales through a convolution layer of a specific size; the extended convolution operation performs a convolution operation on input feature maps of different scales through an extended convolution of a specific rate and a specific size; the global average pooling operation performs global average pooling on input feature maps of different scales.

Images