A heterogeneous remote sensing image object-level change detection method

Abstract

The application discloses a kind of heterologous remote sensing image object level change detection method, through design a kind of multiscale feature extraction difference module to realize the extraction of multiscale feature difference map, and as the supplementary input of UNet++, to improve the learning ability of network to the feature of change area;On this basis, to realize the mapping of pixel level result to geographical object, design different scale adaptive evidence confidence index, and then construct a kind of WDSEF identification strategy based on DS evidence theory, realize the leap of pixel level detection result to object.Experiments are carried out to two multi-temporal heterologous image data sets of ZY-3 / GF-2 and GF-1 / GF-2, and compared with a variety of deep learning methods, the results show that the application can significantly improve the recognition ability to the change area, and effectively reduce the influence of "pseudo change", OA and F1 can reach more than 95% and 63.31% respectively, and visual analysis and quantitative evaluation are significantly better than comparative method.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of heterologous remote sensing image object level change detection method, belong to remote sensing image change detection technical field. BACKGROUND

[0002] Remote sensing image change detection is obtained by comparing and analyzing multi-temporal images to obtain the geographical object of change, and is currently widely used in land use planning, disaster assessment and other fields.Compared with single-source image method, change detection method based on heterologous remote sensing image is helpful to find hidden knowledge and rules by using the difference and complementarity of multi-mode data, and has higher data selection flexibility and method availability.In recent years, with the increasing richness of different earth observation platforms, change detection technology for heterologous remote sensing image has become a research hotspot in the field of remote sensing technology and application.

[0003] At the same time, the differences in spectrum, texture and imaging angle brought by different sensors reduce the spatial consistency of the internal pixels of the object, making the "pseudo-change" problem more prominent in the change detection process, that is, it is more challenging to achieve change detection across multiple sensors than single sensor.To this end, scholars have carried out extensive research and achieved many results.Touati R et al. proposed A Reliable Mixed-Norm-Based Multiresolution Change Detector in Heterogeneous Remote Sensing Images, which combined imaging modal invariant operators with multi-scale features to deal with the differences in high-frequency features of various structural regions in heterogenous satellite images; WAN Ling et al. proposed Object-based method for optical and SAR images change detection, which first divided the image into a uniform set of image objects through segmentation, then defined a discrimination criterion based on the membership of the classified objects to classify the image, and thus obtained the change detection result of heterogenous images; Yang Jin-yi et al. combined visual and semantic features and proposed a change detection method based on Bag-Of-Words (BOW) model and active learning, which helped to reduce the "pseudo-change" caused by the imaging differences of heterogenous images. However, the above methods all use user-defined change features, and the detection performance is easily affected by prior knowledge, imaging environment differences and other constraint factors.

[0004] In recent years, deep learning-based remote sensing image processing techniques have attracted extensive attention from scholars and have shown excellent performance in the task of cross-sensor remote sensing image change detection. Deep learning-based change detection methods start from the image itself, and use the powerful abstract representation ability of neural networks to automatically extract multi-level and multi-dimensional deep change features. LIU Jia et al. proposed a deep convolutional coupled network for optical and SAR images, which is composed of symmetric convolutional layers and coupled layers. By converting multi-temporal images into feature space, more consistent feature representation is obtained. WANG Moyang et al. proposed a cross-sensor image supervised change detection method based on deep Siamese convolutional network and hybrid convolutional feature extraction module (HCFEM), which has an expansion convolutional network structure that can effectively extract deep change features. In addition, change detection models using segmentation networks such as UNet and its improved model UNet++ have received extensive attention from scholars because the extracted features have good consistency with the input image size and have strong ability to extract detailed features. PENG Xueli et al. designed a difference-enhancement dense-attention convolutional neural network (DDCNN) for optical image change detection, which uses the UNet++ structure as the backbone, effectively simulates the internal correlation between high-level and low-level features, and combines the difference enhancement (DE) unit to realize the weighted extraction of aggregated features. ZHENG Zhi et al. proposed a cross-layer network (CLNet) based on UNet, which merges multi-scale features and multi-level context information by embedding the designed cross-layer block (CLB), and achieves good results in cross-sensor change detection.

[0005] However, in the task of cross-sensor change detection, the change regions are usually more dispersed and have significant differences in manifestation, so the accurate extraction of change information is more demanding. On the other hand, the traditional UNet structure ignores the autonomous positioning of change regions and the reinforcement learning of change features. In addition, as a semantic segmentation network, there is a significant semantic gap between the pixel-level change detection results output by UNet and the real geographical objects, which limits its practical application in the field of change detection. SUMMARY

[0006] The technical problem to be solved by this invention is to provide a method for detecting object-level changes in heterogeneous remote sensing images. A multi-scale feature extraction differential module is designed to extract multi-scale feature difference maps. Based on this, a weighted DS evidence fusion strategy is proposed, which combines the pixel-level change detection results output by a deep network, thereby achieving the leap from pixel-level detection results to object-level changes.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A method for detecting object-level changes in heterogeneous remote sensing images includes the following steps:

[0009] Step 1: For the area to be detected, acquire two remote sensing images T1 and T2 at different time phases, and T1 and T2 have different resolutions. Using the remote sensing image with lower resolution as a reference, resample the remote sensing image with higher resolution to obtain two-phase remote sensing images with the same resolution, and perform registration processing on the two-phase remote sensing images.

[0010] Step 2: Perform image segmentation on the registered dual-temporal remote sensing images, and extract the segmentation object set based on the segmentation results;

[0011] Step 3: Use UNet++_MSOF as the base network for semantic segmentation. Design a multi-scale feature extraction differential module (MFED) in the encoder part of UNet++_MSOF. This MFED module includes first and second multi-scale feature extraction modules. The original differential feature map D is obtained by differentially analyzing the registered dual-temporal remote sensing images using the first and second multi-scale feature extraction modules. 0 , for D 0 Perform downsampling and convolution operations to obtain the difference feature map D. 1 , for D 1 Perform downsampling and convolution operations to obtain the difference feature map D. 2 , for D 2 Perform downsampling and convolution operations to obtain the difference feature map D. 3 , for D 3 Perform downsampling and convolution operations to obtain the difference feature map D. 4 This yields the difference feature maps corresponding to the five convolutional layers of the UNet++_MSOF encoder, D. 0 D 1 D 2 D 3 D 4 This corresponds to the input of the 0th convolutional unit in the 0th, 1st, 2nd, 3rd, and 4th downsampling layers of the UNet++_MSOF encoder, thus obtaining the pixel-level change detection results of the five layers of MFED-UNet++;

[0012] Step 4, when the MFED-UNet++ training converges, the prediction accuracy and loss function value of each convolution layer of the MFED-UNet++ are obtained, and the prediction accuracy and loss function value are used to design an adaptive evidence confidence index;

[0013] Step 5, for any segmentation object in the segmentation object set obtained in step 2, the adaptive evidence confidence index is combined, and a weighted DS evidence fusion strategy is used to identify the changed object.

[0014] As a preferred scheme of the present application, in step 1, the resampling and registration processing are performed by using a remote sensing image processing platform ENVI, in order to ensure the registration accuracy, the control points that do not meet the requirements are adjusted and deleted, so that the root mean square error is not more than 0.5.

[0015] As a preferred scheme of the present application, in step 2, the image segmentation is performed by using an intelligent image analysis software eCognition, the parameter adjustment is performed based on the remote sensing image collected later in the dual-phase remote sensing image, and the segmentation boundary is projected to the remote sensing image collected earlier according to the coordinate relationship obtained by registration, so as to obtain a set R containing N objects all .

[0016] As a preferred scheme of the present application, in step 3, the first and second multi-scale feature extraction modules are the same in structure, and the process of obtaining the original differential feature map D 0 from the first and second multi-scale feature extraction modules is as follows: for the registered dual-phase remote sensing image, one of the remote sensing images is sequentially subjected to four channels, the first channel is subjected to maximum pooling and average pooling on the remote sensing image, and then the maximum pooling and average pooling results are input into a 1*1 convolution layer, the result of the 1*1 convolution layer is input into a first convolution attention module to obtain the result of the first channel; the second channel is subjected to 3*3 convolution with a dilation rate of 1, and then enters a second convolution attention module to obtain the result of the second channel; the third channel is subjected to 3*3 convolution with a dilation rate of 1 and 2, respectively, and then the results of the 3*3 convolution with a dilation rate of 1 and 2 are fused by using an Add function, and the fused result is input into a third convolution attention module to obtain the result of the third channel; the fourth channel is subjected to 3*3 convolution with a dilation rate of 2 and 5, respectively, and then the results of the 3*3 convolution with a dilation rate of 2 and 5 are fused by using an Add function, and the fused result is input into a fourth convolution attention module to obtain the result of the fourth channel; finally, the results of the first to fourth channels are combined and then subjected to a 1*1 convolution layer to obtain a first feature map; the other remote sensing image is subjected to the same operation as above to obtain a second feature map, and the first feature map and the second feature map are subtracted to obtain the original differential feature map D 0 ;

[0017] The output feature map of each convolution unit in the UNet++_MSOF encoder part is shown in the following formula:

[0018]

[0019] wherein x MFED i,j represents the output feature map of X i,j , X i,j represents a convolution unit, i represents the i-th down-sampling layer, j represents the j-th convolution unit in the jump direction, C(·) represents a convolution operation followed by an activation function, ▽(·) represents a down-sampling operation, Δ(·) represents an up-sampling operation, and [·] represents a concatenation operation; when j=0, the convolution unit only receives one input from the previous down-sampling layer; when j>0, the convolution unit receives a total of j+1 inputs from the jump connection layer and the up-sampling layer, represents the dense jump connection feature map composed of the k=0,…,j-1 convolution layers in the i-th down-sampling layer path;

[0020] On this basis, the MSOF makes the four output nodes {X 0,1 , X 0,2 , X 0,3 , X 0,4} pass through the sigmoid layer to obtain four side outputs {Y 0 , Y 1 , Y 2 , Y 3}, and then generates the fifth output node by connecting the results of the four side outputs:

[0021]

[0022] wherein, represents a concatenation operation, X 0,5 generates the fifth output Y 4 through the sigmoid layer, thereby obtaining the five outputs {Y 0 , Y 1 , Y 2 , Y 3 , Y 4} of the MFED-UNet++.

[0023] As a preferred scheme of the present application, the adaptive evidence confidence index in step 4 is specifically represented as follows:

[0024]

[0025] wherein s=1,2,…,5 represents different levels of network depth, AECI s represents the adaptive evidence confidence index of the s-th layer, and VAs and L s represents the prediction accuracy and loss function value of the s-th layer, respectively.

[0026] As a preferred embodiment of the present invention, the specific process of step 5 is as follows:

[0027] For the set of segmented objects R all Any segmentation object R in n Let n = 1, 2, 3, ..., N, where N is the number of segmented objects in the set. Define the recognition frame Θ: {P, Q}, where P and Q represent changed and unchanged, respectively. Then, the non-empty subset A of Θ includes {P}, {Q}, and {P, Q}. Based on this, define the basic probability allocation function formula as m:2 Θ →[0,1], and satisfies the following constraints:

[0028]

[0029] The synthesis rules based on the above constraints are as follows:

[0030]

[0031] Where K is a normalization constant, which reflects the degree of conflict between pieces of evidence. A' is a set; the smaller K is, the less conflict there is between the pieces of evidence, and the more important the synthesis rule mentioned above is. It plays a major role, and the synthesis result approximates the DS evidence theory; when K→1, that is, when there is a high degree of conflict between the evidence, the above synthesis rules... Plays a major role, AECI s Let m(P) represent the adaptive evidence confidence index of the s-th layer. Finally, if m(P) ≥ m(Q), then it is determined to be a changed object; otherwise, it is an unchanged object. Traverse all objects to obtain the object-level change detection results.

[0032] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0033] 1. This invention designs a multi-scale feature extraction differential module to extract multi-scale feature difference maps, which are then used as supplementary inputs to UNet++ to improve the network's ability to learn features of changing regions and effectively reduce "spurious changes". Based on this, to achieve the mapping from pixel-level results to geographic objects, an adaptive evidence confidence index at different scales is designed, and a WDSEF recognition strategy based on DS evidence theory is constructed, thus ultimately realizing the leap from pixel-level detection results to objects.

[0034] 2, The method has more excellent performance than other advanced methods in the change detection of heterogeneous high-resolution remote sensing images, and the overall accuracy (Overall Accuracy, OA) of two heterogeneous data sets is more than 95%, and the F1 score (F1-score, F1) is 63.31%, 68.30% respectively. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a general flowchart of the object-level change detection method for heterogeneous remote sensing images of the application;

[0036] Figure 2 It is a data set, wherein (a) is data set 1, (b) is data set 2;

[0037] Figure 3 It is an original image, image segmentation and annotation result corresponding to a sample;

[0038] Figure 4 It is a pixel-level change detection framework based on MFED-UNet + +;

[0039] Figure 5 It is a MFEM module display;

[0040] Figure 6 It is the experimental result of different networks on data set 1, wherein (a) is T1 image, (b) is T2 image, (c) is true value graph, (d) is the method of the application, (e) is UNet ++_MSOF, (f) is CLNet, (g) is OB-DSCNH, (h) is MPFF_CNN;

[0041] Figure 7 It is the experimental result of different networks on data set 2, wherein (a) is T1 image, (b) is T2 image, (c) is true value graph, (d) is the method of the application, (e) is UNet ++_MSOF, (f) is CLNet, (g) is OB-DSCNH, (h) is MPFF_CNN;

[0042] Figure 8 It is the evaluation result of pixel to object based on different scale output;

[0043] Figure 9 It is the influence of image spatial resolution on change detection accuracy, wherein (a) is the down-sampling ratio of data set 1, (b) is the accuracy curve of data set 1 under different resolutions, (c) is the down-sampling ratio of data set 2, (d) is the accuracy curve of data set 2 under different resolutions;

[0044] Figure 10is the effectiveness analysis of data augmentation, wherein (a) is the effect after data set 1 is augmented, and (b) is the effect after data set 2 is augmented. DETAILED DESCRIPTION

[0045] Embodiments of the present application are described in detail below with reference to examples shown in the accompanying drawings. The embodiments described below by reference to the drawings are exemplary and are for the purpose of explanation only and are not to be construed as limiting the present application.

[0046] The heterogenous remote sensing image change detection proposed in the present application mainly includes three parts: ① data preprocessing; ② pixel-level change detection based on MFED-UNet++; and ③ change object recognition based on joint multi-scale evidence fusion. The specific process is shown in Figure 1 , wherein T1 and T2 respectively represent images at different time phases.

[0047] 1 Process method

[0048] 1.1 Data preprocessing

[0049] 1.1.1 Heterogeneous image registration and object set extraction

[0050] Since there is usually a difference in spatial resolution between double-time-phase heterogenous remote sensing images, the present application uses commercial software ENVI to perform resampling and registration processing. Among them, both data sets are based on low-resolution images, and the high-resolution images are resampled. In order to ensure the registration accuracy, by adjusting and deleting the control points that do not meet the requirements, it is ensured that the root mean square error (RMSE) does not exceed 0.5.

[0051] Since object-level change detection requires extracting a unified object set in multi-time-phase images, the present application further segments the registered images. The segmentation uses the multi-resolution segmentation in the currently popular commercial software eCognition, which is developed from the fractal network evolution algorithm (FNEA). After visual interpretation and parameter adjustment based on the image collected later in time, the scale parameter, shape parameter and compactness parameter in the two groups of data are set to [35, 0.5, 0.3] and [34, 0.6, 0.3], respectively. Finally, the segmentation boundary is projected into the other time phase image according to the coordinate correspondence, thereby obtaining a set R all containing N objects.

[0052] 1.1.2 Data set annotation

[0053] To verify the accuracy and reliability of the proposed method, two sets of heterogeneous high-resolution remote sensing image datasets were used in the experiment. To address the resolution differences, the strategy described in Section 1.1.1 was employed to obtain multi-temporal images with the same resolution. Dataset 1 consisted of heterogeneous Gaofen-1 (GF-1) and Gaofen-2 (GF-2) images from different temporal phases in Nanjing and Yinchuan, with spatial resolutions of 2m and 0.81m, respectively. (Details are as follows...) Figure 2 As shown in (a); Dataset 2 uses ZY-3 and GF-2 heterogeneous imagery from Nanjing and Yinchuan at different time points, with spatial resolutions of 2.1m and 0.81m, respectively, as detailed below. Figure 2 As shown in (b).

[0054] Based on this, the original image was cropped into a set of 128×128 pixel sub-images, with each sub-image treated as a sample and used as input for subsequent deep networks. In sample annotation, each sample was manually interpreted using a combination of visual analysis and Google Earth, and each object in the sample was labeled as "changed" or "unchanged." Ultimately, Dataset 1 yielded 1232 samples, and Dataset 2 yielded 1280 samples. Figure 3 As shown in (a)-(d), the original T1 phase image, the original T2 phase image, the image segmentation result, and the annotation result corresponding to a sample are displayed respectively (where black represents "unchanged" and white represents "changed").

[0055] 1.2 Pixel-level change detection based on MFED-UNet++

[0056] The heterogeneous change detection task is essentially a binary classification problem of images. Therefore, this invention uses UNet++_MSOF as the base network for semantic segmentation. UNet++ is an encoder-decoder network with a dense skip connection structure, possessing strong feature extraction and detail preservation capabilities. Building upon this, it combines high-level semantic information with low-level texture features using a Multiple Side-output Fusion (MSOF) strategy, better preserving detail edges and avoiding confusion effects. However, when faced with more complex heterogeneous image change feature extraction, UNet++ does not specifically enhance the learning of differential information, making it difficult for the model to effectively handle prominent "pseudo-change" problems. Therefore, this invention designs a novel multi-scale feature extraction differential module (MFED) in the encoder part to enhance the learning of change-type features, thereby improving the model's ability to distinguish real change information. The basic framework of MFED-UNet++ is as follows: Figure 4 As shown, MFEM (Multi-scale Feature Extraction Module) is the multi-scale feature extraction module, Y0 ,Y 1 , ..., Y 4 The five corresponding outputs represent the changing features extracted by MFED-UNet++ in different depth networks.

[0057] 1.2.1 Basic Framework of UNet++_MSOF

[0058] like Figure 4 As shown, UNet++ consists of convolutional units, downsampling modules, upsampling modules, and dense skip connections between convolutional units. Among them, the convolutional unit X... i,j Using the block structure in VGG16, x i,j X represents i,j The output feature map is given by i, which represents the i-th downsampling layer, and j, which represents the j-th convolutional unit in the jump direction, as shown in Equation (1).

[0059]

[0060] C(·) represents a convolution operation followed by an activation function, ▽(·) represents a downsampling operation, Δ(·) represents an upsampling operation, and [·] represents a concatenation operation. When j = 0, the convolutional unit receives only one input from the previous downsampling layer; when j > 0, the convolutional unit receives a total of j+1 inputs from the skip connection layer and the upsampling layer. This represents the dense skip connection feature map formed by the k=0,…,j-1 convolutional layers in the i-th layer path. Let convolutional module X... 0,4 For example, x 0,4 =C([x 0,0 ,x 0,1 ,x 0,2 ,x 0,3 ,Δ(x 1,3 )]) indicates that x 0,0 ,x 0,1 ,x 0,2 ,x 0,3 and x 1,3 The upsampling results are concatenated, and then x is generated through convolution and activation functions. 0,4 .

[0061] Based on this, MSOF sets the four output nodes {X 0,1 ,X 0,2 ,X 0,3 ,X 0,4 The four side outputs {Y} are obtained after passing through the sigmoid layer. 0 ,Y 1 ,Y 2 ,Y 3} and then generate the fifth output node by connecting the results of the four side outputs:

[0062]

[0063] wherein represents a series operation. 0,5 Generate the fifth output Y by a sigmoid layer 4 , so as to obtain five outputs {Y 0 ,Y 1 ,Y 2 ,Y 3 ,Y 4} of UNet++_MSOF.

[0064] 1.2.2 MFED module

[0065] The MFED module designed in the application is composed of a MFEM and a multi-level difference module, and the MFEM is as shown in the figure. Figure 5

[0066] The MFEM combines four channels composed of different convolution layers: channel 1 increases the perception of global information by connecting the maximum pooling and average pooling in parallel before the 1x1 convolution layer; channel 2 adopts a 3x3 convolution module with a dilation rate of 1; channel 3 fuses the output results of the 3x3 convolution modules with dilation rates of 1 and 2 through an Add function; and channel 4 fuses the output results of the 3x3 convolution modules with dilation rates of 2 and 5 through an Add function, so as to obtain different depth abstract features provided by channels 2, 3 and 4. On this basis, a convolution attention module (CBAM) is added at the output position of the four channels, so that the model can focus on learning the key features of the heterogeneous images.

[0067] In the MFED module, the original difference feature map can be obtained by performing difference on the multi-scale features, and down-sampling and convolution unit operations are performed, so as to obtain difference features D i :{D 0 ,D 1 ,D 2 ,D 3 ,D 4} corresponding to the five convolution layers of the UNet++_MSOF encoder part respectively. At this time, x i,j After adding the designed MFED module, x MFED i ,j , as shown in formula (3).

[0068] ​

[0069] When j = 0, the difference eigenmap D i This is used as part of the input to the convolutional units of the encoder. This allows us to obtain the five-layer pixel-level change detection results output by MFED-UNet++.

[0070] 1.3 Change Object Identification through Joint Multi-Scale Evidence Fusion

[0071] The output of MFED-UNet++ reflects the multi-scale variation information of each pixel at different network depths. However, since it originates from the learning results of networks with different depths, its reliability (i.e., confidence level) varies. Therefore, an Adaptive Evidence Credibility Indicator (AECI) is designed to quantify the reliability of the output results at each layer. Based on this, and combining the object set extracted from image segmentation with DS evidence theory, a joint multi-scale evidence fusion method for changing object recognition, WDSEF, is proposed.

[0072] 1.3.1 Adaptive Extraction of Evidence Confidence Index (AECI)

[0073] During the training of MFED-UNet++, the prediction accuracy and loss function values ​​of different convolutional layers can be obtained by comparing the prediction results with the validation set. When the deep learning model converges, if the prediction accuracy of a certain layer is higher or the loss function is smaller, the credibility of its output result is usually higher. In addition, as the network depth increases, it usually means better nonlinear expression ability, thus enabling the learning of more complex changing features. To this end, based on the prediction accuracy, loss function value and network depth, an evidence confidence index AECI for DS fusion is proposed, as shown in formula (4).

[0074]

[0075] In the formula, s = 1, 2, ..., 5 represent different network depths. During training, this invention uses the minimum loss function value based on the validation set as the basis for model convergence. At this time, the prediction accuracy and loss function value of the s-th layer are recorded and respectively calculated using VA. s and L s This indicates that, as a normalization metric, AECI can be adaptively determined based on the model training results. The higher the AECI, the more reliable the prediction results.

[0076] 1.3.2 Recognition Framework Based on DS Theory

[0077] Let R all Any object in R n(n = 1, 2, 3…, N), define the recognition framework Θ: {P, Q}, where P and Q represent change and non-change respectively, then the non-empty subset A of Θ includes {P}, {Q} and {P, Q}. On this basis, the basic probability assignment function formula (BPAF) is defined as m: 2 Θ → [0, 1], and satisfies the following conditions:

[0078]

[0079] Then the synthesis rule based on this constraint condition is as follows:

[0080]

[0081] Where the normalization constant reflects the degree of conflict between evidences. When K is small, that is, the conflict between evidences is small, the in formula (6) plays a major role, and the synthesis result is approximately equal to the classical DS evidence theory. When K→1, that is, the evidences are highly conflicting, the synthesis result is mainly determined by , and AECI s is 1.3.1 section adaptive extraction of evidence confidence. Finally, if m(P) ≥ m(Q), it is determined as a changed object, otherwise as an unchanged object. By traversing all objects, the object-level change detection result can be obtained.

[0082] 2 Experiment and result analysis

[0083] 2.1 Experimental setup and comparison methods

[0084] The present application adopts two sets of heterogeneous high-resolution remote sensing image data sets in section 1.1.2 to carry out experiments, and each set of data set is divided into training set, verification set and test set according to the sample ratio of 6:1:3. In addition, the experiment is realized by Keras, TensorFlow is used as the back end, and the loss function based on cross entropy

[24] is used, and the convolution kernel size of all convolution layers is set to 3x3 pixels.

[0085] On this basis, the present application selects four advanced comparison methods for comparison experiments, which specifically include: ① UNet++_MSOF, which adopts an MSOF strategy in the feature fusion stage of UNet++, which can effectively capture the change information of different scales in complex scenes

[23] . ② CLNet, by embedding CLB module in UNet structure, can effectively fuse multi-scale features and multi-level context information. ③ MPFF-CNN, the framework fully utilizes the spatial feature information of the sample by fusing the features obtained under different scale inputs, and combines image segmentation and voting method to obtain change detection results. ④ OB-DSCNH, the framework faces the change detection of heterogeneous remote sensing images, and proposes a feature extraction module based on Siamese network and hybrid convolution. Among them, the first two are pixel-level change detection methods, and the application determines the object-level change detection result according to the pixel class with the proportion of the pixel; the latter two can directly obtain the object-level change detection result. In addition, the four kinds of comparison methods all use the object set extracted by the application as the basic unit for carrying out object-level precision evaluation.

[0086] 2.2 Performance comparison of different networks

[0087] 2.2.1 Experimental results and visual analysis

[0088] In order to facilitate visual analysis, 2 representative regions are selected in each of the dataset 1 and dataset 2 experiments. In the two groups of data, region 1 is mainly the new construction or demolition of individual buildings in the building-dense area, which corresponds to the partial area renewal and reconstruction application scenario in the built-up area; region 2 is mainly the area where the large area of wasteland is changed into the densely arranged dense building area, which corresponds to the original development scenario of wasteland in the process of urban rapid development. On this basis, further adopt orange frame to represent the true change area, and purple frame to represent the area which has no change but has great difference in spectrum, texture and the like, such as Figure 6 (a)-(h) and Figure 7 (a)-(h) shown.

[0089] As can be seen from Figure 6-7 , the change detection effect of the method of the application is significantly better than the other four comparison methods in two common practical application scenarios. Compared with the method of the application, UNet++_MSOF, CLNet and MPFF_CNN are all single-source image change detection methods. These methods combine multi-temporal images into channels and input them into the network for training, without targeted enhancement of the feature extraction capability of the possible change area in the image, so they are easily affected by the "pseudo change" in the heterogeneous image. As shown in the light gray frame area of region 1 and region 2 of Figure 6 (f) and (h), CLNet and MPFF_CNN have obvious missed detection; and in the two regions of Figure 7 (e) and Figure 7The UNet++_MSOF and MPFF_CNN in the dark gray box region of region 1 of (h) appeared false detection, mainly detecting shadows as change regions. The OB-DSCNH and the method of the application are both heterogeneous detection methods, and the difference features obtained by performing multi-scale feature extraction on the images of the before and after time phases are used to deal with the serious problem of "pseudo change" in the heterogeneous images. However, the OB-DSCNH only relies on the difference features between the images, and ignores the abstract features of the original single-phase images, so it is easy to miss detection in the regions where the change features are not obvious, such as Figure 6 (g) of (g) and Figure 7 the dark gray box region in region 2 of (g) is shown. Compared with this, the MFED module of the application introduces an attention mechanism in the feature extraction stage, and the difference features are only used as auxiliary information to determine whether there is a change or not, so the effect is better.

[0090] 2.2.2 Quantitative evaluation

[0091] In order to verify the effectiveness of the proposed method, the accuracy (Precision, P), recall (Recall, R), F1, OA and Kappa coefficient are used for precision evaluation. Among them, P and R reflect the single-class precision of the change object detection result, the larger the P is, the lower the false alarm rate is, and the larger the R is, the lower the missed detection rate is; F1 reflects the balanced evaluation result of P and R, and OA and Kappa coefficient reveal the overall performance of change detection, the larger the value is, the higher the detection accuracy is. The definitions are as follows:

[0092]

[0093]

[0094]

[0095]

[0096] Among them

[0097] Among them, TP, TN, FP and FN represent true positive, true negative, false positive and false negative, respectively. Taking the change object as an example, TP is the probability that the change is correctly predicted as change; TN is the probability that the non-change is predicted as non-change; FP is the probability that the non-change is predicted as change; and FN is the probability that the change is predicted as non-change. Combined with the above evaluation indexes and the object set R all extracted by the application, the quantitative evaluation of the object-level change detection results of different methods is shown in Table 1:

[0098] Table 1 Quantitative evaluation of object-level change detection results of different methods

[0099]

[0100] As shown in the above table, in the two sets of data, the OA of the method of the application respectively reached 95.74% and 95.67%, and the Kappa coefficient respectively reached 0.5593 and 0.6593, which were significantly better than other comparative methods. At the same time, due to the proportion of non-changing objects in the two sets of data reaching 94% and 92% respectively, the precision index was artificially high to a certain extent. Therefore, based on P and R, the detection performance of the changing object of the method of the application was further evaluated. In the first set of data, it was 60.91% and 65.90% respectively, and in the second set of data, it was 63.25% and 74.23% respectively, which increased by more than 9% and 3% respectively compared with OB-DSCNH. In addition, compared with the method of the application, the F1 of the three single-source image methods UNet++_MSOF, CLNet and MPFF_CNN decreased by more than 9%, 5% and 8% respectively. Therefore, the detection accuracy of the method of the application is significantly better than other comparative methods, which is consistent with the conclusion of visual analysis.

[0101] 3 Verification

[0102] 3.1 Effect analysis of embedding MFED module

[0103] In order to verify the effectiveness of the MFED module separately, under the condition of using WDSEF, the detection effect of the base network UNet++_MSOF before and after embedding the MFED module was compared by the application, and the results are shown in Table 2.

[0104] Table 2 Effect analysis of embedding MFED module, "√" and "-" respectively represent that the module is used and not used

[0105]

[0106]

[0107] It can be seen that after embedding the MFED module, the OA in the experiments of the two sets of data increased by more than 1%, the Kappa and F1 increased by more than 8%, the P increased by 5.8% and 1.8% respectively, and the R increased by more than 10%. Therefore, in the UNet++ semantic segmentation network, by introducing the multi-scale feature difference mechanism, it is helpful to improve the feature extraction capability of the changing area, which proves that the constructed MFED module is feasible and effective.

[0108] 3.2 Comparative analysis of pixel-level results output by deep network

[0109] To verify the performance of the deep network part in the constructed model, the pixel-level prediction results of different networks are quantitatively evaluated based on the same training settings, and the results are shown in Table 3. Among them, the output results of the deepest network of the method are used for evaluation.

[0110] Table 3 Quantitative evaluation of pixel-level prediction results of different methods

[0111]

[0112] As can be seen from Table 3, similar to the analysis conclusion of the object-level detection results, the deep network output results of the method are still significantly better than the four comparative methods. Among them, the OA reaches 95.73% and 95.01% respectively, and the F1 reaches 60.06% and 63.74% respectively. It can be seen that without using pixel-to-object mapping, the network designed by the method can more accurately identify the changed areas in the multi-source remote sensing images. On the other hand, by comparing Table 1 and Table 3, it can be seen that after the pixel-to-object mapping, the various precision evaluation indexes of all methods are improved, which shows that using objects as the analysis unit can help improve the classification accuracy of the changed areas.

[0113] 3.3 Multi-scale difference feature effectiveness analysis

[0114] In order to further verify the necessity of the extracted multi-scale difference features, based on WDSEF, the F1 precision evaluation is carried out on the object-level output results of all scales and each scale, and the results are shown in Table 4. Figure 8

[0115] Figure 8 Among them, Y 0 , Y 1 , Y 2 , Y 3 and Y 4 represent different scales, and Y 0～4 represents the set of all scales. As can be seen from the figure, the prediction accuracy does not necessarily increase with the increase of network depth (such as the accuracy of Y 3 is slightly lower than that of Y 2 ), but Y 0 -Y 4 generally shows a gradually rising trend. In addition, the method based on WDSEF comprehensively utilizes the change features of different scales, and the prediction accuracy is significantly better than that of a single scale. Therefore, the difference features of each scale have complementarity, and the strategy of combining multi-scale features is feasible and necessary.

[0116] 3.4 AECI effectiveness analysis

[0117] ​In the WDSEF proposed in the present application, the AECI adaptively extracted as a key parameter reflects the reliability of each scale difference feature. In order to prove its effectiveness, the present application directly assigns a consistent reliability to all scales, and compares the effect with the use of AECI, and the results are shown in Table 4.

[0118] Table 4 AECI effectiveness analysis, "√" and "-" respectively represent that the module is used and not used

[0119]

[0120] As shown in Table 4, after using AECI, the accuracy of each item is improved to a certain extent, especially in dataset 1, F1 is improved by 1.5%. Therefore, although the use of different scales helps to describe the difference features from multiple angles, there are differences in their reliability. On the other hand, it also shows that the strategy of the present application to determine AECI by combining the loss function and accuracy of the validation set is reasonable and effective.

[0121] 3.5 Image resolution

[0122] When the spatial resolution of the image is higher, the same ground object contains more pixels, so it contains more detailed change information. In order to analyze the influence of training samples of different resolutions on network training effect, the present application respectively down-samples the original image by 0.4, 0.6 and 0.8 times. On this basis, two groups of heterogeneous data sets are used for training, and the accuracy is evaluated. The relationship curve between spatial resolution and change detection accuracy is shown in Figure 9 (a), (b), (c) and (d).

[0123] As can be seen from the figure, with the increase of resolution, P, R, F1 of the method of the present application all show a gradually rising trend. Compared with the original image, in the two data sets, after 0.4 times down-sampling, P decreases by 10.83%, 13.91%; R decreases by 5.61%, 8.94%; F1 decreases by 8.6%, 12.03% respectively. Therefore, more detailed information is helpful to improve the detection accuracy of the changed area, and the higher the resolution of the data set, the better the network training effect.

[0124] 3.6 Data set expansion

[0125] In order to further analyze the influence of data set expansion on model performance, on the basis of the original data set, rotation, mirror and translation operations are used to increase the training samples. After expansion, the two data sets contain 2864 and 3920 samples respectively. The accuracy evaluation results are shown in Figure 10 (a) and (b) of.

[0126] As shown in Figure 10As shown, with the increase of the number of labeled samples, P, R, F1, OA and Kappa are all significantly improved, which are increased by 0.1%, 6%, 2.14%, 0.28%, 2.43% in dataset 1, and 1.41%, 2.85%, 1.98%, 0.87%, 3.12% in dataset 2, respectively. Therefore, by expanding the sample set, the performance of the proposed model can be further improved.

[0127] For object-level change detection of heterogeneous remote sensing images, the application proposes a change detection method of heterogeneous remote sensing images combined with UNet++ and multi-level difference module. Among them, a MFED module is designed to realize the extraction of multi-scale feature difference map, and it is used as the supplementary input of UNet++ to improve the learning ability of the network to the features of the change area, so as to effectively reduce the "pseudo change"; on this basis, in order to realize the mapping from the pixel-level result to the geographical object, an adaptive confidence index AECI of different scales is designed, and then a WDSEF recognition strategy based on DS evidence theory is constructed. Through experiments on two multi-temporal high-resolution heterogeneous image datasets of ZY-3 / GF-2 and GF-1 / GF-2, and compared with a variety of advanced deep learning methods, the results show that the proposed method can significantly improve the recognition ability of the change area, and effectively reduce the influence of "pseudo change", and the OA and F1 can reach more than 95% and 63.31% respectively, and the visual analysis and quantitative evaluation are significantly better than the comparative methods. However, the application only studies the heterogeneous optical remote sensing images with relatively small imaging difference, and future research will focus on more complex optical and SAR image change detection.

[0128] The above examples only illustrate the technical idea of the application, and cannot limit the protection scope of the application. Any modification made according to the technical idea of the application on the basis of the technical scheme falls within the protection scope of the application.

Claims

1. A method for object-level change detection in heterogeneous remote sensing imagery, characterized in that, Comprising the following steps: Step 1, for the region to be detected, obtain two remote sensing images T1 and T2 at different time phases, and the resolutions of T1 and T2 are different, take the remote sensing image with lower resolution as the reference, resample the remote sensing image with higher resolution to obtain double-time-phase remote sensing images with the same resolution, and perform registration processing on the double-time-phase remote sensing images; Step 2, perform image segmentation on the registered double-time-phase remote sensing images, and extract a segmentation object set according to the segmentation result; Step 3: Use UNet++_MSOF as the base network for semantic segmentation. Design a multi-scale feature extraction differential module (MFED) in the encoder part of UNet++_MSOF. This MFED module includes first and second multi-scale feature extraction modules. The original differential feature map D is obtained by differentially analyzing the registered dual-temporal remote sensing images using the first and second multi-scale feature extraction modules. 0 , for D 0 Perform downsampling and convolution operations to obtain the difference feature map D. 1 , for D 1 Perform downsampling and convolution operations to obtain the difference feature map D. 2 , for D 2 Perform downsampling and convolution operations to obtain the difference feature map D. 3 , for D 3 Perform downsampling and convolution operations to obtain the difference feature map D. 4 This yields the difference feature maps corresponding to the five convolutional layers of the UNet++_MSOF encoder, D. 0 D 1 D 2 D 3 D 4 This corresponds to the input of the 0th convolutional unit in the 0th, 1st, 2nd, 3rd, and 4th downsampling layers of the UNet++_MSOF encoder, thus obtaining the pixel-level change detection results of the five layers of MFED-UNet++; Step 4, after the MFED-UNet++ training converges, the prediction accuracy and loss function value of each convolution layer of the MFED-UNet++ are obtained, and an adaptive evidence confidence index is designed using the prediction accuracy and loss function value; Step 5, for any segmentation object in the segmentation object set obtained in step 2, combine the adaptive evidence confidence index, and use a weighted DS evidence fusion strategy to identify the changed object.

2. The heterogeneous remote sensing imagery object-level change detection method according to claim 1, wherein, In step 1, the resampling and registration processing are performed by using a remote sensing image processing platform ENVI, in order to ensure the registration accuracy, the control points that do not meet the requirements are adjusted and deleted, so that the root mean square error is not more than 0.

5.

3. The heterogeneous remote sensing image object-level change detection method according to claim 1, wherein, In step 2, the image segmentation is performed by using intelligent image analysis software eCognition, visual interpretation is performed based on the remote sensing image collected later in the dual-phase remote sensing image, and the segmentation boundary is projected to the remote sensing image collected earlier according to the coordinate relationship obtained by registration, so as to obtain a set R containing N objects all .

4. The heterogeneous remote sensing image object-level change detection method of claim 1, wherein, The first and second multi-scale feature extraction modules are the same in structure in the step 3, and the original difference feature map D 0 is obtained through the first and second multi-scale feature extraction modules The process is as follows: for the registered double-time-phase remote sensing image, one of the remote sensing images is sequentially subjected to four channels, the first channel is subjected to maximum pooling and average pooling, and then the maximum pooling and average pooling results are input into a 1*1 convolution layer, the result of the 1*1 convolution layer is input into a first convolution attention module to obtain the result of the first channel; the second channel is subjected to 3*3 convolution with a dilation rate of 1, and then enters a second convolution attention module to obtain the result of the second channel; the third channel is subjected to 3*3 convolution with a dilation rate of 1 and 2, respectively, the results of the 3*3 convolution with the dilation rates of 1 and 2 are fused by using an Add function, and then the fused result is input into a third convolution attention module to obtain the result of the third channel; the fourth channel is subjected to 3*3 convolution with a dilation rate of 2 and 5, respectively, the results of the 3*3 convolution with the dilation rates of 2 and 5 are fused by using an Add function, and then the fused result is input into a fourth convolution attention module to obtain the result of the fourth channel; finally, the results of the first to fourth channels are combined, and then subjected to a 1*1 convolution layer to obtain a first feature map; the other remote sensing image is subjected to the same operation to obtain a second feature map, and the first feature map and the second feature map are subtracted to obtain the original difference feature map D 0 ; The output feature map of each convolution unit in the UNet++_MSOF encoder part is as follows: where x MFED i,j denotes the output feature map of X i,j i,j denotes the convolution unit, i denotes the i-th down-sampling layer, j denotes the j-th convolution unit in the skip direction, C(·) denotes the convolution operation followed by an activation function, denotes the down-sampling operation, Δ(·) denotes the up-sampling operation, [·] denotes the concatenation operation; when j = 0, the convolution unit only receives one input from the previous down-sampling layer; when j > 0, the convolution unit receives a total of j + 1 inputs from the skip connection layer and the up-sampling layer, denotes the dense skip connection feature map consisting of the k = 0, …, j - 1 convolution layers in the i-th down-sampling layer path;​ On this basis, MSOF makes four output nodes {X 0,1 , X 0,2 , X 0,3 , X 0,4} pass through the sigmoid layer to obtain four side outputs {Y 0 , Y 1 , Y 2 , Y 3}, and then generate the fifth output node by connecting the results of the four side outputs. wherein, represents a series operation, X 0,5 generates a fifth output Y 4 , thereby obtaining five outputs {Y 0 ,Y 1 ,Y 2 ,Y 3 ,Y 4} of the MFED-UNet++.

5. The heterogeneous remote sensing image object-level change detection method of claim 1, wherein, The adaptive evidence confidence index in step 4 is specifically represented as follows: where s = 1, 2,..., 5 represents different levels of network depth, AECI s denotes the adaptive evidence confidence indicator of the s-th layer, VA s and L s denote the prediction accuracy and loss function value of the s-th layer, respectively.

6. The heterogeneous remote sensing image object-level change detection method of claim 1, wherein, The specific process of step 5 is as follows: For any segmentation object R all in the segmentation object set R n , n = 1, 2, 3…, N, N is the number of segmentation objects in the set, define the identification framework Θ: {P, Q}, wherein P and Q represent change and non-change respectively, then the non-empty subset A of Θ includes {P}, {Q} and {P, Q}, on this basis, define the basic probability assignment function formula m: 2 Θ → [0, 1], and satisfy the following constraint conditions: Then the synthesis rule based on the above constraint condition is as follows: where K is a normalization constant, which reflects the conflict degree between evidences, A' is a set; when K is smaller, i.e. the conflict between evidences is smaller, the plays a major role, and the synthesis result is similar to the DS evidence theory; when K→1, i.e. the evidences are highly conflicting, the plays a major role, and the AECI s represents the adaptive evidence confidence index of the s-th layer. Finally, if m(P)≥m(Q), the object is determined as a changed object; otherwise, it is determined as an unchanged object. The object-level change detection result is obtained by traversing all objects.

Images