Multi-temporal live-action three-dimensional geographic scene change detection method based on deep learning

Abstract

The invention discloses a multi-temporal live-action three-dimensional geographic scene change detection method based on deep learning, and relates to the technical field of image processing. The method comprises the following steps: acquiring remote sensing images of different time phases, and preprocessing the remote sensing images to obtain a first input image and a second input image; taking the first input image and the second input image as input of a pre-trained deep learning model to obtain a binary mask image; the deep learning model is a deep learning model based on a double-branch main and auxiliary branch structure multi-dense feature enhancement network; and determining a geographical scene change area according to the binary mask image. The feature map of the down-sampling part is connected with the up-sampling part in a jumping manner, so that detail features are effectively reserved, the problem of information loss in the down-sampling process is relieved, the sensitivity to geometric shape and texture changes is enhanced, and the accuracy of change region recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a multi-temporal real-scene three-dimensional geographical scene change detection method based on deep learning. Background Art

[0002] Geographical scene change detection is to utilize multi-source remote sensing images and related geospatial data covering the same surface area at different times, combine the corresponding features of ground objects and the remote sensing imaging mechanism, and adopt image, graphics processing theories and mathematical models to determine and analyze the changes in the positions and ranges of ground objects in this area, as well as the changes in the properties and states of ground objects. Currently, the change detection methods based on deep learning mainly rely on the statistical consistency of observational data, can automatically and multi-levelly perform non-linear feature recognition and extraction, and obtain image features. Its implementation mechanisms include generative adversarial networks, joint sparse representation, spatial structure extraction based on neural networks and random forests, etc.

[0003] Remote sensing images obtained at different times are vulnerable to the influence of conditions such as seasons, climate, and lighting. The spectral features of the same entity may change at different times. The texture features of crops in different growth cycles vary greatly. The imaging mechanisms of different remote sensing sensors are different, and the same entity has large differences on different types of remote sensing images. The texture features of the same entity in different terrain categories are significantly different. For example, the texture differences between mountain dry land and plain dry land are relatively large, and it is easily interfered during multi-sample training. These differences lead to the problem of spatio-temporal heterogeneity of the same entity, which is not only reflected in the changes in spectral features, but also involves the changes in information such as the geometric shape and texture of the entity. Traditional methods can extract spatial and spectral integrated features, establish non-linear correlation relationships between target features, and suppress the differences in entity texture features to a certain extent. However, such methods are still difficult to consider the correlation and complementarity of multi-temporal and multi-type semantic features, and the accuracy of the methods is low and the applicability is not strong. Summary of the Invention

[0004] The object of the present invention is to solve the problem of low accuracy mentioned in the above background art, and to propose a multi-temporal real-scene three-dimensional geographical scene change detection method based on deep learning.

[0005] In the first aspect of the implementation of the present invention, a multi-temporal real-scene three-dimensional geographical scene change detection method based on deep learning is provided. The method includes:

[0006] Obtain remote sensing images at different times, preprocess the remote sensing images to obtain a first input image and a second input image;

[0007] Taking the first input image and the second input image as the inputs of a pre-trained deep learning model to obtain a binary mask image; the deep learning model is a deep learning model based on a multi-dense feature enhancement network with a dual-branch main and auxiliary branch structure, including a downsampling part and an upsampling part; several feature maps of the downsampling part are skip-connected to the upsampling part;

[0008] Determining the geographical scene change area according to the binary mask image.

[0009] Optionally, the downsampling part includes a first auxiliary branch, a second auxiliary branch and a main branch; where:

[0010] The first auxiliary branch is used to extract feature information from the first input image, gradually compress the features, and obtain feature maps of different sizes at different layers;

[0011] The second auxiliary branch is used to extract feature information from the second input image, gradually compress the features, and obtain feature maps of different sizes at different layers;

[0012] The main branch is used to fuse and enhance the feature maps of the first auxiliary branch and the second auxiliary branch at different layers and extract feature information, and gradually compress the features, and obtain feature maps of different sizes at different layers;

[0013] The upsampling part is used to perform an upsampling operation on the feature map output by the last layer of the main branch, gradually restore the spatial resolution of the feature map; and during the upsampling process, perform channel splicing with the feature map output by the corresponding layer of the main branch to enhance the semantics; finally output a binary mask image.

[0014] Optionally, the first auxiliary branch and the second auxiliary branch have the same structure; the first auxiliary branch includes four dense feature extraction modules DFEM11, DFEM12, DFEM13 and DFEM14, and each dense feature extraction module is composed of a dense connection block and a transition layer; the number of layers of the dense connection blocks in DFEM11, DFEM12, DFEM13 and DFEM14 are 2, 2, 3 and 2 respectively;

[0015] The calculation process of the first auxiliary branch includes:

[0016]

[0017] Among them, P1 is the first input image; B11, B12, B13, B14 and B15 are feature maps output at different layers; f Conv represents a convolution operation; f DFEM11 , f DFEM12 , f DFEM13 and f DFEM14Represents the operation of the dense feature extraction module.

[0018] Optionally, the main branch includes four dense feature extraction modules DFEM31, DFEM32, DFEM33, DFEM34 and three feature fusion modules FFM1, FFM2 and FFM3; the numbers of layers of the dense connection blocks in DFEM31, DFEM32, DFEM33 and DFEM34 are 2, 3, 4 and 5 respectively;

[0019] The calculation process of the main branch includes:

[0020]

[0021] Among them, M1, M2, M3, M4, M5, M6, M7 and M8 are the feature maps output by different layers of the main branch; B11, B13, B14 and B15 are the feature maps output by different layers of the first auxiliary branch; B21, B23, B24 and B25 are the feature maps output by different layers of the second auxiliary branch; f DFEM31 , f DFEM32 , f DFEM33 and f DFEM34 Represents the operation of the dense feature extraction module; f FFM1 , f FFM2 and f FFM3 Represents the operation of the feature fusion module.

[0022] Optionally, the target feature fusion module receives the corresponding main branch feature map and double auxiliary branch feature maps, fuses and enhances them, and outputs the feature map to the next module; the target feature fusion module is any one of the feature fusion modules; the target feature fusion module includes a splicing module, a convolution module and a feature enhancement module;

[0023] The calculation process of the target feature fusion module includes:

[0024]

[0025] Among them, Y1 is the feature map transmitted by the first auxiliary branch, corresponding to B13, B14 or B15; Y2 is the feature map transmitted by the second auxiliary branch, corresponding to B23, B24 or B25; Y3 is the feature map transmitted by the main branch, corresponding to M3, M5 or M7; X1 and X2 are the feature maps generated during the operation process; X3 is the feature map output by the target feature fusion module; concat represents channel splicing; f Conv Represents the convolution operation; f IEM Represents the operation of the feature enhancement module.

[0026] Optionally, the calculation process of the feature enhancement module includes:

[0027]

[0028] Among them, X2 and Y3 are the inputs of the feature enhancement module; f GAP represents global average pooling; represents a 1×1 convolution operation; ReLU and Sigmoid are activation functions; X4 is the feature map generated during the calculation process; X5 and X6 are the feature vectors generated during the calculation process; X7 is the output feature map of the feature enhancement module.

[0029] Optionally, the upsampling part includes six dense connection blocks DB1, DB2, DB3, DB4, DB5, and DB6; the number of layers of DB1, DB2, DB3, DB4, DB5, and DB6 are 6, 5, 4, 3, 2, and 2 respectively;

[0030] The calculation process of the upsampling part includes:

[0031]

[0032] Among them, M2, M4, M6, and M8 are the feature maps output by different layers of the main branch; Z1, Z2, Z3, Z4, Z5, Z6, Z7, Z8, and Z9 are the feature maps generated during the calculation process; upsample represents the upsampling operation; concat represents channel concatenation; f DB1 、f DB2 、f DB3 、f DB4 、f DB5 and f DB6 represent the operations of the dense connection block; Z10 is the output of the upsampling part.

[0033] Advantages of the present invention:

[0034] The present invention proposes a multi-temporal real-scene three-dimensional geographical scene change detection method based on deep learning. The method includes: acquiring remote sensing images of different time phases, preprocessing the remote sensing images to obtain a first input image and a second input image; using the first input image and the second input image as the inputs of a pre-trained deep learning model to obtain a binary mask image; the deep learning model is a deep learning model based on a multi-dense feature enhancement network with a double-branch main and sub-branch structure; determining the geographical scene change area according to the binary mask image.

[0035] By making the feature maps of the downsampling part and the upsampling part skip-connected, the detailed features are effectively retained, the problem of information loss during the downsampling process is alleviated, and the sensitivity to geometric shape and texture changes is enhanced, thereby improving the accuracy of identifying the change area. Description of the Drawings

[0036] The present invention will be further described below with reference to the accompanying drawings.

[0037] Figure 1 The flowchart of a multi-temporal real-scene three-dimensional geographic scene change detection method based on deep learning is provided for an embodiment of the present invention;

[0038] Figure 2 The network structure diagram of a deep learning model is provided for an embodiment of the present invention;

[0039] Figure 3 The structural schematic diagram of a feature fusion module is provided for an embodiment of the present invention;

[0040] Figure 4 The structural schematic diagram of a dense module is provided for an embodiment of the present invention. Specific embodiments

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0042] An embodiment of the present invention provides a multi-temporal real-scene three-dimensional geographic scene change detection method based on deep learning. Refer to Figure 1 , Figure 1 The flowchart of a multi-temporal real-scene three-dimensional geographic scene change detection method provided for an embodiment of the present invention. The method includes the following steps:

[0043] S101, obtain remote sensing images of different time phases, preprocess the remote sensing images to obtain a first input image and a second input image.

[0044] S101, take the first input image and the second input image as the inputs of a pre-trained deep learning model to obtain a binary mask image.

[0045] S101, determine the geographic scene change area according to the binary mask image.

[0046] Among them, the deep learning model is a deep learning model based on a multi-dense feature enhancement network with a double-branch main and sub-branch structure, including a downsampling part and an upsampling part; several feature maps of the downsampling part are skip-connected with the upsampling part.

[0047] Based on a multi-temporal real-scene three-dimensional geographic scene change detection method based on deep learning provided by an embodiment of the present invention, by jump-connecting the feature maps of the downsampling part with the upsampling part, the detailed features are effectively retained, the problem of information loss in the downsampling process is alleviated, and the sensitivity to geometric shape and texture changes is enhanced, thereby improving the accuracy of identifying the changed area.

[0048] In one implementation, the preprocessing includes spatial registration and size cropping of the image to conform to the input of the model. For example, the image resolution is adjusted to 448×448.

[0049] In one embodiment, the downsampling part includes a first auxiliary branch, a second auxiliary branch, and a main branch; see Figure 2 , Figure 2 which is a network structure diagram of a deep learning model provided by an embodiment of the present invention. In the figure, Image_t1 represents the first input image, and its corresponding branch is the first auxiliary branch; Image_t2 represents the second input image, and its corresponding branch is the second auxiliary branch. Among them:

[0050] The first auxiliary branch is used to extract feature information from the first input image and gradually compress the features to obtain feature maps of different sizes at different layers.

[0051] The second auxiliary branch is used to extract feature information from the second input image and gradually compress the features to obtain feature maps of different sizes at different layers.

[0052] The main branch is used to fuse and enhance the feature maps of the first auxiliary branch and the second auxiliary branch at different layers and extract feature information, and gradually compress the features to obtain feature maps of different sizes at different layers.

[0053] The upsampling part is used to perform upsampling operations on the feature maps output by the last layer of the main branch, gradually restore the spatial resolution of the feature maps; and during the upsampling process, channel splicing is performed with the feature maps output by the corresponding layers of the main branch to enhance semantics; finally, a binary mask image is output.

[0054] In one implementation, the first auxiliary branch and the second auxiliary branch have the same structure; the first auxiliary branch includes four dense feature extraction modules DFEM11, DFEM12, DFEM13, and DFEM14, and each dense feature extraction module is composed of a dense connection block and a transition layer; the number of layers of the dense connection blocks in DFEM11, DFEM12, DFEM13, and DFEM14 are 2, 2, 3, and 2 respectively;

[0055] The calculation process of the first auxiliary branch includes:

[0056]

[0057] Among them, P1 is the first input image, i.e., Image_t1; B11, B12, B13, B14, and B15 are the feature maps output by different layers of the first auxiliary branch; f Conv represents a convolution operation, where the convolution kernel size is 7×7 and the stride is 1; f DFEM11 , f DFEM12 , f DFEM13 and f DFEM14 represent the operations of the dense feature extraction module.

[0058] Correspondingly, the calculation process of the second auxiliary branch includes:

[0059]

[0060] Among them, P2 is the second input image, i.e., Image_t2; B21, B22, B23, B24, and B25 are the feature maps output by different layers of the second auxiliary branch; f Conv represents a convolution operation; f DFEM21 , f DFEM22 , f DFEM23 and f DFEM24 represent the operations of the dense feature extraction module.

[0061] In one implementation, the main branch includes four dense feature extraction modules DFEM31, DFEM32, DFEM33, DFEM34 and three feature fusion modules FFM1, FFM2, and FFM3; the number of layers of the dense connection blocks in DFEM31, DFEM32, DFEM33, and DFEM34 are 2, 3, 4, and 5 respectively;

[0062] The calculation process of the main branch includes:

[0063]

[0064] Among them, M1, M2, M3, M4, M5, M6, M7, and M8 are the feature maps output by different layers of the main branch; B11, B13, B14, and B15 are the feature maps output by different layers of the first auxiliary branch; B21, B23, B24, and B25 are the feature maps output by different layers of the second auxiliary branch; f DFEM31 , f DFEM32 , f DFEM33 and f DFEM34 represent the operations of the dense feature extraction module; f FFM1 , f FFM2 and f FFM3 represent the operations of the feature fusion module.

[0065] Based on a deep learning model provided by an embodiment of the present invention, downsampling is performed through a mechanism of decoupling and progressive advancement of the main and auxiliary branches. In this mechanism, the rates at which the number of densely connected layers in the main and auxiliary branches change during the downsampling process are different. The number of dense layers in the main branch increases in the form of two layers, three layers, four layers, and five layers, and the two auxiliary branches perform downsampling with the number of two layers, two layers, three layers, and two layers. By setting different growth rates of the number of layers, higher-level features can be obtained from the main branch, and the low-level features of the auxiliary branches can be used for auxiliary feature learning. This loosely coupled network structure and densely connected layer design can better improve the accuracy of identifying changing regions. The feature information of the auxiliary branch is added to the main branch through the feature fusion module. This feature enhancement module fuses the features of the three branches together to form an extraction module for changing features, which can guide the extraction of semantic feature information. A skip connection is made between the semantic information of the main branch after feature fusion and the semantics during the upsampling process. Also, when the main branch first performs downsampling, it also makes a skip connection with the corresponding information of the upsampling. In this way, the model can retain detailed image feature information and reduce the loss of low-level detail information.

[0066] In one implementation, the feature fusion module receives the feature maps of the three branches and performs fusion; the structures of each feature fusion module are the same, and any one of them is denoted as the target feature fusion module for description.

[0067] The target feature fusion module receives the corresponding main branch feature map and double auxiliary branch feature maps, fuses and enhances them, and outputs the feature map to the next module; see Figure 3 , Figure 3 is a schematic structural diagram of a feature fusion module provided by an embodiment of the present invention. The target feature fusion module includes a splicing module, a convolution module, and a feature enhancement module;

[0068] As Figure 3 shown in (a) of

[0069]

[0070] Among them, Y1 is the feature map transmitted by the first auxiliary branch, corresponding to B13, B14, or B15; Y2 is the feature map transmitted by the second auxiliary branch, corresponding to B23, B24, or B25; Y3 is the feature map transmitted by the main branch, corresponding to M3, M5, or M7; X1 and X2 are the feature maps generated during the operation process; X3 is the feature map output by the target feature fusion module; concat represents channel splicing; f Conv represents convolution operation; f IEM represents the operation of the feature enhancement module.

[0071] As Figure 3As shown in (b) of [], the calculation process of the feature enhancement module IEM includes:

[0072]

[0073] Among them, X2 and Y3 are the inputs of the feature enhancement module; f GAP represents global average pooling; represents a 1×1 convolution operation; ReLU and Sigmoid are activation functions; X4 is the feature map generated during the calculation process; X5 and X6 are the feature vectors generated during the calculation process; X7 is the output feature map of the feature enhancement module; the operator represents multiplication, that is, channel weighting of the feature map.

[0074] Through the operation of the feature fusion module, more attention can be paid to the feature information of the changes between two periods of images during network training and inference, allowing the network to obtain more auxiliary information during downsampling to guide semantic information and detect changes.

[0075] In one implementation, as Figure 1 shown, the upsampling part includes six dense connection blocks DB1, DB2, DB3, DB4, DB5, and DB6; the number of layers of DB1, DB2, DB3, DB4, DB5, and DB6 are 6, 5, 4, 3, 2, and 2 respectively; after the resolution recovery process of the upsampling part, the model outputs a binary mask image (BMI, Binary Mask Image).

[0076] The calculation process of the upsampling part includes:

[0077]

[0078] Among them, M2, M4, M6, and M8 are the feature maps output by different layers of the main branch; Z1, Z2, Z3, Z4, Z5, Z6, Z7, Z8, and Z9 are the feature maps generated during the calculation process; upsample represents the upsampling operation, which can be implemented using bilinear interpolation; concat represents channel concatenation; f DB1 、f DB2 、f DB3 、f DB4 、f DB5 、and f DB6 represent the operations of the dense connection block; Z10 is the output of the last layer of the upsampling part, that is, BMI.

[0079] In one implementation, each dense feature extraction module has a similar structure. Any one of them is denoted as the target dense feature extraction module for description. See Figure 4 , Figure 4 which is the structural schematic diagram of a dense module provided by an embodiment of the present invention. AsFigure 4 As shown in (a) in [reference], the target dense feature extraction module DFEM is composed of a dense connection block DB and a transition layer TL. Figure 4 (b) in [reference] shows the structure of the dense connection block DB, where the number of layers n of the dense connection block DB is the same as the number of convolutional modules BRC therein; Figure 4 (c) in [reference] shows the structure of the convolutional module BRC, where BN is the batch normalization operation, ReLU is the activation function, and Conv is the convolutional operation; Figure 4 (d) in [reference] shows the structure of the transition layer, and the feature map is compressed through average pooling with a stride of 2. It should be noted that during the convolution process, if the size and stride of the convolution kernel are not specified, the default convolution kernel size is 3×3, the stride is 1, and no size compression is performed.

[0080] The above has described a specific embodiment of the present invention in detail, but the described content is only a preferred embodiment of the present invention and cannot be considered as limiting the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of the present invention application should still fall within the scope covered by the patent of the present invention.

Claims

1. A multi-temporal real-life three-dimensional geographic scene change detection method based on deep learning, characterized in that: The method comprises: Acquire remote sensing images of different phases, and preprocess the remote sensing images to obtain a first input image and a second input image; The first input image and the second input image are used as inputs of a pre-trained deep learning model to obtain a binary mask image; the deep learning model is a deep learning model based on a dual-branch main-sub-branch structure multi-dense feature enhancement network, including a downsampling part and an upsampling part; a plurality of feature maps of the downsampling part are jump-connected with the upsampling part; The geographic scene change area is determined according to the binary mask image.

2. According to the method for detecting changes in multi-temporal real-life three-dimensional geographic scenes based on deep learning in claim 1, it is characterized in that: The down sampling part includes a first auxiliary branch, a second auxiliary branch and a main branch; wherein: The first auxiliary branch is used to extract feature information from the first input image and gradually compress features to obtain feature maps of different sizes at different layers; The second auxiliary branch is used to extract feature information from the second input image and gradually compress features to obtain feature maps of different sizes at different layers; The main branch is used to perform fusion enhancement and feature information extraction on the feature maps of the first auxiliary branch and the second auxiliary branch at different layers, and gradually compress the features to obtain feature maps of different sizes at different layers; The upsampling part is used to perform an upsampling operation on the feature map output by the last layer of the main branch, and gradually restore the spatial resolution of the feature map; and during the upsampling process, channel splicing is performed with the feature map output by the corresponding layer of the main branch to enhance semantics; and finally a binary mask image is output.

3. The method for detecting changes in multi-temporal real-life three-dimensional geographic scenes based on deep learning according to claim 2, characterized in that: The first auxiliary branch and the second auxiliary branch have the same structure; the first auxiliary branch includes four dense feature extraction modules DFEM11, DFEM12, DFEM13 and DFEM14, each dense feature extraction module is composed of a dense connection block and a transition layer; the number of layers of the dense connection blocks in DFEM11, DFEM12, DFEM13 and DFEM14 are 2, 2, 3 and 2 respectively; The calculation process of the first auxiliary branch includes: Wherein, P1 is the first input image; B11, B12, B13, B14 and B15 are feature maps output by different layers; f Conv represents the convolution operation; f DFEM11 、f DFEM12 、f DFEM13 and f DFEM14 Represents the operation of the dense feature extraction module.

4. The method for detecting changes in multi-temporal real-life three-dimensional geographic scenes based on deep learning according to claim 3, characterized in that: The main branch includes four dense feature extraction modules DFEM31, DFEM32, DFEM33, DFEM34 and three feature fusion modules FFM1, FFM2 and FFM3; the number of layers of dense connection blocks in DFEM31, DFEM32, DFEM33 and DFEM34 are 2, 3, 4 and 5 respectively; The calculation process of the main branch includes: Among them, M1, M2, M3, M4, M5, M6, M7 and M8 are the feature maps of the output of different layers of the main branch; B11, B13, B14 and B15 are the feature maps of the output of different layers of the first auxiliary branch; B21, B23, B24 and B25 are the feature maps of the output of different layers of the second auxiliary branch; f DFEM31 、f DFEM32 、f DFEM33 and f DFEM34 represents the operation of the dense feature extraction module; f FFM1 、f FFM2 and f FFM3 Represents the operation of the feature fusion module.

5. The method for detecting changes in multi-temporal real-life three-dimensional geographic scenes based on deep learning according to claim 4, characterized in that: The target feature fusion module receives the corresponding main branch feature map and the dual auxiliary branch feature map, fuses and enhances them, and outputs the feature map to the next module; the target feature fusion module is any feature fusion module; The target feature fusion module includes a splicing module, a convolution module and a feature enhancement module; The calculation process of the target feature fusion module includes: Among them, Y1 is the feature map transmitted by the first auxiliary branch, corresponding to B13, B14 or B15; Y2 is the feature map transmitted by the second auxiliary branch, corresponding to B23, B24 or B25; Y3 is the feature map transmitted by the main branch, corresponding to M3, M5 or M7; X1 and X2 are the feature maps generated during the operation; X3 is the feature map output by the target feature fusion module; concat represents channel concatenation; f Conv represents the convolution operation; f IEM Represents the operation of the feature enhancement module.

6. The method for detecting changes in multi-temporal real-life three-dimensional geographic scenes based on deep learning according to claim 5, characterized in that: The calculation process of the feature enhancement module includes: Wherein, X2 and Y3 are the inputs of the feature enhancement module; GAP represents global average pooling; represents a 1×1 convolution operation; ReLU and Sigmoid are activation functions; X4 is a feature map generated by the calculation process; X5 and X6 are feature vectors generated by the calculation process; and X7 is an output feature map of the feature enhancement module.

7. The method for detecting changes in multi-temporal real-life three-dimensional geographic scenes based on deep learning according to claim 4, characterized in that: The upsampling part includes six densely connected blocks DB1, DB2, DB3, DB4, DB5 and DB6; the number of layers of DB1, DB2, DB3, DB4, DB5 and DB6 are 6, 5, 4, 3, 2 and 2 respectively; The calculation process of the upsampling part includes: Among them, M2, M4, M6 and M8 are the feature maps output by different layers of the main branch; the feature maps generated by the calculation process of Z1, Z2, Z3, Z4, Z5, Z6, Z7, Z8 and Z9; upsample represents upsampling operation; concat represents channel concatenation; f DB1 、f DB2 、f DB3 、f DB4 、f DB5 and f DB6 represents the operation of the densely connected block; Z10 is the output of the upsampling part.

Images