Remote sensing image change detection system and method based on depth separable convolution module
By introducing a depthwise separable convolutional module and an efficient channel attention module into the UNet network, the remote sensing image change detection method is improved, solving the problems of complex network structure and large computational load, and achieving real-time and high-precision lightweight remote sensing image change detection.
Patent Information
- Application Number
- CN202310228942.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-10
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-03-10
AI Technical Summary
Existing remote sensing image change detection methods have complex network structures and high computational costs, making it difficult to achieve a balance between real-time performance and high accuracy on lightweight mobile devices.
A depthwise separable convolution module is introduced to improve the UNet network structure. A Siamese network structure is used to extract features from dual-temporal images. An efficient channel attention module and a lightweight ASPP+ module are introduced between the encoder and decoder for feature extraction and fusion.
The number of network model parameters and computational overhead were reduced, enabling real-time detection of changes in remote sensing images while maintaining high detection accuracy and meeting the requirements for lightweight design.
Smart Images
Figure CN116229283B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image processing, and particularly relates to a remote sensing image change detection system and method based on a deep separable convolution module. BACKGROUND
[0002] Remote sensing images have become the main data source for obtaining ground information, and remote sensing image change detection research plays a crucial role in natural disaster assessment, land use dynamic detection, forest vegetation change monitoring and urban planning. With the continuous development of satellite remote sensing technology, the resolution of remote sensing images has been greatly improved. Although the change detection method based on the convolutional neural network has achieved satisfactory results, a common problem is that the network structure of the above-mentioned methods is too complex, and the network calculation amount is also relatively large. The current remote sensing image change detection task is gradually applied to lightweight mobile devices, and real-time change detection network research has also become a current hot direction, but it is difficult to achieve both high accuracy in the change detection task and lightweight effect of the network model. The deep separable convolution module is introduced for feature extraction, which can greatly reduce the parameter amount and calculation cost of the network, so as to make the network model more lightweight. SUMMARY
[0003] The application improves the UNet network model in view of the real-time requirement of the current remote sensing image change detection research, introduces a deep separable convolution module, and proposes a remote sensing image change detection system and method based on the deep separable convolution module.
[0004] The application is implemented through the following technical scheme. The remote sensing image change detection system based on the deep separable convolution module comprises a remote sensing image change detection lightweight model based on the deep separable convolution module constructed by taking the Unet network as a basic network structure, wherein the remote sensing image change detection lightweight model comprises an encoder end and a decoder end; a twin network structure is adopted at the encoder end to extract the features of double-time images, and the absolute values of the differences between the double-time images are connected; the deep separable convolution module is used for feature extraction at the encoder end and the decoder end; an effective and lightweight efficient channel attention module is introduced before the skip connection between the encoder end and the decoder end; then, the lightweight ASPP + module is followed by the decoder end; finally, the change map is output through convolution.
[0005] Further preferably, the encoder end and the decoder end of the UNet network and the ASPP + module all adopt the deep separable convolution module.
[0006] Further preferably, the encoder end of the UNet network is fused with a twin network structure, the dual-phase images are respectively input into four consecutive deep separable convolution modules connected in series, and the absolute value of the difference between them is connected; the difference feature map output by each layer of the encoder end is subjected to an efficient channel attention module, and then fused with the deep separable convolution module of the corresponding layer of the decoder end.
[0007] Further preferably, the twin network structure comprises a first branch for extracting a previous-phase remote sensing image and a second branch for extracting a next-phase remote sensing image, the first branch comprises a plurality of deep separable convolution modules in series, the second branch also comprises the same number of deep separable convolution modules in series, and the decoder end also comprises the same number of deep separable convolution modules; the output of the last deep separable convolution module of the second branch is connected to the first deep separable convolution module of the decoder end after passing through an intermediate deep separable convolution module; the first deep separable convolution module of the first branch is connected to the first deep separable convolution module of the second branch in a skip connection manner, and the last deep separable convolution module of the decoder end is connected before the skip connection through a first efficient channel attention module; the second deep separable convolution module of the first branch is connected to the second deep separable convolution module of the second branch in a skip connection manner, and the second-to-last deep separable convolution module of the decoder end is connected before the skip connection through a second efficient channel attention module, and so on; the last deep separable convolution module of the decoder end is connected to an ASPP + module; and a convolution module is connected after the ASPP + module.
[0008] Further preferably, the deep separable convolution module comprises, in series, a 3*3 deep convolution layer, a first BN layer, a first ReLU activation function layer, a 1*1 pointwise convolution layer, a second BN layer, and a second ReLU activation function layer.
[0009] Further preferably, the ASPP + module comprises four branches, which are a 1*1 convolution, a 3*3 deep separable convolution module with a void rate of 6, a 3*3 deep separable convolution module with a void rate of 12, and a 3*3 deep separable convolution module with a void rate of 18.
[0010] The present application proposes a remote sensing image change detection method based on a deep separable convolution module, and the steps are as follows:
[0011] S1: selecting a high-resolution remote sensing image dataset for change detection, and performing a preprocessing operation on the selected high-resolution remote sensing image dataset to make a training set, a validation set, and a test set;
[0012] S2: constructing a remote sensing image change detection lightweight model based on a deep separable convolution module;
[0013] S3: training a remote sensing image change detection lightweight model based on a deep separable convolution module using a training set, verifying using a verification set, and testing using a test set;
[0014] S4: using the trained remote sensing image change detection lightweight model based on the deep separable convolution module to perform remote sensing image change detection.
[0015] More specifically, the process of the remote sensing image change detection lightweight model based on the deep separable convolution module to perform remote sensing image change detection is as follows: a twin network structure is adopted at the encoder end to extract feature information of double-time-phase images, and the absolute value of the difference between them is connected, and a deep separable convolution module is introduced in the feature extraction process; the difference feature map at the encoder end before the jump connection is first highlighted through an efficient channel attention module, and then fused with the corresponding module at the decoder end to obtain global context information; an ASPP module is connected after the decoder end to further improve the detection ability of the network model on different scale target objects; and the enhanced feature map is output after convolution to output the final change map. +
[0016] More specifically, the deep separable convolution module includes two convolution operations, namely deep convolution and pointwise convolution; wherein a deep convolution with a convolution kernel of 3x3 is used to perform a single filter convolution operation on each channel of the input feature map; then, BN regularization and ReLU activation operations are performed; then, a pointwise convolution with a convolution kernel of 1x1 is used to combine the output channel number of the deep convolution; finally, the feature map generated by the pointwise convolution is also subjected to regularization and ReLU activation operations.
[0017] More specifically, in the efficient channel attention module, the input feature map is subjected to a global average pooling operation without dimension reduction; then, local cross-channel information interaction is realized by using a fast one-dimensional convolution; and then a Sigmoid function is used to generate channel weights.
[0018] Compared with the prior art, the present application has the following advantages and beneficial effects:
[0019] In view of the real-time requirement of remote sensing image change detection, the present application proposes a remote sensing image change detection lightweight model based on a deep separable convolution module. First, the model takes UNet network as the backbone network, and introduces a deep separable convolution module for feature extraction in view of the lightweight requirement of the remote sensing image change detection network model, to reduce the redundancy and computational complexity of the network model; then, an efficient channel attention module is used before the jump connection between the encoder and the decoder to suppress irrelevant change information; secondly, a lightweight ASPP+ The small target missing detection problem is solved by the module, and the detection capability of the network model on different scale targets is improved. Experimental results show that the number of parameters of the network model is only 1.08MB, and the FLOPs is only 7.14G, so that the network model further meets the real-time requirement of remote sensing image change detection, and the detection results on the CDD and SYSU-CD data sets still maintain high accuracy. BRIEF DESCRIPTION OF DRAWINGS
[0020] Figure 1 It is a lightweight model structure diagram of remote sensing image change detection based on a deep separable convolution module of the application.
[0021] Figure 2 It is a deep separable convolution module structure diagram.
[0022] Figure 3 It is a high-efficiency channel attention module structure diagram.
[0023] Figure 4 It is a structure diagram of an ASPP + Module.
[0024] Figure 5 It is a flowchart of the application. DETAILED DESCRIPTION
[0025] The application will be further described below in combination with the drawings and specific embodiments.
[0026] The remote sensing image change detection device based on a deep separable convolution module comprises a lightweight model of remote sensing image change detection based on a deep separable convolution module constructed by taking an Unet network as a basic network structure, wherein the lightweight model of remote sensing image change detection comprises an encoder end and a decoder end; firstly, a twin network structure is used to extract features of double-time images at the encoder end, and the absolute value of the difference between them is connected; the deep separable convolution module is used for feature extraction at the encoder end and the decoder end respectively; an efficient and lightweight high-efficiency channel attention module is introduced before the jump connection between the encoder end and the decoder end; then, a lightweight ASPP + Module is followed after the decoder end; finally, a 1x1 convolution is used to output a change map.
[0027] The twin network structure refers to using the network with the same structure at the encoder end to extract features of double-time images; for example Figure 1As shown, the twin network structure includes a first branch for extracting T1 (previous time) phase remote sensing images and a second branch for extracting T2 (next time) phase remote sensing images, the first branch includes four deep separable convolution modules in turn, the second branch also includes four deep separable convolution modules in turn, and the decoder end also includes four deep separable convolution modules, the output of the fourth deep separable convolution module of the second branch is connected to the first deep separable convolution module of the decoder end after passing through an intermediate deep separable convolution module, the first deep separable convolution module of the first branch is jump connected to the first deep separable convolution module of the second branch, and the fourth deep separable convolution module of the decoder end is connected before the jump connection through a first efficient channel attention module; the second deep separable convolution module of the first branch is jump connected to the second deep separable convolution module of the second branch, and the third deep separable convolution module of the decoder end is connected before the jump connection through a second efficient channel attention module; the third deep separable convolution module of the first branch is jump connected to the third deep separable convolution module of the second branch, and the second deep separable convolution module of the decoder end is connected before the jump connection through a third efficient channel attention module; the fourth deep separable convolution module of the first branch is jump connected to the fourth deep separable convolution module of the second branch; the first deep separable convolution module of the decoder end is connected before the jump connection through a fourth efficient channel attention module; the fourth deep separable convolution module of the decoder end is connected to an ASPP + module; the ASPP + module is connected to a 1x1 convolution module.
[0028] In this example, in order to meet the lightweight demand of remote sensing image change detection, the encoder end and the decoder end of the UNet network and the ASPP + module all use deep separable convolution modules for feature extraction, as shown in Figure 2 The deep separable convolution module decomposes the conventional convolution module into two layers of convolution operations, namely deep convolution and pointwise convolution, and includes a 3x3 deep convolution layer, a first BN layer, a first ReLU activation function layer, a 1x1 pointwise convolution layer, a second BN layer, and a second ReLU activation function layer in turn. Among them, the deep convolution with a convolution kernel of 3x3 performs single filter convolution operation on each channel of the input feature map; then, BN regularization and ReLU activation operation are performed; then, the pointwise convolution with a convolution kernel of 1x1 is used to combine the output channel number of the deep convolution; finally, the feature map generated by the pointwise convolution is also subjected to BN regularization and ReLU activation operation.
[0029] In order to better highlight the features of the change area and suppress some irrelevant information, this embodiment introduces a lightweight efficient channel attention module, as shown in Figure 3As shown, the high-efficiency channel attention module removes the full connection layer after the global average pooling layer and uses a 1x1 fast one-dimensional convolution kernel. Among them, the input feature map first performs a global average pooling operation without dimension reduction; then, a fast one-dimensional convolution with a kernel k value of 5 is used to realize local cross-channel information interaction, where the kernel k represents the coverage of cross-channel interaction; and then a Sigmoid function is used to generate channel weights.
[0030] It is known that different scale feature information is essential for remote sensing image change detection, and in order to meet the lightweight demand of remote sensing image change detection, the embodiment proposes a lightweight ASPP + module, such as Figure 4 As shown, the four branches of the ASPP + module are 1x1 convolution, 3x3 depth separable convolution module with a void rate of 6, 3x3 depth separable convolution module with a void rate of 12, and 3x3 depth separable convolution module with a void rate of 18. The ASPP + module first extracts features by parallelizing three depth separable convolution modules and a 1x1 convolution, where the void convolution is to add a void rate parameter in the convolution layer, that is, the convolution kernel is expanded by adding a space according to the void rate, and the space is filled with 0 values; then, the feature maps processed by the 1x1 convolution and the three parallel void convolutions are spliced in the channel dimension, where the void rates of the three void convolutions are 6, 12 and 18 respectively; then, the spliced feature maps are input to a 1x1 convolution for feature fusion.
[0031] In view of the real-time demand of remote sensing image change detection research, the embodiment proposes a remote sensing image change detection method based on a depth separable convolution module, which is described in detail with reference to Figure 5 , the steps are as follows,
[0032] S1: Select a high-resolution remote sensing image dataset for change detection, and perform preprocessing operations on the selected high-resolution remote sensing image dataset to make training set, validation set and test set.
[0033] Taking an image in the change detection high-resolution remote sensing image dataset as an example, there are several typical change types, such as vegetation change, offshore construction, road expansion and building change, etc., where white pixels represent change areas and black pixels represent unchanged areas.
[0034] The preprocessing process mainly includes image cropping and data enhancement. Considering the factor of GPU memory limit, random cropping operation is performed on each original image pair, the images in the change detection dataset are uniformly cropped to 256x256 pixel size images, and data enhancement is performed by horizontal and vertical flipping. The processed change detection dataset is divided into training set: validation set: test set=6:2:2 ratio.
[0035] S2: Constructing a lightweight model for remote sensing image change detection based on a deep separable convolution module;
[0036] S3: Training the lightweight model for remote sensing image change detection based on the deep separable convolution module using the training set, verifying it using the validation set, and testing it using the test set;
[0037] S4: Using the trained lightweight model for remote sensing image change detection based on the deep separable convolution module to perform remote sensing image change detection.
[0038] In this embodiment, the process of using the lightweight model for remote sensing image change detection based on the deep separable convolution module to perform remote sensing image change detection is as follows:
[0039] First, a twin network structure is used in the encoder end of the UNet network to extract the features of the dual temporal images. Deep separable convolution modules are introduced in the feature extraction process, and the absolute values of the differences between them are connected. The twin network structure can better preserve the image feature information of the dual temporal images and effectively alleviate the impact of image information loss during feature extraction. By introducing deep separable convolution modules, the computational load of the network model can be reduced, thereby further meeting the lightweight requirements of remote sensing image change detection.
[0040] Then, the difference map output by the encoder end is first passed through an efficient channel attention module, and then fused with the corresponding module of the decoder end to obtain global context information. The introduction of the efficient channel attention module before the skip connection between the encoder end and the decoder end can better focus on the change information and suppress some irrelevant information.
[0041] After that, an ASPP + module is connected after the decoder end to further improve the detection ability of the network model for different scale target objects. In order to meet the lightweight requirements of remote sensing image change detection, the ASPP +The module first extracts features through parallel three cavity convolutions using a deep separable convolution module and a 1x1 ordinary convolution; then, the feature maps after 1x1 convolution and parallel processing of the three cavity convolutions are spliced in the channel dimension, wherein the cavity rates of the three cavity convolutions are 6, 12 and 18 respectively; finally, the spliced feature maps are input to a 1x1 convolution for feature fusion. This operation improves the detection capability of the network model for different scale target objects without significantly increasing the computational load of the network model.
[0042] Finally, the enhanced feature map is output after 1x1 convolution to output the final experimental effect diagram.
[0043] The embodiment trains and tests the remote sensing image change detection lightweight model based on the deep separable convolution module on two published seasonal change detection data sets (CDD) and Sun Yat-sen University Change Detection Data Set (SYSU-CD), and uses four commonly used objective quantities as evaluation indexes of change detection, namely precision P, recall rate R, comprehensive index F1 value and OA value.
[0044] Table 1 is the objective performance evaluation of the remote sensing image change detection lightweight model based on the deep separable convolution module of the present application on the CDD data set. Table 2 is the objective performance evaluation of the remote sensing image change detection lightweight model based on the deep separable convolution module of the present application on the SYSU-CD data set.
[0045] Table 1 is the evaluation of the present application on the CDD data set
[0046]
[0047] Table 2 is the evaluation of the present application on the SYSU-CD data set
[0048]
[0049] In order to better highlight the feasibility of the remote sensing image change detection lightweight model based on the deep separable convolution module of the present application, five representative change types are selected as test targets on the CDD data set, such as buildings, roads, vehicles, small-scale and complex objects, etc.; at the same time, four typical change types are selected on the SYSU-CD data set, such as vegetation change, offshore construction, road expansion and building change, etc. By analyzing the change maps of typical scenes in the two public data sets, the results show that the experimental effect diagram obtained by using the remote sensing image change detection lightweight model based on the deep separable convolution module can have better detection results on different change types, and can further meet the real-time demand of remote sensing image change detection.
[0050] Finally, it should be noted that the above only describes the preferred examples of the present application, and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that modifications can be made to the technical solutions described in the foregoing embodiments, or some of the technical features thereof can be replaced equivalently, without departing from the spirit and principle of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A remote sensing image change detection system based on depthwise separable convolution module, characterized in that, The remote sensing image change detection lightweight model based on the deep separable convolution module includes an encoder end and a decoder end; the twin network structure is adopted at the encoder end to extract the features of the double-time-phase images, and the absolute values of the differences between the double-time-phase images are connected; the deep separable convolution module is adopted at the encoder end and the decoder end to extract the features; the effective and lightweight efficient channel attention module is introduced before the jump connection between the encoder end and the decoder end; then, the lightweight ASPP + module follows the decoder end; finally, the change map is output through convolution. The twin network structure comprises a first branch for extracting a previous phase remote sensing image and a second branch for extracting a next phase remote sensing image, the first branch comprises a plurality of deep separable convolution modules in sequence, the second branch also comprises a same number of deep separable convolution modules in sequence, and the decoder end also comprises a same number of deep separable convolution modules; an output of a last deep separable convolution module of the second branch is connected to a first deep separable convolution module of the decoder end through an intermediate deep separable convolution module; a first deep separable convolution module of the first branch is jump-connected to a first deep separable convolution module of the second branch, and before the jump connection, the first deep separable convolution module of the first branch is connected to a last deep separable convolution module of the decoder end through a first efficient channel attention module; a second deep separable convolution module of the first branch is jump-connected to a second deep separable convolution module of the second branch, and before the jump connection, the second deep separable convolution module of the first branch is connected to a penultimate deep separable convolution module of the decoder end through a second efficient channel attention module; and the like; and a last deep separable convolution module of the decoder end is connected to an ASPP + module + ; the ASPP + module comprises four branches, and the four branches are respectively a 1x1 convolution, a 3x3 deep separable convolution module with a hollow rate of 6, a 3x3 deep separable convolution module with a hollow rate of 12, and a 3x3 deep separable convolution module with a hollow rate of 18.
2. The deep separable convolution module based remote sensing image change detection system according to claim 1, wherein, The encoder end and the decoder end of the UNet network, ASPP + All the modules adopt deep separable convolution modules.
3. The deep separable convolution module based remote sensing image change detection system according to claim 2, wherein, The encoder end of the UNet network is fused with a twin network structure, and the dual-time images are respectively input into four deep separable convolution modules connected with four continuous down-sampling, and the absolute value of the difference between them is connected; the difference feature map output by each layer of the encoder end is subjected to an efficient channel attention module, and then the deep separable convolution module of the corresponding layer of the decoder end is fused.
4. The deep separable convolution module based remote sensing image change detection system according to claim 1, wherein, The deep separable convolution module sequentially comprises a 3*3 deep convolution layer, a first BN layer, a first ReLU activation function layer, a 1*1 pointwise convolution layer, a second BN layer, and a second ReLU activation function layer.
5. A remote sensing image change detection method based on a depthwise separable convolution module, characterized in that, The steps are as follows: S1: selecting a high-resolution remote sensing image dataset for change detection, and performing a preprocessing operation on the selected high-resolution remote sensing image dataset to make a training set, a validation set, and a test set; S2: constructing the remote sensing image change detection lightweight model based on the deep separable convolution module according to any one of claims 1-4; S3: training the remote sensing image change detection lightweight model based on the deep separable convolution module using the training set, verifying it using the validation set, and testing it using the test set; S4: using the trained remote sensing image change detection lightweight model based on the deep separable convolution module to perform remote sensing image change detection.
6. The method of claim 5, wherein the deep separable convolution module-based remote sensing image change detection method is characterized by, The process of remote sensing image change detection by the remote sensing image change detection lightweight model based on the deep separable convolution module is as follows: the twin network structure is used at the encoder end to extract the feature information of the dual-time images, and the absolute value of the difference between them is connected, and the deep separable convolution module is introduced in the feature extraction process; before the jump connection, the difference feature map of the encoder end is first subjected to an efficient channel attention module to highlight the change feature information, and then fused with the module of the corresponding layer of the decoder end to obtain the global context information; Connecting an ASPP + The module is followed by a decoder end, which further improves the detection capability of the network model for different scale target objects; and the enhanced feature map is output after convolution as the final change map.
7. The method of claim 6, wherein the deep separable convolution module-based remote sensing image change detection method is characterized by, The depth separable convolution module comprises two layers of convolution operations, namely a depth convolution and a point-wise convolution; wherein a convolution kernel with a size of is used to perform a single filter convolution operation on each channel of the input feature map; then, a BN regularization and a ReLU activation operation are performed; then, a convolution kernel with a size of is used to combine the output channel number of the depth convolution; finally, the feature map generated by the point-wise convolution is also subjected to a regularization and a ReLU activation operation.
8. The method of claim 7, wherein the deep separable convolution module-based remote sensing image change detection method is characterized by, In the efficient channel attention module, the input feature map is subjected to a global average pooling operation without dimension reduction; then, a fast one-dimensional convolution is used to realize local cross-channel information interaction, and then a Sigmoid function is used to generate channel weights.