Building semi-supervised change detection method and device under consistency loss guidance
By using a lightweight encoder-decoder network and a consistency loss-guided method for generating pseudo-labels, the problems of redundant model parameters and insufficient data utilization in building change detection of remote sensing images are solved, achieving efficient building change detection suitable for small sample scenarios.
Patent Information
- Application Number
- CN202310255751.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-16
- Publication Date
- 2025-12-23
- Estimated Expiration
- 2043-03-16
AI Technical Summary
Existing remote sensing image building change detection models suffer from redundant parameters and insufficient data utilization, making them unsuitable for small sample scenarios. Furthermore, deep learning models rely on a large number of labeled samples, leading to overfitting and poor generalization ability.
A lightweight encoder-decoder network is adopted, which combines a pseudo-twin encoder and multiple decoders. Pseudo-labels are generated by guiding consistency loss, and semi-supervised learning is performed using labeled and unlabeled samples to optimize the network model.
It improves the accuracy and efficiency of building change detection, enhances the model's application capability in small sample scenarios, reduces the number of network parameters, and is suitable for devices with limited memory.
Smart Images
Figure CN116343033B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of remote sensing image information extraction, and relates to a high-resolution remote sensing image building semi-supervised change detection method and device guided by consistency loss. BACKGROUND
[0002] The continuous progress of space-to-ground observation technology promotes the continuous improvement of the spectral resolution, spatial resolution and temporal resolution of remote sensing data. Massive remote sensing data has the characteristics of rich types, diversification and strong uncertainty. Compared with manual data collection, remote sensing data has the advantages of wide coverage, low cost and high efficiency, and has been widely used in the research fields of urban ecological environment destruction and land surface feature change detection. Buildings, as the most basic constituent unit in the city, play an important role in the process of urbanization. Precise detection of building change information can provide scientific data reference for urban land resource management, illegal information investigation and urban sustainable development.
[0003] However, the remote sensing technology faces the following difficulties in the change detection task: first, the data deep feature extraction is insufficient; second, the data processing model parameters are redundant and inefficient; third, the number of labeled change samples is limited, and the model performance is limited. Although the change detection method based on deep learning can alleviate the shortcomings of traditional algorithms to some extent, it also depends on a large number of labeled change samples. When the number of labeled samples is small, the model is prone to overfitting, and the generalization ability is poor. In addition, compared with traditional algorithms, the parameter quantity of deep learning network model increases dramatically, which makes it difficult to deploy in small memory hardware devices. SUMMARY
[0004] In view of the deficiencies of the existing remote sensing image information extraction technology, the purpose of the present application is to provide a new encoder-decoder lightweight model based on residual structure, and realize a high-resolution remote sensing image building semi-supervised change detection scheme guided by consistency loss.
[0005] The technical scheme of the present application provides a building semi-supervised change detection method guided by consistency loss, comprising the following steps,
[0006] A lightweight encoder-decoder change detection network model is constructed, which comprises a pseudo-twin encoder with independent weights at the encoder end, and three parallel decoders at the decoder end, namely a main decoder, an auxiliary decoder 1 and an auxiliary decoder 2, for feature size recovery and accurate extraction of change detection results.
[0007] To achieve network model optimization based on labeled samples, the network model is optimized by inputting labeled dual-temporal remote sensing images into a lightweight encoder-decoder network, using the main decoder to obtain the final building change detection results, and using two auxiliary encoders only to assist in optimizing encoder network parameters and improve the network's performance in extracting deep features.
[0008] Augmentation of unlabeled image sample data;
[0009] A network model optimization based on unlabeled samples is achieved by inputting unlabeled samples from the same domain into the network model, constructing high-quality pseudo-labels by combining the output consistency of multiple decoders and the confidence threshold of the master decoder, and applying the pseudo-labels to the change detection results obtained from supervised unenhanced and enhanced image pairs.
[0010] The network model is optimized as a whole, and building changes are detected based on the optimization results.
[0011] Moreover, the pseudo-twin encoder includes two encoders with the same structure. Each encoder consists of four consecutive feature extraction modules. The first three feature extraction modules consist of two convolutional layers and a max pooling layer for downsampling, while the fourth feature extraction module contains only two convolutional layers.
[0012] Furthermore, depthwise separable convolutions are used in the third and fourth feature extraction modules of the encoder, and residual structures are added to optimize the feature extraction effect.
[0013] Moreover, each decoder at the decoder end consists of four consecutive feature recovery modules. The first three feature recovery modules each consist of a deconvolution layer for upsampling and two convolutional layers, while the fourth feature extraction module contains only two convolutional layers. The output features are used to obtain the building change detection results.
[0014] Furthermore, depthwise separable convolutional modules with residual structures are used in the first and second feature recovery modules.
[0015] Furthermore, the depth-separable convolution module includes a depthwise convolution operation D_Conv and a pointwise convolution operation P_Conv.
[0016] Furthermore, the overall optimization of the network model adopts the following network loss function.
[0017] L total =L sup +λ(L semi +L semi_aug )
[0018] The formula includes the supervised loss L for labeled samples. sup Unlabeled sample semi-supervised loss L semi And unlabeled augmented sample semi-supervised loss Lsemi_aug wherein, is a weight parameter, used to balance the importance between the labeled sample supervised loss and the unlabeled sample semi-supervised loss, to improve the network model building change detection performance.
[0019] In another aspect, the present application provides a building semi-supervised change detection device under consistency loss guidance, used to implement a building semi-supervised change detection method under consistency loss guidance as described above.
[0020] Furthermore, it comprises a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a building semi-supervised change detection method under consistency loss guidance as described above.
[0021] Alternatively, it comprises a readable storage medium, and the readable storage medium has a computer program stored thereon, and the computer program is executed to implement a building semi-supervised change detection method under consistency loss guidance as described above.
[0022] The present application overcomes the problems of parameter redundancy and insufficient data utilization of existing remote sensing image building change detection models, and realizes more accurate building change detection effect through lightweight network structure and semi-supervised learning under consistency loss guidance.
[0023] Compared with existing methods, the advantages and positive effects of the present application are: the existing building change detection method based on convolutional neural network has parameter redundancy, heavy network model and cannot effectively play the advantages of unlabeled data, and is difficult to apply to small sample application scenarios. The advantage of the present application is that the pseudo-label data is generated by consistency comparison and confidence threshold, and the semi-supervised model is learned, so as to improve the building change detection precision and improve the change detection result.
[0024] The present application scheme is simple and convenient to implement, has strong practicability, solves the problems of low practicability and inconvenience in actual application of related technologies, can improve user experience, and has important market value. BRIEF DESCRIPTION OF DRAWINGS
[0025] Figure 1 The method flowchart of the embodiment of the present application.
[0026] Figure 2 The deep separable convolution framework diagram of the embodiment of the present application. DETAILED DESCRIPTION
[0027] The technical scheme of the present application will be specifically described below in combination with the drawings and embodiments.
[0028] The application provides a high-resolution remote sensing image building semi-supervised change detection method under consistency loss guidance, adopts an end-to-end encoder-decoder model to realize model parameter optimization based on labeled samples, inputs unlabeled samples in the same domain into the network model, combines the output consistency of multiple decoders and the confidence threshold of the main decoder to construct high-quality pseudo labels, realizes cross-supervision between different decoders by using the pseudo labels, realizes consistency regularization supervision on enhanced samples, and realizes model parameter optimization based on unlabeled samples.
[0029] The application provides a high-resolution remote sensing image building semi-supervised change detection method under consistency loss guidance.
[0030] The application uses the environment: the CPU of the server is Intel Xeon E5-2665, the GPU is NVIDIA GTX1080Ti, the operating system is Ubuntu 16.04, the compiling environment is PyTorch 1.1.0, Python 3.5, CUDA9.0 and CUDNN7.1. The embodiment of the application is remote sensing image information extraction, and the specific process refers to Figure 1 The specific steps of the embodiment of the application are as follows:
[0031] Step a, constructing a lightweight encoder-decoder change detection network model
[0032] The application sets an independent-weight pseudo twin encoder at the encoder end of the lightweight encoder-decoder change detection network model, and sets three parallel decoders at the decoder end.
[0033] The bi-temporal image sample data are input into the independent-weight pseudo twin encoder, so that the deep features of the different temporal remote sensing image data are fully extracted. In order to reduce the parameter order of the network model, the depth separable convolution is used to realize the extraction of the space-time features, and the maximum pooling operation is used to realize the down-sampling of the encoder feature map, so as to reduce the memory occupation of the network features and improve the processing efficiency of the network model.
[0034] In the embodiment, the pseudo-twin encoder includes two encoders with the same structure, each of which is composed of four consecutive feature extraction modules. The first three feature extraction modules are composed of two convolution layers with a convolution kernel of 3*3 and a max-pooling layer for down-sampling. The feature channel number output by the first feature extraction module is 64, the feature channel number output by the second feature extraction module is 128, the feature channel number output by the third feature extraction module is 256, and the fourth feature extraction module only includes two convolution layers with a convolution kernel of 3*3, and the output feature channel number is 512. In order to reduce the number of network parameters required when extracting multi-dimensional features, deep separable convolution is used in the third and fourth feature extraction modules, and a residual structure is added to optimize the feature extraction effect.
[0035] The lightweight change detection network decoder end is provided with three parallel decoders for feature size recovery and accurate extraction of change detection results, which are respectively referred to as a main decoder, an auxiliary decoder 1 and an auxiliary decoder 2. Each decoder is composed of four consecutive feature recovery modules. The first three feature recovery modules are each composed of a deconvolution layer for up-sampling and two convolution layers with a convolution kernel of 3*3. The feature channel number output by the first feature extraction module is 256, the feature channel number output by the second feature extraction module is 128, the feature channel number output by the third feature extraction module is 64, and the fourth feature extraction module only includes two convolution layers with a convolution kernel of 3*3, and the output feature channel number is 2, which is used to obtain the building change detection result. Similarly, deep separable convolution modules with residual structures are used in the first and second feature recovery modules with higher dimensions.
[0036] Deep separable convolution is a prior art and is used in the model proposed in the present application. The residual structure is an auxiliary addition connection added outside the backbone network, which optimizes the model prediction performance by using the residual connection method, reduces the loss of information, reduces the risk of gradient vanishing and explosion, and reduces the loss of information.
[0037] Referring to Figure 2To reduce the number of parameters required to obtain high-dimensional feature maps, the embodiment provides a lightweight encoder-decoder change detection network model which is designed and introduced with a residual connection deep separable convolution module when the number of feature map output channels is large. The structure of the deep separable convolution module can be referred to the prior art. Through deep convolution and pointwise convolution, 128 filters (3x3), feature maps (128xHxW), 256 filters (256x1x1), and output feature maps (HxW) are gradually obtained from the input feature maps (128xHxW). In the residual connection part, the main branch first uses a max pooling layer MaxPool to realize the down-sampling of the feature map, and then uses two consecutive deep separable convolution operations (composed of deep convolution D_Conv and pointwise convolution P_Conv in series) to realize the effective extraction of high-dimensional features of the image. In addition, the structure also uses a 1x1 convolution operation Conv on the branch to transform the channel dimension of the input feature, and then realizes the effective combination of different branch feature maps based on the residual structure to improve the network performance and avoid the occurrence of gradient vanishing and explosion problems. In this module, the deep separable convolution module is used on the main path to replace the ordinary convolution operation to ensure that the same dimension feature map is obtained while greatly reducing the model parameter quantity. The deep separable convolution is composed of two consecutive operation steps, which are deep convolution operation D_Conv and pointwise convolution operation P_Conv. Among them, the deep convolution operation D_Conv completes the operation in the two-dimensional plane, and the number of convolution kernels is the same as the number of channels of the input feature map. Each convolution kernel is responsible for completing the convolution operation of the corresponding channel of the feature map, and the number of channels of the output feature map is the same as that of the input feature map. After completing the deep convolution operation, the feature map is further input into the pointwise convolution P_Conv. The feature maps of different branches of the main path and the branch are weighted and summed in the channel dimension, which can effectively reduce the network model parameter quantity and significantly increase the model depth.
[0038] Step b, realizing network model optimization based on labeled samples
[0039] The labeled dual-phase remote sensing image (x t1 ,x t2 ) is input into the lightweight encoder-decoder network, and the final building change detection result can be obtained by using the main decoder. The two auxiliary encoders are only used to assist in optimizing the encoder network parameters and improving the performance of the network in extracting deep features. The Adam optimizer and the Cross Entropy Loss cross entropy loss function are combined to realize the back propagation and optimization of the network parameters. Assuming that y is the expected output, the probability distribution a is the actual output, and C is the cross entropy, the calculation formula is:
[0040]
[0041] Wherein, x is a sample, n represents the total number of samples, the cross entropy C is the distance between the actual output and the expected output, and the smaller indicates that the probability distribution between the two is closer.
[0042] In order to highlight the importance of the main decoder, while making the auxiliary decoder play a regularization role to improve the encoder feature extraction capability, the present application further sets different weights for the cross entropy losses of the three decoders. The supervised loss L sup is obtained for the labeled samples:
[0043] L sup = mC main +n(C aux1 +C aux2 ) (2)
[0044] Wherein, C main is the cross entropy loss obtained by the main decoder, C aux1 and C aux2 are the cross entropy losses obtained by the two auxiliary decoders, and m and n are weight parameters, and m is greater than n.
[0045] Step c, data enhancement of unlabeled image sample
[0046] In order to fully play the role of unlabeled image samples in improving the performance of change detection, the present application further proposes to perform data enhancement processing on the unlabeled samples, and the enhanced mode preferably includes vertical flip, horizontal flip, 90 degree rotation, transposition and random grid disorder. Data enhancement can increase the diversity of input network data, and further improve the generalization ability of the model processing.
[0047] Step d, realizing network model optimization based on unlabeled samples
[0048] The unlabeled and enhanced dual-phase remote sensing image pairs are respectively input into the lightweight encoder-decoder network model, and the unlabeled image pairs are processed by the three parallel decoders to obtain three change detection results. In the three results, there are the same prediction values, and the pixels in the main decoder prediction result whose confidence obtained by the Softmax activation function exceeds the set threshold are selected as high-quality pseudo labels y pseo .
[0049] The pseudo labels are applied to the change detection results obtained by the supervised unlabeled and enhanced image pairs to improve the consistency of the output results of different decoders, optimize the processing ability of the model for the enhanced images, and improve the generalization effect of the network model. For the unlabeled samples, the semi-supervised Cross Entropy Loss cross entropy loss function L semi is:
[0050] L semi = mCmain (a main ,y pseo )+n(C aux1 (a aux1 ,y pseo )+C aux2 (a aux2 ,y pseo )) (3)
[0051] Wherein, a main , a aux1 , a aux2 are the change detection prediction results of the main decoder, the auxiliary decoder 1 and the auxiliary decoder 2 in the model respectively.
[0052] The generated pseudo label y pseo is subjected to an image enhancement operation, and the enhancement mode is consistent with the unlabeled sample. After enhancement, the image and the pseudo label pixel position maintain the original one-to-one correspondence. The pseudo label y pseo_aug after enhancement is used to supervise the detection result of the unlabeled enhanced sample, and the Cross Entropy Loss cross entropy loss function L semi_aug of the unlabeled enhanced sample is:
[0053] L semi_aug =mC main (a main ,y pseo_aug )+n(C aux1 (a aux1 ,y pseo_aug )+C aux2 (a aux2 ,y pseo )) (4)
[0054] Step e, the whole network model is optimized, and the building change detection is carried out according to the optimization result.
[0055] The network loss function includes three parts, the supervision loss L sup of the labeled sample (i.e. step b), the semi-supervised loss L semi of the unlabeled sample (i.e. step d) and the semi-supervised loss L semi_aug of the unlabeled enhanced sample.
[0056] In summary, the total loss function is:
[0057] L total =L sup +λ(L semi +L semi_aug ) (5)
[0058] In the formula, λ is a weight parameter, used to balance the importance between the labeled sample supervised loss and the unlabeled sample semi-supervised loss, so as to improve the building change detection performance of the network model. In a specific implementation, a user can set the value according to an actual situation.
[0059] In a specific implementation, the method proposed in the technical solution of the present application can be automatically run by a person skilled in the art using computer software technology, and the system device of the method, such as a computer readable storage medium storing the corresponding computer program of the technical solution of the present application and a computer device including the corresponding computer program, should also be within the protection scope of the present application.
[0060] In some possible embodiments, a building semi-supervised change detection device under consistency loss guidance is provided, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a building semi-supervised change detection method under consistency loss guidance as described above.
[0061] In some possible embodiments, a building semi-supervised change detection device under consistency loss guidance is provided, including a readable storage medium, and the readable storage medium has a computer program stored thereon, and the computer program is executed to implement a building semi-supervised change detection method under consistency loss guidance as described above.
[0062] The specific embodiments described herein merely illustrate the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or replace them with similar ways, without deviating from the spirit of the present application or exceeding the scope defined by the appended claims.
Claims
1. A method for building semi-supervised change detection under consistency loss guidance, characterized in that: The method comprises the following steps, A lightweight encoder-decoder change detection network model is constructed, which comprises a pseudo-twin encoder with independent weights at the encoder end, and three parallel decoders at the decoder end, namely a main decoder, an auxiliary decoder 1 and an auxiliary decoder 2, for feature size restoration and accurate extraction of change detection results; Network model optimization based on labeled samples is implemented, including inputting labeled dual-phase remote sensing images into the lightweight encoder-decoder network, using the main decoder to obtain the final building change detection result, and using the two auxiliary encoders to assist in optimizing the encoder network parameters and improving the performance of network extraction of deep features; Unlabeled image sample data enhancement is performed; Network model optimization based on unlabeled samples is implemented, including inputting unlabeled samples into the network model, constructing high-quality pseudo-labels based on the output consistency of the multiple decoders and the confidence threshold of the main decoder, and applying the pseudo-labels to the change detection results obtained from the unlabeled and enhanced image pairs; The network model is optimized as a whole, and building change detection is performed according to the optimization results.
2. The method according to claim 1, wherein the consistency loss guided building semi-supervised change detection method is characterized in that: The pseudo-twin encoder comprises two encoders with the same structure, each of which is composed of four consecutive feature extraction modules, the first three feature extraction modules are composed of two convolutional layers and one max-pooling layer for downsampling, and the fourth feature extraction module only contains two convolutional layers.
3. The method according to claim 2, wherein the consistency loss guided building semi-supervised change detection method is characterized in that: Depth separable convolution is used in the third and fourth feature extraction modules of the encoder, and a residual structure is added to optimize the feature extraction effect.
4. The method according to claim 3, wherein the consistency loss guided building semi-supervised change detection method is characterized in that: Each decoder at the decoder end is composed of four consecutive feature restoration modules, the first three feature restoration modules are composed of one deconvolutional layer for upsampling and two convolutional layers, and the fourth feature extraction module only contains two convolutional layers, and the output features are used to obtain building change detection results.
5. The method according to claim 4, wherein the consistency loss guided building semi-supervised change detection method is characterized by: The depth separable convolution module includes a depth convolution operation D_Conv and a point-by-point convolution operation P_Conv.
6. The method according to claim 5, wherein the consistency loss guided building semi-supervised change detection method is characterized by: The overall network model optimization adopts the following network loss function 7. The method according to claim 1 or 2 or 3 or 4 or 5 or 6, characterized in that: The method is used for implementing the building semi-supervised change detection method guided by the consistency loss according to any one of claims 1-7. L total = L sup + λ(L semi + L semi_aug ) wherein the labeled sample supervised loss includes a labeled sample supervised loss the unlabeled sample semi-supervised loss and the unlabeled augmented sample semi-supervised loss , is a weight parameter for balancing the importance between the labeled sample supervised loss and the unlabeled sample semi-supervised loss to improve the network model building change detection performance.
8. A building semi-supervised change detection apparatus under consistency loss guidance, characterized in that: The method comprises a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the stored instructions in the memory to execute the building semi-supervised change detection method guided by the consistency loss according to any one of claims 1-7.
9. A building semi-supervised change detection device under consistency loss guidance, characterized by: The computer program is stored on the readable storage medium, and when the computer program is executed, the building semi-supervised change detection method guided by the consistency loss according to any one of claims 1-7 is implemented.
10. A readable storage medium, characterized by: