A semi-supervised building change detection method and system based on CutMix-ResNet

Abstract

The present invention discloses a semi-supervised building change detection method and system based on CutMix-ResNet, which divides a remote sensing data set into a labeled training data set and an unlabeled test data set; performs data preprocessing on the training data set; uses the training data set to perform CutMix to obtain an enhanced data set to train a regional teacher model and a sample-level teacher model; and uses two teacher models and a small amount of labeled data sets to jointly train a student model. The present invention uses CutMix technology to perform data enhancement on the original labeled data set, and optimizes the self-supervision loss and mutual cross pseudo-supervision loss of the two teacher models, so that the jointly trained student model is more robust and complete, and the generalization performance and accuracy of the change detection model are improved, which can be used for change detection of remote sensing data.

Description

Technical Field

[0001] The present invention belongs to the technical field of change detection, and in particular relates to a semi-supervised building change detection method and system based on CutMix-ResNet. Background Art

[0002] With the accelerated development of urbanization in my country, the scale of urbanization and construction land is constantly expanding, and the land census for construction land is becoming more and more important. However, with the continuous development of my country's economy, the method of using a large amount of human resources for census is limited by low efficiency and high cost, and it is difficult to meet the actual application needs; therefore, it is crucial to quickly and accurately discover the information of surface cover changes.

[0003] Building change detection aims to extract building change information through two dual-temporal remote sensing data, through a variety of image processing and pattern recognition methods, and quantitatively analyze the characteristics and processes of surface building changes. It uses digital image processing technology to extract pixels where ground objects have changed from multi-temporal remote sensing images and assign change labels, providing qualitative and quantitative information on the spatial distribution of ground objects and their changes.

[0004] However, it is limited by the small number of labeled samples, and the number of unlabeled samples to be detected is much larger than the number of labeled samples. In addition, the traditional change detection method based on fully convolutional neural network (FNN) has low accuracy and poor detection efficiency, and cannot efficiently identify the change pattern information in the image, so as to improve the ability to identify rapid changes in remote sensing data in the construction of spatial information network.

[0005] Change detection based on semi-supervised learning requires data based on two time phases, and the training of the neural network model can be completed with only a small number of samples. The network used for general semi-supervised change detection is a fully convolutional neural network (FNN), but FNN will encounter two problems during training:

[0006] (1) The general FNN structure is shallow, such as Figure 1 As shown in the figure, it is impossible to effectively extract deep image information, resulting in poor detection effect; and as the forward propagation proceeds, more and more signal information is lost, so the features finally extracted are not complete.

[0007] (2) The receptive field of FNN detection change points is relatively wide, the generalization ability is low, and the robustness is poor. Although the FNN-based detection method can be executed in parallel, it cannot capture local change features.

[0008] The above reasons lead to the problem of incomplete extraction of local change information in change detection, and the problem of low generalization ability has not been effectively solved. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a semi-supervised building change detection method and system based on CutMix-ResNet in response to the deficiencies in the above-mentioned prior art, which uses a ResNet-based teacher-student model and CutMix technology to judge the changes of surface buildings, thereby improving the generalization performance and accuracy of the change detection model, and solving the problem that the existing deep learning change detection technology requires a large amount of supervision information but is limited by the high cost of manual labeling.

[0010] The present invention adopts the following technical solutions:

[0011] A semi-supervised building change detection method based on CutMix-ResNet, comprising the following steps:

[0012] S1. Collect three-channel RGB remote sensing data from different time phases of the same area through remote sensing satellites, perform data normalization preprocessing on each channel of the three-channel RGB remote sensing data, and obtain remote sensing dual-time image data;

[0013] S2: Divide the remote sensing dual-time image data obtained in step S1 into labeled training data sets x train And a labeled validation dataset x val ;

[0014] S3, perform CutMix operation on the remote sensing dual-time image data obtained in step S1 to obtain enhanced data, and construct a semi-supervised building change detection regional teacher model M based on CutMix-ResNet RL and the sample-level teacher model M SL ;

[0015] S4, the regional teacher model M obtained by integrating step S3 RL and the sample-level teacher model M SL , and the labeled samples x pre-labeled in step S2 train Jointly train the student model M S , and then the labeled verification data set x divided in step S2 val Input the trained student model M S In this paper, change detection based on ResNet teacher-student network is implemented.

[0016] Specifically, step S3 is as follows:

[0017] S301, the data x of the two phases t1 and x t2 Perform CutMix operation to obtain enhanced data x t1cut and x t2cut , enhanced data x t1cutand x t2cut Collectively referred to as x rl and rl ;

[0018] S302, building a basic structure of a residual block, connecting shallow and deep information through a shortcut connection operation, so that the initial input and the result processed by the convolutional network are directly added, and the result of the addition is then processed by layer normalization to form the basic structure of the residual block;

[0019] S303, building a residual block, specifically a first convolutional layer, a first normalization layer, an activation layer, a second convolutional layer, a shortcut connection of the second normalization layer and the first convolutional layer;

[0020] S304, build an activation layer connecting the residual blocks, use softmax as the activation function, and add nonlinear factors;

[0021] S305, repeat steps S303 to S304 N times to complete the process of a single encoder processing data, connect the residual block and activation layer built in step S303 to obtain a single encoder module, and then connect multiple encoder modules to obtain multiple encoder networks as the decoder of the network;

[0022] S306, build a residual neural network for classification, and use a fully connected layer after the average pooling layer to modify the output dimension of the last Encoder in the decoder of the network in step S205 to a pseudo label;

[0023] S307, using a binary cross entropy loss function to calculate the cross entropy loss of the result of the self-supervised loss item after processing in step S306, and optimizing the residual neural network;

[0024] S308, using the residual neural network built in step S306, the input data is x rl and rl , build regional teacher model M RL ;

[0025] S309, using the residual neural network built in step S306, the input data is x sl and sl , build a sample-level teacher model M SL ;

[0026] S310: The data x enhanced by CutMix in step S301 is rl and rl Input the regional teacher model M constructed in step S308 RL Calculate the cross pseudo-supervision loss Lcps1 ;

[0027] S311, the original data x sl and sl Input the regional teacher model M constructed in step S309 SL Calculate the cross pseudo-supervision loss L cps2 ;

[0028] S312, select the Adam optimization method, perform each batch of training and verification once, and save the cross entropy loss, training accuracy, verification cross entropy loss, and verification accuracy of the training process.

[0029] Furthermore, in step S301, CutMix operates as follows:

[0030] x t1cut =M⊙x t1 +(1-M)⊙x t2

[0031] x t2cut =M⊙x t2 +(1-M)⊙x t1

[0032] y t1cut =M⊙y t1 +(1-M)⊙y t2

[0033] y t2cut =M⊙y t2 +(1-M)⊙y t1

[0034] Among them, M represents Mask, ⊙ is pixel-level multiplication, x t1cut and x t2cut are two-phase sample data enhanced by CutMix, x t1cut and x t2cut Collectively referred to as x rl ,y t1cut and t2cut are the two-phase label data enhanced by CutMix, y t1cut and t2cut Collectively referred to as y rl .

[0035] Furthermore, in step S302, the shortcut connection operation is as follows:

[0036] H(x)=F(x)+x

[0037] Among them, H(x) is the observation value, F(x) is the residual mapping function of the solution network, and x is the feature map output by the previous layer of ResNet.

[0038] Furthermore, in step S304, the softmax is calculated as follows:

[0039]

[0040] Among them, S i is the predicted probability, ∑ j e j is the sum of the input values, e i For each input value.

[0041] Furthermore, in step S307, the cross entropy loss is as follows:

[0042]

[0043] x n =M teacher (x)

[0044] Among them, S represents the number of samples in the batch, ∑ represents the sum of the subsequent operation results, y represents the label, x represents the input data, and M teacher () represents the Enconder-Deconder operation of the teacher model, x n Represents the classification result predicted by the neural network.

[0045] Specifically, in step S310, the regional teacher model M RL and the sample-level teacher model M SL The objective function L RL and L SL They are:

[0046] L RL =L ce (M RL (x rl ),y rl )+L ce (M SL (x sl ),y rl )

[0047] L SL =L ce (M SL (x sl ),y sl )+L ce (M RL (x rl ),y sl )

[0048] Regional Teacher Model M RL and the sample-level teacher model M SLThe optimization objectives are respectively:

[0049] argmin(L RL )

[0050] argmin(L SL )

[0051] where x t1 and x t2 The data and corresponding labels at two time phases are x sl , y sl , L RL and L SL respectively represent the objective functions of the regional teacher model M RL and the sample-level teacher model M SL , L ce represents the cross-entropy loss function, L ce (M RL (x rl ), y rl ) and L ce (M SL (x sl ), y sl ) represent the self-supervised loss, L ce (M SL (x sl ), y rl ) and L ce (M RL (x rl ), y sl ) represent the cross-pseudo-supervised loss.

[0052] Specifically, step S4 is specifically as follows:

[0053] S401. Repeat the process of a single Encoder processing data in step S3 for M times, where M < N and N is the number of times of a single Encoder processing data, to construct a student residual neural network;

[0054] S402. Integrate the regional teacher model M RL and the sample-level teacher model M SL , and average the outputs of the two complementary networks to be M mean , and perform knowledge distillation on the student model in step S403;

[0055] S403. The outputs of the regional teacher model M RL and the sample-level teacher model M SL are integrated into stronger guidance to supervise the training of the student model M S on unlabeled target data, and the optimization objective of the student model is argmin(L S );

[0056] S404, select the cross entropy loss function, use the Adam optimization algorithm, perform validation once after each batch of training, and save the cross entropy loss, training accuracy, validation cross entropy loss, and validation accuracy of the training process.

[0057] Further, in step S402, the regional teacher model M RL and the sample-level teacher model N SL The integration is expressed as:

[0058]

[0059] Among them, E() means taking the average of the two teachers’ outputs, x u Represents a large number of unlabeled samples, q represents the probability of each category output, and μ represents the weight.

[0060] Another technical solution of the present invention is a semi-supervised building change detection system based on CutMix-ResNet, comprising:

[0061] The processing module collects three-channel RGB remote sensing data from remote sensing satellites at different time phases of the same area, performs data normalization preprocessing on each channel of the three-channel RGB remote sensing data, and obtains remote sensing dual-time image data;

[0062] The division module divides the remote sensing dual-time image data obtained by the processing module into labeled training data sets x train And a labeled validation dataset x val ;

[0063] The enhancement module performs CutMix operation on the remote sensing dual-time image data obtained by the processing module to obtain enhanced data and constructs a regional-level teacher model M for semi-supervised building change detection based on CutMix-ResNet. RL and the sample-level teacher model M SL ;

[0064] Detection module, integrated enhancement module to obtain the regional teacher model M RL and the sample-level teacher model M SL , and the labeled samples x pre-labeled by the partitioning module train Jointly train the student model M S , and then divide the labeled verification data set x divided by the partition module val Input the trained student model M S In this paper, change detection based on ResNet teacher-student network is implemented.

[0065] Compared with the prior art, the present invention has at least the following beneficial effects:

[0066] The invention discloses a semi-supervised building change detection method based on CutMix-ResNet. CutMix is ​​used to enhance the original data so that the network can learn the change features in a smaller area, and the sensitivity of the network to the change points can be effectively improved. Thanks to the global feature extraction capability of ResNet and the good anti-degradation advantage of deep network, the network structure of ResNet is deeper than that of ordinary CNN. After the accuracy of ordinary CNN system saturates and starts to decline, the system accuracy of ResNet can continue to increase, and the computing power of GPU can be fully utilized.

[0067] Furthermore, in step S3, a regional teacher model M is constructed. RL and the sample-level teacher model M SL , making the overall change detection effect more robust after knowledge distillation in step S4.

[0068] Furthermore, in step S301, a CutMix operation is performed on the original data to obtain enhanced data, so that the model is more robust and the regional teacher model M RL Focus on changes in smaller buildings.

[0069] Furthermore, in step S302, in ResNet, a deep neural network structure is used to extract features, which improves the network's ability to extract deep features, but problems follow: as the number of network layers increases, gradient diffusion / explosion will occur. Some methods can compensate for this, such as normalized initialization processing, normalization of inputs at each layer, so that the depth of the network that can converge is increased tenfold. However, although the network converges, the network is greatly degraded, so that increasing the number of network layers leads to greater errors, so a shortcut-connection operation is needed to connect the shallow and deep features.

[0070] Furthermore, in step S304, softmax is added to map the output to the interval [0,1] to facilitate classification.

[0071] Furthermore, in step S307, when the mean square error loss function is used, the gradient is proportional to the gradient of the activation function. However, for the sigmoid activation function, the input is z and the output is σ(z). When σ(z) is large, σ′(z) is very small, almost 0, resulting in slow parameter updates. However, for the cross entropy loss function, the gradient at this time is independent of the activation function, and when the difference between the predicted result σ(z) and the true result y is larger, the larger the gradient, the faster the parameter update. When the difference between the predicted result and the true result is small, the gradient becomes smaller, and the parameter is basically in a stable state, which helps the neural network converge quickly.

[0072] Further, in step S310, the regional teacher model M RL and the sample-level teacher model M SL The objective function L RL and L SL They each contain two items, the self-supervised loss term and the cross-pseudo-supervised loss term. The introduction of the cross-pseudo-supervised loss term can make the model more resistant to interference and act as a regular term.

[0073] Furthermore, in step S311, the Adam optimization algorithm is used to ensure that the gradient has a momentum value, which indirectly ensures that the optimization will continue and will not reach a saddle point. In addition, the historical gradient information is used to correct the current gradient to eliminate the oscillation during variable update, thereby obtaining a stable gradient update value, and the second-order moment estimate of the gradient is corrected to obtain the learning rate weight parameter of each weight parameter. At the same time, the exponential moving average of the square of the gradient is combined to adjust the change of the learning rate. That is to say, if the gradient changes too quickly, the learning rate is reduced, and if the gradient changes too slowly, the learning rate is increased.

[0074] Further, in step S4, the regional teacher model M RL and the sample-level teacher model M SL For students network S By performing knowledge distillation, the student network integrates the advantages of the two teacher networks and its own parameters are simpler than those of the teacher network.

[0075] Further, in step S402, through the regional teacher model M RL and the sample-level teacher model M SL The integration of knowledge distillation guides the student network M S During the training process, the outputs of the two ResNet teacher models are integrated as stronger guidance, along with a small number of labeled samples, to guide the student model and obtain prediction results.

[0076] In summary, the present invention integrates CutMix technology into the teacher-student model for change detection through a deep learning method, thereby achieving accurate judgment of changes in surface buildings.

[0077] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a typical fully convolutional neural network diagram;

[0079] Figure 2 Training graphs for the region-level teacher model and sample-level teacher model;

[0080] Figure 3This is a schematic diagram of Mask;

[0081] Figure 4 This is the residual neural network architecture diagram;

[0082] Figure 5 This is a schematic diagram of the shortcut connection operation;

[0083] Figure 6 Training graph for student model;

[0084] Figure 7 Schematic diagram of CutMix operation on remote sensing dual-temporal image data. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0087] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0088] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship.

[0089] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0090] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0091] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0092] The present invention provides a semi-supervised building change detection method based on CutMix-ResNet, which divides a remote sensing data set into a labeled training data set and an unlabeled test data set; performs data preprocessing on the training data set; uses the training data set to perform CutMix to obtain an enhanced data set to train a regional teacher model and a sample-level teacher model; and uses two teacher models and a small amount of labeled data sets to jointly train a student model. The present invention uses CutMix technology to perform data enhancement on the original labeled data set, and optimizes the self-supervision loss and mutual cross pseudo-supervision loss of the two teacher models, so that the jointly trained student model is more robust and complete, and the generalization performance and accuracy of the change detection model are improved, which can be used for change detection of remote sensing data.

[0093] See also Figure 2 The present invention provides a semi-supervised building change detection method based on CutMix-ResNet, comprising the following steps:

[0094] S1. Collect three-channel RGB remote sensing data from remote sensing satellites at different phases of the same area. The image size is 1024×1024×3 dual-phase data. Each channel of the image is divided by 255 for data normalization preprocessing and uniformly processed into a 1024×1024×3 labeled training data set x l And divided into x according to different phases t1 and x t2 ;

[0095] The processing of dividing each RGB channel of remote sensing data by 255 is mainly considered to perform normalization processing and then deep convolution processing, which can make the network converge faster.

[0096] S2, remote sensing dual-time image data is divided into labeled training data sets x train And a labeled validation dataset x val ;

[0097] The collected labeled training data set x l 80% of the data is divided into labeled training data set x train , divide the remaining 20% ​​of the dataset into a labeled validation dataset x val ; Use 80% of the data for training and the other 20% for validation. This 20% does not participate in training but is used to evaluate the performance of the model. The performance of the parameters is verified in continuous "training-validation" iterations to achieve the purpose of selecting the correct hyperparameters.

[0098] S3. Construct a semi-supervised building change detection region-level M based on CutMix-ResNet RL Teacher model and sample level M SL Teacher Model;

[0099] S301, x t1 and x t2 The data of the two phases are cut mixed to obtain the enhanced data x t1cut and x t2cut ;

[0100] See also Figure 3 and Figure 7 , the CutMix operation is described as follows:

[0101]

[0102] Among them, M represents Mask, ⊙ is pixel-level multiplication, x t1cut and x t2cut are two-phase sample data enhanced by CutMix, x t1cut and x t2cut Collectively referred to as x rl ,y t1cut and t2cut are the two-phase label data enhanced by CutMix, y t1cut and t2cut Collectively referred to as y rl .

[0103] S302, build a residual neural network, connect the shallow and deep information through the shortcut connection operation, add the initial input and the result processed by the convolutional network directly, and then use layer normalization to process the added result. The normalization process makes the parameters of the hidden layer independent and identically distributed, which can speed up the training speed and convergence. Figure 4 and Figure 5 As shown;

[0104] The shortcut connection operation is described as follows:

[0105] H(x)=F(x)+x (2)

[0106] Among them, H(x) is the observation value, F(x) is the residual mapping function of the solution network, and x is the feature map output by the previous layer of ResNet.

[0107] S303, build a residual block, which is essentially a shortcut connection between convolution layer 1-standardization layer 1-activation layer-convolution layer 2-standardization layer 2 and convolution layer 1;

[0108] S304, constructing an activation layer connecting the residual blocks, using softmax as an activation function to add nonlinear factors and improve the expression ability of the neural network to the model;

[0109] Among them, the softmax calculation method is:

[0110]

[0111] S305, repeating steps S303 to S304 N times. Steps S303 to S304 are the flow of a single encoder processing data. The residual network block and the activation layer are connected to obtain a single encoder, and then multiple such encoders are connected to obtain multiple encoder networks. Multiple encoder networks are connected to make the network have stronger generalization ability;

[0112] S306. Build a classification network and use a fully connected layer after the average-pooling layer to modify the output dimension of the last Encoder to a pseudo label.

[0113] Since the fully connected layer is prone to overfitting and heavily relies on dropout regularization, a global average pooling layer is used for regularization to prevent overfitting of the overall structure.

[0114] Among them, the fully connected layer is described as:

[0115] z=W Tx+b (4)

[0116] z is the output of the fully connected layer, x is the input of the fully connected layer, b is the bias vector, and W is the weight matrix.

[0117] S307. The result of the processed self-supervisory loss term is used to calculate the cross entropy loss using a binary cross entropy loss function, as shown in formula (5):

[0118]

[0119] x n =M teacher (x) (6)

[0120] Among them, S represents the number of samples in the batch processing, ∑ represents the sum of the subsequent operation results, y represents the label, which is 1 or 0 here, 1 means the building has changed, 2 means it remains unchanged, ln represents the logarithm, x represents the input data, and M teacher () represents the Enconder-Deconder operation of the teacher model, x n Represents the classification result predicted by the neural network.

[0121] S308. Build regional teacher model M RL : Use the residual neural network built in step S306, and the input data is x rl and rl ;

[0122] S309. Build a sample-level teacher model M SL : Use the residual neural network built in step S306, and the input data is x sl and sl ;

[0123] S310, the CutMix enhanced data x rl and rl Enter the regional teacher model M RL Calculate the cross pseudo-supervision loss L cps1 :

[0124] L cps1 =L ce (M RL (x rl ),y rl ) (7)

[0125] S311, the original data x sl and sl Enter the regional teacher model M SL Calculate the cross pseudo-supervision loss L cps2 :

[0126] Lcps2 =L ce (M SL (x sl ),y sl ) (8)

[0127] Thus, the regional teacher model M RL and the sample-level teacher model M SL The objective function L RL and L SL They are:

[0128]

[0129] Furthermore, the regional teacher model M RL and the sample-level teacher model M SL The optimization goal is:

[0130]

[0131] Among them, x t1 and x t2 The data of the two phases and the corresponding labels are collectively referred to as x sl ,y sl , L RL and L SL Represents the regional teacher model M RL and the sample-level teacher model M SL The objective function, L ce represents the cross entropy loss function, L ce (M RL (x rl ),y rl ) and L ce (M SL (x sl ),y sl ) represents the self-supervision loss, L ce (M SL (x sl ),y rl ) and L ce (M RL (x rl ),y sl ) represents the cross pseudo-supervision loss.

[0132] S312, select the Adam optimization method, such as formula (11):

[0133]

[0134] Among them, w t represents the original gradient, α represents the learning rate, represents the correction to the first-order moment estimate of the gradient, Represents the correction of the second moment estimation of the gradient, and ε represents a constant added to maintain numerical stability.

[0135] The cross-entropy loss function is selected, the Adam optimization algorithm is used, and it is verified once every time a batch of training is completed. The cross-entropy loss, training accuracy, validation cross-entropy loss, and validation accuracy during the training process are saved.

[0136] S4. By integrating the region-level teacher model M RL and the sample-level teacher model M SL , as well as a small number of labeled samples x sl , jointly train the student model (M S ), and then input the validation data set x val divided in step S2 into the trained student model to implement change detection based on the ResNet teacher-student network.

[0137] S401. Repeat the work in steps S303 - S304 for M times, where M < N, to construct the student residual neural network;

[0138] S402. Perform an integration operation on the region-level teacher model M RL and the sample-level teacher model M SL , and average the outputs of the two complementary networks to obtain M mean ;

[0139] M RL , M SL The integration (knowledge distillation) is expressed as:

[0140]

[0141] where E() represents taking the average of the two teacher outputs, x u represents a large number of unlabeled samples, q represents the probability of each class output, and μ represents the weight.

[0142] S403. The outputs of the two teacher models M RL , M SL are integrated into stronger guidance to supervise the training of the student model M S on the unlabeled target data; in addition, the student model M S is also supervised by the labels on a small number of true labels label;

[0143] Please refer to Figure 6 , the student model M S obtains knowledge by minimizing the KL-divergence between the outputs of the two models M RL , M SL . At the same time, the student model M SAlso received a small amount of marked label y l supervision.

[0144] Student Model M S The objective function L S for:

[0145] L S =λ kl L KL (E(M RL (x u ),M SL (x u )), M S (x u ))+λ ce L ce (M S (x l ),y l ) (13)

[0146] Among them, L KL represents the KL divergence loss, λ kl and λ ce are KL divergence weight and cross entropy loss weight respectively, x u A large amount of unlabeled data.

[0147] Furthermore, the optimization goal of the final student model is:

[0148] argmin(L S ) (14)

[0149] S404, select the Adam optimization method, select the cross entropy loss function, use the Adam optimization algorithm, complete each batch of training and verification once, save the cross entropy loss, training accuracy, verification cross entropy loss, and verification accuracy of the training process.

[0150] In another embodiment of the present invention, a semi-supervised building change detection system based on CutMix-ResNet is provided, which can be used to implement the above-mentioned semi-supervised building change detection method based on CutMix-ResNet. Specifically, the semi-supervised building change detection system based on CutMix-ResNet includes a processing module, a division module, an enhancement module and a detection module.

[0151] Among them, the processing module collects three-channel RGB remote sensing data from different phases of the same area through remote sensing satellites, performs data normalization preprocessing on each channel of the three-channel RGB remote sensing data, and obtains remote sensing dual-time image data;

[0152] The division module divides the remote sensing dual-time image data obtained by the processing module into labeled training data sets xtrain And a labeled validation dataset x val ;

[0153] The enhancement module performs CutMix operation on the remote sensing dual-time image data obtained by the processing module to obtain enhanced data and constructs a regional-level teacher model M for semi-supervised building change detection based on CutMix-ResNet. RL and the sample-level teacher model M SL ;

[0154] Detection module, integrated enhancement module to obtain the regional teacher model M RL and the sample-level teacher model M SL , and the labeled samples x pre-labeled by the partitioning module train Jointly train the student model M S , and then divide the labeled verification data set x divided by the partition module val Input the trained student model M S In this paper, change detection based on ResNet teacher-student network is implemented.

[0155] In summary, the present invention provides a semi-supervised building change detection method and system based on CutMix-ResNet. CutMix is ​​used to enhance the original data so that the network can learn the change features in a smaller area, which can effectively improve the network's sensitivity to change points. ResNet is used to effectively solve the learning degradation problem of deep networks and effectively reduce the output error of the network.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0160] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A semi-supervised building change detection method based on CutMix-ResNet, characterized in that: It includes the following steps: S1. Collect three-channel RGB remote sensing data for different time phases of the same area through a remote sensing satellite, and perform data normalization preprocessing on each channel of the three-channel RGB remote sensing data to obtain remote sensing double-temporal image data; S2: Divide the remote sensing dual-time image data obtained in step S1 into labeled training data sets And a labeled validation dataset ; S3, perform CutMix operation on the remote sensing dual-time image data obtained in step S1 to obtain enhanced data, and construct a semi-supervised building change detection regional teacher model M based on CutMix-ResNet RL and the sample-level teacher model M SL , specifically: S301, the data of the two phases and Perform CutMix operation to obtain enhanced data and , augmented data and Collectively referred to as and ; S302. Build the basic structure of the residual block, connect the shallow and deep information through the shortcut connection operation, directly add the initial input and the result processed by the convolutional network, and then use layer normalization processing on the added result to form the basic structure of the residual block; S303. Build the residual block, specifically including the first convolutional layer, the first normalization layer, the activation layer, the second convolutional layer, the second normalization layer, and the shortcut connection of the first convolutional layer; S304, build the activation layer connecting the residual blocks, using As an activation function, nonlinear factors are added; S305. Repeat the work in steps S303 - S304 for N times to complete the process of a single Encoder processing data. Connect the residual block and the activation layer built in step S303 to obtain a single Encoder module, and then connect multiple Encoder modules to obtain multiple Encoder networks as the decoder of the network; S306. Build a residual neural network for classification, and use a fully connected layer after the average pooling layer to modify the output dimension of the last Encoder in the decoder of the network in step S205 to a pseudo-label; S307. Calculate the cross-entropy loss using the binary cross-entropy loss function for the result of the self-supervised loss term processed in step S306 to optimize the residual neural network; S308, using the residual neural network built in step S306, the input data is and , build a regional teacher model ; S309, using the residual neural network built in step S306, the input data is and , build a sample-level teacher model ; S310: The data enhanced by CutMix in step S301 and Enter the regional teacher model built in step S308 Calculating cross-pseudo-supervision loss ; S311, the original data and Enter the regional teacher model built in step S309 Calculating cross-pseudo-supervision loss ; S312. Select the Adam optimization method, verify once every time a batch of training is completed, and save the cross-entropy loss, training accuracy, validation cross-entropy loss, and validation accuracy during the training process; S4, the regional teacher model M obtained by integrating step S3 RL and the sample-level teacher model M SL , and the labeled samples pre-labeled in step S2 Jointly train the student model , and then the labeled verification data set divided in step S2 Input the trained student model In this paper, change detection based on ResNet teacher-student network is implemented.

2. The semi-supervised building change detection method based on CutMix-ResNet according to claim 1, characterized in that: In step S301, the CutMix operation is as follows: in, Indicates Mask, is pixel-level multiplication, and They are two-phase sample data enhanced by CutMix. and Collectively referred to as , and They are the two-phase label data enhanced by CutMix. and Collectively referred to as .

3. The semi-supervised building change detection method based on CutMix-ResNet according to claim 1, characterized in that: In step S302, the shortcut connection operation is as follows: in, is the observed value, To solve the residual mapping function of the network, It is the feature map output by the previous layer of ResNet.

4. The semi-supervised building change detection method based on CutMix-ResNet according to claim 1, characterized in that: In step S304, The calculation is as follows: in, is the predicted probability, is the sum of the input values, For each input value.

5. The semi-supervised building change detection method based on CutMix-ResNet according to claim 1, characterized in that: In step S307, the cross-entropy loss is as follows: in, represents the number of samples in the batch, ∑ represents the sum of the subsequent operation results, Indicates the label, Represents input data, Represents the Enconder-Deconder operation of the teacher model, Represents the classification result predicted by the neural network.

6. The semi-supervised building change detection method based on CutMix-ResNet according to claim 1, characterized in that: In step S310, the regional teacher model and sample-level teacher model The objective function and They are: Regional Teacher Model M RL and the sample-level teacher model M SL The optimization objectives are: in, and The data of the two phases and their corresponding labels are , , and Represents the regional teacher model and sample-level teacher model The objective function is represents the cross entropy loss function, and represents the self-supervised loss, and represents the cross pseudo-supervision loss.

7. The semi-supervised building change detection method based on CutMix-ResNet according to claim 1, characterized in that: Step S4 is specifically: S401. Repeat the work in the process of a single Encoder processing data in step S3 for M times, where M < N and N is the number of times of a single Encoder processing data, to build a student residual neural network; S402, regional teacher model and sample-level teacher model Perform an integration operation and average the output of the two complementary networks to be , in step S403, knowledge distillation is performed on the student model; S403, Regional Teacher Model and sample-level teacher model The output of is integrated as a stronger guide to supervise the student model When training on unlabeled target data, the optimization objective of the student model is ; S404. Select the cross-entropy loss function, use the Adam optimization algorithm, verify once every time a batch of training is completed, and save the cross-entropy loss, training accuracy, validation cross-entropy loss, and validation accuracy during the training process.

8. The semi-supervised building change detection method based on CutMix-ResNet according to claim 7, characterized in that: In step S402, the regional teacher model and sample-level teacher model The integration is expressed as: in, It means taking the average output of two teachers. represents a large number of unlabeled samples, represents the probability of each category output, Represents weight.

9. A semi-supervised building change detection system based on CutMix-ResNet, characterized in that: It includes: A processing module that collects three-channel RGB remote sensing data for different time phases of the same area through a remote sensing satellite, and performs data normalization preprocessing on each channel of the three-channel RGB remote sensing data to obtain remote sensing double-temporal image data; The division module divides the remote sensing dual-time image data obtained by the processing module into labeled training data sets And a labeled validation dataset ; The enhancement module performs CutMix operation on the remote sensing dual-time image data obtained by the processing module to obtain enhanced data and constructs a regional-level teacher model M for semi-supervised building change detection based on CutMix-ResNet. RL and the sample-level teacher model M SL , specifically: The data of two phases and Perform CutMix operation to obtain enhanced data and , augmented data and Collectively referred to as and ; Build the basic structure of the residual block, connect the shallow and deep information through the shortcut connection operation, directly add the initial input and the result processed by the convolutional network, and then use layer normalization processing on the added result to form the basic structure of the residual block; Build a residual block, specifically the first convolutional layer, the first normalization layer, the activation layer, the second convolutional layer, the second normalization layer and the shortcut connection of the first convolutional layer; Build the activation layer connected between the residual blocks, using As an activation function, nonlinear factors are added; Repeat the above steps N times to complete the process of a single Encoder processing data, connect the residual block and activation layer built in step S303 to obtain a single Encoder module, and then connect multiple Encoder modules to obtain multiple Encoder networks as the decoder of the network; Build a residual neural network for classification, and use a fully connected layer after the average pooling layer to modify the output dimension of the last encoder in the decoder of the network to a pseudo label; The processed self-supervised loss term is used to calculate the cross entropy loss using a binary cross entropy loss function to optimize the residual neural network. Using the built residual neural network, the input data is and , build a regional teacher model ; Using the built residual neural network, the input data is and , build a sample-level teacher model ; CutMix enhanced data and Enter the regional teacher model Calculating cross-pseudo-supervision loss ; The original data and Enter the regional teacher model Calculating cross-pseudo-supervision loss ; Select the Adam optimization method, complete each batch of training and verification, and save the cross entropy loss, training accuracy, verification cross entropy loss, and verification accuracy of the training process; Detection module, integrated enhancement module to obtain the regional teacher model M RL and the sample-level teacher model M SL , and the labeled samples pre-annotated by the partitioning module Jointly train the student model , and then divide the labeled verification data set divided by the division module Input the trained student model In this paper, change detection based on ResNet teacher-student network is implemented.

Images