Lightweight remote sensing change detection method based on non-back propagation learning

Abstract

The invention provides a lightweight remote sensing change detection method based on non-back propagation learning, and the method comprises the steps: carrying out the uniform segmentation of a to-be-reasoned remote sensing image before and after the change, and obtaining a sub-image set; using an affinity matrix method based on block dimension to select changed sub-images before and after change and non-changed sub-images before and after change from the sub-image set as training samples of the neural network; a training sample is used, a local learning method is adopted for fine tuning of the neural network, during fine tuning, a loss function is allocated to a convolutional layer needing to be trained for fine tuning, parameters of other convolutional layers are kept unchanged, the other convolutional layers are made to execute forward propagation, and the neural network after fine tuning is obtained; and inputting to-be-reasoned remote sensing images before and after change into the neural network for reasoning, and binarizing a reasoning result to obtain a binary change result. According to the scheme, the method is suitable for heterogeneous remote sensing image change detection when computing resources are limited.

Description

Technical Field

[0001] This application relates to the technical field of heterogeneous remote sensing image change detection, and particularly to a lightweight remote sensing change detection method and device based on non-backpropagation learning. Background Art

[0002] Change detection is of great significance in military detection, land use planning, disaster rescue, etc. It can timely detect the information of the actual affected areas after disasters such as debris flows, earthquakes, and typhoons and guide rescue operations.

[0003] In recent years, many scholars have proposed many methods for performing change detection tasks using traditional machine learning techniques. For example, the pixel-level homogeneous transformation method based on K-nearest neighbors and the change detection method based on dictionary learning. However, the above methods are relatively traditional with low detection accuracy and usually require a large amount of computing resources. This further limits the actual deployment and application of such methods. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems in the related art to some extent.

[0005] To this end, the first object of this application is to propose a lightweight remote sensing change detection method based on non-backpropagation learning, which is applicable to heterogeneous remote sensing image change detection when computing resources are limited.

[0006] The second object of this application is to propose a lightweight remote sensing change detection device based on non-backpropagation learning.

[0007] The third object of this application is to propose a computer device.

[0008] The fourth object of this application is to propose a non-transitory computer-readable storage medium.

[0009] To achieve the above object, the first aspect embodiment of this application proposes a lightweight remote sensing change detection method based on non-backpropagation learning, including:

[0010] Uniformly slice the remote sensing images before and after the change to be inferred to obtain a set of sub-images;

[0011] Use the affinity matrix method based on block dimensions to select the sub-images before and after the change that have changed and the sub-images before and after the change that have not changed from the set of sub-images as the training samples of the neural network;

[0012] Use the training samples and adopt a local learning method to fine-tune the neural network. When fine-tuning, assign a loss function to the convolutional layer to be trained for fine-tuning, and keep the parameters of other convolutional layers unchanged, so that other convolutional layers perform forward propagation to obtain the fine-tuned neural network;

[0013] Input the remote sensing images before and after the change to be inferred into the neural network for inference, and binarize the inference result to obtain the binary change result.

[0014] Optionally, in an embodiment of the present application, using the affinity matrix method based on the block dimension, select the pre-change and post-change sub-images that have changed and the pre-change and post-change sub-images that have not changed from the sub-image set as the training samples of the neural network, including:

[0015] Solve the distance between each sub-image and other sub-images on the remote sensing images before and after the change, expressed as:

[0016]

[0017] where the superscript m is pre or post, pre and post represent before and after the change respectively, and M i is the i-th sub-image;

[0018] Measure the possibility of sub-image change according to the exponential function of the distance. When using the affinity matrix for measurement, the expression is:

[0019]

[0020] where, is the normalization parameter, represented by the maximum value of all :

[0021] ;

[0022] Sum up the possibility of change for each sub-image and normalize it to obtain the change probability of each sub-image. Among them, summing up the possibility of change for the i-th sub-image is expressed as:

[0023]

[0024] where N is the number of splits of the remote sensing image;

[0025] Based on the change probability of each sub-image, select several pre-change and post-change sub-images that are most likely to change and several pre-change and post-change sub-images that are least likely to change from the sub-image set as the training samples of the neural network.

[0026] Optionally, in an embodiment of the present application, assign a loss function to the convolutional layer that needs to be trained, including:

[0027] Take the deep convolutional layer as the convolutional layer that needs to be trained. For the deep convolutional layer, construct a contrastive learning loss function to increase the output feature difference of the changed samples and reduce the output feature difference of the unchanged samples. Among them, the contrastive learning loss function is:

[0028]

[0029] Among them, L m represents the loss function assigned to the m-th convolutional layer to be trained, processing the output features of the samples with changes and the samples without changes respectively, t 1 and t 2 are sub-networks for processing the image before change and the image after change respectively, represents the output feature of the image before change, represents the output feature of the image after change,

[0030] (x, y, z) represents the output coordinates, W m and H m and C m represent the width, length, and number of channels of the output feature;

[0031] Fine-tuning the deep convolutional layer includes:

[0032] Using the contrastive learning loss function and the output feature values to calculate the gradient of the output feature, expressed as:

[0033]

[0034] Among them, gO is the gradient of the output feature O, and s(·) is the sign function, expressed as:

[0035]

[0036] Calculating the weight gradient based on the gradient of the output feature, expressed as:

[0037]

[0038] Among them, is the weight gradient of the m-th layer of the sub-network t p , * represents the convolution operation, is the input of the m-th layer of the sub-network t p ;

[0039] Updating the weights based on the weight gradient, expressed as:

[0040]

[0041] Among them, k represents the number of iterations.

[0042] Optionally, in an embodiment of the present application, the remotely sensed images before and after the change to be inferred are input into the neural network for inference, and the inference result is obtained, expressed as:

[0043]

[0044] I d = D x + U(D x+1 ) +... + U(D x+n )

[0045] where C m is the number of channels of the m-th convolutional layer, x, y, z are the coordinates of the output, t 1 , t 2 are sub-networks for processing the pre-changed image and the post-changed image respectively, pre and post represent before change and after change respectively, represents the output features of the pre-changed image, represents the output features of the post-changed image, D x+n , D x+1 ,..., D x+n are convolutional layers to be trained, x + n is the number of convolutional layers of the neural network, I d is the changed image;

[0046] Using the Otsu method of maximum inter-class variance to obtain the binarization result, which is expressed as:

[0047] I b = Otsu(I d ).

[0048] To achieve the above object, the second aspect embodiment of the present invention proposes a lightweight remote sensing change detection device based on non-backpropagation learning, including:

[0049] An image segmentation module for uniformly segmenting the remote sensing images before and after change to be inferred to obtain a sub-image set;

[0050] A sample selection module for using the affinity matrix method based on block dimension to select the pre-and post-changed sub-images with changes and the pre-and post-changed sub-images without changes from the sub-image set as the training samples of the neural network;

[0051] A neural network fine-tuning module for using the training samples to fine-tune the neural network by using the local learning method. During fine-tuning, a loss function is assigned to the convolutional layers to be trained for fine-tuning, and the parameters of other convolutional layers are kept unchanged, so that other convolutional layers perform forward propagation to obtain the fine-tuned neural network;

[0052] A change detection module for inputting the remote sensing images before and after change to be inferred into the neural network for inference and binarizing the inference result to obtain a binary change result.

[0053] Optionally, in an embodiment of the present application, a method based on a block - dimension affinity matrix is used to select the pre - and post - change sub - images that have changed and the pre - and post - change sub - images that have not changed from the sub - image set as the training samples of the neural network, including:

[0054] Solve the distance between each sub - image and other sub - images on the remote - sensing image before and after the change, expressed as:

[0055]

[0056] where the superscript m is pre or post, pre and post represent before and after the change respectively, and W i is the i - th sub - image;

[0057] Measure the possibility of sub - image change according to the exponential function of the distance. When using the affinity matrix for measurement, the expression is:

[0058]

[0059] where, is the normalization parameter, represented by the maximum value of all :

[0060] ;

[0061] Sum and normalize the possibility of change for each sub - image to obtain the change probability of each sub - image. Among them, the sum of the possibility of change for the i - th sub - image is expressed as:

[0062]

[0063] where N is the number of segments of the remote - sensing image;

[0064] Based on the change probability of each sub - image, select several pre - and post - change sub - images that are most likely to change and several pre - and post - change sub - images that are least likely to change from the sub - image set as the training samples of the neural network.

[0065] Optionally, in an embodiment of the present application, assign a loss function to the convolutional layer that needs to be trained, including:

[0066] Regard the deep convolutional layer as the convolutional layer that needs to be trained. For the deep convolutional layer, construct a contrastive learning loss function to increase the output feature difference of the changed samples and reduce the output feature difference of the unchanged samples. Among them, the contrastive learning loss function is:

[0067]

[0068]

[0069] Among them, L m represents the loss function assigned to the m-th convolutional layer to be trained, processing the output features of the samples with changes and the samples without changes respectively, t 1 and t 2 are sub-networks for processing the image before change and the image after change respectively, represents the output feature of the image before change, represents the output feature of the image after change,

[0070] (x, y, z) represents the output coordinates, W m and H m and C m represent the width, length and number of channels of the output feature;

[0071] Fine-tuning the deep convolutional layer includes:

[0072] Using the contrastive learning loss function and the output feature values to calculate the gradient of the output feature, expressed as:

[0073]

[0074] Among them, gO is the gradient of the output feature O, and s(·) is the sign function, expressed as:

[0075]

[0076] Calculating the weight gradient based on the gradient of the output feature, expressed as:

[0077]

[0078] Among them, is the weight gradient of the m-th layer of the sub-network t p , * represents the convolution operation, is the input of the m-th layer of the sub-network t p ;

[0079] Updating the weight based on the weight gradient, expressed as:

[0080]

[0081] Among them, k represents the number of iterations.

[0082] Optionally, in an embodiment of the present application, the remotely sensed images before and after the change to be inferred are input into the neural network for inference, and the inference result is obtained, expressed as:

[0083]

[0084] I d= D x + U(D x+1 ) +... + U(D x+n )

[0085] where C m is the number of channels of the m-th convolutional layer, x, y, z are the coordinates of the output, and t 1 , t 2 are sub-networks for processing the image before change and the image after change respectively, pre and post represent before change and after change respectively, represents the output feature of the image before change, represents the output feature of the image after change, D x+n , D x+1 ,..., D x+n are convolutional layers to be trained, x + n is the number of convolutional layers of the neural network, and I d is the changed image;

[0086] Using the Otsu method of maximum inter-class variance to obtain the binarization result, which is expressed as:

[0087] I b = Otsu(I d ).

[0088] To achieve the above object, an embodiment of the third aspect of the present invention proposes a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above lightweight remote sensing change detection method based on non-backpropagation learning is implemented.

[0089] To achieve the above object, an embodiment of the fourth aspect of the present invention proposes a non-temporary computer-readable storage medium, which can execute the lightweight remote sensing change detection method based on non-backpropagation learning when the instructions in the storage medium are executed by the processor.

[0090] For the lightweight remote sensing change detection method and device based on non-backpropagation learning in the embodiments of the present application, first, the affinity matrix method based on block dimension is used to quickly select changed samples and unchanged samples as neural network training samples. Secondly, the local learning method is used to fine-tune the neural network. The fine-tuning method based on local learning assigns a loss function to each convolutional layer to be trained. After forward propagation through this layer, the output of this layer is directly used to calculate the weight gradient, thus avoiding the large computational amount of the backpropagation algorithm. Finally, the entire image is input into the neural network to perform the inference operation, and the result is binarized to obtain the binary change result.

[0091] The additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. Brief Description of the Drawings

[0092] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the drawings, where:

[0093] Figure 1 It is a schematic flowchart of a lightweight remote sensing change detection method based on non-backpropagation learning provided in Embodiment 1 of the present application;

[0094] Figure 2 It is a schematic diagram of the basic network structure of an embodiment of the present application;

[0095] Figure 3 For the detection effect example diagram of the existing affinity matrix + S 3 N method;

[0096] Figure 4 It is a detection effect example diagram of an embodiment of the present application;

[0097] Figure 5 It is a schematic structural diagram of a lightweight remote sensing change detection device based on non-backpropagation learning provided in an embodiment of the present application. Detailed Description of the Embodiments

[0098] The following details the embodiments of the present application. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0099] The following describes a lightweight remote sensing change detection method and device based on non-backpropagation learning according to an embodiment of the present application with reference to the drawings.

[0100] Figure 1 It is a schematic flowchart of a lightweight remote sensing change detection method based on non-backpropagation learning provided in Embodiment 1 of the present application.

[0101] As Figure 1 shown, the lightweight remote sensing change detection method based on non-backpropagation learning includes the following steps:

[0102] Step 101: Uniformly divide the remote sensing images before and after the change to be inferred to obtain a set of sub-images;

[0103] In this embodiment, the process of image division includes:

[0104] Obtain the remote sensing images before and after the change, with a size of W×H, where W represents the number of rows and H represents the number of columns.

[0105] The size of the small pieces cut is p×p (usually set to 64×64).

[0106] Step 102: Using the affinity matrix method based on the block dimension, select the sub-images before and after the change and the sub-images before and after the change that have not changed from the set of sub-images as the training samples of the neural network;

[0107] In this embodiment, the process of selecting training samples includes:

[0108] Step 1: Uniformly cut two complete heterogeneous remote sensing images into small pieces of size p×p without overlap. Directly solve the arithmetic mean of each small piece, denoted as and where pre and post represent that the small pieces come from the images before and after the change. i represents the i-th small piece.

[0109] Step 2: Solve the distance between each small piece and other small pieces on the two images before and after the change respectively:

[0110]

[0111] m represents the image before or after the event. The greater the distance obtained, the smaller the correlation between the two small pieces. The smaller the correlation represents a greater possibility of change.

[0112] Step 3: Measure the possibility of change according to the exponential function of the distance. Here, the affinity matrix is used for measurement, and the formula is expressed as:

[0113]

[0114] where, is the normalization parameter, represented by the maximum value of all :

[0115]

[0116] Then use the formula to sum and normalize to obtain the change probability of each small piece. In this embodiment, 8 samples with the least likely to change and 2 samples with the most likely to change are selected to fine-tune the neural network.

[0117] Step 103: Using the training samples, adopt the local learning method to fine-tune the neural network. During fine-tuning, assign a loss function to the convolutional layer to be trained for fine-tuning, and keep the parameters of other convolutional layers unchanged, so that other convolutional layers perform forward propagation to obtain the fine-tuned neural network;

[0118] In this embodiment, Mobilenet is a very lightweight neural network that can effectively extract features. The Mobilenet network is used as the base network. As Figure 2 shown, in this embodiment, only the deeper convolutional layers Conv 3-2, Conv4-2, and Conv 5-2 are fine-tuned to achieve the purpose of deep content learning. At the same time, during the fine-tuning process, other convolutional layers only perform forward propagation without updating parameters to achieve the purpose of lightweight.

[0119] In this embodiment, the data used for fine-tuning is the selected training samples and the Mobilenet neural network that has been pre-trained using the natural image dataset Cifar10. The fine-tuning process includes:

[0120] Step 1: First, perform the forward propagation of the shallow convolutional layers Conv1-1 to DwC3-2 using the training samples selected by the previous module.

[0121] Step 2: For each convolutional layer {Convm-2, m = 3, 4, 5} to be fine-tuned, a loss function L m is assigned in this embodiment, as Figure 2 shown. The loss function used for training is

[0122]

[0123] where L m represents the loss function assigned to {Convm-2, m = 3, 4, 5}. Each L m contains two contrastive learning loss functions and to process the output features of the varying samples and the invariant samples respectively. represents the feature output of the {Convm-2, m = 3, 4, 5} layers of the sub-network p when the input is a varying sample (v = +) or an invariant sample (v = -). (x, y, z) represents the output coordinates. Among them, W m , H m , C m represent the width, length, and number of channels of the output features. By using the contrastive learning loss function, the output feature differences of the varying samples will gradually increase during the training process, while the feature differences of the invariant samples will gradually decrease, so as to achieve the ability to highlight the varying features and hide the invariant features.

[0124] Step 3: Directly calculate the gradient using the loss function and the output feature values.

[0125]

[0126] where gO is the gradient of the output feature O. s(·) is the sign function:

[0127]

[0128] After obtaining the gradient of the output feature in the previous solution, directly use the formula:

[0129]

[0130] Calculate the gradient of the weight. Where is the weight gradient of the {Convm-2, m = 3, 4, 5} layer of sub-network p. * represents the convolution operation, is the input of the m-th layer of sub-network t p . Finally, through the formula:

[0131]

[0132] Update the weight. Where k represents different iteration times. In this embodiment, this method is used to fine-tune the neural network 1000 times to achieve the effect of sufficient fine-tuning.

[0133] Step 104, input the remote sensing images before and after the change to be inferred into the neural network for inference, and binarize the inference result to obtain the binary change result.

[0134] In this embodiment, the entire heterogeneous remote sensing image is input into the fine-tuned mobilenet neural network for inference and binarized to obtain the binary change result. Among them, the neural network inference and binarization process includes:

[0135] Step 1: Input the entire heterogeneous remote sensing image into the neural network for inference. Use the formula

[0136]

[0137] I d = D 3 + U(D 4 ) + U(D 5 )

[0138] to perform inference to obtain the change image I d . Where is the feature output by neural network p when the input image is I τ , τ ∈ {pre, post} the pre (pre) or post (post) image before and after the change. U is the bilinear upsampling function that upsamples the smaller outputs D 4 , D 5 to the same size as the previous layer output D 3 . Finally, in this embodiment, the Otsu method is used to obtain the binarization result:

[0139] I b = Otsu(I d )

[0140] In the lightweight remote sensing change detection method based on non-backpropagation learning according to the embodiments of the present application, first, a method of affinity matrix based on block dimension is used to quickly select changed samples and unchanged samples as neural network training samples. Secondly, a local learning method is used to fine-tune the neural network. The fine-tuning method based on local learning assigns a loss function to each convolutional layer to be trained. After forward propagation through this layer, the output of this layer is directly used to calculate the weight gradient, thus avoiding the backpropagation algorithm with a large amount of computation. Finally, the entire image is input into the neural network to perform an inference operation, and the result is binarized to obtain a binary change result.

[0141] Table 1 shows the comparison results between the existing affinity matrix + S 3 N method and the lightweight remote sensing change detection method based on non-backpropagation learning of this embodiment.

[0142] Method Kappa Time <![CDATA[Affinity matrix + S 3 N]]> 0.8028 74s This embodiment 0.8823 21s

[0143] Table 1

[0144] Figure 3 , Figure 4 respectively show the example diagrams of the detection effects of this embodiment, the existing affinity matrix + S 3 N method and the lightweight remote sensing change detection method based on non-backpropagation learning of this embodiment.

[0145] To implement the above embodiments, the present application also proposes a lightweight remote sensing change detection device based on non-backpropagation learning.

[0146] Figure 5 shows the structural schematic diagram of a lightweight remote sensing change detection device provided by the embodiments of the present application.

[0147] As Figure 5 shown, the lightweight remote sensing change detection device based on non-backpropagation learning includes:

[0148] An image segmentation module, configured to uniformly segment the remote sensing images before and after the change to be inferred to obtain a set of sub-images;

[0149] A sample selection module, configured to use a method of affinity matrix based on block dimension to select the changed sub-images before and after the change and the unchanged sub-images before and after the change from the set of sub-images as the training samples of the neural network;

[0150] A neural network fine-tuning module, which is used to fine-tune a neural network using training samples and a local learning method. During fine-tuning, a loss function is assigned to the convolutional layer to be trained for fine-tuning, and the parameters of other convolutional layers are kept unchanged, enabling other convolutional layers to perform forward propagation to obtain the fine-tuned neural network;

[0151] A change detection module, which is used to input remotely sensed images before and after change to be inferred into the neural network for inference, and binarize the inference result to obtain a binary change result.

[0152] Optionally, in an embodiment of the present application, a method of affinity matrix based on block dimension is used to select pre-change and post-change sub-images that have changed and pre-change and post-change sub-images that have not changed from a set of sub-images as training samples for the neural network, including:

[0153] Solve the distance between each sub-image and other sub-images on remotely sensed images before and after change, expressed as:

[0154]

[0155] where the superscript m is pre or post, pre and post respectively represent before change and after change, and M i is the i-th sub-image;

[0156] Measure the possibility of sub-image change according to the exponential function of the distance. When using the affinity matrix for measurement, the expression is:

[0157]

[0158] where is the normalization parameter, represented by the maximum value of all :

[0159] ;

[0160] Sum the possibility of change of each sub-image and normalize it to obtain the change probability of each sub-image. Among them, the sum of the possibility of change of the i-th sub-image is expressed as:

[0161]

[0162] where N is the number of segments of the remotely sensed image;

[0163] Based on the change probability of each sub-image, select several pre-change and post-change sub-images that are most likely to change and several pre-change and post-change sub-images that are least likely to change from the set of sub-images as training samples for the neural network.

[0164] Optionally, in an embodiment of the present application, assigning a loss function to the convolutional layer to be trained includes:

[0165] Take the deep convolutional layer as the convolutional layer to be trained. For the deep convolutional layer, construct a contrastive learning loss function to increase the output feature differences of the varying samples and reduce the output feature differences of the invariant samples. Among them, the contrastive learning loss function is:

[0166]

[0167] Among them, L m represents the loss function assigned to the m-th convolutional layer to be trained, processes the output features of the varying samples and the invariant samples respectively. t 1 , t 2 are the sub-networks that process the image before change and the image after change respectively, represents the output feature of the image before change, represents the output feature of the image after change,

[0168] (x, y, z) represents the output coordinates, and W m , H m , C m represent the width, length, and number of channels of the output feature;

[0169] Fine-tune the deep convolutional layer, including:

[0170] Use the contrastive learning loss function and the output feature values to calculate the gradient of the output feature, expressed as:

[0171]

[0172] Among them, gO is the gradient of the output feature O, and s(·) is the sign function, expressed as:

[0173]

[0174] Calculate the weight gradient based on the gradient of the output feature, expressed as:

[0175]

[0176] Among them, is the weight gradient of the m-th layer of the sub-network t p , and * represents the convolution operation, is the input of the m-th layer of the sub-network t p ;

[0177] Update the weights based on the weight gradient, expressed as:

[0178]

[0179] Among them, k represents the number of iterations.

[0180] Optionally, in one embodiment of the present application, the remote sensing images before and after the change to be inferred are input into a neural network for inference, and the inference result is obtained, which is expressed as:

[0181]

[0182] I d = D x + U(D x+1 ) +... + U(D x+n )

[0183] where C m is the number of channels of the m-th convolutional layer, x, y, z are the coordinates of the output, t 1 , t 2 are sub-networks for processing the image before the change and the image after the change respectively, pre and post represent before the change and after the change respectively, represents the output feature of the image before the change, represents the output feature of the image after the change, D x+n , D x+1 ,..., D x+n are convolutional layers to be trained, x + n is the number of convolutional layers of the neural network, I d is the changed image;

[0184] The binarization result is obtained using the Otsu method of maximum inter-class variance, which is expressed as:

[0185] I b = Otsu(I d ).

[0186] It should be noted that the foregoing explanation of the embodiment of the lightweight remote sensing change detection method based on non-backpropagation learning also applies to the lightweight remote sensing change detection device based on non-backpropagation learning in this embodiment, and will not be repeated here.

[0187] To implement the above embodiment, the present invention also proposes a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method described in the above embodiment is implemented.

[0188] To implement the above embodiment, the present invention also proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the above embodiment is implemented.

[0189] In the description of this specification, the descriptions with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0190] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0191] Any process or method description shown in a flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions may be executed in a manner that is not in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0192] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable sequence list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then storing it in a computer memory.

[0193] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0194] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0195] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0196] The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A lightweight remote sensing change detection method based on non-backpropagation learning, characterized in that: include: The remote sensing images before and after the reasoning change are evenly divided to obtain a set of sub-images; Using a block dimension-based affinity matrix method, selecting the changed sub-images before and after the change and the unchanged sub-images before and after the change from the sub-image set as training samples for the neural network; Using the training samples, a local learning method is used to fine-tune the neural network. During fine-tuning, a loss function is assigned to the convolution layer to be trained for fine-tuning, and the parameters of other convolution layers are kept unchanged, so that the other convolution layers perform forward propagation to obtain a fine-tuned neural network; The remote sensing images before and after the change to be inferred are input into a neural network for inference, and the inference result is binarized to obtain a binary change result.

2. The method according to claim 1, characterized in that The method of using the block dimension-based affinity matrix to select the changed sub-images before and after the change and the unchanged sub-images before and after the change from the sub-image set as training samples for the neural network includes: The distance between each sub-image and other sub-images is solved on the remote sensing images before and after the change, expressed as: The superscript m stands for pre or post. Pre and post respectively represent before and after the change. i is the i-th sub-image; The possibility of sub-image change is measured according to the exponential function of distance, where the expression when using affinity matrix measurement is: in, As the normalization parameter, all The maximum value of is: The probability of change of each sub-image is summed and normalized to obtain the probability of change of each sub-image, where the probability of change of the i-th sub-image is summed, expressed as: Where N is the number of remote sensing image segments; Based on the change probability of each sub-image, several sub-images before and after the change that are most likely to change and several sub-images before and after the change that are least likely to change are selected from the sub-image set as training samples for the neural network.

3. The method according to claim 1, characterized in that Assign loss functions to the convolutional layers that need to be trained, including: The deep convolution layer is used as the convolution layer to be trained. For the deep convolution layer, a contrastive learning loss function is constructed to increase the output feature difference of the changed samples and reduce the output feature difference of the unchanged samples, wherein the contrastive learning loss function is: L=ΣL m Among them, L m represents the loss function assigned to the mth convolutional layer that needs to be trained, The output features of the samples that have changed and the samples that have not changed are processed respectively. t1 and t2 are sub-networks that process the images before and after the changes respectively. represents the output features of the image before the change, represents the output features of the changed image, (x, y, z) represents the output coordinates, W m , H m , C m Represents the width, length and number of channels of the output features; Fine-tune the deep convolutional layers, including: The gradient of the output feature is calculated using the contrastive learning loss function and the output feature value, expressed as: Among them, gO is the gradient of the output feature O, s(·) is the sign function, expressed as: The weight gradient is calculated based on the gradient of the output feature, expressed as: in, is the subnetwork t p The weight gradient of the mth layer, * represents the convolution operation, For subnetwork t p The input of the mth layer; Update the weights based on the weight gradient, expressed as: Here, k represents the number of iterations.

4. The method according to claim 1, characterized in that The remote sensing images before and after the change to be inferred are input into the neural network for inference, and the inference result is obtained, which is expressed as: I d =D x +U(D x+1 )+...+U(D x+n ) Among them, C m is the number of channels of the mth convolutional layer, x, y, z are the output coordinates, t1 and t2 are subnetworks that process the image before and after the change, respectively, pre and post represent before and after the change, respectively. represents the output features of the image before the change, Denotes the output features of the changed image, D x+n , D x+1 ,...,D x+n is the convolutional layer that needs to be trained, x+n is the number of convolutional layers of the neural network, I d For changing images; The maximum inter-class variance method Otsu is used to obtain the binarization result, which is expressed as: I b =Otsu(I d ).

5. A lightweight remote sensing change detection device based on non-backpropagation learning, characterized in that: include: An image segmentation module is used to evenly segment the remote sensing image before and after the change to be inferred to obtain a set of sub-images; A sample selection module, used for selecting the changed sub-images before and after the change and the unchanged sub-images before and after the change from the sub-image set as training samples for the neural network by using a block dimension-based affinity matrix method; A neural network fine-tuning module is used to use the training samples to fine-tune the neural network using a local learning method. During fine-tuning, a loss function is assigned to the convolution layer to be trained for fine-tuning, and the parameters of other convolution layers are kept unchanged, so that the other convolution layers perform forward propagation to obtain a fine-tuned neural network; The change detection module is used to input the remote sensing images before and after the change to be inferred into the neural network for inference, and binarize the inference result to obtain a binary change result.

6. The device according to claim 5, characterized in that The method of using the block dimension-based affinity matrix to select the changed sub-images before and after the change and the unchanged sub-images before and after the change from the sub-image set as training samples for the neural network includes: The distance between each sub-image and other sub-images is solved on the remote sensing images before and after the change, expressed as: The superscript m stands for pre or post. Pre and post respectively represent before and after the change. i is the i-th sub-image; The possibility of sub-image change is measured according to the exponential function of distance, where the expression when using affinity matrix measurement is: in, As the normalization parameter, all The maximum value of is: The probability of change of each sub-image is summed and normalized to obtain the probability of change of each sub-image, where the probability of change of the i-th sub-image is summed, expressed as: Where N is the number of remote sensing image segments; Based on the change probability of each sub-image, several sub-images before and after the change that are most likely to change and several sub-images before and after the change that are least likely to change are selected from the sub-image set as training samples for the neural network.

7. The device according to claim 5, characterized in that Assign loss functions to the convolutional layers that need to be trained, including: The deep convolution layer is used as the convolution layer to be trained. For the deep convolution layer, a contrastive learning loss function is constructed to increase the output feature difference of the changed samples and reduce the output feature difference of the unchanged samples, wherein the contrastive learning loss function is: L=ΣL m Among them, L m represents the loss function assigned to the mth convolutional layer that needs to be trained, The output features of the samples that have changed and the samples that have not changed are processed respectively. t1 and t2 are sub-networks that process the images before and after the changes respectively. represents the output features of the image before the change, represents the output features of the changed image, (x, y, z) represents the output coordinates, W m , H m , C m Represents the width, length and number of channels of the output features; Fine-tune the deep convolutional layers, including: The gradient of the output feature is calculated using the contrastive learning loss function and the output feature value, expressed as: Among them, gO is the gradient of the output feature O, s(·) is the sign function, expressed as: The weight gradient is calculated based on the gradient of the output feature, expressed as: in, is the subnetwork t p The weight gradient of the mth layer, * represents the convolution operation, For subnetwork t p The input of the mth layer; Update the weights based on the weight gradient, expressed as: Here, k represents the number of iterations.

8. The device according to claim 5, characterized in that The remote sensing images before and after the change to be inferred are input into the neural network for inference, and the inference result is obtained, which is expressed as: I d =D x +U(D x+1 )+...+U(D x+n ) Among them, C m is the number of channels of the mth convolutional layer, x, y, z are the output coordinates, t1 and t2 are subnetworks that process the image before and after the change, respectively, pre and post represent before and after the change, respectively. represents the output features of the image before the change, Denotes the output features of the changed image, D x+n , D x+1 ,...,D x+n is the convolutional layer that needs to be trained, x+n is the number of convolutional layers of the neural network, I d For changing images; The maximum inter-class variance method Otsu is used to obtain the binarization result, which is expressed as: I b =Otsu(I d ).

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images