A Multi-Scale Remote Sensing Change Detection Method and Device Based on Attention Mechanism

Abstract

The present application provides a multi-scale remote sensing change detection method and device based on an attention mechanism, which relates to the field of remote sensing image detection and includes: obtaining images of different scales of a to-be-detected plot, where the images include a first-scale image and a second-scale image; respectively preprocessing the first-scale image and the second-scale image to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map; calculating the change result of the to-be-detected plot according to the formula S = A1 × Sn1 + (1 - A1)Sn2 to obtain a change detection result. The technical solution of the present application is based on an attention mechanism, that is, the allocation of input weights is added during the weighted fusion at different scales. A multi-scale resolution remote sensing change detection network is used to replace the traditional multi-scale prediction. When the speed is consistent with that of the traditional multi-scale prediction, the accuracy of the model is effectively improved, and the accuracy of the plot change detection result is also greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image detection, and particularly relates to a multi-scale remote sensing change detection method and device based on an attention mechanism. Background Technique

[0002] Change detection is of great significance for applications such as the "red line of cultivated land" and land use supervision. Using multi-temporal remote sensing data and adopting various image processing and pattern recognition methods to extract change information, and quantitatively analyzing and determining the characteristics and processes of surface changes is the essence of remote sensing change detection. The traditional method in the remote sensing industry of judging the temporal changes of ground objects based on manual annotation of two-phase images is limited by problems such as low efficiency and high cost, and it is difficult to meet the actual application requirements. This paper hopes to select an efficient remote sensing image change detection algorithm model to efficiently identify the change patch information in the image and improve the ability to quickly identify changes in remote sensing images in the construction of the spatial information network. Multi-scale prediction is a method commonly used to improve performance in semantic segmentation. However, traditional multi-scale prediction considers that the predictions at all scales are equally important. But in fact, fine details are usually best predicted at higher scales, large objects are better predicted at lower scales, and at lower scales, the receptive field of the network can better understand the scene, and the results at different scales should not be treated equally. Therefore, in the prior art, treating the results at different scales equally and processing them in the same way will lead to the problem of low accuracy of the detection results. Summary of the Invention

[0003] The present application provides a multi-scale remote sensing change detection method and device based on an attention mechanism, aiming to solve the problem of low accuracy of the detection results caused by treating different scales in the same way as described above.

[0004] To achieve the above object, the present application adopts the following technical solutions, including:

[0005] A multi-scale remote sensing change detection method based on an attention mechanism, including:

[0006] Obtain images of different scales of the plot to be measured, where the images include a first-scale image and a second-scale image;

[0007] Preprocess the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map;

[0008] Calculate the change result of the plot to be measured according to the formula S = A1×Sn1+(1 - A1)Sn2 to obtain a change detection result, where S is the change detection result, A1 is the first attention map, Sn1 is the first prediction image, and Sn2 is the second prediction image.

[0009] Preferably, the preprocessing of the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map includes:

[0010] The first-scale image includes a first-phase image and a second-phase image. The first-phase image and the second-phase image are respectively input into a high-resolution network for feature extraction to obtain a first feature and a second feature;

[0011] According to the formula feature = |feat1 - feat2|, the result feature of the first-scale image is calculated to obtain a third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature.

[0012] Preferably, the preprocessing of the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map further includes:

[0013] The third feature is input into an OCR context model for segmentation prediction of the first-scale image to obtain the first prediction image;

[0014] After the third feature is input into the OCR context model for processing, it is then input into an attention model for calculating the pixel weights of the first-scale image to obtain the first attention map.

[0015] Preferably, the second-scale image is used to replace the first-scale image, and the operations in weights 2 and 3 are repeated to obtain the second prediction image and the second attention map.

[0016] Preferably, the technologies used in the process of preprocessing the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map include a siamese neural network and a periodic learning rate technique.

[0017] A multi-scale remote sensing change detection device based on an attention mechanism includes:

[0018] An acquisition module: used to acquire images of different scales of a to-be-detected plot, where the images include a first-scale image and a second-scale image;

[0019] An image preprocessing module: used to preprocess the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map;

[0020] The plot change detection module: It is used to calculate the change result of the plot to be measured according to the formula S = A1×Sn1+(1 - A1)Sn2 to obtain the change detection result, where S is the change detection result, A1 is the first attention map, Sn1 is the first prediction image, and Sn2 is the second prediction image.

[0021] Preferably, the image preprocessing module includes:

[0022] The feature extraction module: For the first-scale image including the first-phase image and the second-phase image, the first-phase image and the second-phase image are respectively input into a high-resolution network for feature extraction to obtain the first feature and the second feature;

[0023] The result feature calculation module: It is used to calculate the result feature of the first-scale image according to the formula feature = |feat1 - feat2| to obtain the third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature.

[0024] Preferably, the image preprocessing module further includes:

[0025] The prediction image acquisition module: It is used to input the third feature into the OCR context model for segmentation prediction of the first-scale image to obtain the first prediction image;

[0026] The attention map acquisition module: It is used to input the third feature into the OCR context model for processing and then input it into the attention model for pixel weight calculation of the first-scale image to obtain the first attention map.

[0027] A multi-scale remote sensing change detection device based on the attention mechanism includes a memory and a processor. The memory is used to store one or more computer instructions. Among them, the one or more computer instructions are executed by the processor to implement a multi-scale remote sensing change detection method according to any one of the above.

[0028] A computer-readable storage medium storing a computer program, where the computer program, when executed by a computer, implements a multi-scale remote sensing change detection method according to any one of the above.

[0029] The technical solution has the following beneficial effects:

[0030] (1) This technical solution is based on the attention mechanism, that is, the distribution of input weights is added during the weighted fusion at different scales. A multi-scale resolution remote sensing change detection network is used to replace the traditional multi-scale prediction. When the speed is the same as that of the traditional multi-scale prediction, the accuracy of the model is effectively improved, and the accuracy of the plot change detection result is also greatly improved;

[0031] (2) Regarding the accuracy of the plot change detection result, this technical solution divides the same plot into different scales for separate calculations and introduces the attention mechanism to assign weights to the pixels in each image, making the detection result image after weighted fusion more accurate. After feature extraction of the two temporal images at the same scale, the absolute value of the difference is taken and then subsequent operations are performed. The purpose of this solution is to detect image changes. Subtracting the two features is to find the differences, and taking the absolute value is to prevent the gradient from falling into the negative interval, both of which can improve the accuracy and precision of the detection result;

[0032] (3) In the process of processing the image to generate the final change detection result, this technical solution uses the Siamese neural network and the periodic learning rate technique, effectively improving the detection ability of the model, thereby improving the accuracy of the change detection result. Description of the Drawings

[0033] Figure 1 It is a flowchart for implementing a multi-scale remote sensing change detection method based on the attention mechanism in an embodiment of the present invention

[0034] Figure 2 It is a flowchart for subsequent processing of a third feature in an embodiment of the present invention

[0035] Figure 3 It is a display diagram of the code design of an attention model in an embodiment of the present invention

[0036] Figure 4 It is a display diagram of the effect of the periodic learning rate technique in an embodiment of the present invention

[0037] Figure 5 It is a display diagram of the plot change detection result in an embodiment of the present invention

[0038] Figure 6 It is a schematic structural diagram of an apparatus for implementing a multi-scale remote sensing change detection based on the attention mechanism in an embodiment of the present invention

[0039] Figure 7 It is a schematic structural diagram of the image preprocessing module 20 in an apparatus for implementing a multi-scale remote sensing change detection based on the attention mechanism in an embodiment of the present invention

[0040] Figure 8Schematic diagram of an electronic device for implementing a multi-scale remote sensing change detection device based on an attention mechanism in an embodiment of the present invention Detailed implementation manners

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0042] The terms "first", "second", etc. in the claims and the description of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances. This is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the description of the present application herein are only for the purpose of describing specific embodiments, and are not intended to limit this application.

[0044] Embodiment 1

[0045] As Figure 1 shown, a multi-scale remote sensing change detection method based on an attention mechanism includes the following steps:

[0046] S11. Obtain images of different scales of the plot to be measured, where the images include a first-scale image and a second-scale image;

[0047] S12. Preprocess the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map;

[0048] S13. Calculate the change result of the plot to be measured according to the formula S = A1 × Sn1 + (1 - A1)Sn2 to obtain a change detection result, where S is the change detection result, A1 is the first attention map, Sn1 is the first prediction image, and Sn2 is the second prediction image.

[0049] In this embodiment, the "first-scale image" and the "second-scale image" are images of the plot to be measured at different scales. The "first prediction image" and the "second prediction image" are two result prediction images generated by subsequent processing of the first-scale image and the second-scale image respectively. The "first attention map" and the "second attention map" are maps of the weight values of each pixel generated by processing the first-scale image and the second-scale image through an attention model;

[0050] First, select a fixed plot, and then obtain images of the plot at different scales. In this embodiment, images at two scales are obtained, namely the first-scale image and the second-scale image. And at each scale, two images of different time phases of the plot are obtained. In the first scale, these two time-phase images are the "first time-phase image" and the "second time-phase image". Then, input the first time-phase image and the second time-phase image into HRNet18 respectively to obtain feature 1 and feature 2, namely the "first feature" and the "second feature". After subtracting the features of the two time phases and taking the absolute value, the processed feature is obtained, and this feature is the "third feature". This process is "calculating the result feature of the first-scale image according to the formula feature = |feat1 - feat2| to obtain the third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature". Then, send the third feature into the OCR context model for detection, (this process is as Figure 2 shown), and the segmentation map prediction result is obtained, and this result is the first prediction image. After obtaining the image, continue with the detection and processing of the attention model, (the code design of this attention model is as Figure 3 shown), and the first attention map at the first scale is obtained. To sum up, the processing process of the second-scale image is the same as that of the first-scale image. After performing the above operations on the second-scale image, the second prediction image and the second attention map are obtained. Finally, according to the formula S = A1×Sn1+(1 - A1)Sn2, substitute the parameters of the first prediction image (Sn1), the second prediction image (Sn2), and the first attention map (A1) into this formula for calculation to obtain the change detection result (S).

[0051] Among them, extracting features from the first time-phase image and the second time-phase image respectively and performing subsequent operations adopt the form of a siamese neural network, and in the specific process, a periodic learning rate technique is adopted. This technique makes the learning rate change periodically within the range of the upper limit and the lower limit. With this learning rate method, the model is not easily overfitted and does not require much parameter tuning. The effect of this technique is as Figure 4 shown.

[0052] The following gives an example for detailed illustration to facilitate understanding:

[0053] 1. We used the dataset of the Remote Sensing Change Detection Competition of the 4th China Conference on Pattern Recognition and Computer Vision. There are a total of 10,000 pairs of all image data, with a size of 512x512, mainly distributed in cities such as Beijing, Shanghai, Guangzhou, and Hangzhou;

[0054] 2. Input the real image 1 and real image 2 into HRNet18 respectively to obtain feature 1 and feature 2. After subtracting the features of the two real images and taking the absolute value, send them into the OCR model to obtain the segmentation map prediction result and the attention map at the current scale. This attention map is used for weighted fusion during the prediction of the next scale, and the weight of each pixel is the corresponding value in the attention map;

[0055] 3. During the training process, we set scale = [0.5, 1.0], and in the test stage, scale = [0.25, 0.5, 1.0, 1.5, 2.0]. Therefore, during the training process, only 2 scales are used, so the training process does not significantly increase the training time. In the test stage, it is equivalent to using multi-scale inference. Because the number of scales used in the test stage is the same as that in the multi-scale network, the speed of our network is consistent with that of multi-scale inference;

[0056] Next, list several groups of experimental data compared with the prior art to demonstrate the beneficial effect of effectively improving the model detection accuracy achieved by the technical solution of this application:

[0057] Specifically, the accuracy results of 4 groups of experiments are shown in Table 1, which includes the improvement of the model accuracy by each module during the entire training process.

[0058] Table 1

[0059]

[0060] Experiment 1 is the baseline (reference, starting point). The native HRNet18_OCR is used. After concatenating (function, merging arrays) the real image 1 and real image 2, they are sent into the model, and the model is obtained using the SGD (Stochastic Gradient Descent) and cosine annealing learning rate strategies.

[0061] In Experiment 2, the model form is changed to the form of a siamese network, and the Cycling Learning Rate (periodic learning rate technique) is used, with the accuracy improved by 1.16.

[0062] Experiment 3 is based on Experiment 2 with multi-scale prediction added, and the accuracy is improved by 0.9. The scheme adopted in this experiment is the scheme commonly used in the prior art.

[0063] Experiment 4 is to add a multi-scale attention network, that is, the network proposed in the technical solution of this application, and the accuracy is improved by 2.78.

[0064] The change detection results are as Figure 5 shown below.

[0065] The scales of both experiments are [0.25, 0.5, 1.0, 1.5, 2.0]. Since the prediction speeds of Experiment 3 and Experiment 4 are basically the same, and the accuracy of the multi-scale attention network is 1.88 higher than that of multi-scale prediction, with a significant improvement, the technical solution of this application has greatly improved the accuracy of model detection while maintaining basically the same speed.

[0066] The beneficial effects of this embodiment are as follows:

[0067] (1) This technical solution is based on the attention mechanism, that is, the distribution of input weights is added during the weighted fusion at different scales. The multi-scale resolution remote sensing change detection network is used to replace the traditional multi-scale prediction. When the speed is the same as that of the traditional multi-scale prediction, the accuracy of the model is effectively improved, and the accuracy of the plot change detection results is also greatly improved;

[0068] (2) Regarding the accuracy of the plot change detection results, this technical solution divides the same plot into different scales for separate calculations and introduces the attention mechanism to assign weights to the pixels in each image, making the detection result image after weighted fusion more accurate. After feature extraction of the two temporal images at the same scale, the absolute value of the difference is taken and then subsequent operations are performed. The purpose of this solution is to detect image changes. Subtracting the two features is to find differences, and taking the absolute value is to prevent the gradient from falling into the negative interval, both of which can improve the accuracy and precision of the detection results;

[0069] (3) In the process of processing the image to generate the final change detection results, this technical solution uses the siamese neural network and the periodic learning rate technique, effectively improving the detection ability of the model, thereby improving the accuracy of the change detection results.

[0070] Embodiment 2

[0071] As Figure 6 shown below, a multi-scale remote sensing change detection device based on the attention mechanism includes:

[0072] An acquisition module 10: used to acquire images of different scales of the plot to be measured, and the images include a first-scale image and a second-scale image;

[0073] An image preprocessing module 20: used to preprocess the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map;

[0074] The plot change detection module 30: is used to calculate the change result of the plot to be measured according to the formula S = A1×Sn1+(1 - A1)Sn2, and obtain the change detection result, where S is the change detection result, A1 is the first attention map, Sn1 is the first prediction image, and Sn2 is the second prediction image.

[0075] One implementation of the above device is that in the acquisition module 10, images of different scales of the plot to be measured are acquired. The images include the first-scale image and the second-scale image. In the image preprocessing module 20, the first-scale image and the second-scale image are respectively preprocessed to obtain the first prediction image, the first attention map, the second prediction image, and the second attention map. In the plot change detection module 30, the change result of the plot to be measured is calculated according to the formula S = A1×Sn1+(1 - A1)Sn2, and the change detection result is obtained, where S is the change detection result, A1 is the first attention map, Sn1 is the first prediction image, and Sn2 is the second prediction image.

[0076] Embodiment 3

[0077] As Figure 7 shown, the image preprocessing module 20 in a multi-scale remote sensing change detection device based on the attention mechanism includes:

[0078] The feature extraction module 21: is used for the first-scale image including the first-phase image and the second-phase image. The first-phase image and the second-phase image are respectively input into the high-resolution network for feature extraction to obtain the first feature and the second feature;

[0079] The result feature calculation module 22: is used to calculate the result feature of the first-scale image according to the formula feature = |feat1 - feat2| to obtain the third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature;

[0080] The prediction image acquisition module 23: is used to input the third feature into the OCR context model for segmentation prediction of the first-scale image to obtain the first prediction image;

[0081] The attention map acquisition module 24: is used to input the third feature into the OCR context model for processing, and then input it into the attention model for pixel weight calculation of the first-scale image to obtain the first attention map.

[0082] One embodiment of the above device is that in the feature extraction module 21, the first-scale image includes a first-phase image and a second-phase image. The first-phase image and the second-phase image are respectively input into a high-resolution network for feature extraction to obtain a first feature and a second feature. In the result feature calculation module 22, the result feature of the first-scale image is calculated according to the formula feature = |feat1 - feat2| to obtain a third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature. In the predicted image acquisition module 23, the third feature is input into an OCR context model for segmentation prediction of the first-scale image to obtain the first predicted image. In the attention map acquisition module 24, after the third feature is input into the OCR context model for processing, it is then input into an attention model for pixel weight calculation of the first-scale image to obtain the first attention map.

[0083] Embodiment 4

[0084] As Figure 8 shown, an electronic device includes a memory 401 and a processor 402. The memory 401 is used to store one or more computer instructions. Among them, the one or more computer instructions are executed by the processor 402 to implement any one of the above methods.

[0085] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described electronic device can refer to the corresponding process in the foregoing method embodiments and will not be elaborated herein.

[0086] A computer-readable storage medium storing a computer program, where the computer program causes a computer to execute to implement any one of the above methods.

[0087] Exemplarily, the computer program can be divided into one or more modules / units. One or more modules / units are stored in the memory 401 and executed by the processor 402, and the data I / O interface transmission is completed by the input interface 405 and the output interface 406 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0088] The computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device may include, but is not limited to, a memory 401 and a processor 402. Those skilled in the art can understand that this embodiment is only an example of the computer device, and does not constitute a limitation on the computer device. It may include more or fewer components, or combine certain components, or different components. For example, the computer device may also include an input device 407, a network access device, a bus, etc.

[0089] The processor 402 can be a central processing unit (CPU), or can also be other general-purpose processors 402, digital signal processors 402 (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 402 can be a microprocessor 402 or the processor 402 can also be any conventional processor 402, etc.

[0090] The memory 401 can be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. The memory 401 can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 401 can also include both the internal storage unit and the external storage device of the computer device. The memory 401 is used to store computer programs and other programs and data required by the computer device. The memory 401 can also be used to temporarily store data in the output device 408, and the aforementioned storage media include various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory ROM403, random access memory RAM404, diskettes, or optical discs.

[0091] The above are only specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any changes or modifications made by those skilled in the art within the scope of the present invention are covered by the patent scope of the present invention.

Claims

1. A multi-scale remote sensing change detection method based on the attention mechanism, characterized in that, Including: Obtain images of the plot to be measured at different scales, where the images include a first-scale image and a second-scale image; Preprocess the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map; According to the formula calculate the change result of the to-be-measured plot to obtain a change detection result, where S is the change detection result, is the first attention map, is the first predicted image, is the second predicted image; The preprocessing the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map includes: The first-scale image includes a first-phase image and a second-phase image. Input the first-phase image and the second-phase image into a high-resolution network for feature extraction respectively to obtain a first feature and a second feature; According to the formula calculate the result features of the first-scale image to obtain a third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature; Input the third feature into an OCR context model for segmentation prediction of the first-scale image to obtain the first prediction image; After inputting the third feature into the OCR context model for processing, input it into an attention model for pixel weight calculation of the first-scale image to obtain the first attention map; The preprocessing process of the second-scale image is the same as that of the first-scale image.

2. The multi-scale remote sensing change detection method based on the attention mechanism according to claim 1, wherein, The technologies used in the process of preprocessing the first-scale image and the second-scale image respectively to obtain a first prediction image, a first attention map, a second prediction image, and a second attention map include a siamese neural network and a periodic learning rate technique.

3. A multi-scale remote sensing change detection device based on an attention mechanism, which is used to implement a multi-scale remote sensing change detection method based on an attention mechanism as described in claim 1, and is characterized in that, Including: An acquisition module: used to acquire images of the plot to be measured at different scales, where the images include a first-scale image and a second-scale image; A feature extraction module: for the first-scale image including a first-phase image and a second-phase image, input the first-phase image and the second-phase image into a high-resolution network for feature extraction respectively to obtain a first feature and a second feature; Result feature calculation module: used to calculate the result feature of the first-scale image according to the formula to obtain a third feature, where feature is the third feature, feat1 is the first feature, and feat2 is the second feature; A prediction image acquisition module: used to input the third feature into an OCR context model for segmentation prediction of the first-scale image to obtain the first prediction image; An attention map acquisition module: used to input the third feature into the OCR context model for processing, and then input it into an attention model for pixel weight calculation of the first-scale image to obtain the first attention map; A preprocessing module for the second-scale image: used for preprocessing the second-scale image, and the preprocessing process of the second-scale image is the same as that of the first-scale image; Parcel change detection module: used to calculate the change result of the parcel to be measured according to the formula to obtain the change detection result, where S is the change detection result, is the first attention map, is the first predicted image, is the second predicted image.

4. A multi-scale remote sensing change detection device based on an attention mechanism, characterized in that, Including a memory and a processor, where the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the processor to implement a multi-scale remote sensing change detection method based on an attention mechanism as claimed in claim 1 or 2.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it implements a multi-scale remote sensing change detection method based on an attention mechanism as claimed in claim 1 or 2.

Images