A high-resolution remote sensing image change detection method based on self-supervised learning
Through self-supervised learning and twin feature extraction network, the problem of labeling sample dependence and subjective uncertainty in high-resolution remote sensing image change detection is solved, and more accurate and efficient change detection is achieved, expanding the scope of application.
Patent Information
- Application Number
- CN202210509781.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-11
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-05-11
AI Technical Summary
In the detection of changes in high-resolution remote sensing images, the performance of the deep learning method depends on a large number of labeling samples, and the labeling process relies on artificial subjective experience, with subjective uncertainty and large human and material consumption, which limits the application scope of the method. .
Using a self-supervised learning method, multiple feature maps of high-resolution remote sensing images are obtained through a preset twin feature extraction network, the differences between feature maps are calculated, and the changes are fusion is carried out to realize change detection. The training samples are generated by automatic calculations. Label information construction to reduce manual intervention.
Improve the accuracy of change detection, reduce the missed detection rate and false detection rate, avoid the "salt and pepper" phenomenon, improve the labeling efficiency, expand the application scope of the method, and realize the automation and efficiency of high-resolution remote sensing image change detection change.
Smart Images

Figure CN114926511B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of remote sensing image processing, and in particular relates to a high-resolution remote sensing image change detection method based on self-supervised learning. Background Art
[0002] With the rapid development of remote sensing technology, new satellites equipped with various advanced sensors have been launched and put into orbit, and the temporal, spatial and spectral resolution of satellite imaging of the earth has been significantly improved. As an important source of remote sensing data, high-resolution remote sensing images intuitively reflect the distribution and information representation of ground objects in a specific geographic space, and contain rich spectral, texture and spatial structure characteristics. Therefore, using high-resolution remote sensing images for change detection is of great significance for a deep understanding of the relationship between human activities and the natural environment and their interaction mechanism.
[0003] Remote sensing image change detection aims to use remote sensing image data obtained from different historical periods in the same geographical area, using image processing theory and statistical learning modeling methods to determine and analyze the changes in the state of objects in the area. With the open sharing of remote sensing platform data and the introduction of new change detection algorithms, remote sensing image change detection technology has been widely used in land and resources surveys, agricultural and forestry monitoring, urban expansion, natural disaster assessment and other fields.
[0004] For medium / low resolution remote sensing images, the spectral information of a single pixel is sufficient to characterize the types of different ground objects. Therefore, directly comparing two remote sensing images pixel by pixel can effectively highlight the actual changes in the observation area. Related detection methods include image difference / ratio method, image transformation method or change vector analysis method. For high-resolution remote sensing images, due to factors such as imaging under different spectra, a single target on the ground is usually composed of a group of pixels, and the targets are spatially related to each other. Therefore, compared with medium / low resolution remote sensing images, high-resolution remote sensing images usually contain richer ground object information. At the same time, due to the existence of the characteristics of "same object, different spectrum, same spectrum, different objects", the method of comparing the differences between two remote sensing images pixel by pixel is difficult to fully utilize the ground object information in the image, which makes it very easy for the "salt and pepper" phenomenon to appear in the change detection results (such as Figure 1 shown).
[0005] In order to achieve high-resolution remote sensing image change detection, deep learning technologies represented by convolutional neural networks have been gradually introduced into the field of remote sensing image processing in recent years. However, the performance of the methods based on deep learning methods to achieve remote sensing image processing in the prior art is often based on sufficient labeled samples. Among them, some methods require the use of open sample libraries, and some methods require manual labeling of a large number of samples, that is, the labeling information of the samples must be predicted based on human subjective experience. However, the samples in the open sample library sometimes do not match the type or size of the images to be processed in the detection task, resulting in poor actual use of the trained model. In the method of manually labeling a large number of samples, the detection results rely on manual labeling, there is a certain subjective uncertainty, and timely and accurate acquisition of a large amount of labeled data usually requires a lot of manpower, material resources and time, which greatly limits the scope of application of such methods in real scenes. Summary of the invention
[0006] In order to solve the above problems existing in the prior art, the present invention provides a high-resolution remote sensing image change detection method based on self-supervised learning.
[0007] The technical problem to be solved by the present invention is achieved through the following technical solutions:
[0008] A high-resolution remote sensing image change detection method based on self-supervised learning, comprising:
[0009] Acquire two high-resolution remote sensing images taken at different times for the same area, and perform radiation correction and registration on the two high-resolution remote sensing images to obtain two registered target images;
[0010] Comparing the spectral characteristics of the two target images to obtain a pixel-level difference map;
[0011] Obtaining multiple feature maps corresponding to each target image through a preset twin feature extraction network; wherein the multiple feature maps respectively correspond to multiple different operation layers of a single branch of the twin feature extraction network;
[0012] Calculating the difference between the feature maps of the two target images from the same-level operation layer to obtain a plurality of target-level difference maps;
[0013] fusing the pixel-level difference map and the multiple object-level difference maps, and obtaining a change detection result according to the fused image;
[0014] Among them, the twin feature extraction network is a feature extraction subnetwork in a pre-trained deep twin convolutional network model; the training samples used to train the deep twin convolutional network model are all from the aligned high-resolution sample remote sensing image pairs, and the annotation information of the training samples is automatically calculated and generated by inputting the sample remote sensing image pairs into a preset annotation algorithm.
[0015] Optionally, the deep twin convolutional network model is used to predict overlap of paired image blocks;
[0016] The labeling algorithm includes:
[0017] Comparing the spectral characteristics of the sample remote sensing image pairs to obtain a sample pixel-level difference map;
[0018] Performing superpixel segmentation on the sample pixel-level difference map to obtain a first segmentation result;
[0019] Taking the centroid position coordinates of each superpixel in the first segmentation result as the center of the image block and taking a preset size as the size of the image block, calculating the overlap of each pair of image blocks between the sample remote sensing image pairs;
[0020] The training samples and the annotation information are constructed according to each pair of image blocks between the sample remote sensing image pairs and the overlap between the image blocks.
[0021] Optionally, based on a preset twin feature extraction network, multiple feature maps corresponding to each target image are obtained, including:
[0022] Performing superpixel segmentation on the pixel-level difference map to obtain a second segmentation result;
[0023] Taking the centroid position coordinates of each superpixel in the second segmentation result as the center of the image block and taking the preset size as the size of the image block, generating a plurality of template windows;
[0024] Dividing each target image into image blocks according to the multiple template windows to obtain multiple pairs of image blocks with one-to-one corresponding positions between the two target images;
[0025] Using the twin feature extraction network, extracting multi-layer features of each image block from each pair of obtained image blocks;
[0026] The multiple layers of features belonging to the same target image are concatenated at the same layer to obtain multiple feature maps corresponding to each target image.
[0027] Optionally, the superpixel segmentation is achieved using a simple linear iterative clustering method.
[0028] Optionally, the calculation method of the overlap degree includes:
[0029]
[0030] Among them, image block p and image block q come from two different images in the sample remote sensing image pair respectively; y(p,q) represents the overlap between image block p and image block q; IoU represents the intersection over union ratio between image blocks.
[0031] Optionally, before using the twin feature extraction network to extract multi-layer features of each image block from each pair of obtained image blocks, the method further includes:
[0032] Normalize the obtained image blocks;
[0033] Furthermore, the image blocks constituting the training samples are also normalized image blocks.
[0034] Optionally, calculating the difference between the feature maps of the two target images from the same-level operation layer includes:
[0035] The Euclidean distance between the feature maps of the two target images from the same operation layer is calculated.
[0036] Optionally, fusing the pixel-level difference map and the multiple object-level difference maps comprises:
[0037] A weighted average result of the pixel-level difference map and the multiple object-level difference maps is obtained to obtain the fused image.
[0038] Optionally, obtaining a change detection result according to the fused image includes:
[0039] The fused image is binary segmented using the Otsu threshold method, and the obtained binary segmentation result is the change detection result; wherein 1 in the binary segmentation result indicates that there is a change, and 0 indicates that there is no change.
[0040] Optionally, the deep twin convolutional network model includes, along the data flow direction: the twin feature extraction network, a feature comparison layer for obtaining absolute differences of features output at the end of the twin feature extraction network, and a fully connected layer;
[0041] Each branch of the twin feature extraction network includes, along the data flow direction, a first convolutional layer, a first BN layer, a maximum pooling layer, and a residual layer, as multiple different operation layers of the branch; wherein the residual layer includes two residual modules connected in series;
[0042] The residual module includes, in sequence along the data flow direction: a second convolutional layer, a second BN layer, a third convolutional layer, a third BN layer, and an addition module; wherein the input features of the second convolutional layer and the output features of the third BN layer are sent to the addition module for summation;
[0043] The first BN layer and the second BN layer are also connected to a Relu activation layer respectively;
[0044] The first convolutional layer includes 64 convolution kernels of size 3×3 and a convolution step of 2;
[0045] The convolution kernel size in the maximum pooling layer is 3×3, and the convolution step size is 2;
[0046] The second convolution layer and the third convolution layer each include 64 convolution kernels of size 3×3, and a convolution step size of 1.
[0047] In the high-resolution remote sensing image change detection method based on self-supervised learning provided by the present invention, a plurality of feature maps corresponding to each target image are obtained through a twin feature extraction network, so that the target in the target image can be used as the analysis unit to effectively extract the characteristic information such as spectrum, texture, geometry, etc. contained in the target image, which is helpful to better mine the key information in the target image and make full use of the image context information. In this way, after calculating the difference between the feature maps of the two target images from the same-level operation layer, the obtained multiple target-level difference maps are fused with the pixel-level difference maps of the two target images, and the obtained change detection result is more accurate. In addition, the twin feature extraction network used in the present invention is a feature extraction subnetwork in a pre-trained deep twin convolutional network model, and the training samples used to train the deep twin convolutional network model are all from the aligned high-resolution sample remote sensing image pairs, which match the detection task, ensuring that the actual use performance of the twin feature extraction network is consistent with the performance obtained when the training is completed. Moreover, the labeling information of the training samples in the present invention is automatically calculated and generated by inputting the sample remote sensing image pairs into a preset labeling algorithm. It does not rely on human subjective experience, does not introduce subjective uncertainty, and has higher labeling accuracy. Moreover, the efficiency of the mechanized labeling method has been greatly improved, and it does not require a lot of manpower, material resources, and time, and can obtain useful samples in a timely and accurate manner, which makes the application scope of the present invention in real scenes more extensive.
[0048] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a schematic diagram of the salt and pepper phenomenon;
[0050] Figure 2is a flow chart of a high-resolution remote sensing image change detection method based on self-supervised learning provided by an embodiment of the present invention;
[0051] Figure 3 is a schematic diagram of constructing annotation information for a sample remote sensing image pair in an embodiment of the present invention;
[0052] Figure 4 It is a schematic diagram of the network structure of a deep twin convolutional network model used in an embodiment of the present invention.
[0053] Figure 5 A set of test data sets used in experimental verification of the embodiments of the present invention;
[0054] Figure 6 It is another set of test data sets used in experimental verification of the embodiments of the present invention;
[0055] Figure 7 The method of the present invention is used to analyze the Figure 5 The detection results obtained from the test data set in ;
[0056] Figure 8 The method of the present invention is used to analyze the Figure 6 The detection results obtained from the test data set in ;
[0057] Fig. 9 It is shown in Figure 7 and Figure 8 Quantitative evaluation index of the detection results. DETAILED DESCRIPTION
[0058] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0059] In order to effectively and efficiently use deep learning technology to improve the performance of high-resolution remote sensing image change detection, the embodiment of the present invention provides a high-resolution remote sensing image change detection method based on self-supervised learning. Figure 2 As shown, the method comprises the following steps:
[0060] S10: Acquire two high-resolution remote sensing images taken at different times for the same area, and perform radiation correction and registration on the two high-resolution remote sensing images to obtain two registered target images.
[0061] Among them, radiation correction is performed on two high-resolution remote sensing images to eliminate the image radiation error caused by differences in sensor sensitivity characteristics, solar altitude angle, atmospheric conditions, terrain undulations and other factors; the specific radiation correction method is the same as the relevant existing technology, and the embodiment of the present invention will not be repeated.
[0062] After the two high-resolution remote sensing images are radiometrically corrected, the two corrected high-resolution remote sensing images are further registered so that the pixels of the same object correspond to the same spatial position. There are many specific registration methods, for example, the existing SIFT (Scale-invariant Feature Transform) method can be used to achieve this, which will not be described in detail in the embodiments of the present invention.
[0063] It can be understood that the two high-resolution remote sensing images obtained after registration are two target images.
[0064] S20: Compare the spectral features of the two target images to obtain a pixel-level difference map.
[0065] Specifically, the change vector analysis method can be used to compare the spectral characteristics of two target images. The implementation process is expressed by the following formula:
[0066]
[0067] in, and They represent the pixel values of the target image I1 and the target image I2 in the bth band of the spectrum respectively, B represents the number of spectral bands corresponding to the target image; X0 represents the pixel level difference map.
[0068] S30: Obtain multiple feature maps corresponding to each target image through a preset twin feature extraction network; wherein the multiple feature maps correspond to multiple different operation layers of a single branch of the twin feature extraction network; the twin feature extraction network is a feature extraction subnetwork in a pre-trained deep twin convolutional network model; the sample remote sensing image pairs used to train the deep twin convolutional network model are all from the registered high-resolution sample remote sensing image pairs, and the annotation information of the training samples is automatically calculated and generated by inputting the sample remote sensing image pairs into a preset annotation algorithm.
[0069] It is understandable that high-resolution remote sensing images usually contain richer ground object information, but as the scale of data increases, they gradually show the characteristics of massive, high-dimensional and diversified. When processing such data, traditional shallow learning models are prone to overfitting and underfitting problems due to limited model capacity, resulting in weak model feature expression capabilities and poor generalization performance. In contrast, deep learning models have a deeper network structure and can use training samples to automatically learn more essential feature representations in the original data, with stronger feature extraction capabilities and generalization performance. For the task of high-resolution remote sensing image change detection, learning higher-level feature representations by deep neural networks can amplify the factors in the perceptual input that are important for classification and suppress some irrelevant changes (such as illumination and noise interference), thereby promoting the recognition of real change areas on the ground. Therefore, in the embodiment of the present invention, a feature extraction subnetwork in a deep twin convolutional network model is used to extract feature maps from the target image.
[0070] In practical applications, there are many possibilities for the functions specifically implemented by the deep twin convolutional network model. Exemplarily, in one implementation, the deep twin convolutional network model can be used to predict the overlap of paired image blocks; that is, after a pair of image blocks are input into the deep twin convolutional network model, the model can output the overlap of the pair of image blocks. Correspondingly, its feature extraction subnetwork, that is, the above-mentioned twin feature extraction network, specifically extracts features from paired image blocks in layers through two twin branches. In this way, the features of each image block belonging to the same image are spliced at the same layer to obtain multiple feature maps corresponding to the image.
[0071] Alternatively, in another implementation, the deep twin convolutional network model can be used to segment similar regions of a pair of images; that is, after a pair of images are input into the deep twin convolutional network model, the model can output the coordinates of similar regions between the pair of images, which is also possible. Correspondingly, its feature extraction subnetwork, namely the twin feature extraction network mentioned above, can specifically extract features from paired images in layers through two twin branches.
[0072] It can be understood that the network structure of the deep twin convolutional network model is related to the specific functions that the deep twin convolutional network model is to be implemented. In order to make the layout of the specification clear, the network structure of the deep twin convolutional network model is illustrated below.
[0073] S40: Calculate the difference between the feature maps of the two target images from the same-level operation layer to obtain multiple target-level difference maps.
[0074] For example, the Euclidean distance between the feature maps of two target images from the same operation layer can be calculated:
[0075] X j =d(x 1j ,x 2j )=||x 1j -x 2j ||2;
[0076] Among them, x 1j represents the feature map output by the jth operation layer of the branch into which the target image I1 enters, x 2j represents the feature map output by the jth operation layer of the branch into which the target image I2 enters, ||·||2 is the binary norm, d is the Euclidean distance function, and X j represents the calculated j-th target-level difference map, that is, j is used to number the operation layers of the twin feature extraction network.
[0077] S50: Fusing the pixel-level difference map and the multiple object-level difference maps, and obtaining a change detection result according to the fused image.
[0078] Specifically, the pixel-level difference map and the multiple object-level difference maps are fused, and the pixel-level difference map and the multiple object-level difference maps can be averaged to obtain an average fused image.
[0079] Alternatively, in a preferred implementation, the weighted average result of the pixel-level difference map and multiple target-level difference maps (as shown in the following formula) can be obtained to obtain a weighted average fused image. In this way, their deep fusion can be achieved, thereby retaining more useful change information.
[0080]
[0081] In this formula, X D Represents the fused image, X when k takes different values k Represents pixel-level difference maps and multiple target-level difference maps, μ k Yes X k Corresponding weight factors. Preferably, the weight factors corresponding to the pixel-level difference maps may be slightly higher than the weight factors corresponding to the target-level difference maps.
[0082] After obtaining the fused image, there are many specific implementation methods for obtaining the change detection result based on the fused image. For example, in one implementation method, the area where the pixels in the fused image with pixel values greater than a preset threshold are located can be determined as the area where the change occurs. The threshold can be determined based on manual experience.
[0083] In another implementation, in order to reduce the impact of artificially set thresholds on the detection results, the Otsu threshold method can be used to perform binary segmentation on the fused image, and the obtained binary segmentation result is used as the change detection result; wherein 1 in the binary segmentation result indicates that there is a change, and 0 indicates that there is no change.
[0084] Here, for the fused image X D The Otsu threshold method is used, which is based on the fusion image X D The grayscale histogram is used to calculate the division threshold T that maximizes the inter-class variance. Specifically, if w0 represents the proportion of foreground points in the image, and its average grayscale is represented by u0, and w1 represents the proportion of background points in the image, and its average grayscale is represented by u1; the threshold T is traversed from the minimum grayscale value to the maximum grayscale value. When a certain T value in the traversal makes the variance δ=w0*w1*(u0-u1)*(u0-u1) the maximum, then T at this time is the optimal segmentation threshold determined by the Otsu threshold method.
[0085] Then, the difference map is binarized using the threshold T, which is expressed as follows:
[0086]
[0087] Among them, X D (r,c) represents the fused image X D CM(r,c) represents the pixel value at position (r,c) in the change detection result. 1 represents the pixel value at position (r,c) and 0 represents the pixel value at position (r,c) in the change detection result.
[0088] In the high-resolution remote sensing image change detection method based on self-supervised learning provided by the embodiment of the present invention, a plurality of feature maps corresponding to each target image are obtained through a twin feature extraction network, so that the target in the target image can be used as the analysis unit to effectively extract the characteristic information such as spectrum, texture, geometric shape contained in the target image, which is helpful to better mine the key information in the target image and make full use of the image context information. In this way, after calculating the difference between the feature maps of the two target images from the same level operation layer, the obtained multiple target-level difference maps are fused with the pixel-level difference maps of the two target images, and the obtained change detection result is more accurate, which is specifically manifested in that the missed detection rate and the false detection rate are low, and the occurrence of the "salt and pepper" phenomenon is effectively avoided. In addition, the twin feature extraction network used in the embodiment of the present invention is a feature extraction subnetwork in a pre-trained deep twin convolutional network model. The training samples used to train the deep twin convolutional network model are all from the registered high-resolution sample remote sensing image pairs, which match the detection task, ensuring that the actual use performance of the twin feature extraction network is consistent with the performance when the training is completed. Moreover, the labeling information of the training samples in the embodiment of the present invention is automatically calculated and generated by inputting the sample remote sensing image pair into a preset labeling algorithm. It does not rely on human subjective experience, does not introduce subjective uncertainty, and has higher labeling accuracy. Moreover, the efficiency of the mechanized labeling method has been greatly improved, and it does not require a lot of manpower, material resources and time, and can obtain useful samples in a timely and accurate manner, which makes the application scope of the present invention in real scenes more extensive.
[0089] In summary, the embodiment of the present invention is a completely unsupervised method, which can realize the automation and efficiency of change detection of high-resolution remote sensing images in complex scenes, and further improve the performance of change detection.
[0090] It is understandable that, according to the differences in analysis units and feature extraction methods, remote sensing image change detection methods can be divided into pixel-based change detection methods and target-based change detection methods. In the embodiment of the present invention, the idea of the pixel-based change detection method is used to calculate the pixel-level difference map, while the idea of the target-based change detection method is used to extract the feature map using the twin feature extraction network and calculate the target-level difference map.
[0091] Among them, the target-based change detection method is usually composed of three links: image segmentation, feature extraction and change recognition. Among them, image segmentation is the core of the target-based change detection method, which aims to divide the remote sensing image into multiple targets with similar spectra and adjacent spaces, laying the foundation for subsequent feature extraction and change analysis. Therefore, how to choose a suitable segmentation scale and accurately characterize the key information in the image is a key factor in determining the quality of the detection effect. In order to achieve this goal, an annotation algorithm is adopted in the embodiment of the present invention to realize automatic segmentation of the image, thereby constructing an annotated sample data set for image block classification tasks, without manual annotation, and applicable to any type of remote sensing image. Correspondingly, the above-mentioned deep twin convolutional network model is designed as a self-supervised learning model for predicting the overlap of paired image blocks.
[0092] Specifically, the preferred annotation algorithm proposed in the embodiment of the present invention includes:
[0093] (1) Compare the spectral characteristics of the sample remote sensing image pairs to obtain the sample pixel-level difference map.
[0094] The specific implementation method of this step can refer to the above step S20.
[0095] (2) Perform superpixel segmentation on the sample pixel-level difference map to obtain a first segmentation result.
[0096] It can be understood that a superpixel refers to an irregular pixel block with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other characteristics.
[0097] Exemplarily, in this step, a simple linear iterative clustering method can be used to segment the sample pixel-level difference map into superpixels. Specifically, a segmentation scale s is set (for example, it can be set to 10), and a simple linear iterative clustering method is used to segment the sample pixel-level difference map, so as to obtain superpixels with similar spectra and adjacent spaces, and obtain the segmentation mask I of the corresponding superpixels. M .
[0098] In this step, the method for implementing superpixel segmentation is not limited thereto, for example, the FS algorithm may also be used to perform superpixel segmentation on the sample pixel-level difference map. The FS algorithm is a method for generating superpixel pairs using a fuzzy C-Means (FCM) clustering algorithm.
[0099] (3) Taking the centroid position coordinates of each superpixel in the first segmentation result as the center of the image block and a preset size as the size of the image block, the overlap degree of each pair of image blocks between the sample remote sensing image pairs is calculated.
[0100] Specifically, for each superpixel, the segmentation mask I is used MCalculate the centroid position coordinates of the superpixel; then, in the two sample images included in the sample remote sensing image pair, take an image block of size S×S with the left side of the centroid position as the center, S=2s+2; in this way, the number of image blocks can be taken from each sample remote sensing image as many as the number of superpixels. Then, calculate the overlap of each pair of image blocks between the sample remote sensing image pairs. For example, assuming that the segmentation result includes 10 superpixels, each sample remote sensing image can take 10 image blocks, and it is necessary to calculate the overlap of 10×10=100 pairs of image blocks between the sample remote sensing image pairs.
[0101] The calculation of overlap is mainly to calculate the intersection and union ratio of two image blocks, and the calculation method is as follows:
[0102]
[0103] Among them, image block p and image block q come from two different images in the sample remote sensing image pair respectively; y(p,q) represents the overlap between image block p and image block q, that is, the category labels of image block p and image block q; IoU represents the intersection over union ratio between image blocks.
[0104] For example, Figure 3 Schematic diagram of constructing annotation information for a sample remote sensing image pair is shown in . The left and right are two images contained in the sample remote sensing image pair; when image block 1 and image block 3 partially overlap and their intersection-and-union ratio is between [0,1], the category label of the pair of image blocks (1,3) is marked as 1; when image block 1 and image block 4 completely overlap and their intersection-and-union ratio is 1, the category label of the pair of image blocks (1,4) is marked as 2; when image block 1 and image block 2 do not overlap at all and their intersection-and-union ratio is 0, the category label of the pair of image blocks (1,2) is marked as 0.
[0105] After labeling each pair of image blocks according to the above process, these pairs of image blocks can be used as training samples, and the labels annotated for them are the corresponding annotation information.
[0106] Therefore, in step S30, a plurality of feature maps corresponding to each target image are obtained through the twin feature extraction network, including:
[0107] (1) Perform superpixel segmentation on the pixel-level difference map obtained in step S20 to obtain a second segmentation result.
[0108] Here, superpixel segmentation is performed in the same way as that used to construct training samples above.
[0109] (2) Taking the centroid position coordinates of each superpixel in the second segmentation result as the center of the image block and taking a preset size as the size of the image block, a plurality of template windows are generated.
[0110] Here, the size of the preset size is the same as the size of the preset size used when constructing the training sample above.
[0111] (3) Each target image is divided into image blocks according to the multiple template windows, thereby obtaining multiple pairs of image blocks with one-to-one correspondence between two target images.
[0112] It can be understood that, since the change detection task is to be performed at this time, only pairs of image blocks with one-to-one corresponding positions between the two target images need to be taken, and there is no need to consider pairs of image blocks with non-corresponding positions.
[0113] (4) Using the twin feature extraction network, multi-layer features of each image block are extracted from each pair of image blocks obtained.
[0114] Specifically, each pair of image blocks obtained in step (3) is respectively input into the twin feature extraction network, so that the twin feature extraction network outputs the multi-layer features of each image block. It can be understood that the multi-layer features mentioned here correspond one-to-one to the multiple different operation layers of the branches of the image blocks entering the twin feature extraction network.
[0115] (5) Multiple layers of features belonging to the same target image are concatenated at the same layer to obtain multiple feature maps corresponding to each target image.
[0116] Specifically, for each target image, features corresponding to the same operation layer extracted from each image block contained therein are spliced to obtain a complete feature map of the target image corresponding to each operation layer.
[0117] In the embodiment of the present invention, there may be multiple deep twin convolutional network models suitable for the above-mentioned annotation algorithm. For example, the designed deep twin convolutional network model may be as follows: Figure 4 As shown, along the data flow direction, there are: a twin feature extraction network, a feature comparison layer for calculating the absolute difference of the features output at the end of the twin feature extraction network, and a fully connected layer.
[0118] Among them, Figure 4 As shown, each branch of the twin feature extraction network includes, along the data flow direction, the first convolution layer, the first BN layer, the maximum pooling layer and the residual layer, as multiple different operation layers of the branch; wherein the residual layer includes two residual modules connected in series; the residual module includes, along the data flow direction, the second convolution layer, the second BN layer, the third convolution layer, the third BN layer and an addition module; wherein the input features of the second convolution layer and the output features of the third BN layer are sent to the addition module for summation; the first BN layer and the second BN layer are also connected to Relu activation layers, respectively.
[0119] Among them, in order to reduce the number of parameters of the model, the first convolution layer preferably includes 64 convolution kernels of size 3×3 and a convolution step of 2; the convolution kernel size in the maximum pooling layer is 3×3 and a convolution step of 2; the second convolution layer and the third convolution layer both include 64 convolution kernels of size 3×3 and a convolution step of 1.
[0120] It should be noted that the above-mentioned settings such as the convolution kernel size, number, and convolution step size are merely examples and do not constitute a limitation on the embodiments of the present invention. In practice, they can be adaptively adjusted according to factors such as the actual resolution size of the target image.
[0121] based on Figure 4 It can be seen from the deep twin convolutional network model shown that the twin feature extraction network has 4 different operation layers, and the corresponding step S30 has 4 feature maps for each target image, so there are 4 target-level difference maps calculated in step S40, plus the pixel-level difference map, when they are fused, the weight factor is preferably set to {0.4, 0.1, 0.3, 0.1, 0.1}. Among them, 0.4 is the weight factor corresponding to the pixel-level difference map, 0.1 is the weight factor corresponding to the target-level difference map obtained according to the features output by the two first convolutional layers, the subsequent 0.3 is the weight factor corresponding to the target-level difference map obtained according to the feature maps output by the two first BN layers, the subsequent 0.1 is the weight factor corresponding to the target-level difference map obtained according to the feature maps output by the two maximum pooling layers, and the subsequent 0.1 is the weight factor corresponding to the target-level difference map obtained according to the feature maps output by the two residual layers.
[0122] In addition, the network model applicable to the above-mentioned annotation algorithm and capable of realizing the above-mentioned overlap prediction function is not limited to Figure 4 The network model shown in ; for example, you can also use the ResNet18 network or other residual networks as the backbone network to build other networks with similar functions, or you can use two VGG16 networks to build the above-mentioned twin feature extraction network, etc., which are all feasible.
[0123] After building the network model and data set, you can train the model.
[0124] First, set the initial parameters for the model. Here, in order to speed up the convergence of the model and improve the feature extraction ability of the model, the weights trained on the network model of the existing dataset can be used as the initial parameters of the built model. For example, when using the ResNet18 network as the backbone network to build a deep twin convolutional network model, the weights of the ResNet18 network pre-trained on the ImageNet dataset can be used as the initial parameters of the deep twin convolutional network model.
[0125] Then, the training samples in the training sample library are input into the built model, so that the model outputs the prediction results. Then, the loss value is calculated based on the prediction results and the annotation information of the input training samples. When the loss value is less than the preset loss threshold, the model converges and a trained deep twin convolutional network model is obtained.
[0126] Among them, since the model performs a multi-classification task, the Softmax loss function is used as the loss function of the network, which can be expressed as:
[0127]
[0128] Where N represents the number of training samples. It is used to indicate whether the nth training sample belongs to the i-th category. When its value is 0, it means that the nth training sample does not belong to the i-th category. When its value is 1, it means that the nth training sample belongs to the i-th category. The specific value of is determined by the labeling information of the nth training sample; represents the probability that the nth training sample output by the network model belongs to the i-th category, w i represents the weight of the sample belonging to the i-th category.
[0129] During the training process, the Adam algorithm (Adaptive Momentum Estimation) is used to optimize the network. The learning rate parameter can be set to 0.001, the weight decay coefficient can be set to 0.00005, and the training generation can be set to 200. Of course, these settings can be adjusted.
[0130] In other implementations, if the deep twin convolutional network model is a self-supervised learning model for similarity region segmentation of the input sample remote sensing image pair, the common image segmentation models such as FCN, SegNet, U-Net or PSPNet can be used to build its main structure. Among them, when constructing annotation information for the sample remote sensing image pair, a preset size of image block can be used as a template to traverse each image in the sample remote sensing image pair, and the traversal step can be adjusted, which can be equal to the diameter of the image block or less than the diameter, so that each image is divided into multiple image blocks; then, by calculating the similarity between the image blocks corresponding to each two positions between the sample remote sensing image pairs, and determining the changed area and unchanged area in the sample remote sensing image pair according to the similarity threshold, the automatic construction of annotation information is realized, and self-supervised learning is realized.
[0131] Optionally, in one implementation, in order to speed up the convergence of the model, before using the twin feature extraction network to extract the multi-layer features of each image block from each pair of image blocks obtained, the method provided by the embodiment of the present invention may further include: normalizing the obtained image blocks. Moreover, when training the deep twin convolutional network model, the image blocks in the training samples used are also normalized image blocks.
[0132] Here, the normalization process can be expressed by the formula as follows:
[0133]
[0134] Among them, X(r,c) is the pixel value of the image block at position (r,c), X out is the corresponding normalized output pixel value, X max and X min are the maximum and minimum values of the pixels in the target image, which are 255 and 0 respectively.
[0135] In summary, it can be seen that the embodiments of the present invention can solve various problems such as the inability of the prior art to effectively represent the object information of the image, insufficient high-quality annotated samples, and insufficient extraction of change information, thereby practically and effectively utilizing deep learning technology in high-resolution remote sensing image change detection, thereby improving the performance of change detection.
[0136] The beneficial effects of the embodiments of the present invention can be further illustrated by the following simulation:
[0137] Simulation conditions: AMD Ryzen 5800X CPU, 3.8GHz Windows 10 system and Python3.7 running platform are used to complete the simulation comparison experiment of the method provided by the present invention and the existing IR-MAD (Iteratively Reweighted Multivariate Alteration Detection) method, PCANet (based on multi-channel principal component analysis network) method and DCVA (deep change vector analysis) algorithm.
[0138] Evaluation indicators:
[0139] (1) Precision: refers to the ratio of the number of pixels correctly identified as changed in the change detection results to the number of pixels predicted as changed in the results. The calculation formula is as follows:
[0140]
[0141] Among them, TP and FP represent the number of pixels correctly classified as the change class and the number of pixels mistakenly classified as the change class in the change detection results, respectively.
[0142] (2) Recall: refers to the ratio of the number of pixels correctly identified as changed pixels to the number of actual changed pixels in the change detection results. The calculation formula is as follows:
[0143]
[0144] Among them, FN represents the number of pixels that are not correctly identified as change classes in the change detection results.
[0145] (3) Accuracy: refers to the ratio of the number of correctly detected pixels in the change detection results to the number of pixels in the entire image. The calculation formula is as follows:
[0146]
[0147] Among them, TN represents the number of pixels correctly classified as unchanged in the change detection results.
[0148] (4) Kappa coefficient: It is used to measure the overall classification ability of the change detection algorithm. The larger the value, the better the classification performance of the algorithm. The calculation formula is as follows:
[0149]
[0150]
[0151] Among them, RC and RU represent the number of pixels of the changed and unchanged classes in the reference image manually annotated according to prior knowledge, respectively.
[0152] Experimental data:
[0153] The first set of test data sets is a data set shot in a certain place in the country, such as Figure 5 As shown in sub-images (a) and (b), it mainly reflects the land type changes caused by ground vegetation. Each image contains three bands: red, green and near-infrared. The image size is 877×738 and the spatial resolution is 2.5 meters. Figure 5 Sub-image (c) in the figure is a reference image that is manually annotated by experts based on prior knowledge of the image, where white represents the area that has changed and black represents the area that has not changed.
[0154] The second set of test data sets is a data set shot in another place in China, such as Figure 7As shown in sub-images (a) and (b) in the figure, it mainly reflects the changes in land types during the urbanization process. Each image contains three bands: red, green and blue. The image size is 1431×1431 and the spatial resolution is 2 meters. Figure 7 Sub-image (c) in the figure is the reference image manually annotated based on image visual interpretation.
[0155] Experimental content and results:
[0156] Figure 6 Sub-figures (a) to (d) in FIG. 1 are detection results obtained for the first set of experimental data using the existing IR-MAD method, the PCANet method, the DCVA method, and the method provided in the embodiment of the present invention, respectively.
[0157] By comparison, it can be seen that the IR-MAD method mistakenly identifies unchanged roads as changed ones, resulting in poor detection results. By considering the neighborhood information of pixels, the PCANet method can detect the main changed areas, but many small changes cannot be effectively identified. The DCVA method can detect most of the changed areas on the surface, but a large number of white noise points appear in the results. The method provided by the embodiment of the present invention can more accurately identify the changed areas, and the change results are cleaner overall.
[0158] Figure 8 Sub-figures (a) to (d) in FIG. 1 are detection results obtained for the second set of experimental data using the existing IR-MAD method, the PCANet method, the DCVA method, and the method provided in the embodiment of the present invention.
[0159] By comparison, it can be seen that the IR-MAD method cannot effectively identify the changed areas, and the detection results are full of spots; the PCANet method can only detect a small part of the changed areas, and some unchanged areas are also identified as changed categories; the DCVA method can identify the main changed areas, but loses a lot of change details; and the method provided in the embodiment of the present invention can effectively highlight the changes in the surface, and the results contain less noise.
[0160] Fig. 9 The quantitative evaluation indicators of different change detection methods on the above two sets of experimental data are shown in Fig. 9It can be seen that the Accuracy and Kappa values of the method provided by the embodiment of the present invention on the first test data set are 0.9754 and 0.9013 respectively, which are much better than the existing methods. For the second test data set, since it contains more types of objects, change detection is more difficult. The Recall, Accuracy and Kappa of the method provided by the embodiment of the present invention on this data set are the highest, which further illustrates the effectiveness and feasibility of the method of the present invention.
[0161] In summary, the high-resolution remote sensing image change detection method based on self-supervised learning provided by the embodiment of the present invention uses superpixels as the basic analysis unit, extracts useful features in the image through self-supervised learning, and combines feature comparison and threshold segmentation to obtain the final change detection result. Compared with the existing methods, the method provided by the embodiment of the present invention can achieve better performance in the problem of high-resolution remote sensing image change detection, and has a wider range of applications and stronger robustness.
[0162] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, a terminal device, a server, etc., which is not limited here. Any electronic device that can implement the present invention belongs to the protection scope of the present invention.
[0163] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0164] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.
[0165] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, a person skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims.
[0166] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, apparatuses (devices) and computer program products of the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0169] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A high-resolution remote sensing image change detection method based on self-supervised learning, characterized in that: include: Acquire two high-resolution remote sensing images of the same area taken at different times, and perform radiometric correction and registration on the two high-resolution remote sensing images to obtain two registered target images; Comparing the spectral characteristics of the two target images to obtain a pixel-level difference map; Obtaining multiple feature maps corresponding to each target image through a preset twin feature extraction network; wherein the multiple feature maps respectively correspond to multiple different operation layers of a single branch of the twin feature extraction network; Calculating the difference between the feature maps of the two target images from the same-level operation layer to obtain a plurality of target-level difference maps; fusing the pixel-level difference map and the plurality of object-level difference maps, and obtaining a change detection result based on the fused image; The twin feature extraction network is a feature extraction subnetwork in a pre-trained deep twin convolutional network model; the training samples used to train the deep twin convolutional network model are all from registered high-resolution sample remote sensing image pairs, and the annotation information of the training samples is automatically calculated by inputting the sample remote sensing image pairs into a preset annotation algorithm; The deep Siamese convolutional network model is used to predict the overlap of paired image blocks; The labeling algorithm includes: Comparing the spectral characteristics of the sample remote sensing image pairs to obtain a sample pixel-level difference map; Performing superpixel segmentation on the sample pixel-level difference map to obtain a first segmentation result; Taking the centroid position coordinates of each superpixel in the first segmentation result as the center of the image block and a preset size as the size of the image block, calculating the overlap of each pair of image blocks between the sample remote sensing image pairs; The training samples and the annotation information are constructed according to each pair of image blocks between the sample remote sensing image pairs and the overlap between the image blocks.
2. The method according to claim 1, characterized in that Based on the preset twin feature extraction network, multiple feature maps corresponding to each target image are obtained, including: Performing superpixel segmentation on the pixel-level difference map to obtain a second segmentation result; Using the centroid position coordinates of each superpixel in the second segmentation result as the center of the image block and using the preset size as the size of the image block to generate multiple template windows; Dividing each target image into image blocks according to the multiple template windows to obtain multiple pairs of image blocks with one-to-one correspondence between the two target images; Using the twin feature extraction network, extracting multi-layer features of each image block from each pair of obtained image blocks; The multi-layer features belonging to the same target image are spliced at the same layer to obtain multiple feature maps corresponding to each target image.
3. The method according to claim 1 or 2, characterized in that The superpixel segmentation is achieved using a simple linear iterative clustering method.
4. The method according to claim 1, wherein The calculation method of the overlap degree includes: Among them, image block p and image block q come from two different images in the sample remote sensing image pair respectively; y(p,q) represents the overlap between image block p and image block q; IoU represents the intersection over union ratio between image blocks.
5. The method according to claim 2, characterized in that Before respectively extracting multi-layer features of each image block from each pair of obtained image blocks using the twin feature extraction network, the method further includes: Normalize the obtained image blocks; Furthermore, the image blocks constituting the training samples are also normalized image blocks.
6. The method according to claim 1, wherein Calculating the difference between the feature maps of the two target images from the same level operation layer, including: Calculate the Euclidean distance between the feature maps of the two target images from the same level operation layer.
7. The method according to claim 1, characterized in that The pixel-level difference map and the plurality of object-level difference maps are fused, comprising: A weighted average result of the pixel-level difference map and the multiple object-level difference maps is obtained to obtain the fused image.
8. The method according to claim 1, characterized in that The change detection results are obtained based on the fused image, including: The fused image is binary segmented using the Otsu threshold method, and the obtained binary segmentation result is the change detection result; wherein 1 in the binary segmentation result indicates that there is a change, and 0 indicates that there is no change.
9. The method according to claim 1, characterized in that The deep twin convolutional network model includes, along the data flow direction, the twin feature extraction network, a feature comparison layer for calculating the absolute difference of the features outputted at the end of the twin feature extraction network, and a fully connected layer; Each branch of the twin feature extraction network includes, along the data flow direction, the first convolutional layer, the first BN layer, the maximum pooling layer, and the residual layer, as multiple different operation layers of the branch; wherein the residual layer includes two residual modules connected in series; The residual module includes, in order along the data flow direction: a second convolutional layer, a second BN layer, a third convolutional layer, a third BN layer, and an addition module; wherein the input features of the second convolutional layer and the output features of the third BN layer are sent to the addition module for summation; The first BN layer and the second BN layer are further connected to a Relu activation layer respectively; The first convolutional layer includes 64 convolution kernels of size 3×3 and a convolution step of 2; The convolution kernel size in the maximum pooling layer is 3×3, and the convolution step size is 2; The second convolutional layer and the third convolutional layer each include 64 convolution kernels of size 3×3, and a convolution step size of 1.
Citation Information
Patent Citations
High-resolution remote sensing image change detection method based on twin convolutional neural network
CN111161218A
Unsupervised change detection method and system for homologous or heterologous remote sensing image
CN113901900A
Multi-temporal remote sensing image registration method and system based on multi-scale receptive field
CN114022522A
