Small sample iteration driven hyperspectral remote sensing image change discovery method
Through the iteratively driven change feature extraction model of small samples, the multi-dimensional feature extraction and symmetric scale difference modules are used to generate pseudo-samples, which solves the problem of limited sample size in hyperspectral remote sensing image change detection and improves detection accuracy and feature extraction capabilities.
Patent Information
- Application Number
- CN202411183722.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2025-01-17
AI Technical Summary
The prior art requires a large number of manually labeled samples in the detection of hyperspectral remote sensing image change, resulting in low detection accuracy in the case of limited sample size, making it difficult to effectively extract deep features.
Using a small sample iteratively driven change feature extraction model, by establishing pre-event image feature extraction branches, differential feature expression branches, and post-event image feature extraction branches, and using multi-dimensional feature extraction module and symmetric scale difference module to generate pseudo-samples to improve detection accuracy .
In the case of a small number of samples, the accuracy of the change detection of hyperspectral remote sensing images was improved, the differential characteristics of the two-temporal images were extracted, and effective pseudo-samples were generated, which improved the detection ability of the network model and improved the accuracy of change discovery.
Smart Images
Figure CN119323678A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of remote sensing image change detection, and in particular relates to a small sample iteration driven hyperspectral remote sensing image change detection method. Background Art
[0002] In recent years, with the rapid development of aerospace remote sensing technology, the number of satellite launches has continued to increase, and the amount of hyperspectral remote sensing data that can be used for research and analysis has continued to increase. Hyperspectral remote sensing images have received widespread attention. Compared with other images, hyperspectral images contain rich spectral information and high spectral resolution, which can obtain more obvious spectral features of ground objects. Through the analysis and comparison of spectral features, the classification and identification of surface cover, such as vegetation, water bodies, buildings, etc., can be achieved, providing important data support for land use planning, resource management and environmental monitoring. It has been widely used in agricultural development, military layout, environmental monitoring, urban planning, natural disaster assessment and other fields. In traditional methods, model training requires a large number of manually labeled samples, and sample labeling requires a lot of manpower and material resources. Summary of the invention
[0003] The purpose of the present invention is to provide a small sample iteration driven hyperspectral remote sensing image change detection method, which can effectively extract the deep spatial-spectral joint features of hyperspectral remote sensing images under the condition of limited sample size to improve the accuracy of hyperspectral remote sensing image change detection.
[0004] The present invention adopts the following technical solution: a training method for a small sample iteratively driven change feature extraction model, comprising the following steps:
[0005] Step 1: Establish a change feature extraction model: including a pre-event image feature extraction branch, a difference feature expression branch, and a post-event image feature extraction branch;
[0006] Step 2: Construct a sample set: collect and manually annotate pre-event images and post-event images, including 0.5% of the sample points with real changes and the remaining unannotated sample points with unknown changes, as sample set S1;
[0007] Step 3: Change feature extraction model training:
[0008] Input the samples in the sample set in step 2 into the change feature extraction model to obtain a change amplitude map based on the dual-temporal image, and output a change probability map;
[0009] In the change probability graph, select the top 1% of sample points with the largest change probability as the changed pseudo sample C1, and the top 1% of sample points with the smallest change probability as the unchanged pseudo sample U1. Merge the changed pseudo sample C1 and the unchanged pseudo sample U1 with the sample set S1 to obtain a new sample set.
[0010] Step 4: Use the new sample set in step 3 as the new input of the change feature extraction model;
[0011] Step 5: Repeat steps 3 and 4 to obtain a trained change feature extraction model.
[0012] Furthermore, the pre-event image feature extraction branch and the post-event image feature extraction branch each include three serial multi-dimensional feature extraction modules; each multi-dimensional feature extraction module includes a feature extraction branch of three dimensions and a first convolutional layer, the feature extraction branches of the three dimensions are in parallel and are all connected to the first convolutional layer, and the feature extraction branch of each dimension is composed of two serially connected second convolutional layers;
[0013] The difference feature expression branch includes three serial symmetric scale difference modules, each of which is composed of a parallel difference information extraction branch at the current scale and a difference information extraction branch at the previous scale.
[0014] Furthermore, the convolution kernel sizes of the two second convolutional layers are different.
[0015] Further, in step three, the pre-event remote sensing image is input into the pre-event image feature extraction branch; the post-event remote sensing image is input into the post-event image feature extraction branch;
[0016] Based on the pre-event image feature extraction branch, three multi-dimensional feature extraction modules extract the three-dimensional features of the pre-event remote sensing image to obtain the feature information of the pre-event remote sensing image;
[0017] Based on the post-event image feature extraction branch, the multi-dimensional feature extraction module extracts and fuses the feature information of the post-event remote sensing image in multiple dimensions to obtain the feature information of the post-event remote sensing image;
[0018] The feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event, and the feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event are subtracted, and the feature information obtained is used as the input of the first symmetric scale difference module. Based on the first symmetric scale difference module, the first differential feature information is obtained;
[0019] The first differential feature information, the feature information of the remote sensing image before the event, and the feature information of the remote sensing image after the event are input to the second symmetric scale differential module, and the second differential feature information is obtained based on the input of the second symmetric scale differential module;
[0020] The second differential feature information, the feature information of the remote sensing image before the event, and the feature information of the remote sensing image after the event are input to the third symmetric scale differential module. Based on the input of the third symmetric scale differential module, the third differential feature information is obtained.
[0021] Furthermore, the loss function of the change feature extraction model is a cross entropy loss function.
[0022] Furthermore, the feature extraction branches in three dimensions are respectively a feature extraction branch in a horizontal dimension, a feature extraction branch in a vertical dimension, and a feature extraction branch in a feature channel dimension.
[0023] The present invention also discloses a small sample iteration driven hyperspectral remote sensing image change discovery method, based on the trained change feature extraction model in the above-mentioned small sample iteration driven change feature extraction model training method, including the following:
[0024] After the entire image before and after the event is input into the trained change feature extraction model, the change amplitude map is obtained, which is converted into a change probability map using the Sigmoid function. Through threshold segmentation, pixels with a change probability greater than 0.5 are segmented as changed pixels, and pixels with a change probability less than 0.5 are segmented as unchanged pixels. Different colors are given to changed pixels and unchanged pixels, and a binary change detection map is output.
[0025] The beneficial effects of the present invention are: 1. Through the change discovery method of the iterative network model, the difference features of the dual-temporal hyperspectral remote sensing images are extracted, and the difference features are used to generate pseudo samples to help the network model training, so that detection can be completed with a small number of samples and the accuracy of change discovery can be improved.
[0026] 2. For hyperspectral remote sensing images, a multidimensional feature extraction module is used to extract features of highlight data along different dimensions, and then the feature information from different dimensions is fused to enhance the model's ability to extract image features and provide more effective and robust deep features for subsequent differential expression.
[0027] 3. A learning module with multi-scale symmetric difference features is adopted, which utilizes the dual-temporal hyperspectral difference features at different scales and effectively complements the difference feature information between different scales, so as to use the difference features to generate pseudo samples that can improve the network detection ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is a structural diagram of a multi-dimensional feature extraction module;
[0029] Figure 2 It is a schematic diagram of the structure of the differential feature expression branch;
[0030] Figure 3 Schematic diagram of example data, where: (a) is the remote sensing image before the event; (b) is the remote sensing image after the event; (c) is the ground reference truth;
[0031] Figure 4 This is a visualization comparison result diagram of different methods, where:
[0032] (a) is the result diagram under the MSDFFN method; (b) is the result diagram under the TriTF method, (c) is the result diagram under the SSTFormer method; (d) is the result diagram under the method of the present invention. DETAILED DESCRIPTION
[0033] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] The present invention discloses a training method for a small sample iteratively driven variation feature extraction model, comprising the following steps:
[0035] Step 1: Establish a change feature extraction model: including a pre-event image feature extraction branch, a difference feature expression branch, and a post-event image feature extraction branch;
[0036] Step 2: Construct a sample set: collect pre-event images and post-event images and manually annotate them, including 0.5% of the sample points with real changes and the remaining unlabeled sample points with unknown changes, as sample set S1; and cut the sample images into 9x9 image blocks.
[0037] Step 3: Change feature extraction model training:
[0038] The samples in the sample set in the step 2 are input into the change feature extraction model to obtain a change amplitude map based on the dual-temporal image, and a change probability map is output; the change amplitude map reflects the change amplitude of each pixel in the image, and the change amplitude map is converted into a probability value between 0 and 1 by a sigmoid function to obtain a change probability map used to reflect the change probability of each pixel point.
[0039] In the change probability graph, select the top 1% of sample points with the largest change probability as the changed pseudo samples C1, and the top 1% of sample points with the smallest change probability as the unchanged pseudo samples U1, and merge the changed pseudo samples C1 and the unchanged pseudo samples U1 with the sample set S1 to obtain a new sample set;
[0040] The loss function of the above change feature extraction model is the cross entropy loss function:
[0041]
[0042] Where N is the total number of pixels of the image to be processed; y i is the predicted label of the i-th pixel; x i is the reference true value, which is 1 or 0;
[0043] is a Sigmoid function, which is used to convert the output of the loss function of the feature extraction model into a probability value between 0 and 1.
[0044] Step 4: using the new sample set in step 3 as a new input of the change feature extraction model;
[0045] Step 5: Repeat steps 3 and 4 to obtain a trained change feature extraction model. Repeat the above process 20 times. The change feature extraction model is well trained and can recognize and extract features, and has the performance of obtaining representative change features.
[0046] like Figure 1 and 2 As shown, the above-mentioned pre-event image feature extraction branch and post-event image feature extraction branch both include three serial multi-dimensional feature extraction modules, which are used to enhance the hyperspectral multi-dimensional information extraction capability, so that the subsequent change characteristics can be better mined. Since remote sensing objects have a certain directionality in space and have different spectral characteristics in different spectral ranges, it is necessary to extract multi-dimensional features of remote sensing objects from different dimensions. Each of the multi-dimensional feature extraction modules includes feature extraction branches in three dimensions and a first convolutional layer. The feature extraction branches in three dimensions are parallel and connected to the first convolutional layer. The feature extraction branches in each dimension are composed of two second convolutional layers connected in series.
[0047] The feature extraction branches in three dimensions are respectively the feature extraction branch in the horizontal dimension, the feature extraction branch in the vertical dimension and the feature extraction branch in the feature channel dimension.
[0048] In each branch, the convolution kernels of the two second convolution layers connected in series have different sizes, which can take into account objects at different scales. One of the convolution kernels is 1x1, and the other is 3x3. The second convolution layer with a convolution kernel size of 1x1 is used for the detailed information of objects in the remote sensing image at a small scale of 1x1, and the second convolution layer with a convolution kernel size of 3x3 is used for the feature information of objects and their neighborhood at a scale of 3x3. Finally, the image feature information obtained from the three branches is connected, and the first convolution layer with a convolution kernel size of 1x1 is used to fuse the feature information from different dimensions, and downsample to obtain image information of 7x7 size.
[0049] The difference feature expression branch includes three serial symmetric scale difference modules, which are used to improve the differential feature capability of the model's dual-temporal hyperspectral image and enhance the ability to identify changed areas. Each of the symmetric scale difference modules is composed of a parallel differential information extraction branch at the current scale and a differential information extraction branch at the previous scale.
[0050] In the step 3, the remote sensing image before the event is divided into two paths, one path is input into the feature extraction branch of the image before the event, and the other path is input into the difference feature expression branch; the remote sensing image after the event is divided into two paths, one path is input into the feature extraction branch of the image after the event, and the other path is input into the difference feature expression branch; at the same time, the outputs of the feature extraction branch of the image before the event and the feature extraction branch of the image after the event are used as inputs to input into the difference feature expression branch;
[0051] Based on the pre-event image feature extraction branch, the multi-dimensional feature extraction module extracts and fuses feature information of multiple dimensions to obtain feature information of the pre-event remote sensing image;
[0052] Based on the post-event image feature extraction branch, the multi-dimensional feature extraction module extracts and fuses feature information of multiple dimensions to obtain feature information of post-event remote sensing images;
[0053] The difference feature expression branch has three inputs. Based on the difference feature expression branch, the first symmetric scale difference module has the following inputs: the feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event obtained by the multidimensional feature extraction module are two 7x7 small-scale feature information. At the same time, the feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event are subtracted to obtain 9x9 large-scale differential feature information. After downsampling by the first symmetric scale difference module, the 9x9 large-scale differential feature information is channel-fused with the 7x7 small-scale differential feature to obtain complementary feature 1; at the same time, the 7x7 small-scale differential feature is upsampled and channel-fused with the 9x9 large-scale differential feature information to obtain 9x9 scale complementary feature 2. The 9x9 scale complementary feature 2 is downsampled and channel-fused with the 7x7 scale complementary feature 1 to obtain the final 7x7 size first differential feature information. The first differential feature information of size 7x7 will be used as guiding information in the second symmetric scale differential module. Similar to the first symmetric scale differential module, the feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event, i.e., two 5x5 small-scale feature information, will be input into the second symmetric scale differential module to obtain the second differential feature information of size 5x5; the 5x5 differential feature information will be used as guiding information in the third symmetric scale differential module. At the same time, the feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event, i.e., two 3x3 small-scale feature information, will be input into the third symmetric scale differential module to obtain the third differential feature information of size 3x3.
[0054] The three serial multi-scale symmetric scaling modules gradually extract and utilize information of different scales, reduce the heterogeneity of information up and down sampling between scales, and can more smoothly complement feature information between different scales, thereby better expressing the differences between features.
[0055] The present invention also discloses a small sample iteration driven hyperspectral remote sensing image change discovery method, based on the trained change feature extraction model in the above-mentioned small sample iteration driven change feature extraction model training method, including the following:
[0056] After the entire image before and after the event is input into the trained change feature extraction model, the change amplitude map is obtained, which is converted into a change probability map using the Sigmoid function. Through threshold segmentation, pixels with a change probability greater than 0.5 are segmented as changed pixels, and pixels with a change probability less than 0.5 are segmented as unchanged pixels. Different colors are given to changed pixels and unchanged pixels, and a binary change detection map is output.
[0057] In order to verify the invention of the present invention, a farmland in a certain area is taken as an example for verification. Figure 3 As shown in Figure 1, farmland is a dataset used for change detection. It is a public dataset containing 155 bands with typical "change" characteristics and dual-phase image false color pictures. The results obtained by using the method in the present invention and other comparison methods are as follows: Figure 4 Table 1 shows the accuracy comparison results of different methods.
[0058] Table 1 Accuracy comparison results of different methods
[0059] Methods OA Kappa AA FA MA TE MSDF 96.66 0.9190 95.97 2.377 5.690 3.338 TriTF 94.85 0.8752 93.80 3.696 8.699 5.148 SSTFormer 96.92 0.9266 96.79 3.446 2.183 3.079 Proposed 97.80 0.9467 97.36 1.581 3.704 2.197
[0060] The calculation formula for the accuracy evaluation index in the table is as follows:
[0061] Overall Accuracy (OA),
[0062] Average user's Accuracy (AA),
[0063] Kappa Coefficient (Ka), in
[0064] False Alarm (FA),
[0065] Missed Alarm (MA),
[0066] Total Error (TE),
[0067] Among them, TP represents the true change class, FP represents the false change class, TN represents the true unchanged class, and FN represents the false unchanged class.
[0068] By using the method of the present invention and other comparative methods, it can be seen from Table 1 that the best accuracy is achieved in the three positive statistical indicators of OA, AA, and Kappa coefficient. The higher the value, the better the detection effect. The best accuracy is also achieved in the two negative statistical indicators of FA and TE. The lower the value, the better the detection effect. At the same time, a low FA is also maintained. This shows that the method proposed in the present invention has a low error rate when distinguishing changes from non-changes. The experimental results show that the method of the present invention can effectively detect surface cover changes under the condition of limited labeled samples, and the detection accuracy is better than similar methods, which improves the detection effect of change discovery.
Claims
1. A training method for a small sample iteratively driven variation feature extraction model, characterized in that: The steps include: Step 1: Establish a change feature extraction model: including a pre-event image feature extraction branch, a difference feature expression branch, and a post-event image feature extraction branch; Step 2: Construct a sample set: collect and manually annotate pre-event images and post-event images, including 0.5% of the sample points with real changes and the remaining unannotated sample points with unknown changes, as sample set S1; Step 3: Change feature extraction model training: Input the samples in the sample set in step 2 into the change feature extraction model to obtain a change amplitude map based on the dual-temporal image, and output a change probability map; In the change probability graph, select the top 1% of sample points with the largest change probability as the changed pseudo samples C1, and the top 1% of sample points with the smallest change probability as the unchanged pseudo samples U1, and merge the changed pseudo samples C1 and the unchanged pseudo samples U1 with the sample set S1 to obtain a new sample set; Step 4: using the new sample set in step 3 as a new input of the change feature extraction model; Step 5: Repeat steps 3 and 4 to obtain a trained change feature extraction model.
2. The training method of a small sample iteratively driven variation feature extraction model as claimed in claim 1, characterized in that: The pre-event image feature extraction branch and the post-event image feature extraction branch each include three serial multi-dimensional feature extraction modules; each of the multi-dimensional feature extraction modules includes a feature extraction branch of three dimensions and a first convolutional layer, the feature extraction branches of the three dimensions are parallel and connected to the first convolutional layer, and the feature extraction branch of each dimension is composed of two serially connected second convolutional layers; The difference feature expression branch includes three serial symmetric scale difference modules, and each of the symmetric scale difference modules is composed of a parallel difference information extraction branch at the current scale and a difference information extraction branch at the previous scale.
3. The training method of a small sample iteratively driven variation feature extraction model as claimed in claim 2, characterized in that: The convolution kernel sizes of the two second convolutional layers are different.
4. The training method of a small sample iteratively driven variation feature extraction model as claimed in claim 3, characterized in that: In the step 3, the pre-event remote sensing image is input into the pre-event image feature extraction branch; Post-event remote sensing image input post-event image feature extraction branch; Based on the pre-event image feature extraction branch, three multi-dimensional feature extraction modules extract the features of the three dimensions of the pre-event remote sensing image to obtain feature information of the pre-event remote sensing image; Based on the post-event image feature extraction branch, the multi-dimensional feature extraction module extracts and fuses the feature information of the post-event remote sensing image in multiple dimensions to obtain the feature information of the post-event remote sensing image; The feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event, and the feature information of the remote sensing image before the event and the feature information of the remote sensing image after the event are subtracted, and the feature information obtained is used as the input of the first symmetric scale difference module. Based on the first symmetric scale difference module, the first differential feature information is obtained; The first differential feature information, the feature information of the remote sensing image before the event, and the feature information of the remote sensing image after the event are input to the second symmetric scale differential module, and the second differential feature information is obtained based on the input of the second symmetric scale differential module; The second differential feature information, the feature information of the remote sensing image before the event, and the feature information of the remote sensing image after the event are input to the third symmetric scale differential module. Based on the input of the third symmetric scale differential module, the third differential feature information is obtained.
5. A training method for a small sample iteratively driven variation feature extraction model as described in claim 4, characterized in that: The loss function of the change feature extraction model is a cross entropy loss function.
6. A training method for a small sample iteratively driven variation feature extraction model as described in claim 5, characterized in that: The feature extraction branches in the three dimensions are respectively a feature extraction branch in the horizontal dimension, a feature extraction branch in the vertical dimension and a feature extraction branch in the feature channel dimension.
7. A small sample iteration driven hyperspectral remote sensing image change detection method, characterized in that: The trained change feature extraction model in the training method of a change feature extraction model driven by small sample iteration according to any one of claims 1 to 6 comprises the following: After the entire image before and after the event is input into the trained change feature extraction model, the change amplitude map is obtained, which is converted into a change probability map using the Sigmoid function. Through threshold segmentation, pixels with a change probability greater than 0.5 are segmented as changed pixels, and pixels with a change probability less than 0.5 are segmented as unchanged pixels. Different colors are given to changed pixels and unchanged pixels, and a binary change detection map is output.