Methods, devices, equipment, and media for small-scale deep learning-based change detection

Abstract

The present invention provides a small-scale deep learning change detection method, device, equipment and medium, including: obtaining two-period remote sensing images corresponding to a small-scale monitoring area; performing multi-scale transformation processing on the two-period remote sensing images to generate a difference probability image, and performing semantic segmentation processing on the two-period remote sensing images through a deep learning model to generate two-period ground feature image spots; based on the difference probability image and the two-period ground feature image spots, generating a change image spot result corresponding to the small-scale monitoring area. The present invention realizes change detection of small-scale monitoring areas, and provides support for the improvement of ecological environment supervision and governance capabilities in small-scale monitoring areas.

Description

Technical Field

[0001] The present invention relates to the field of change detection technology, and in particular to a small-scale deep learning change detection method, device, equipment and medium. Background Art

[0002] Remote sensing change detection is a key technology in Earth observation. It involves comparing and analyzing two sets of imagery, using relevant methods to determine the extent of change. The goal is to filter out non-change information and extract change information. Remote sensing change monitoring has a wide range of applications, including disaster monitoring, urban surveillance, forest protection, military operations, and topographic and geomorphological changes.

[0003] Deep learning, a branch of machine learning, has rapidly developed due to its multi-level feature learning capabilities. Deep learning change detection algorithms have become the mainstream algorithm for remote sensing change detection. Deep learning change detection algorithms generally utilize deep neural networks such as convolutional neural networks (CNNs), generative adversarial networks (GANs), and the Transformer attention mechanism to construct change detection network frameworks for change detection. These methods generally require a large number of labeled samples and are suitable for provincial and national monitoring areas where significant changes occur. However, due to the small number of changes in small-scale areas at the city and county level within a short period of time, it is difficult to obtain samples for deep learning. Furthermore, deep learning models trained using provincial and national regional samples are difficult to directly apply to the city and county scale due to differences in geographical characteristics such as phenology and topography.

[0004] In summary, for change detection in small-scale areas, the deep learning method cannot obtain enough change samples in the monitoring area, which seriously restricts the application of change detection in small-scale areas. There is an urgent need for a deep learning change detection algorithm for small-scale areas. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a small-scale deep learning change detection method, device, equipment and medium, which realizes change detection in small-scale monitoring areas and provides support for the improvement of ecological environment supervision and governance capabilities in small-scale monitoring areas.

[0006] In a first aspect, an embodiment of the present invention provides a small-scale deep learning change detection method, comprising:

[0007] Obtain two phases of remote sensing images corresponding to the small-scale monitoring area;

[0008] The two phases of remote sensing images are processed through multi-scale transformation to generate difference probability images, and the two phases of remote sensing images are processed through semantic segmentation using a deep learning model to generate two phases of ground feature patch images. The difference probability images are used to represent the confidence level corresponding to the sub-regions that have changed within the small-scale monitoring area, and the ground feature patch images are used to represent the distribution of different ground features within the small-scale monitoring area.

[0009] Based on the difference probability image and the two-phase ground feature patch images, the change patch results corresponding to the small-scale monitoring area are generated; among them, the change patch results are used to characterize the changes of different ground features in the small-scale monitoring area.

[0010] In one embodiment, performing multi-scale transformation on two phases of remote sensing images to generate a difference probability image includes:

[0011] Perform band-by-band difference processing on the two phases of remote sensing images to obtain the difference image;

[0012] Constructing a multi-scale transformation model based on the difference image to perform multi-scale transformation processing on the difference image through the multi-scale transformation model;

[0013] The difference image after multi-scale transformation is stretched to generate a difference probability image.

[0014] In one embodiment, semantic segmentation processing is performed on two phases of remote sensing images using a deep learning model to generate two phases of ground feature feature patch images, including:

[0015] Perform semantic information annotation on any one or two phases of remote sensing images to construct semantic samples corresponding to different ground features;

[0016] The deep learning model is trained using the semantic samples corresponding to each ground feature to obtain a semantic segmentation model corresponding to each ground feature;

[0017] The semantic segmentation model corresponding to each land feature is used to perform semantic segmentation processing on the two phases of remote sensing images to generate two phases of land feature patch images corresponding to each land feature.

[0018] In one embodiment, the method further comprises:

[0019] During the training of the deep learning model, target points are randomly sampled from the image output by the deep learning model, and the prediction results of the ground features to which the target points belong are determined. The loss value is calculated based on the semantic information of the target point annotations, and the model parameters of the deep learning model are adjusted using the loss value.

[0020] In one embodiment, based on the difference probability image and the two-phase ground feature image, a change image result corresponding to the small-scale monitoring area is generated, including:

[0021] Based on the confidence level of the difference probability image representation and the preset confidence threshold, candidate sub-regions are extracted from the small-scale monitoring area;

[0022] Using spatial indexing algorithm, the distribution of different land features in the candidate sub-region is determined based on the two-phase land feature patch images.

[0023] Based on the distribution of different ground features in the candidate sub-areas, determine the target sub-areas that have changed within the small-scale monitoring area and their corresponding change categories;

[0024] Based on the target sub-region and its corresponding change category, the change patch results corresponding to the small-scale monitoring area are generated.

[0025] In one embodiment, a spatial index algorithm is used to determine the distribution of different land features within a candidate sub-region based on two phases of land feature imagery, including:

[0026] For each ground feature, a spatial index algorithm is used to determine the first number of pixels corresponding to the ground feature in the candidate sub-area based on a ground feature image of one period, and to determine the second number of pixels corresponding to the ground feature in each candidate sub-area based on a ground feature image of another period;

[0027] The distribution of the land feature in the candidate sub-region is described using a first pixel ratio between the first number of pixels and the total number of pixels in the candidate sub-region, and a second pixel ratio between the second number of pixels and the total number of pixels in the candidate sub-region.

[0028] In one embodiment, based on the distribution of different ground features in the candidate sub-areas, determining the target sub-areas that have changed in the small-scale monitoring area and their corresponding change categories includes:

[0029] Determine whether the first pixel ratio and the second pixel ratio are less than a first ratio threshold;

[0030] If so, it is determined that the candidate subregion has not changed;

[0031] If not, if the first pixel ratio is greater than or equal to the first ratio threshold, and the second pixel ratio is less than the second ratio threshold, then it is determined that the candidate sub-region belongs to the changed target sub-region and the change category is a reduction in ground objects; if the first pixel ratio is less than the second ratio threshold, and the second pixel ratio is greater than or equal to the first ratio threshold, then it is determined that the candidate sub-region belongs to the changed target sub-region and the change category is an increase in ground objects.

[0032] In a second aspect, an embodiment of the present invention further provides a small-scale deep learning change detection device, comprising:

[0033] Image acquisition module, used to obtain two phases of remote sensing images corresponding to the small-scale monitoring area;

[0034] The image processing module is used to perform multi-scale transformation processing on the two remote sensing images to generate difference probability images, and to perform semantic segmentation processing on the two remote sensing images through a deep learning model to generate two ground feature patch images. Among them, the difference probability image is used to represent the confidence level corresponding to the sub-regions that have changed within the small-scale monitoring area, and the ground feature patch image is used to represent the distribution of different ground features within the small-scale monitoring area.

[0035] The change detection module is used to generate change patch results corresponding to the small-scale monitoring area based on the difference probability image and the two-phase ground feature patch images; among them, the change patch results are used to characterize the changes of different ground features in the small-scale monitoring area.

[0036] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0037] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0038] The embodiments of the present invention provide a small-scale deep learning change detection method, device, equipment and medium. First, two remote sensing images corresponding to the small-scale monitoring area are obtained; then, the two remote sensing images are multi-scale transformed to generate difference probability images, and at the same time, the two remote sensing images are semantically segmented through a deep learning model to generate two ground feature patch images. The difference probability images are used to characterize the confidence corresponding to the sub-areas that have changed in the small-scale monitoring area, and the ground feature patch images are used to characterize the distribution of different ground features in the small-scale monitoring area; finally, based on the difference probability images and the two-phase ground feature patch images, the change patch results corresponding to the small-scale monitoring area are generated, and the change patch results are used to characterize the changes of different ground features in the small-scale monitoring area. The embodiment of the present invention constructs a deep learning change detection algorithm suitable for small-scale monitoring areas, combines multi-scale transformation and semantic segmentation algorithms, uses multi-scale transformation processing to extract high-confidence change areas, and then uses the semantic segmentation algorithm to determine the change areas and change categories. The embodiment of the present invention can be applied to multiple categories of land features such as woodlands, water bodies, and buildings, and does not require change detection samples. It has the advantages of strong applicability and fast computing speed, and provides support for the improvement of ecological environment supervision and governance capabilities in small-scale monitoring areas.

[0039] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A schematic diagram of a process flow of a small-scale deep learning change detection method provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a remote sensing image preprocessing process provided by an embodiment of the present invention;

[0044] Figure 3A schematic diagram of a multi-scale transformation process provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a semantic segmentation process provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the Mask2Former structure provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of a change detection process provided by an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of the overall process of a small-scale deep learning change detection method provided by an embodiment of the present invention;

[0049] Figure 8 A schematic diagram of a two-phase remote sensing image provided by an embodiment of the present invention;

[0050] Figure 9 A schematic diagram of a difference probability image provided by an embodiment of the present invention;

[0051] Figure 10 A schematic diagram of building distribution in two phases provided by an embodiment of the present invention;

[0052] Figure 11 A schematic diagram of the structure of a small-scale deep learning change detection device provided by an embodiment of the present invention;

[0053] Figure 12 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] At present, for change detection in small-scale areas, the application of deep learning methods in change detection of small-scale areas is seriously restricted because they cannot obtain sufficient change samples in the monitoring area. There is an urgent need for a deep learning change detection algorithm for small-scale areas. Based on this, the present invention implements a small-scale deep learning change detection method, device, equipment and medium, which realizes change detection in small-scale monitoring areas and provides support for the improvement of ecological environment supervision and governance capabilities in small-scale monitoring areas.

[0056] To facilitate understanding of this embodiment, a small-scale deep learning change detection method disclosed in an embodiment of the present invention is first introduced in detail. Figure 1 The flowchart of a small-scale deep learning change detection method shown in FIG. 1 mainly includes the following steps S102 to S106:

[0057] Step S102: Acquire two phases of remote sensing images corresponding to the small-scale monitoring area.

[0058] Among them, the small-scale monitoring area is also the monitoring area at the city and county levels, and the two remote sensing images can be two adjacent remote sensing images, which are called the front remote sensing image and the back remote sensing image.

[0059] Step S104 , performing multi-scale transformation processing on the two phases of remote sensing images to generate difference probability images, and performing semantic segmentation processing on the two phases of remote sensing images through a deep learning model to generate two phases of ground feature patch images.

[0060] The difference probability image is used to characterize the confidence level of sub-regions that have changed within the small-scale monitoring area. A higher confidence level indicates a greater likelihood of change in the sub-region. The ground feature patch image is used to characterize the distribution of different ground features within the small-scale monitoring area. In one example, two remote sensing imagery phases can be subjected to band-by-band interpolation, multi-scale transformation, and image stretching to generate a difference probability image. In another example, semantic samples corresponding to different ground features can be constructed using either or both phases of remote sensing imagery. A deep learning model can then be trained using these semantic samples to generate semantic segmentation models corresponding to different ground features. The semantic segmentation model can then be used to output ground feature patch images corresponding to each of the two phases of remote sensing imagery.

[0061] Step S106 , generating a change patch result corresponding to the small-scale monitoring area based on the difference probability image and the two-phase ground feature patch images.

[0062] The change patch results are used to characterize the changes in different ground features within the small-scale monitoring area. In one example, the confidence level represented by the difference probability image can be used to screen candidate sub-regions with a high probability of change. Combined with the distribution of different ground features represented by the ground feature patch images from the two phases of remote sensing imagery, the target sub-regions with changes can be further screened, along with the change categories within each sub-region, such as ground feature reduction or increase, to ultimately generate the change patch results.

[0063] The embodiment of the present invention constructs a deep learning change detection algorithm suitable for small-scale monitoring areas, combines multi-scale transformation and semantic segmentation algorithms, uses multi-scale transformation processing to extract high-confidence change areas, and then uses the semantic segmentation algorithm to determine the change areas and change categories. The embodiment of the present invention can be applied to multiple categories of land features such as woodlands, water bodies, and buildings, and does not require change detection samples. It has the advantages of strong applicability and fast computing speed, and provides support for the improvement of ecological environment supervision and governance capabilities in small-scale monitoring areas.

[0064] For ease of understanding, an embodiment of the present invention provides a specific implementation of a small-scale deep learning change detection method.

[0065] Regarding the aforementioned step S102, after obtaining two phases of remote sensing images corresponding to the small-scale monitoring area, it is necessary to perform geometric correction and cropping on the two phases of images, such as Figure 2 The figure shows a schematic diagram of a remote sensing image preprocessing process. Specifically, two images of a small-scale monitoring area are obtained and geometrically corrected to minimize the geometric offset of the images for change detection. The geometrically corrected images are then cropped using the small-scale monitoring vector to obtain two preprocessed images of the small-scale monitoring area, denoted as image A and image B. Image A and image B have the same number of bands n and scale (xsize, ysize). The small-scale monitoring vector is used to represent the scope of the small-scale monitoring area. It can be the administrative boundary of a city, county, or township, or a customized small-scale boundary of interest to the customer.

[0066] Regarding the aforementioned step S104, the embodiment of the present invention provides an implementation method for performing multi-scale transformation on two remote sensing images to generate a difference probability image. In practical applications, since remote sensing images have multi-spectral characteristics and the spectral characteristics of each ground feature are inconsistent, the redundancy of remote sensing image information and the inability to measure the spectra of different ground features with a unified standard are caused. Therefore, it is necessary to perform multi-scale transformation on the data of image A and image B to reduce data redundancy and obtain the difference area as much as possible. Figure 3 The figure shows a multi-scale transformation process flow diagram. First, the difference between A and B is processed band by band. Then, the difference image is subjected to multi-scale transformation for spectral compression. Finally, the difference probability image is constructed by image stretching, which is recorded as image C.

[0067] Please refer to steps 1A to 1C below for details:

[0068] Step 1A: Perform band-by-band difference processing on the two remote sensing images to obtain the difference image. The expression for band-by-band difference processing is as follows:

[0069] C1=[B1-A1,B i -Ai ,…,B n -A n ];

[0070] Among them, A i 、B i is the i-th band image between image A and image B, C1 is the difference image, and the size of C1 is (n, xsize, ysize).

[0071] Step 1B: construct a multi-scale transformation model based on the difference image to perform multi-scale transformation processing on the difference image through the multi-scale transformation model. The expression of the multi-scale transformation processing is as follows:

[0072] C2 = MSD (C1);

[0073] Among them, MDS (Multi-dimensional scaling) is a multi-scale transformation, C2 is a difference image processed by the multi-scale transformation, and C2 is (xsize, ysize).

[0074] The MSD algorithm process is as follows:

[0075] Transform the array size of C1 to generate C ′ 1, the size is (n, m), m = xsize * ysize. In order to obtain the MSD model, it speeds up the operation. ′ In 1, randomly select pixels with a percentage of p to generate a sample S with a size of (n, m1), m1 = INT(p*m).

[0076] First calculate the pairwise distance dist of S:

[0077]

[0078] dist=add(add(-2*(sum_S·sum_S).T),sum_S);

[0079] Where add is the matrix element accumulation, sum_S·sum_S is the dot product operation between sum_S, and sum_S is the cumulative sum of squares in the band direction. The size of the dist array is (m1,m1).

[0080] Calculate the inner product matrix Be of dist, and let one be the initial matrix with the value of (m1,m1) equal to 1:

[0081] Be=-(T1-T2-T3+dist) / 2;

[0082] T1 = one*sum(dist) / (m1*m1);

[0083] T2 = sum(dist0) / (m1);

[0084] T3 = sum(dist1) / (m1);

[0085] Where sum(dist0) and sum(dist1) are the accumulated values ​​of row and column vectors.

[0086] Perform eigendecomposition on Be, obtain the eigensolution and eigenvector, sort the eigenfeatures, select the index value where the maximum value is located, and use the eigensolution, eigenvector and the maximum eigenindex as the update parameters of the MSD model. The MSD model is recorded as model.

[0087] Using model to C ′ 1 performs dimensionality reduction processing to generate a 1-dimensional vector C3. The size (m, 1) is converted to the array size of C3 to generate C, the size is (xsize, ysize), and the threshold range is 0-100.

[0088] Step 1C: Perform image stretching on the difference image after multi-scale transformation to generate a difference probability image. The expression for image stretching is as follows:

[0089]

[0090] Among them, max(C2) and min(C2) are the maximum and minimum values ​​of C2, and C is the difference probability image.

[0091] Regarding the aforementioned step S104, the embodiment of the present invention also provides an implementation method for generating two-phase ground feature feature patch images by performing semantic segmentation processing on two-phase remote sensing images using a deep learning model. In practical applications, although small-scale regions have fewer changing areas, they contain a large amount of semantic information. For example, forest land, cultivated land, and buildings have fewer changing areas but are distributed over a large area. This basic fact is used to perform semantic segmentation of ground features to optimize the boundaries of changing areas and assign attribute information. Figure 4 The figure shows a flow chart of semantic segmentation processing. Semantic information is annotated based on the distribution information of ground features, massive semantic samples are generated, and the Mask2Former (Masked-attention MaskTransformer) deep learning network framework is used for training to obtain the mask2former model (also known as the semantic segmentation model). Then, the A and B images are predicted to output two ground feature patch images D and E.

[0092] For details, please refer to steps 2A to 2C below:

[0093] Step 2A: perform semantic information annotation on any one or two phases of remote sensing images to construct semantic samples corresponding to different land features; or perform semantic information annotation on existing samples outside the monitoring area to construct semantic samples corresponding to different land features.

[0094] Step 2B: Use the semantic samples corresponding to each land feature to train the deep learning model to obtain a semantic segmentation model corresponding to each land feature.

[0095] Among them, Mask2Former is built on a simple meta-framework (MaskFormer) and a new Transformer decoder. Its key component is masked-attention, which extracts local features by limiting cross attention to the predicted mask area. The framework consists of three parts, see Figure 5 The figure shows a schematic diagram of the Mask2Former structure, which includes a backbone feature extractor, a pixel decoder, and a Transformer decoder. The backbone feature extractor extracts low-resolution features from the image; the pixel decoder gradually upsamples the low-resolution features from the output of the backbone to generate a high-resolution per-pixel embedding; the Transformer decoder operates on the image features to process the object query and finally obtains the predicted image (i.e., the ground feature patch image).

[0096] Since Mask2Former is designed for visual images, and remote sensing has unique multi-scale and multi-spectral features and requires large amounts of stored data, it needs to be modified to be suitable for remote sensing images. The modifications are as follows: (1) Modify the number and order of bands in the pre-trained weights to be suitable for remote sensing multi-spectral features; (2) Optimize Transformer; use random sampling methods to improve computational efficiency and save memory to be suitable for remote sensing large storage features.

[0097] By optimizing the Transformer, the embodiment of the present invention can randomly sample target points from the image output by the deep learning model during the training of the deep learning model, determine the prediction results of the ground features to which the target points belong, calculate the loss value in combination with the semantic information annotated by the target points, and use the loss value to adjust the model parameters of the deep learning model. During the training process, in order to solve the problem that remote sensing images require a large amount of computation compared to natural images, the sampling method for calculating the loss of Mask2Former during training is modified. The original method of calculating the loss for all pixels of an image of size img_size*img_size is changed to randomly selecting points for calculation according to a certain ratio (50%). This can reduce the amount of computation to 1 / 4 of the original amount while ensuring the accuracy of the model.

[0098] Step 2C: Perform semantic segmentation processing on the two phases of remote sensing images using the semantic segmentation model corresponding to each ground feature to generate two phases of ground feature patch images corresponding to each ground feature.

[0099] Based on steps 2A to 2C above, the specific process of generating patch images of land features is as follows: first, use the previous and next images to create samples of a certain land feature, and feed the samples into the modified Mask2Former deep learning framework. Then, set the main hyperparameters of Mask2Former, namely the number of semantic segmentation categories num, slice size img_szie, number of iterations epochs, loss function loss, optimizer optimizer, and learning rate lr. Train Mask2Former to obtain a deep learning model. Finally, slice images A and B, perform predictions, and finally splice them to generate patch images of a single land feature, which are recorded as images D and E.

[0100] Regarding step S106, the present invention also provides an implementation method for generating a change patch result corresponding to a small-scale monitoring area based on the difference probability image and the two-phase ground feature feature patch image. In practical applications, because the semantically segmented land class boundaries are relatively accurate and the land class attributes are clear, they can be used as a basis for optimizing the boundaries of the change patch and determining the change category. This determines the ground feature change boundaries and the types of reduction and increase, and generates a change patch result, which is recorded as the F image.

[0101] For details, please refer to steps 3A to 3D below:

[0102] Step 3A: Based on the confidence of the difference probability image representation and the preset confidence threshold, candidate sub-regions are extracted from the small-scale monitoring area. Figure 6 The diagram shows a change detection process. First, the difference probability image C is divided according to the empirical threshold. The candidate sub-regions with probabilities greater than the preset confidence threshold v1 are high-confidence candidate sub-regions, that is, suspected change spots. The suspected change spots are marked and numbered. If the number of suspected change spots is s, the numbering starts from 1 and ends at s, with an interval of 1.

[0103] Step 3B: Using a spatial index algorithm, the distribution of different land features in the candidate sub-region is determined based on the two phases of land feature patch images.

[0104] In one example, for each land feature, a spatial index algorithm is used to determine the first number of pixels corresponding to the land feature in the candidate sub-region based on a land feature spot image of one period, and the second number of pixels corresponding to the land feature in each candidate sub-region is determined based on another land feature spot image; then, the first pixel ratio between the first pixel number and the total number of pixels contained in the candidate sub-region, and the second pixel ratio between the second pixel number and the total number of pixels contained in the candidate sub-region, are used to describe the distribution of the land feature in the candidate sub-region.

[0105] Please continue to see Figure 6 Step 3B is the process of overlaying and analyzing the suspected change spots with the two phases of ground feature spot images. In the specific implementation, the suspected change spots are numbered k, and the number of pixels is nk. Based on the spatial indexing technology, the distribution of ground features in the D and E images within the k region can be obtained. The number of pixels identified as a ground feature in the k region is calculated, where the first pixel count in the D image is nd, and the second pixel count in the E image is ne. The proportion is calculated as follows:

[0106] p1 = nd / nk;

[0107] p2=ne / nk;

[0108] Among them, p1 is the first pixel ratio, and p2 is the second pixel ratio.

[0109] Step 3C: Based on the distribution of different ground features in the candidate sub-areas, determine the target sub-areas with changes in the small-scale monitoring area and their corresponding change categories;

[0110] In one example, it is determined whether the first pixel ratio and the second pixel ratio are less than the first ratio threshold; if so, it is determined that the candidate sub-region has not changed; if not, if the first pixel ratio is greater than or equal to the first ratio threshold, and the second pixel ratio is less than the second ratio threshold, then it is determined that the candidate sub-region belongs to the target sub-region that has changed and the change category is a reduction in ground objects; if the first pixel ratio is less than the second ratio threshold, and the second pixel ratio is greater than or equal to the first ratio threshold, then it is determined that the candidate sub-region belongs to the target sub-region that has changed and the change category is an increase in ground objects.

[0111] Please continue to see Figure 6 , using the decision tree to hierarchically determine the change area and change category, the specific process is as follows:

[0112] (1) When p1 or p2 is greater than or equal to the first ratio threshold v2, this condition is recorded as condition 1. If condition 1 is met, the change occurs; if condition 1 is not met, the change does not occur.

[0113] (2) When the further category judgment of condition 1 is met, when condition 1 is met and p1 is greater than or equal to the first ratio threshold v2 and p2 is less than the second ratio threshold v3, this condition is recorded as condition 2. If this condition is met, the ground object is reduced;

[0114] (3) When condition 1 is satisfied and p2 is greater than or equal to the first ratio threshold v2 and p1 is less than the second ratio threshold v3, this condition is recorded as condition 3, and the satisfaction of this condition is ground feature increase;

[0115] (4) When condition 1 is met but conditions 2 and 3 are not met, it should be a changing noise and is recorded as unchanged.

[0116] In step 3D, based on the target sub-region and its corresponding change category, a change map result corresponding to the small-scale monitoring area is generated. In one example, all suspected change spots are traversed according to step 3C, and a change map image is generated after screening and judgment. The raster-to-vector conversion process is then performed to generate the change map result.

[0117] For ease of understanding, the present invention provides an application example of a small-scale deep learning change detection method. Figure 7 The figure shows an overall process diagram of a small-scale deep learning change detection method, which includes preprocessing and multi-scale transformation of two-phase images to obtain difference probability images, semantic sample annotation of the two-phase images to obtain semantic samples, using the mask2former network framework suitable for the purpose to output ground feature patch images, combining the difference probability images and ground feature patch images to determine change information and change categories, and finally outputting the change patch results.

[0118] Taking Qibin District of Hebi City as an example of a small-scale monitoring area, the deep learning change detection method for small scales mainly includes the following steps:

[0119] (1) Remote sensing image processing:

[0120] The embodiment of the present invention selects Qibin District of Hebi City, a small-scale area, to monitor common features in northern cities and counties, including water bodies, cultivated land, buildings, and woodlands, which are typical and representative. GF1 images of this area were obtained in July 2022 and March 2023. After geometric correction and vector cropping, the images were generated into images A and B, such as Figure 8 A schematic diagram of a two-phase remote sensing image is shown, where the number of bands of image A and image B is 4, and the size is (20164, 14066).

[0121] (2) Construction of difference probability image:

[0122] Perform difference processing on image A and image B, perform multi-scale transformation on the difference image to perform spectral compression processing, and then construct the difference probability image C by image stretching, such as Figure 9 The figure shows a schematic diagram of a difference probability image. The number of samples selected by MSD spectrum compression is 5%, and the number of sample data is 40993412.

[0123] (3) Semantic Deep Learning Segmentation:

[0124] Select buildings for semantic segmentation, mark 569 patches in the building area, and slice them into 256*256 size, generating 2160 sample sets. Based on the slice set, the Mask2Former network is trained to extract the building distribution (that is, the aforementioned ground feature patch image), such as Figure 10 The figure shows a schematic diagram of building distribution in two phases, including images D and E. The Mask2Former semantic segmentation model uses 2 categories, a slice size of 256, 1000 iterations, and a learning rate of 0.01.

[0125] (4) Change monitoring area extraction:

[0126] The difference probability image C was divided according to empirical thresholds. Areas with a probability greater than 50 were identified as high-confidence change regions. These change regions were labeled and numbered. The total number of change patches was 40,268, and the numbering started at 1 and ended at 40,268, with an interval of 1. The change patches were superimposed with the building distribution images D and E. The v2 and v3 thresholds for the decision tree layers were 0.75 and 0.15, respectively.

[0127] In summary, the embodiments of the present invention construct a deep learning change detection algorithm suitable for small-scale areas, combining multi-scale transformation with the mask2former semantic segmentation algorithm, using the multi-scale transformation algorithm to extract high-confidence change areas, and then using the mask2former semantic information to determine the change areas and change categories. This method can be applied to multiple categories of land features such as woodlands, water bodies, and buildings, and does not require change detection samples. It has the advantages of strong applicability and fast computing speed, providing support for the improvement of ecological environmental supervision and governance capabilities in small-scale areas.

[0128] Based on the above embodiments, the present invention provides a small-scale deep learning change detection device. Figure 11 The schematic diagram of the structure of a small-scale deep learning change detection device is shown in FIG. The device mainly includes the following parts:

[0129] Image acquisition module 1102, used to acquire two phases of remote sensing images corresponding to the small-scale monitoring area;

[0130] Image processing module 1104 is used to perform multi-scale transformation processing on the two remote sensing images to generate difference probability images, and to perform semantic segmentation processing on the two remote sensing images using a deep learning model to generate two ground feature patch images; wherein the difference probability image is used to represent the confidence level corresponding to the sub-regions that have changed within the small-scale monitoring area, and the ground feature patch image is used to represent the distribution of different ground features within the small-scale monitoring area;

[0131] The change detection module 1106 is used to generate a change patch result corresponding to the small-scale monitoring area based on the difference probability image and the two-phase ground feature patch image; wherein the change patch result is used to characterize the change of different ground features in the small-scale monitoring area.

[0132] The embodiment of the present invention constructs a deep learning change detection device suitable for small-scale monitoring areas, combines multi-scale transformation and semantic segmentation algorithms, uses multi-scale transformation processing to extract high-confidence change areas, and then uses the semantic segmentation algorithm to determine the change areas and change categories. The embodiment of the present invention can be applied to multiple categories of land features such as woodlands, water bodies, and buildings, and does not require change detection samples. It has the advantages of strong applicability and fast computing speed, and provides support for the improvement of ecological environment supervision and governance capabilities in small-scale monitoring areas.

[0133] In one embodiment, the image processing module 1104 is specifically configured to:

[0134] Perform band-by-band difference processing on the two phases of remote sensing images to obtain the difference image;

[0135] Constructing a multi-scale transformation model based on the difference image to perform multi-scale transformation processing on the difference image through the multi-scale transformation model;

[0136] The difference image after multi-scale transformation is stretched to generate a difference probability image.

[0137] In one embodiment, the image processing module 1104 is specifically configured to:

[0138] Perform semantic information annotation on any one or two phases of remote sensing images to construct semantic samples corresponding to different ground features;

[0139] The deep learning model is trained using the semantic samples corresponding to each ground feature to obtain a semantic segmentation model corresponding to each ground feature;

[0140] The semantic segmentation model corresponding to each land feature is used to perform semantic segmentation processing on the two phases of remote sensing images to generate two phases of land feature patch images corresponding to each land feature.

[0141] In one embodiment, the image processing module 1104 is specifically configured to:

[0142] During the training of the deep learning model, target points are randomly sampled from the image output by the deep learning model, and the prediction results of the ground features to which the target points belong are determined. The loss value is calculated based on the semantic information of the target point annotations, and the model parameters of the deep learning model are adjusted using the loss value.

[0143] In one embodiment, the change detection module 1106 is specifically configured to:

[0144] Based on the confidence level of the difference probability image representation and the preset confidence threshold, candidate sub-regions are extracted from the small-scale monitoring area;

[0145] Using spatial indexing algorithm, the distribution of different land features in the candidate sub-region is determined based on the two-phase land feature patch images.

[0146] Based on the distribution of different ground features in the candidate sub-areas, determine the target sub-areas that have changed within the small-scale monitoring area and their corresponding change categories;

[0147] Based on the target sub-region and its corresponding change category, the change patch results corresponding to the small-scale monitoring area are generated.

[0148] In one embodiment, the change detection module 1106 is specifically configured to:

[0149] For each ground feature, a spatial index algorithm is used to determine the first number of pixels corresponding to the ground feature in the candidate sub-area based on a ground feature image of one period, and to determine the second number of pixels corresponding to the ground feature in each candidate sub-area based on a ground feature image of another period;

[0150] The distribution of the land feature in the candidate sub-region is described using a first pixel ratio between the first number of pixels and the total number of pixels in the candidate sub-region, and a second pixel ratio between the second number of pixels and the total number of pixels in the candidate sub-region.

[0151] In one embodiment, the change detection module 1106 is specifically configured to:

[0152] Determine whether the first pixel ratio and the second pixel ratio are less than a first ratio threshold;

[0153] If so, it is determined that the candidate subregion has not changed;

[0154] If not, if the first pixel ratio is greater than or equal to the first ratio threshold, and the second pixel ratio is less than the second ratio threshold, then it is determined that the candidate sub-region belongs to the changed target sub-region and the change category is a reduction in ground objects; if the first pixel ratio is less than the second ratio threshold, and the second pixel ratio is greater than or equal to the first ratio threshold, then it is determined that the candidate sub-region belongs to the changed target sub-region and the change category is an increase in ground objects.

[0155] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0156] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0157] Figure 12 A structural diagram of an electronic device provided in an embodiment of the present invention, wherein the electronic device 100 includes: a processor 120, a memory 121, a bus 122 and a communication interface 123, wherein the processor 120, the communication interface 123 and the memory 121 are connected via the bus 122; the processor 120 is used to execute an executable module stored in the memory 121, such as a computer program.

[0158] The memory 121 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 123 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0159] The bus 122 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 12 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0160] Among them, the memory 121 is used to store programs, and the processor 120 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 120 or implemented by the processor 120.

[0161] The processor 120 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 120 or by software instructions. The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 121, and processor 120 reads the information in memory 121 and, in conjunction with its hardware, completes the steps of the above method.

[0162] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0163] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0164] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A small-scale deep learning change detection method, characterized in that: include: Obtain two phases of remote sensing images corresponding to the small-scale monitoring area; Performing multi-scale transformation processing on the two phases of remote sensing images to generate difference probability images, and performing semantic segmentation processing on the two phases of remote sensing images through a deep learning model to generate two phases of ground feature image spots; wherein the difference probability image is used to characterize the confidence corresponding to the sub-regions that have changed in the small-scale monitoring area, and the ground feature image spot is used to characterize the distribution of different ground features in the small-scale monitoring area; Based on the difference probability image and the two phases of the ground object element spot images, a change spot result corresponding to the small-scale monitoring area is generated; wherein the change spot result is used to characterize the change of different ground object elements in the small-scale monitoring area; Based on the difference probability image and the two-phase ground object element spot images, a change spot result corresponding to the small-scale monitoring area is generated, including: based on the confidence represented by the difference probability image and a preset confidence threshold, a candidate sub-area is extracted from the small-scale monitoring area; using a spatial index algorithm, based on the two-phase ground object element spot images, the distribution of different ground objects in the candidate sub-area is determined; based on the distribution of different ground objects in the candidate sub-area, the target sub-area that has changed in the small-scale monitoring area and its corresponding change category are determined; based on the target sub-area and its corresponding change category, a change spot result corresponding to the small-scale monitoring area is generated; Determine the distribution of different land features in the candidate sub-region based on the land feature feature spot images of two phases using a spatial index algorithm, including: for each land feature element, determine the first number of pixels corresponding to the land feature element in the candidate sub-region based on the land feature feature spot images of one phase, and determine the second number of pixels corresponding to the land feature element in each candidate sub-region based on the land feature feature spot images of another phase using a spatial index algorithm; describe the distribution of the land feature element in the candidate sub-region using a first pixel ratio between the first pixel ratio and the total number of pixels in the candidate sub-region, and a second pixel ratio between the second pixel ratio and the total number of pixels in the candidate sub-region; Based on the distribution of different ground objects in the candidate sub-area, the target sub-area that has changed in the small-scale monitoring area and its corresponding change category are determined, including: judging whether the first pixel ratio and the second pixel ratio are less than a first ratio threshold; if yes, determining that the candidate sub-area has not changed; if not, if the first pixel ratio is greater than or equal to the first ratio threshold, and the second pixel ratio is less than the second ratio threshold, then determining that the candidate sub-area belongs to the target sub-area that has changed and the change category is a reduction in ground objects; if the first pixel ratio is less than the second ratio threshold, and the second pixel ratio is greater than or equal to the first ratio threshold, then determining that the candidate sub-area belongs to the target sub-area that has changed and the change category is an increase in ground objects.

2. The small-scale deep learning change detection method according to claim 1, characterized in that: The remote sensing images of the two phases are subjected to multi-scale transformation processing to generate difference probability images, including: Performing band-by-band difference processing on the remote sensing images of the two phases to obtain a difference image; Constructing a multi-scale transformation model based on the difference image, so as to perform multi-scale transformation processing on the difference image through the multi-scale transformation model; The difference image after the multi-scale transformation is subjected to image stretching processing to generate a difference probability image.

3. The small-scale deep learning change detection method according to claim 1, characterized in that: The two phases of remote sensing images are processed by semantic segmentation through a deep learning model to generate two phases of ground feature image spots, including: Annotating the remote sensing images of any one or both periods with semantic information to construct semantic samples corresponding to different ground features; The deep learning model is trained using the semantic samples corresponding to each ground feature to obtain a semantic segmentation model corresponding to each ground feature; The semantic segmentation model corresponding to each of the land feature elements is used to perform semantic segmentation processing on the two phases of remote sensing images to generate two phases of land feature element spot images corresponding to each of the land feature elements.

4. The small-scale deep learning change detection method according to claim 3, characterized in that: The method further comprises: During the training of the deep learning model, target points are randomly sampled from the image output by the deep learning model, and the prediction results of the ground features to which the target points belong are determined. The loss values ​​are calculated in combination with the semantic information annotated with the target points, and the model parameters of the deep learning model are adjusted using the loss values.

5. A small-scale deep learning change detection device, characterized in that: The device is used to implement the small-scale deep learning change detection method according to claim 1, and the device includes: Image acquisition module, used to obtain two phases of remote sensing images corresponding to the small-scale monitoring area; An image processing module is used to perform multi-scale transformation processing on the two phases of remote sensing images to generate difference probability images, and to perform semantic segmentation processing on the two phases of remote sensing images through a deep learning model to generate two phases of ground feature image spots; wherein the difference probability image is used to characterize the confidence corresponding to the sub-region that has changed in the small-scale monitoring area, and the ground feature image is used to characterize the distribution of different ground features in the small-scale monitoring area; The change detection module is used to generate a change spot result corresponding to the small-scale monitoring area based on the difference probability image and the two-phase ground feature spot images; wherein the change spot result is used to characterize the change of different ground features in the small-scale monitoring area.

6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 4.

Images