Multi-temporal remote sensing image change detection method and system based on interactive prompts

Abstract

The application discloses a multi-temporal remote sensing image change detection method and system based on interactive prompts, belongs to the field of remote sensing images and AI cross combination, obtains paired multi-temporal remote sensing images and corresponding binary change label maps, forms a sample set containing a front-time-phase image, a later-time-phase image and a label map; three interactive modes of clicking, frame selecting and line drawing are adopted to simulate generation of positive sample prompt maps and negative sample prompt maps; in the first iteration, the simulated positive sample prompt maps, the negative sample prompt maps and the sample set are input into an interactive prompt remote sensing change detection model; in subsequent iterations, the currently generated positive and negative sample prompt maps and the previous segmentation result maps are jointly input into the interactive prompt remote sensing change detection model; the iteration step is repeated at least 3 times to optimize the parameters of the interactive prompt remote sensing change detection model; and the multi-temporal remote sensing images to be detected are input into the trained interactive prompt remote sensing change detection model, and a change detection result map is output in combination with real-time interactive prompts of a user.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image and AI cross combination, and particularly relates to a multi-temporal remote sensing image change detection method and system based on interactive prompts. BACKGROUND

[0002] Remote sensing image is a digital image data that records the characteristics and spatial information of the earth's surface (or other targets) generated by receiving electromagnetic wave information (visible light, infrared, microwave, etc.) without physical contact with the target through sensors (remote sensing technology) on platforms such as satellites and aircraft. Remote sensing image change detection can provide key technical support for monitoring land cover change, climate change and evaluating the impact of human activities. In recent years, intelligent change detection methods based on deep learning have significantly improved the automation level, but existing methods generally rely on a large number of labeled samples for model training. Sample collection requires manual comparison of multiple images and pixel-by-pixel labeling, which consumes a lot of time and labor cost, seriously restricting the application of this technology in high-time-efficiency scenarios.

[0003] To reduce sample dependence, existing technologies propose interactive remote sensing image change detection models based on clicking and box selection, allowing users to generate detection results through a small amount of interactive prompts. Although the above two methods can reduce the amount of manual interaction to a certain extent and improve work efficiency. However, this kind of method has obvious limitations: (1) The interaction mode is single, only one of clicking or box selection is supported, and only two-phase remote sensing image change detection can be achieved, and the best interaction strategy cannot be flexibly selected according to the target form; (2) Lack of iterative optimization mechanism, the result is directly output after single interaction, and the detection accuracy of complex change areas is insufficient; (3) No use of historical segmentation information, no cooperative optimization relationship between interactive prompts and intermediate results, leading to error accumulation. In addition, for irregular change targets such as building demolition and farmland occupation, the line drawing interaction mode proposed by the existing technology can provide more accurate guidance.

[0004] However, the existing technology has not realized the unified integration of the three interaction modes of clicking, box selection and line drawing to complete the multi-temporal remote sensing change detection task; at the same time, it lacks a gradual optimization mechanism that integrates the last segmentation result and the current interactive prompt, making it difficult to realize the closed-loop detection process of "interaction-feedback-optimization", which is a technical bottleneck that needs to be broken through in the current field. SUMMARY

[0005] The present application is aimed at the above problems, and provides a multi-temporal remote sensing image change detection method and system based on interactive prompts to make up for the shortcomings of the existing technology.

[0006] To achieve the above purpose, the present application adopts the following technical solutions.

[0007] In a first aspect, the present application provides a multi-temporal remote sensing image change detection method based on interactive prompts, comprising the following steps:

[0008] S1. Constructing a training sample set: obtaining a pair of multi-temporal remote sensing images and corresponding binary change label maps to form a sample set containing a pre-phase image, a post-phase image, and a label map;

[0009] S2. Computer simulation of interactive prompts: using three interactive methods of clicking, box selection, and line drawing to simulate the generation of positive sample prompt maps and negative sample prompt maps;

[0010] S3. Progressive iterative training of interactive prompt remote sensing change detection model: in the first iteration, the positive sample prompt maps and negative sample prompt maps simulated in step S2 and the sample set containing the pre-phase image, the post-phase image, and the label map in step S1 are input into the interactive prompt remote sensing change detection model; in subsequent iterations, the currently generated positive and negative sample prompt maps and the previous segmentation result map are jointly input into the interactive prompt remote sensing change detection model; the iteration step is repeated at least 3 times to optimize the parameters of the interactive prompt remote sensing change detection model;

[0011] S4. Performing change detection: inputting the multi-temporal remote sensing images to be detected into the trained interactive prompt remote sensing change detection model, and outputting a change detection result map in combination with real-time interactive prompts from the user.

[0012] Further, in step S2, the simulation method of clicking interaction is to generate a circular prompt area with a specified coordinate point as the center; the simulation method of box selection interaction is to fill the inside of a rectangular frame with a foreground value to generate a prompt map; the simulation method of line drawing interaction includes: extracting and smoothing the boundary line through an edge detection operator, extracting the center axis line through center axis transformation, and smoothing the Bezier curve based on the center axis line.

[0013] Further, in step S3, the generation method of positive and negative sample prompt maps is: the positive sample prompt map is taken from the largest connected region in the label true value that is not covered by the previous segmentation; the negative sample prompt map is taken from the largest false detection region in the previous segmentation result that exceeds the label true value, and the largest region unit is a pixel.

[0014] Further, in step S3, the fusion method of progressive iterative training of the interactive prompt remote sensing change detection model is: taking the positive sample prompt map, the negative sample prompt map, and the previous segmentation result map as independent channels, performing three-channel band stacking to form multi-band data input.

[0015] Further, in step S3, the interactive prompt remote sensing change detection model includes a segmentation backbone network and an optimization module.

[0016] Further, the segmentation backbone network adopts any one of SegNeXt, ResNet, ResNeXt, SegFormer or SwinTransformer, and the optimization module adopts a large kernel convolution layer for feature extraction.

[0017] Further, the sample set type in step S1 includes building change, cultivated land change or forest change detection data set, and the change sample set is any other change category in application.

[0018] Further, the boundary line of the line drawing interaction is generated through smoothing processing after the change region boundary is extracted by a Canny operator.

[0019] Further, the maximum connected region is the largest region in a binary image.

[0020] In a second aspect, the application further provides a multi-temporal remote sensing image change detection system based on interactive prompts, which is used to implement the multi-temporal remote sensing image change detection method based on interactive prompts, and includes a sample construction module, an interactive simulation module, an iterative training module and a detection execution module.

[0021] The sample construction module is used to obtain paired multi-temporal remote sensing images and corresponding binary change label maps, and form a sample set containing a pre-phase image, a post-phase image and a label map.

[0022] The interactive simulation module is used to simulate generation of positive sample prompt maps and negative sample prompt maps by using three interactive modes of clicking, frame selection and line drawing.

[0023] The iterative training module is used to input the positive sample prompt maps, the negative sample prompt maps simulated in step S2 and the sample set containing the pre-phase image, the post-phase image and the label map in step S1 into the interactive prompt remote sensing change detection model in the first iteration, input the positive sample prompt maps and the negative sample prompt maps generated currently and the previous segmentation result map into the interactive prompt remote sensing change detection model in subsequent iterations, and repeat the iteration step at least 3 times to optimize the parameters of the interactive prompt remote sensing change detection model.

[0024] The detection execution module is used to input the multi-temporal remote sensing images to be detected into the trained interactive prompt remote sensing change detection model, and output a change detection result map in combination with real-time interactive prompts of a user.

[0025] The application has the following beneficial effects:

[0026] 1.The application realizes the significant reduction of artificial annotation cost and the breakthrough improvement of complex scene detection accuracy by the fusion application of computer simulation of three types of interactive prompts, click, box selection and line drawing, combined with the progressive iterative training mechanism. In the training stage, the interactive prompt diagram is automatically generated to replace the artificial annotation, and only a small amount of real interaction is needed to construct a high-quality sample set. After at least three iterations of training, the three-channel waveband superposition fusion of the historical segmentation result and the positive and negative sample prompts is performed, so that the model can independently identify and correct the missed or false detection areas.

[0027] 2.At the same time, for irregular targets such as building contours and farmland boundaries, the multi-modal guidance mode of boundary lines, central axes and Bezier curves provided by the line drawing interaction, combined with the dynamic selection strategy of the maximum connected region, effectively improves the recognition accuracy of complex and variable targets. The application can adapt to the change analysis of non-fixed types of targets, and can expand the application range of remote sensing change detection. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0029] Figure 1 The flowchart of a multi-temporal remote sensing image change detection method based on interactive prompts of the present application.

[0030] Figure 2 The structural schematic diagram of a multi-temporal remote sensing image change detection system based on interactive prompts of the present application.

[0031] In the figure, 200 is a sample construction module, 201 is an interactive simulation module, 202 is an iterative training module, 203 is a detection execution module, and 204 is a multi-temporal remote sensing image change detection system based on interactive prompts. DETAILED DESCRIPTION

[0032] In order to make the technical problems, technical solutions and beneficial effects solved by the present application more clear, the following will further describe the present application in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

[0033] As shown in the figure, the multi-temporal remote sensing image change detection method based on interactive prompts provided by the embodiments of the present application includes the following steps: Figure 1

[0034] ​S1. Constructing a training sample set: obtaining pairs of multi-temporal remote sensing images and corresponding binary change label maps to form a sample set containing a pre-phase image, a post-phase image and a label map;

[0035] The sample set type in step S1 includes a building change, a cultivated land change or a forest change detection data set.

[0036] The sample set is mainly labeled by using a desktop GIS software to mark the change target of interest of the two-phase remote sensing image to generate a sample data set conforming to the change detection task, such as a building change detection data set, a cultivated land change detection data set, a forest change detection data set or other kinds of change detection data sets, and the main format is two-phase images plus a binary label map, and three image maps are added together as an input of the interactive remote sensing change detection model.

[0037] S2. Computer simulation interaction: three interactive modes of clicking, box selecting and line drawing are used to simulate the generation of positive sample prompt maps and negative sample prompt maps; the computer simulation of artificial interaction is realized by using the three interactive modes of clicking, box selecting and line drawing, so that the interactive remote sensing change detection model can be trained gradually and iteratively.

[0038] In the specific training process of the interactive remote sensing change detection model, the algorithm simulation mode needs to be used to realize the human-computer interaction process, because the complete dependence on manual training of the interactive remote sensing change detection model needs to consume great human and material resources; in order to accurately simulate the human-computer interaction mode, the present application constructs three interactive modes of clicking, box selecting and line drawing to simulate the human-computer interaction, and the three interactive modes of clicking, box selecting and line drawing can simulate the human-computer interaction process in most cases.

[0039] In step S2, the simulation mode of clicking interaction is to generate a circular prompt area with a specified coordinate point as the center, and a coordinate point buffer of a certain pixel is used to simulate, generally 5 pixels; the simulation mode of box selecting interaction is to fill the inside of the box with a foreground value to generate a prompt map; the simulation mode of line drawing interaction includes: a boundary line extracted by an edge detection operator and smoothed, a center axis line extracted by center axis transformation, and a Bezier curve based on the smoothed center axis line. As a preferred embodiment, the boundary line of the line drawing interaction is generated by smoothing the boundary of the change area extracted by the Canny operator; specifically, the center axis line is extracted by the center axis transformation in computer vision; specifically, the Bezier curve is first extracted from the center axis line of the binary change area, and then smoothed and burr-removed to obtain a smoothed center axis line.

[0040] S3. Gradually iteratively train the interactive prompt remote sensing change detection model: in the first iteration, input the positive sample prompt image generated in step S2, the negative sample prompt image, and the sample set containing the previous phase image, the later phase image, and the label image in step S1 into the interactive prompt remote sensing change detection model; in subsequent iterations, input the current generated positive and negative sample prompt images and the previous segmentation result image into the interactive prompt remote sensing change detection model; repeat the iteration step at least 3 times to optimize the parameters of the interactive prompt remote sensing change detection model.

[0041] Specifically, in step S3, the positive and negative sample prompt images are generated as follows: the positive sample prompt image is taken from the largest connected region in the label true value that is not covered by the previous segmentation; the negative sample prompt image is taken from the largest false detection region in the previous segmentation result that exceeds the label true value; the determination standard of the largest connected region is that the area ratio of the change region exceeds a preset threshold; for any sample set, only the positive sample simulation is performed in the first interactive process, and the negative sample and the previous segmentation result are replaced by a 0 value image; in the second and subsequent interactive processes, the positive and negative sample prompt images are generated. The formation process of the positive sample prompt image mainly includes: taking the missed part of the automatic extraction result and the label true value, and simulating the region with the largest missed area by using any one of the three interactive prompts to obtain the positive sample prompt image; the formation process of the negative sample prompt image mainly includes: taking the false extraction part of the automatic extraction result and the label true value, and simulating the false extraction part by using any one of the three prompt methods to obtain the negative sample prompt image.

[0042] Specifically, the fusion method of step S3 for gradually iteratively training the interactive prompt remote sensing change detection model is: taking the positive sample prompt image, the negative sample prompt image, and the previous segmentation result image as independent channels, performing three-channel band stacking to form the fusion input data.

[0043] Specifically, in step S3, the interactive prompt remote sensing change detection model includes a segmentation backbone network and an optimization module; as an optimization, the segmentation backbone network uses any one of SegNeXt, ResNet, ResNext, or SegFormer, and can also use other modern backbone neural networks such as Swin Transformer, and the segmentation backbone network has good universality; the optimization module uses a large kernel convolution layer for feature extraction, which can improve the interactive segmentation accuracy.

[0044] The gradually iterative training method of step S3 can fuse the previous segmentation result image into the interactive prompt, and can gradually optimize the segmentation result.

[0045] S4. Perform change detection: input the multi-temporal remote sensing image to be detected into the trained interactive prompt remote sensing change detection model, and output the change detection result image in combination with real-time interactive prompts from the user.

[0046] As Figure 2 The embodiment of the present application provides a multi-temporal remote sensing image change detection system based on interactive prompts, which is used for implementing the multi-temporal remote sensing image change detection method based on interactive prompts, and the multi-temporal remote sensing image change detection system 204 based on interactive prompts comprises a sample construction module 200, an interactive simulation module 201, an iterative training module 202 and a detection execution module 203.

[0047] The sample construction module 200 is used for acquiring a pair of multi-temporal remote sensing images and a corresponding binary change label map, and forming a sample set comprising a front-time image, a rear-time image and a label map.

[0048] The interactive simulation module 201 is used for simulating generation of positive sample prompt maps and negative sample prompt maps by using three interactive modes of clicking, frame selecting and line drawing.

[0049] The iterative training module 202 is used for inputting the positive sample prompt map simulated in step S2, the negative sample prompt map and the sample set comprising the front-time image, the rear-time image and the label map in step S1 into the interactive prompt remote sensing change detection model in the first iteration; in subsequent iterations, the positive sample prompt map and the negative sample prompt map generated currently are inputted into the interactive prompt remote sensing change detection model together with the segmentation result map of the previous time; and the iteration step is repeated at least three times, so that the parameters of the interactive prompt remote sensing change detection model are optimized.

[0050] The detection execution module 203 is used for inputting the multi-temporal remote sensing image to be detected into the trained interactive prompt remote sensing change detection model, and outputting a change detection result map in combination with real-time interactive prompts of a user.

[0051] To sum up, the interactive prompt model trained by the progressive iteration has universality, can well perform interactive segmentation on different tasks, has a simple model structure and good migratability. The interactive segmentation of the present application is realized by using a segmentation backbone network and a pure convolution structure, can support a domestic deep learning framework and has an advantage of running in a secure environment.

[0052] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "preferred embodiment", "specific embodiments" or "preferred embodiments" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are contained in at least one embodiment or example of the present application. In the present application, the exemplary description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0053] It can be understood that the above specific description of the present application is only for illustrating the present application and is not limited to the technical solutions described in the embodiments of the present application. Those skilled in the art should understand that the present application can still be modified or replaced equivalently to achieve the same technical effects. As long as the use needs are met, it is within the protection scope of the present application.

Claims

1. A multi-temporal remote sensing image change detection method based on interactive cues, characterized in that: Comprise the following steps: S1. Constructing a training sample set: obtaining pairs of multi-temporal remote sensing images and corresponding binary change label maps, forming a sample set containing pre-phase images, post-phase images and label maps; S2. Computer simulation interaction prompt: adopt three kinds of interactive ways of clicking, frame selection, line drawing to simulate the generation of positive sample prompt graph and negative sample prompt graph; the simulation method of line drawing interaction includes: extracting and smoothing the boundary line through edge detection operator, extracting the center axis line through center axis transformation, and generating the Bezier curve based on the smoothing processing of the center axis line; specifically, the boundary line of line drawing interaction is generated by smoothing the boundary of the change area extracted by Canny operator; the center axis line is the center axis line of the binary change area extracted by the center axis transformation in computer vision; the Bezier curve is first extracted from the center axis line of the binary change area, and then the smoothed center axis line is obtained by smoothing and burr removal; S3. Gradual iterative training of interactive prompt remote sensing change detection model: in the first iteration, input the positive sample prompt graph and negative sample prompt graph simulated in step S2, and the sample set containing pre-phase images, post-phase images and label maps in step S1 into the interactive prompt remote sensing change detection model; in subsequent iterations, input the generated positive and negative sample prompt graphs and the previous segmentation result graph into the interactive prompt remote sensing change detection model; repeat the iteration step at least 3 times to optimize the parameters of the interactive prompt remote sensing change detection model; S4. Perform change detection: input the multi-temporal remote sensing image to be detected into the trained interactive prompt remote sensing change detection model, and output the change detection result graph combined with real-time interactive prompt of the user.

2. The method of claim 1, wherein: In step S2, the simulation method of clicking interaction is to generate a circular prompt area with a specified coordinate point as the center; the simulation method of frame selection interaction is to fill the inside of the rectangular frame with foreground value to generate a prompt graph.

3. The method of claim 1, wherein: In step S3, the generation method of positive and negative sample prompt graphs is: the positive sample prompt graph is taken from the largest connected region in the label true value that is not covered by the previous segmentation; the negative sample prompt graph is taken from the largest false detection region in the previous segmentation result that exceeds the label true value, and the unit of the largest region is pixel.

4. The method of claim 1, wherein: In step S3, the fusion method of gradual iterative training of interactive prompt remote sensing change detection model is: taking the positive sample prompt graph, the negative sample prompt graph and the previous segmentation result graph as independent channels, performing three-channel band stacking to form multi-band data input.

5. The method of claim 1, wherein: In step S3, the interactive prompt remote sensing change detection model comprises a segmentation backbone network and an optimization module.

6. The method of claim 5, wherein: The segmentation backbone network adopts any one of SegNeXt, ResNet, ResNeXt, SegFormer or Swin Transformer, and the optimization module adopts a large kernel convolution layer for feature extraction.

7. The method of claim 1, wherein: The sample set type in step S1 includes building change, cultivated land change or forest change detection data set, and the change sample set in application is any other change category.

8. The method of claim 2, wherein: The boundary line of line drawing interaction is generated by smoothing the change area boundary extracted by Canny operator.

9. The method of claim 3, wherein: The largest connected region is the largest region in the binary graph.

10. A multi-temporal remote sensing image change detection system based on interactive cues, characterized in that: The method for implementing the multi-temporal remote sensing image change detection based on interactive prompts of claim 1 comprises a sample construction module, an interactive simulation module, an iterative training module, and a detection execution module. The sample construction module is used to obtain paired multi-temporal remote sensing images and corresponding binary change label maps, and form a sample set containing a pre-phase image, a post-phase image, and a label map. The interactive simulation module is used to simulate the generation of positive sample prompt maps and negative sample prompt maps by using three interactive modes of clicking, frame selection, and line drawing. The iterative training module is used to input the positive sample prompt maps, the negative sample prompt maps simulated in step S2, and the sample set containing the pre-phase image, the post-phase image, and the label map in step S1 into the interactive prompt remote sensing change detection model in the first iteration, and input the positive and negative sample prompt maps generated currently and the segmentation result map in the previous iteration into the interactive prompt remote sensing change detection model in subsequent iterations. The iteration step is repeated at least 3 times to optimize the parameters of the interactive prompt remote sensing change detection model. The detection execution module is used to input the multi-temporal remote sensing images to be detected into the trained interactive prompt remote sensing change detection model, and output a change detection result map in combination with real-time interactive prompts of a user.

Images