Remote sensing image change detection method and device, equipment and medium

Abstract

The invention discloses a remote sensing image change detection method and device, equipment and a medium, and relates to the technical field of image processing. The change detection method of the remote sensing image comprises the steps that a front time phase remote sensing image and a rear time phase remote sensing image are obtained, the front time phase remote sensing image and the rear time phase remote sensing image are input into an encoder, a plurality of space-time difference enhancement feature maps of different scales are obtained, the encoder comprises a feature extraction module and a plurality of space-time difference enhancement modules, and the feature extraction module is used for extracting the feature extraction module and the space-time difference enhancement modules; and inputting the plurality of space-time difference enhanced feature maps with different scales into a decoder, and carrying out edge refinement and feature fusion on the plurality of space-time difference enhanced feature maps with different scales through the decoder to obtain a prediction binary image comprising change information between the front time phase remote sensing image and the rear time phase remote sensing image. According to the embodiment of the invention, the accuracy of remote sensing image change detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a remote sensing image change detection method, device, equipment and medium. Background Art

[0002] Remote sensing image change detection is a technical means to identify and quantify changes in surface cover and land use in the same area by analyzing remote sensing images acquired at different times. It is currently widely used in many fields such as natural disaster and environmental monitoring, urban planning, agricultural management and resource surveys.

[0003] The research objects based on the remote sensing image change detection methods in related technologies include pixel-based, object-based, scene-based detection methods, etc. These detection methods still have some problems, resulting in low accuracy in remote sensing image change detection. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a remote sensing image change detection method, device, equipment and medium, aiming to improve the accuracy of remote sensing image change detection.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a change detection method for remote sensing images, comprising the following steps: obtaining a front-phase remote sensing image and a back-phase remote sensing image, wherein the front-phase remote sensing image is used to characterize a surface feature image of a target area at a first time point, and the back-phase remote sensing image is used to characterize a surface feature image of the target area at a second time point; inputting the front-phase remote sensing image and the back-phase remote sensing image into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, wherein the encoder includes a feature extraction module and a plurality of spatiotemporal difference enhancement modules, and the feature extraction module is used to extract the The features in the previous phase remote sensing image and the subsequent phase remote sensing image are used to generate a plurality of dual-phase feature maps of different scales, and the plurality of spatiotemporal difference enhancement modules are used to capture the remote context information and local cross-feature information of the change area in the dual-phase feature maps of multiple scales to generate the spatiotemporal difference enhanced feature maps of multiple scales; the spatiotemporal difference enhanced feature maps of multiple scales are input into a decoder, and the decoder is used to perform edge refinement and feature fusion on the spatiotemporal difference enhanced feature maps of multiple scales to obtain a predicted binary image including the change information between the previous phase remote sensing image and the subsequent phase remote sensing image.

[0006] In some embodiments, the feature extraction module includes two parallel feature extraction branches, and the front-phase remote sensing image and the back-phase remote sensing image are input into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, including: inputting the front-phase remote sensing image and the back-phase remote sensing image into two parallel feature extraction branches respectively, and extracting a plurality of dual-phase feature maps of different scales through the feature extraction network in the feature extraction branches; and inputting the dual-phase feature maps of multiple different scales into the corresponding spatiotemporal difference enhancement modules respectively to obtain the spatiotemporal difference enhanced feature maps of multiple different scales.

[0007] In some embodiments, the spatiotemporal difference enhancement module includes a subtraction branch and a channel alternating splicing branch, and the dual-phase feature map is input into the spatiotemporal difference enhancement module to obtain the spatiotemporal difference enhanced feature map, including: inputting the dual-phase feature map into the subtraction branch, and calculating the absolute difference feature between the dual-phase feature maps in the subtraction branch; inputting the absolute difference feature into the dual attention mechanism module, and capturing the remote context information of the changing area in the dual-phase feature map through the dual attention mechanism module to obtain a spatial attention enhanced feature map and a channel attention enhanced feature map; performing channel splicing on the spatial attention enhanced feature map and the channel attention enhanced feature map to obtain a spatial channel attention enhanced feature map; inputting the dual-phase feature map into the channel alternating splicing branch, and inputting the feature map after channel alternating splicing into the channel adaptive enhancement module to enhance the model's ability to extract cross-feature interaction information to obtain a channel enhanced feature map; adding the spatial channel attention enhanced feature map to the channel enhanced feature map to obtain the spatiotemporal difference enhanced feature map.

[0008] In some embodiments, the dual attention mechanism module includes a spatial attention mechanism module and a channel attention mechanism module, and the absolute difference feature is input into the dual attention mechanism module, and the remote context information of the changing area in the dual-phase feature map is captured through the dual attention mechanism module to obtain a spatial attention enhanced feature map and a channel attention enhanced feature map, including: inputting the absolute difference feature into the spatial attention mechanism module, establishing the contextual relationship of the local feature through the spatial attention mechanism module, and obtaining the spatial attention enhanced feature map; inputting the absolute difference feature into the channel attention mechanism module, and establishing the remote dependency relationship between the feature map channels through the channel attention mechanism module to obtain the channel attention enhanced feature map.

[0009] In some embodiments, the spatiotemporal difference enhancement module includes a channel alternating splicing branch, and the step of inputting the dual-phase feature map into the spatiotemporal difference enhancement module to obtain the spatiotemporal difference enhanced feature maps of multiple different scales includes: inputting the dual-phase feature map into the channel alternating splicing branch, and alternately splicing the dual-phase feature map in the channel alternating splicing branch to obtain splicing features; inputting the splicing features into a channel adaptive enhancement mechanism module, and enhancing the local cross-feature interaction information of the changing area in the dual-phase feature map through the channel adaptive enhancement mechanism module to obtain a channel enhanced feature map; adding the channel enhanced feature map to the spatial channel attention enhanced feature map to obtain the spatiotemporal difference enhanced feature map.

[0010] In some embodiments, the spatiotemporal difference enhancement feature maps of multiple scales are input into a decoder, and the decoder is used to perform edge refinement and feature fusion on the spatiotemporal difference enhancement feature maps of multiple scales, including: inputting the spatiotemporal difference enhancement feature maps of multiple scales into multiple corresponding edge refinement residual modules respectively, and enhancing the edges of the change areas in the spatiotemporal difference enhancement feature maps at different scales by the edge refinement residual modules to obtain multiple edge refinement feature maps; inputting the multiple edge refinement feature maps into an adaptive bidirectional feature fusion module, and performing top-down and bottom-up layer-by-layer fusion on the edge refinement feature maps by the adaptive bidirectional feature fusion module to obtain a fused feature map; upsampling and channel adjustment on the fused feature map to obtain a predicted binary image including the change information between the front-phase remote sensing image and the back-phase remote sensing image.

[0011] In some embodiments, the edge refinement residual module includes a first branch and a second branch, and the multiple spatiotemporal difference enhancement feature maps of different scales are respectively input into multiple corresponding edge refinement residual modules, and the edges of the changing areas in the spatiotemporal difference enhancement feature maps at different scales are enhanced by the edge refinement residual modules to obtain multiple edge refinement feature maps, including: inputting the multiple spatiotemporal difference enhancement feature maps of different scales into the corresponding first branch, unifying the number of channels of the spatiotemporal difference enhancement feature maps at different scales through the first branch, and obtaining multiple standard channel feature maps; inputting the multiple standard channel feature maps into the corresponding second branch, and refining the spatiotemporal difference enhancement feature maps of different scales through the second branch to obtain multiple refined feature maps; and summing the multiple standard channel feature maps and the refined feature maps respectively to obtain multiple edge refinement feature maps.

[0012] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a remote sensing image change detection device, the remote sensing image change detection device comprising: an acquisition module, used to acquire a front-phase remote sensing image and a back-phase remote sensing image, the front-phase remote sensing image is used to characterize the surface feature image of the target area at a first time point, and the back-phase remote sensing image is used to characterize the surface feature image of the target area at a second time point; a feature extraction generation module, used to input the front-phase remote sensing image and the back-phase remote sensing image into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, wherein the encoder comprises a feature extraction module and a plurality of spatiotemporal difference enhancement modules, the feature extraction The module is used to extract features from the front-phase remote sensing image and the back-phase remote sensing image to generate a plurality of dual-phase feature maps of different scales. The plurality of spatiotemporal difference enhancement modules are used to capture the remote context information and local cross-feature information of the change area in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhanced feature maps of different scales. The feature refinement and fusion module is used to input the spatiotemporal difference enhanced feature maps of different scales into the decoder, and the decoder is used to perform edge refinement and feature fusion on the spatiotemporal difference enhanced feature maps of different scales to obtain a predicted binary image including the change information between the front-phase remote sensing image and the back-phase remote sensing image.

[0013] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned remote sensing image change detection method when executing the computer program.

[0014] To achieve the above objectives, another aspect of an embodiment of the present application proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by one or more processors, the steps of the remote sensing image change detection method as described above are implemented.

[0015] The embodiments of the present application include at least the following beneficial effects:

[0016] The present application provides a remote sensing image change detection method, device, equipment and medium. In the embodiment of the present application, first, a front-phase remote sensing image and a back-phase remote sensing image are obtained, the front-phase remote sensing image is used to represent the surface feature image of the target area at the first time point, and the back-phase remote sensing image is used to represent the surface feature image of the target area at the second time point; then, the front-phase remote sensing image and the back-phase remote sensing image are input into an encoder to obtain multiple spatiotemporal difference enhancement feature maps of different scales, wherein the encoder includes a feature extraction module and multiple spatiotemporal difference enhancement modules, and the feature extraction module is used to extract The features in the front-phase remote sensing image and the back-phase remote sensing image are used to generate a plurality of dual-phase feature maps of different scales. A plurality of spatiotemporal difference enhancement modules are used to capture the remote context information and local cross-feature information of the change area in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhanced feature maps of different scales. Finally, the plurality of spatiotemporal difference enhanced feature maps of different scales are input into the decoder, and the decoder performs edge refinement and feature fusion on the plurality of spatiotemporal difference enhanced feature maps of different scales to obtain a predicted binary image including the change information between the front-phase remote sensing image and the back-phase remote sensing image. The embodiment of the present application extracts features from the front-phase remote sensing image and the back-phase remote sensing image through a feature extraction module to generate a plurality of dual-phase feature maps of different scales. Afterwards, the remote context information and local cross-feature information of the change area in the dual-phase feature maps of different scales are captured through a plurality of spatiotemporal difference enhancement modules to generate a plurality of spatiotemporal difference enhanced feature maps of different scales. The global change information and local fine-grained change information between the dual-phase feature maps can be learned at the same scale, and the spatiotemporal difference of the dual-phase image features extracted by the encoder can be enhanced. The edge refinement and feature fusion of the spatiotemporal difference enhanced feature maps of different scales are performed through the decoder, so that the rich position information in the shallow feature map and the rich semantic information in the deep feature map are effectively fused, thereby improving the accuracy of remote sensing image change detection.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0019] Figure 1 is a flow chart of a remote sensing image change detection method provided by some embodiments of the present application;

[0020] Figure 2 is a schematic diagram of the architecture of a change detection model for remote sensing images provided in some embodiments of the present application;

[0021] Figure 3 is a schematic diagram of the network structure of an encoder provided in some embodiments of the present application;

[0022] Figure 4 is a schematic diagram of the network structure of a decoder partial feature edge thinning residual module provided in some embodiments of the present application;

[0023] Figure 5 is a schematic diagram of the network structure of a decoder partial adaptive bidirectional feature fusion module provided in some embodiments of the present application;

[0024] Figure 6 is a schematic diagram comparing visualization results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the WHU-CD dataset;

[0025] Figure 7 is a schematic diagram comparing visualization results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the LEVIR-CD dataset;

[0026] Figure 8 is a schematic diagram comparing visualization results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the SYSU-CD dataset;

[0027] FIG9( a ) is a schematic diagram showing the computational complexity comparison results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the WHU-CD dataset;

[0028] FIG9( b ) is a schematic diagram showing the computational complexity comparison results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the LEVIR-CD dataset;

[0029] FIG9( c ) is a schematic diagram showing the computational complexity comparison results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the SYSU-CD dataset;

[0030] Fig.10 is a schematic block diagram of modules of a remote sensing image change detection device provided in some embodiments of the present application;

[0031] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0033] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0034] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0037] In order to make the inventive concept of the present application easy to understand, before describing the embodiments of the present application in detail, the English abbreviations (terms) / related concepts involved in the embodiments of the present application are first described. The English abbreviations (terms) / related concepts involved in the embodiments of the present application are subject to the following explanations.

[0038] Bi-temporal feature mapping: involves comparing images acquired at two different time points to identify and analyze changes that occurred during this time interval. Phase 1 represents the image acquired at the first time point; Phase 2 represents the image acquired at the second time point.

[0039] LEVIR-CD dataset: is a large-scale remote sensing building change detection dataset. This dataset is designed as a new benchmark for evaluating change detection (CD) algorithms, especially those based on deep learning.

[0040] WHU-CD dataset: is a dataset for remote sensing image change detection, focusing on building change detection, and is often used to evaluate and compare different remote sensing image change detection algorithms.

[0041] SYSU-CD dataset: is a dataset for remote sensing image change detection, used to detect and analyze changes in the earth's surface, such as urban expansion, deforestation, or the impact of natural disasters.

[0042] Remote sensing image change detection is a technical means to identify and quantify changes in surface cover and land use in the same area by analyzing remote sensing images acquired at different times. It is currently widely used in many fields such as natural disaster and environmental monitoring, urban planning, agricultural management and resource survey. With the rapid development of remote sensing earth observation technology, image processing technology and artificial intelligence, change detection based on multi-source, multi-temporal remote sensing image data-model-knowledge joint drive has become an important direction of remote sensing discipline. Multi-source, multi-temporal, multi-spectral and high-resolution remote sensing images have problems such as different representation of ground objects (such as pseudo-changes such as shadow changes and seasonal changes), complex scene changes, and "different spectra for the same object, same spectrum for different objects". As a result, remote sensing image change detection has problems such as missed detection and false detection, which are not ideal detection effects. This brings many challenges to the change detection task.

[0043] The research objects of traditional remote sensing image change detection methods include pixel-based, object-based, scene-based and other methods.

[0044] Pixel-based remote sensing change detection methods mainly obtain remote sensing images at two time points, perform image registration to ensure spatial alignment, and then use thresholds, statistics, time series analysis or machine learning methods to detect changes in each pixel, and finally perform post-processing optimization and analyze the results. However, this method is easily affected by noise, is not accurate enough for detecting changes in small areas in the image, and cannot quantify the relationship between pixels.

[0045] The object-based remote sensing change detection method organizes the pixels in the image into objects, which can better reflect the scenes in the real world. Its detection accuracy is higher than that of the pixel-based method. However, due to the registration errors between multi-temporal images, the complex morphology of the objects and the unclear boundaries of the objects, the change areas extracted by the object-based change detection method are inaccurate.

[0046] The scene-based remote sensing change detection method uses the object category information and scene context to segment and classify the remote sensing image, and finally identifies and analyzes the changes of different object categories over time based on the change analysis technology of the object category, such as the change detection model or classifier. However, this method is more complicated in defining and extracting the scene, and needs to consider the comprehensive influence of various factors. It is not sensitive enough for detecting local subtle changes, and is prone to missed detection or false detection.

[0047] With the continuous development of artificial intelligence technology, deep learning models have been widely used in high-resolution remote sensing image interpretation tasks such as scene classification, target detection and semantic segmentation, and their interpretation accuracy has surpassed traditional methods. In the remote sensing image change detection task, the mainstream networks used are convolutional neural networks (CNNs) and Transformers. Although deep learning has made significant progress in the field of remote sensing image change detection, existing change detection methods still have certain limitations, including: (1) they mainly focus on indicators such as the optimal Iou or F1 score of the detection results, while ignoring the applicability of different lightweight and non-lightweight convolutional network models, Transformer models and network architectures under the same model computational efficiency conditions; (2) they focus too much on feature extraction and fail to fully utilize spatiotemporal difference information, ignoring the substantial differences between dual-phase images; (3) multi-scale feature fusion relies too much on modules with specific structures and fails to fully quantify the importance of features at different scales, ignoring the correlation between features at different scales.

[0048] In view of this, the present application proposes a remote sensing image change detection method, device, equipment and medium. The scheme obtains a front-phase remote sensing image and a back-phase remote sensing image, wherein the front-phase remote sensing image is used to characterize the surface feature image of the target area at the first time point, and the back-phase remote sensing image is used to characterize the surface feature image of the target area at the second time point; then, the front-phase remote sensing image and the back-phase remote sensing image are input into an encoder to obtain multiple spatiotemporal difference enhanced feature maps of different scales, wherein the encoder includes a feature extraction module and multiple spatiotemporal difference enhancement modules, and the feature extraction module is used to extract the front-phase remote sensing image and the back-phase remote sensing image. The features in the bi-phase remote sensing image and the post-phase remote sensing image are used to generate a plurality of bi-phase feature maps of different scales. The plurality of spatio-temporal difference enhancement modules are used to capture the remote context information and local cross-feature information of the change area in the bi-phase feature maps of different scales to generate a plurality of spatio-temporal difference enhanced feature maps of different scales. Finally, the plurality of spatio-temporal difference enhanced feature maps of different scales are input into the decoder, and the decoder performs edge refinement and feature fusion on the plurality of spatio-temporal difference enhanced feature maps of different scales to obtain a predicted binary image including the change information between the front-phase remote sensing image and the back-phase remote sensing image. The embodiment of the present application extracts features from the front-phase remote sensing image and the back-phase remote sensing image through a feature extraction module to generate a plurality of dual-phase feature maps of different scales. Afterwards, the remote context information and local cross-feature information of the change area in the dual-phase feature maps of different scales are captured through a plurality of spatiotemporal difference enhancement modules to generate a plurality of spatiotemporal difference enhanced feature maps of different scales. The global change information and local fine-grained change information between the dual-phase feature maps can be learned at the same scale, and the spatiotemporal difference of the dual-phase image features extracted by the encoder can be enhanced. The edge refinement and feature fusion of the spatiotemporal difference enhanced feature maps of different scales are performed through the decoder, so that the rich position information in the shallow feature map and the rich semantic information in the deep feature map are effectively fused, thereby improving the accuracy of remote sensing image change detection.

[0049] The method provided in the embodiment of the present application can be applied to the electronic device provided in the embodiment of the present application, wherein the electronic device can be a terminal or a server.

[0050] The terminal may be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto.

[0051] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, as well as big data and artificial intelligence platforms.

[0052] The following will describe in detail the implementation steps of a remote sensing image change detection method provided by an embodiment of the present application in conjunction with the accompanying drawings.

[0053] Please refer to Figure 1 , Figure 1 A flowchart of a remote sensing image change detection method provided for some embodiments of the present application. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0054] The method of the embodiment of the present application comprises the following steps:

[0055] Step 101: acquiring a front-phase remote sensing image and a back-phase remote sensing image, wherein the front-phase remote sensing image is used to represent a surface feature image of a target area at a first time point, and the back-phase remote sensing image is used to represent a surface feature image of a target area at a second time point;

[0056] Step 102: inputting the front-phase remote sensing image and the back-phase remote sensing image into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, wherein the encoder includes a feature extraction module and a plurality of spatiotemporal difference enhancement modules, the feature extraction module is used to extract features from the front-phase remote sensing image and the back-phase remote sensing image to generate a plurality of dual-phase feature maps of different scales, and the plurality of spatiotemporal difference enhancement modules are used to capture remote context information and local cross-feature information of the changing regions in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhanced feature maps of different scales;

[0057] Step 103: Input multiple spatiotemporal difference enhancement feature maps of different scales into the decoder, and perform edge refinement and feature fusion on the multiple spatiotemporal difference enhancement feature maps of different scales through the decoder to obtain a predicted binary image including change information between the previous phase remote sensing image and the next phase remote sensing image.

[0058] In steps 101 to 103 shown in the embodiment of the present application, features in the front-phase remote sensing image and the back-phase remote sensing image are extracted by a feature extraction module to generate a plurality of dual-phase feature maps of different scales. Afterwards, remote context information and local cross-feature information of the change area in the dual-phase feature maps of different scales are captured by a plurality of spatiotemporal difference enhancement modules to generate a plurality of spatiotemporal difference enhanced feature maps of different scales. The global change information and local fine-grained change information between the dual-phase feature maps can be learned at the same scale, and the spatiotemporal difference of the dual-phase image features extracted by the encoder can be enhanced. The edge refinement and feature fusion of the spatiotemporal difference enhanced feature maps of different scales are performed by the decoder, so that the rich position information in the shallow feature map and the rich semantic information in the deep feature map are effectively fused, thereby improving the accuracy of remote sensing image change detection.

[0059] Please refer to Figure 2 , Figure 2 A schematic diagram of the architecture of a change detection model for remote sensing images provided in some embodiments of the present application. Figure 2 In the figure, the change detection model of remote sensing images mainly includes an encoder module and a decoder module. The part enclosed by a dotted line on the left is the encoder part, and the part enclosed by a dotted line on the right is the decoder part.

[0060] The encoder part includes a feature extraction module and a spatiotemporal difference enhancement module (STDEM), T1 and T2 are respectively the acquired remote sensing images of the previous phase and the acquired remote sensing images of the later phase. By extracting features from T1 and T2 respectively through parallel feature extraction branches (feature extraction modules), four feature map groups of different scales can be obtained, for example: the first feature map group, the second feature map group, the third feature map group and the fourth feature map group, each feature map group includes two feature maps, corresponding to the dual-phase feature map. The four feature map groups of different scales are respectively input into the four STDEM modules to obtain four spatiotemporal difference enhancement feature maps at different scales.

[0061] The decoder part includes an edge refinement residual module (ERRM) and an adaptive bidirectional feature fusion module (ABiFFM). The spatiotemporal difference enhanced feature maps at four different scales are input into four corresponding ERRMs to obtain feature maps of four different scales after edge refinement. Then, the feature maps of four different scales after edge refinement are input into ABiFFM to obtain fused feature maps of four different scales. Finally, bilinear interpolation is used to perform 4 times upsampling of the feature map and 1×1 convolution to adjust the number of channels of the feature map to 2 to generate the final binary prediction map. Among them, Figure 2 The green rectangle in the figure represents upsampling, and the red rectangle represents 1×1 convolution. Figure 2 In the , labeled images involved in supervised training are also included.

[0062] The specific implementation methods of the above steps are introduced below.

[0063] In step 101, a front-phase remote sensing image and a back-phase remote sensing image are obtained. The front-phase remote sensing image is used to represent the surface feature image of the target area at a first time point, and the back-phase remote sensing image is used to represent the surface feature image of the target area at a second time point.

[0064] The surface feature image of the target area at the first time point may be the surface feature image of the target area before the change occurs, and the surface feature image of the target area at the second time point may be the surface feature image of the target area after the change occurs.

[0065] The remote sensing images may be remote sensing image data provided by space agencies or organizations, or high-resolution remote sensing images provided by commercial satellite companies, or remote sensing images may be obtained through online platforms, data sharing communities or forums. This application does not limit the method of obtaining remote sensing images.

[0066] The embodiment provided in the present application provides image support for subsequent processing of remote sensing images and obtaining change information between remote sensing images by acquiring a front-phase remote sensing image and a back-phase remote sensing image.

[0067] In step 102, the front-phase remote sensing image and the back-phase remote sensing image are input into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, wherein the encoder includes a feature extraction module and a plurality of spatiotemporal difference enhancement modules, the feature extraction module is used to extract features from the front-phase remote sensing image and the back-phase remote sensing image to generate a plurality of dual-phase feature maps of different scales, and the plurality of spatiotemporal difference enhancement modules are used to capture the remote context information and local cross-feature information of the changing areas in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhanced feature maps of different scales.

[0068] In some embodiments, the feature extraction module includes two parallel feature extraction branches, and the front-phase remote sensing image and the back-phase remote sensing image can be respectively input into the two parallel feature extraction branches, and a plurality of dual-phase feature maps of different scales are extracted through the feature extraction network in the feature extraction branch; the plurality of dual-phase feature maps of different scales are respectively input into the corresponding spatiotemporal difference enhancement modules to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales.

[0069] Since in diverse change detection scenarios, feature extraction networks of different types and different parameter amounts will have different effects on the performance and effect of the model in specific scenarios, it is necessary to analyze the adaptability of multiple feature extraction networks of different types and different parameter amounts to the network architecture, and then select the appropriate backbone feature extraction network.

[0070] Exemplarily, the selected feature backbone extraction network may be an end-to-end feature extraction network built using the MobileNet V3-Large architecture, which is a variant of the lightweight neural network architecture series MobileNet.

[0071] Exemplarily, based on the input remote sensing image group, a feature extraction backbone network adapted to the network architecture can be used on two parallel feature extraction branches to extract four dual-phase feature maps of different scales, including a first feature map group, a second feature map group, a third feature map group and a fourth feature map group. Subsequently, the dual-phase feature maps at different scales are respectively input into the spatiotemporal difference enhancement module to obtain multiple spatiotemporal difference enhanced feature maps of different scales, including a fifth feature map, a sixth feature map, a seventh feature map and an eighth feature map.

[0072] The main purpose of selecting a feature extraction backbone network adapted to the network architecture to obtain the fifth feature map, the sixth feature map, the seventh feature map, and the eighth feature map is that different models have different generalization capabilities, and feature extraction backbone networks of different types and parameter amounts have different effects on the performance and effect of the model in a specific scenario. Therefore, a lightweight network of the MobileNet V3-Large series can be selected to reduce the number of parameters, improve feature extraction capabilities and computer efficiency. At the same time, the spatiotemporal difference enhancement module is used to process these dual-phase feature maps separately, thereby enhancing the substantial differences in dual-phase features.

[0073] The main purpose of obtaining spatiotemporal difference enhanced feature maps at various scales is that the channel adaptive enhancement mechanism and the dual attention mechanism can capture the long-range contextual information and local cross-feature information of the changing areas in the dual-temporal feature map, thereby enhancing the spatiotemporal differences of the dual-temporal image features.

[0074] In some embodiments, the spatiotemporal difference enhancement module includes a subtraction branch and a channel alternating splicing branch. The dual-phase feature map can be input into the subtraction branch, and the absolute difference feature between the dual-phase feature maps is calculated in the subtraction branch; the absolute difference feature is input into the dual attention mechanism module, and the remote context information of the changing area in the dual-phase feature map is captured through the dual attention mechanism module to obtain a spatial attention enhanced feature map and a channel attention enhanced feature map; the spatial attention enhanced feature map and the channel attention enhanced feature map are channel-spliced ​​to obtain a spatial channel attention enhanced feature map; the dual-phase feature map is input into the channel alternating splicing branch, and the feature map after channel alternating splicing is input into the channel adaptive enhancement module to enhance the model's ability to extract cross-feature interaction information to obtain a channel enhanced feature map; finally, the spatial channel attention enhanced feature map and the channel enhanced feature map are added to obtain a spatiotemporal difference enhanced feature map.

[0075] Generally, dual-temporal remote sensing images have problems such as different representation of objects, different spectra for the same object, and the same spectrum for different objects. Direct subtraction, addition, or channel splicing of dual-temporal images will introduce a lot of noise into change detection, and the detection results lack interpretability, leading to problems such as missed detection and false detection. Therefore, a dual-branch structured spatiotemporal difference enhancement module is introduced, which uses the channel adaptive enhancement mechanism and dual attention mechanism to capture the remote context information and local cross-feature information of the changing area in the dual-temporal feature map, and enhance the spatiotemporal difference of the dual-temporal image features.

[0076] In the subtraction branch, a dual attention mechanism of spatial attention mechanism and channel attention mechanism is adopted. The spatial attention mechanism can model rich contextual relationships on local features, and the channel attention mechanism can explicitly model the long-range dependencies between feature map channels.

[0077] In some embodiments, the dual attention mechanism module includes a spatial attention mechanism module and a channel attention mechanism module. The absolute difference feature can be input into the spatial attention mechanism module, and the contextual relationship of the local feature is established through the spatial attention mechanism module to obtain a spatial attention enhanced feature map; the absolute difference feature can be input into the channel attention mechanism module, and the long-range dependency relationship between the feature map channels is established through the channel attention mechanism module to obtain a channel attention enhanced feature map.

[0078] Optionally, for the subtraction branch, the absolute difference features between the input first image and the second image group (bi-phase feature map) are first calculated, and then input into the spatial attention mechanism module and the channel attention mechanism module to capture the long-range context information of the changed area, and obtain the spatial attention enhanced feature map and the channel attention enhanced feature map. After that, the two are concatenated on the channel, and then subjected to 3x3 convolution operation, normalization and non-linear activation of the ReLu function to obtain the third image (spatial channel attention enhanced feature map).

[0079] For example, reference may be made to Figure 3 The spatial attention mechanism module in the spatial attention mechanism can model rich contextual relationships on local features. For a given feature map A, A∈N H×W×C , where H represents height, W represents width, and C represents the number of channels. Use 1×1 convolution to generate feature maps B, C, and D of the same size, and reset the sizes of B, C, and D to N respectively. C×(H×W) 、N (H×W)×C , then perform matrix multiplication on B and C, and use the Softmax function to calculate the spatial feature map K, K∈N (H×W)×(H×W) :

[0080]

[0081] Kji represents the elements in the spatial feature map K, and i and j are variables representing the elements.

[0082] Then perform matrix multiplication on K and D and reset the result to N H×W×C , multiply the reset result by the learnable scale parameter α, and then sum it with the original feature map A. The final output is E, E∈N H×W×C :

[0083]

[0084] The output E is the spatial attention enhanced feature map.

[0085] For example, reference may be made to Figure 3 The channel attention mechanism module in the channel attention mechanism module can explicitly model the long-range dependencies between feature map channels. For a given feature map A, A∈N H×W×C , first reset the size of A to N C×(H×W) and N (H×W)×C , perform matrix multiplication on the two, and use the Softmax function to calculate the channel feature map T, T∈N C×C :

[0086]

[0087] t ji Represents the elements in the channel feature map T, and i and j are variables representing the elements.

[0088] Then perform matrix multiplication on T and the reset matrix of A, and reset the result to N H×W×C , multiply the reset result by the learnable scale parameter β, and then sum it with the feature map A. The final output is R, R∈N H×W×C :

[0089]

[0090] The output R is the channel attention enhanced feature map.

[0091] The spatial attention enhanced feature map and the channel attention enhanced feature map are concatenated on the channel, and then subjected to 3x3 convolution operation, normalization, and nonlinear activation of the ReLu function to obtain the third image (spatial channel attention enhanced feature map).

[0092] In some embodiments of the present application, the spatiotemporal difference enhancement module includes a channel alternating splicing branch, and a dual-phase feature map can be input into the channel alternating splicing branch, and the dual-phase feature map is alternately spliced ​​in the channel alternating splicing branch to obtain a splicing feature; the splicing feature is input into the channel adaptive enhancement mechanism module, and the local cross-feature interaction information of the changing area in the dual-phase feature map is enhanced by the channel adaptive enhancement mechanism module to obtain a channel enhanced feature map; the channel enhanced feature map is added to the spatial channel attention enhanced feature map to obtain a spatiotemporal difference enhanced feature map.

[0093] In the channel alternating splicing branch, the channel adaptive enhancement mechanism can enhance the model's ability to extract cross-feature interactive information. In this way, spatiotemporal difference enhanced feature maps at various scales can be obtained. Compared with other direct subtraction and splicing methods, these feature maps have less noise and stronger spatiotemporal differences in dual-phase image features.

[0094] Optionally, for the channel alternating splicing branch, the first image and the second image group (dual-phase feature map) are alternately spliced ​​in the channel dimension, and then a channel adaptive enhancement mechanism is introduced to highlight the local cross-feature interaction information in the changed area, and then it is subjected to 1x1 convolution, normalization and nonlinear activation of the ReLu function to obtain the fourth image (channel enhanced feature map).

[0095] The embodiment provided in this application adopts a channel adaptive enhancement mechanism to enhance the model's ability to extract cross-feature interactive information. H×W×C First, the bi-phase feature maps are spliced ​​alternately in the channel dimension, and then the feature maps are globally averaged pooled to obtain a weight matrix w containing 2C channels, w∈N 1×1×2C :

[0096]

[0097] i, j, and t are variables representing elements in the weight matrix.

[0098] Perform a 1D convolution operation on w with a kernel size of k, and then use the Sigmoid activation function to limit the weight size to the (0,1) interval:

[0099] w′=Sigmoid(Conv1D k (w));

[0100] Among them, the convolution kernel size k is adaptively determined by the number of input channels 2C:

[0101]

[0102] Among them, int means rounding down, odd means calculating the nearest odd number, and γ and b are 2 and 1 respectively.

[0103] Let w′∈N 1×1×2C Reset to w″∈N H×W×2C , multiply w″ by A to obtain a feature map after cross-channel feature interaction. After the feature map is subjected to 1x1 convolution, normalization, and nonlinear activation of the ReLu function, the fourth image (channel enhanced feature map) is obtained.

[0104] The channel enhancement feature map is added to the spatial channel attention enhancement feature map to obtain the spatiotemporal difference enhancement feature map, that is, the third image and the fourth image obtained by the dual-branch structure processing are summed to obtain the spatiotemporal difference enhancement feature map. Among them, the third image is the enhancement feature map of the subtraction branch, and the fourth image is the enhancement feature map of the channel alternating splicing branch. The mathematical formula can be expressed as:

[0105]

[0106] i represents the variable of the element in the feature map.

[0107] The feature maps obtained by the dual-branch structure processing are summed to obtain the spatiotemporal difference enhanced feature map (S i ).

[0108] The embodiment provided in the present application introduces a spatiotemporal difference enhancement module with a dual-branch structure, utilizes a channel adaptive enhancement mechanism and a dual attention mechanism to capture the remote context information and local cross-feature information of the changing area in the dual-phase feature map, and can enhance the spatiotemporal difference of the dual-phase image features.

[0109] In step 103, a plurality of spatiotemporal difference enhancement feature maps of different scales are input into a decoder, and the decoder performs edge refinement and feature fusion on the plurality of spatiotemporal difference enhancement feature maps of different scales to obtain a predicted binary image including change information between the front-phase remote sensing image and the back-phase remote sensing image.

[0110] The decoder introduces an edge refinement residual module and an adaptive bidirectional feature fusion module. By utilizing the residual learning of dual-branch edge refinement, the feature maps at different scales can be refined and the representation and learning capabilities of the network can be enhanced, and the feature maps processed by the edge refinement residual block can be obtained. Subsequently, the features at different scales are fused, and the semantic information in the deep network is fully integrated with the position information of the shallow network through the fusion of the top-down processing layer and the bottom-up processing layer. At the same time, the weight parameter η is added in the fusion process to quantify the importance of feature maps at different levels and scales, and a predicted binary image including the change information between the front-phase remote sensing image and the back-phase remote sensing image is obtained.

[0111] In some embodiments, multiple spatiotemporal difference enhancement feature maps of different scales are respectively input into multiple corresponding edge refinement residual modules, and the edges of the changing areas in the spatiotemporal difference enhancement feature maps at different scales are enhanced by the edge refinement residual modules to obtain multiple edge refinement feature maps; the multiple edge refinement feature maps are input into an adaptive bidirectional feature fusion module, and the edge refinement feature maps are fused layer by layer from top to bottom and from bottom to top by the adaptive bidirectional feature fusion module to obtain a fused feature map; the fused feature map is upsampled and channel adjusted to obtain a predicted binary image including the change information between the previous phase remote sensing image and the next phase remote sensing image.

[0112] The edge refinement residual module can perform dual-branch convolution and basic residual operations on multiple input spatiotemporal difference enhancement feature maps of different scales, such as the fifth feature map, the sixth feature map, the seventh feature map and the eighth feature map, to obtain feature maps after edge refinement: the fifth feature map, the sixth feature map, the seventh feature map and the eighth feature map.

[0113] In the embodiment provided by the present application, an edge refinement residual module is introduced to enable the model to learn the differences between feature maps of different scales, thereby optimizing the edges of the changing areas in the spatiotemporal difference enhancement feature maps at different scales.

[0114] The adaptive bidirectional weighted feature fusion module (adaptive bidirectional feature fusion module) is used to perform feature fusion of the fifth, sixth, seventh and eighth feature maps after edge refinement of the input, so that the semantic information in the deep network and the position information in the shallow network are fully integrated to obtain fused feature maps of different scales.

[0115] Exemplarily, the adaptive bidirectional weighted feature fusion module can adopt the PANet model. The PANet model adopts a path aggregation module, ignoring the feature weights of different levels and scales. Features of different levels and scales have inconsistent effects on the fused output results. Therefore, the top-down processing layer combined with the bottom-up processing layer can fully integrate the semantic information in the deep network with the position information of the shallow network. The weight parameter η is added in the fusion process to quantify the importance of feature maps at different levels and scales.

[0116] In the decoder module, first, the fifth, sixth, seventh, and eighth feature maps at different scales output by the encoder are input into the edge thinning residual module to obtain the fifth, sixth, seventh, and eighth feature maps after edge thinning. Then, they are input into the adaptive bidirectional weighted feature fusion module to perform feature fusion by dynamically adjusting the weights of features at different scales to obtain the output fused feature map. Finally, bilinear interpolation is used to perform feature fusion. Figure 4The number of channels of the feature map is adjusted to 2 by upsampling and 1×1 convolution to generate the final binary prediction map.

[0117] The embodiment provided by the present application constructs a decoder by combining an edge thinning residual module and an adaptive bidirectional feature fusion module. The residual connection and feature thinning operation in the edge thinning residual module can suppress the interference of noise, thereby further improving the representation ability and generalization ability of the network; the adaptive bidirectional feature fusion module performs feature fusion by dynamically adjusting the weights of features of different scales, quantifying the importance of features of different scales, and promoting the interaction between rich semantic information in the deep network and rich position information in the shallow network, so as to optimize the edges of the changing areas in the spatiotemporal difference enhancement feature map at different scales to the greatest extent, and improve the representation ability and learning ability of the network. At the same time, the feature weights of different levels and scales are taken into account, and the features of different levels and scales contribute to the detailed information lost in the fused feature output, generating an accurate predicted binary image with a low missed detection rate.

[0118] In some embodiments of the present application, the edge refinement residual module includes a first branch and a second branch. Multiple spatiotemporal difference enhancement feature maps of different scales can be input into the corresponding first branch, and the number of channels of the spatiotemporal difference enhancement feature maps at different scales can be unified through the first branch to obtain multiple standard channel feature maps; multiple standard channel feature maps are input into the corresponding second branch, and the spatiotemporal difference enhancement feature maps of different scales are refined through the second branch to obtain multiple refined feature maps; and multiple standard channel feature maps and refined feature maps are summed respectively to obtain multiple edge refinement feature maps.

[0119] The feature edge refinement residual module adopts a dual-branch structure, which can further optimize the edges of the changing areas in the spatiotemporal difference enhancement feature map at different scales and improve the representation and learning capabilities of the network.

[0120] Alternatively, see Figure 4 , Figure 4 is a schematic diagram of the network structure of the decoder partial feature edge refinement residual module provided by some embodiments of the present application, wherein a plurality of spatiotemporal difference enhancement feature maps of different scales, such as the fifth feature map, the sixth feature map, the seventh feature map, and the eighth feature map, are respectively input into the dual-branch structure of the edge refinement residual module, wherein the first branch uses a 1×1 convolutional layer to transform the feature maps F at different scales into i The number of channels is unified as C2, and this process can be expressed as:

[0121]

[0122] Among them, Conv1 represents the convolution operation with a convolution kernel size of 1, and i represents the variable of the element in the feature map.

[0123] After that, the feature map that has passed the first branch is input into the second branch. The second branch is a basic residual block operation. The input feature map undergoes a 3×3 convolution operation, batch normalization, ReLU activation function, and 3×3 convolution operation. This can refine the feature maps at different scales and enhance the representation and learning capabilities of the network. The mathematical formula is as follows:

[0124]

[0125] Among them, Conv3 represents the convolution operation with a convolution kernel size of 3, BN and ReLU represent batch normalization and activation operations respectively. i and F2 i Sum them up to get the feature map after edge refinement residual block processing:

[0126]

[0127] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the network structure of a decoder partial adaptive bidirectional feature fusion module provided in some embodiments of the present application. Figure 5 The method includes an input feature map, a top-down branch, and a bottom-up branch. In the adaptive bidirectional weighted feature fusion module, the process of obtaining fused feature maps of different scales can be: the fifth feature map, the sixth feature map, the seventh feature map, and the eighth feature map after edge refinement of four different scales are processed hierarchically from top to bottom to obtain a fused feature map at any scale in the second level.

[0128] like Figure 5 As shown in the figure, from top to bottom, the feature maps after edge refinement between different scales at the first level are calculated and fused layer by layer from top to bottom to obtain the feature fusion map F at the second level. 2,i , the specific formula is:

[0129]

[0130] Among them, Conv1 represents the convolution operation with a convolution kernel size of 1, η is a learnable weight parameter, the value of ∈ is 0.00001 to avoid the sum of the weights being 0, and Down is a downsampling operation.

[0131] Optionally, in the bottom-up branch, the feature fusion maps of different scales in the second level are fused layer by layer from bottom to top through formula calculation, so as to obtain the feature fusion map of the third level. The specific formula is:

[0132]

[0133] Among them, Conv1 represents a convolution operation with a convolution kernel size of 1, η' is a learnable weight parameter, the value of ∈ is 0.00001 to avoid the sum of the weights being 0, and Up is an upsampling operation.

[0134] Exemplarily, the decoder includes four edge refinement residual modules, an adaptive bidirectional weighted feature fusion module, a 1x1 convolution layer and an upsampling module, and the fifth feature map, the sixth feature map, the seventh feature map and the eighth feature map are input into the edge refinement residual module to obtain the fifth feature map, the sixth feature map, the seventh feature map and the eighth feature map after edge refinement; the fifth feature map, the sixth feature map, the seventh feature map and the eighth feature map after edge refinement are input into the adaptive bidirectional weighted feature fusion module, and features of different levels are fused at different scales to output a fused feature map with a size and channels of H / 4×W / 4×C2; the fused feature map is then upsampled by 4 times using bilinear interpolation, and finally the number of channels of the feature map is adjusted to 2 using 1×1 convolution to obtain the final predicted binary image.

[0135] In the embodiments provided in the present application, an edge refinement residual module is introduced in the decoder, so that the remote sensing image change detection model can optimize the edges of the changing areas in the spatiotemporal difference enhancement feature map at different scales, and improve the model's ability to capture multiple image features; an adaptive bidirectional weighted feature fusion module is introduced to quantify the importance of features of different scales by adopting learnable weight parameters to achieve effective fusion of multi-scale features.

[0136] Please refer to Figure 6 , Figure 6 The diagram is a comparison diagram of the visualization results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the WHU-CD dataset. The WHU-CD dataset consists of a pair of remote sensing images with a spatial resolution of 0.7m and an image size of 32507×15354. The original image can be cropped into a non-overlapping image of 512×512 pixels and randomly divided into 1461, 183 and 183 pairs of images for training, verification and testing. Figure 6 The images presented in the figure are remote sensing images at different time points on the WHU-CD dataset, labels, various comparison methods, and predicted binary images of the method of the present invention. The white area of ​​the binary image is the changed area, the black area is the unchanged area, the green area is the missed detection area, and the red area is the false detection area.

[0137] Please refer to Figure 7 , Figure 7The figure is a schematic diagram comparing the visualization results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the LEVIR-CD dataset. The LEVIR-CD dataset is a large-scale dataset for building change detection tasks. The original training set, validation set, and test set contain 445, 64, and 128 pairs of images, respectively, with a spatial resolution of 0.5m and an image size of 1024×1024. Each image can be cropped to a non-overlapping image of 512×512 pixels in size, and the final training set, validation set, and test set sizes are 1780, 256, and 512 pairs of images, respectively. Figure 7 The images presented in the figure are remote sensing images at different time points on the LEVIR-CD dataset, labels, various comparison methods, and predicted binary images of the method of the present invention. The white area of ​​the binary image is the changed area, the black area is the unchanged area, the green area is the missed detection area, and the red area is the false detection area.

[0138] Please refer to Figure 8 , Figure 8 The figure is a comparison diagram of the visualization results of the remote sensing image change detection method provided by some embodiments of the present application and the existing remote sensing image change detection method on the SYSU-CD dataset. The SYSU-CD dataset covers various types of land changes, including buildings, vegetation, roads, and waters, and the size of each pair of images is 256×256. The dataset contains 20,000 pairs of 0.5m aerial images taken between 2007 and 2014, and the sizes of the training set, validation set, and test set are 12,000, 4,000, and 4,000 pairs of images, respectively. Figure 8 The images presented in the figure are remote sensing images at different time points on the SYSU-CD dataset, labels, various comparison methods, and predicted binary images of the method of the present invention. The white area of ​​the binary image is the changed area, the black area is the unchanged area, the green area is the missed detection area, and the red area is the false detection area.

[0139] Figures 6 to 8 In the paper, the comparison methods used are divided into the following types, among which Tiny-CD and RFANet are lightweight networks with difference feature semantic enhancement modules. FC Siam diff and SNUNet are fully convolutional neural networks with skip connection structures. ChangeFormer is a pure Transformer architecture network with a multi-head attention mechanism, and BIT is an integrated deep convolutional neural network and Transformer architecture network. TFI_GR is a network with a spatiotemporal difference feature interaction module, and DMINet introduces a cross-attention mechanism and a self-attention mechanism network as well as the network SEAFNet of the present invention.

[0140] from Figure 6, Figure 7 and Figure 8 It can be seen that compared with the other six methods, SEAFNet has the best extraction effect on large-scale changes in large buildings, large-area surface vegetation, etc., and the edge structure of the extracted change area is relatively complete, without obvious false detection or missed detection. Figure 6 New buildings, Figure 7 New parks and Figure 8 In the extraction of newly added vegetation, except for the method SEAFNet of the present invention, other methods have different degrees of missed detection. Figure 6 In the extraction of newly added buildings, except for the comparison method DMINet and the method SEAFNet of the present invention, other methods have different degrees of false detection.

[0141] In terms of small and medium-scale landform changes, Figure 6 , Figure 7 and Figure 8 It can be seen that the method SEAFNet of the present invention has the lowest degree of missed detection and false detection in changes such as small buildings and narrow buildings. Figure 7 The new slender buildings and Figure 8 In the newly added small building change scene, the missed detection degree of other methods is more serious than that of SEAFNet of the present invention.

[0142] In terms of pseudo-change, Figure 7 and Figure 8 It can be seen that compared with other methods, the method SEAFNet of the present invention has the lowest false detection degree in pseudo-change scenarios, for example Figure 7 In the pseudo-change scene caused by seasonal factors, other methods all showed different degrees of false detection. The method ChangeFormer with long-distance image dependency performed the worst, while the method SEAFNet of the present invention performed the best.

[0143] Please refer to Figure 9(a) to Figure 9(c) , Figure 9(a) to Figure 9(c)The schematic diagrams respectively show the computational complexity comparison results of the remote sensing image change detection method SEAFNet provided by the embodiment of the present application and the existing remote sensing image change detection method on the WHU-CD dataset, LEVIR-CD dataset and SYSU-CD dataset. Among them, the horizontal axis represents the FLOPs of the model, and the vertical axis represents the Iou value of the model. It can be seen that the method of the present invention achieves the best Iou performance with the third best FLOPs, and the parameter amount has certain advantages. Although the FLOPs and parameter amounts of the Tiny-CD and REFNet models are lower than those of SEAFNet, the quantitative and qualitative analysis results of SEAFNet are better. The TFI-GR and DMINet models embedded with the spatiotemporal difference interaction module perform relatively well on all datasets, but they still have certain deficiencies in processing multi-scale feature maps, and the FLOPs are higher than the method in this paper. . Based on the above analysis, it is fully explained that in the remote sensing image change detection task, enhancing the difference between dual-phase images and effectively fusing feature maps at different scales are particularly critical, and the remote sensing image change detection method provided by the embodiment of the present application has achieved the best balance in terms of model calculation efficiency and accuracy.

[0144] The above is an introduction to an embodiment of the remote sensing image change detection method of the present application.

[0145] The implementation of the remote sensing image change detection device provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0146] Regarding the remote sensing image change detection method provided in the above embodiment, the present application embodiment also provides a remote sensing image change detection device for implementing the above method, such as Fig.10 As shown, Fig.10 This is a schematic block diagram of a module of a remote sensing image change detection device according to an embodiment of the present application. The remote sensing image change detection device includes:

[0147] An acquisition module is used to acquire a front-phase remote sensing image and a back-phase remote sensing image, wherein the front-phase remote sensing image is used to represent a surface feature image of a target area at a first time point, and the back-phase remote sensing image is used to represent a surface feature image of a target area at a second time point;

[0148] A feature extraction generation module is used to input the front-phase remote sensing image and the back-phase remote sensing image into an encoder to obtain a plurality of spatiotemporal difference enhancement feature maps of different scales, wherein the encoder includes a feature extraction module and a plurality of spatiotemporal difference enhancement modules, the feature extraction module is used to extract features from the front-phase remote sensing image and the back-phase remote sensing image to generate a plurality of dual-phase feature maps of different scales, and the plurality of spatiotemporal difference enhancement modules are used to capture the remote context information and local cross-feature information of the changing areas in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhancement feature maps of different scales;

[0149] The feature refinement and fusion module is used to input multiple spatiotemporal difference enhancement feature maps of different scales into the decoder, and the decoder performs edge refinement and feature fusion on multiple spatiotemporal difference enhancement feature maps of different scales to obtain a predicted binary image including the change information between the previous phase remote sensing image and the next phase remote sensing image.

[0150] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] like Fig.11 As shown, the embodiment of the present application also provides an electronic device, the electronic device includes a memory, one or more processors ( Fig.11 Only one is shown) and a computer program stored in the memory and executable on the processor. Wherein: the memory is used to store software programs and units, and the processor executes various functional applications and data processing by running the software programs and units stored in the memory to obtain the resources corresponding to the above-mentioned preset events. Optionally, the processor implements the above-mentioned remote sensing image change detection method by running the above-mentioned computer program stored in the memory.

[0152] The memory, as a non-transitory computer-readable medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network.

[0153] It can be understood that the contents of the above method embodiments are all applicable to the electronic device embodiment, the functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0154] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned remote sensing image change detection method is implemented.

[0155] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it can implement the steps of the remote sensing image change detection method.

[0157] It can be understood that the contents of the above method embodiments are all applicable to the present computer program product, the functions specifically implemented by the present computer program product embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0158] The remote sensing image change detection method, device, electronic device, medium and computer program product provided in the embodiments of the present application extract features in the front-phase remote sensing image and the back-phase remote sensing image through a feature extraction module to generate multiple dual-phase feature maps of different scales. After that, the remote context information and local cross-feature information of the change area in the dual-phase feature maps of multiple scales are captured through multiple spatiotemporal difference enhancement modules to generate multiple spatiotemporal difference enhanced feature maps of different scales. The global change information and local fine-grained change information between the dual-phase feature maps can be learned at the same scale, and the spatiotemporal difference of the dual-phase image features extracted by the encoder can be enhanced. The edge refinement and feature fusion of the multiple spatiotemporal difference enhanced feature maps of different scales are performed by the decoder, so that the rich position information in the shallow feature map and the rich semantic information in the deep feature map are effectively fused, while taking into account the feature weights of different levels and scales. The features of different levels and scales contribute to the detailed information lost in the fused feature output, thereby improving the accuracy of remote sensing image change detection.

[0159] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0160] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0161] Some aspects of the present disclosure are described above with reference to the block diagram and flow chart of the system, method, system and / or computer program product according to the exemplary embodiments.It should be understood that the combination of one or more blocks in the block diagram and the flow chart and the blocks in the block diagram and the flow chart can be realized by executing computer executable program instructions respectively.Equally, according to some embodiments, some blocks in the block diagram and the flow chart may not need to be executed in the order shown, or may not need to be executed in full.In addition, additional components and / or operations beyond those components and / or operations shown in the blocks in the block diagram and the flow chart may be present in certain embodiments.

[0162] Therefore, the blocks in the block diagrams and flow charts support the combination of devices for performing the specified functions, the combination of elements or steps for performing the specified functions, and the program instruction devices for performing the specified functions. It should also be understood that each block in the block diagrams and flow charts and the combination of blocks in the block diagrams and flow charts can be implemented by a special-purpose hardware computer system or a combination of special-purpose hardware and computer instructions that performs a specific function, element or step.

[0163] The program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0164] Software components can be encoded with any of various programming languages. An exemplary programming language can be a low-level programming language, such as an assembly language associated with a specific hardware architecture and / or an operating system platform. The software component comprising assembly language instructions may need to be converted to executable machine code by an assembler before being executed by hardware architecture and / or platform. Another exemplary programming language can be a higher-level programming language, which can be transplanted across multiple architectures. The software component comprising a higher-level programming language may need to be converted to an intermediate representation by an interpreter or a compiler before execution. Other examples of programming languages ​​include but are not limited to macro languages, shells or command languages, job control languages, scripting languages, database queries or search languages ​​or report writing languages. In one or more exemplary embodiments, the software component comprising the instruction of one of the above-mentioned programming language examples can be directly executed by an operating system or other software components, without first being converted into another form.

[0165] Software components may be stored as files or other data storage structures. Software components of similar type or related functions may be stored together, such as in a specific directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).

[0166] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A remote sensing image change detection method, characterized in that: The following steps are involved: Acquire a front-phase remote sensing image and a back-phase remote sensing image, wherein the front-phase remote sensing image is used to represent a surface feature image of a target area at a first time point, and the back-phase remote sensing image is used to represent a surface feature image of the target area at a second time point; Input the front-phase remote sensing image and the back-phase remote sensing image into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, wherein the encoder includes a feature extraction module and a plurality of spatiotemporal difference enhancement modules, the feature extraction module is used to extract features from the front-phase remote sensing image and the back-phase remote sensing image to generate a plurality of dual-phase feature maps of different scales, and the plurality of spatiotemporal difference enhancement modules are used to capture remote context information and local cross-feature information of the changing areas in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhanced feature maps of different scales; The spatiotemporal difference enhancement feature maps of multiple different scales are input into a decoder, and the decoder performs edge refinement and feature fusion on the spatiotemporal difference enhancement feature maps of multiple different scales to obtain a predicted binary image including change information between the front-phase remote sensing image and the back-phase remote sensing image.

2. The remote sensing image change detection method according to claim 1, characterized in that: The feature extraction module includes two parallel feature extraction branches, and the front-phase remote sensing image and the back-phase remote sensing image are input into an encoder to obtain a plurality of spatiotemporal difference enhancement feature maps of different scales, including: Inputting the front-phase remote sensing image and the back-phase remote sensing image into two parallel feature extraction branches respectively, and extracting a plurality of dual-phase feature maps of different scales through the feature extraction network in the feature extraction branches; The dual-phase feature maps of multiple different scales are respectively input into the corresponding spatiotemporal difference enhancement modules to obtain the spatiotemporal difference enhancement feature maps of multiple different scales.

3. The remote sensing image change detection method according to claim 2, characterized in that: The spatiotemporal difference enhancement module includes a subtraction branch, and the dual-phase feature map is input into the spatiotemporal difference enhancement module to obtain the spatiotemporal difference enhancement feature map, including: Inputting the dual-phase feature graph into the subtraction branch, and calculating the absolute difference feature between the dual-phase feature graphs in the subtraction branch; Inputting the absolute difference feature into a dual attention mechanism module, capturing the remote context information of the changing area in the dual-phase feature map through the dual attention mechanism module, and obtaining a spatial attention enhanced feature map and a channel attention enhanced feature map; Perform channel concatenation on the spatial attention enhancement feature map and the channel attention enhancement feature map to obtain a spatial channel attention enhancement feature map; The spatial channel attention enhancement feature map is added to the channel enhancement feature map to obtain the spatiotemporal difference enhancement feature map.

4. The remote sensing image change detection method according to claim 3, characterized in that: The dual attention mechanism module includes a spatial attention mechanism module and a channel attention mechanism module, and the absolute difference feature is input into the dual attention mechanism module, and the remote context information of the change area in the dual-phase feature map is captured by the dual attention mechanism module to obtain a spatial attention enhanced feature map and a channel attention enhanced feature map, including: Inputting the absolute difference feature into the spatial attention mechanism module, establishing the contextual relationship of the local feature through the spatial attention mechanism module, and obtaining the spatial attention enhanced feature map; The absolute difference feature is input into the channel attention mechanism module, and the long-range dependency relationship between the feature map channels is established through the channel attention mechanism module to obtain the channel attention enhanced feature map.

5. The remote sensing image change detection method according to claim 2, characterized in that: The spatiotemporal difference enhancement module includes a channel alternating splicing branch, and the dual-phase feature map is input into the spatiotemporal difference enhancement module to obtain a plurality of spatiotemporal difference enhancement feature maps of different scales, including: Inputting the bi-phase feature map into the channel alternating splicing branch, and performing alternating splicing on the bi-phase feature map in the channel alternating splicing branch to obtain a splicing feature; Inputting the spliced ​​features into a channel adaptive enhancement mechanism module, and enhancing the local cross-feature interaction information of the changing area in the dual-phase feature map through the channel adaptive enhancement mechanism module to obtain a channel enhanced feature map; The channel enhancement feature map is added to the spatial channel attention enhancement feature map to obtain the spatiotemporal difference enhancement feature map.

6. The remote sensing image change detection method according to claim 1, characterized in that: The step of inputting the spatiotemporal difference enhancement feature maps of multiple different scales into a decoder, and performing edge refinement and feature fusion on the spatiotemporal difference enhancement feature maps of multiple different scales through the decoder includes: Inputting the spatiotemporal difference enhancement feature maps of multiple different scales into multiple corresponding edge thinning residual modules respectively, and enhancing the edges of the changing areas in the spatiotemporal difference enhancement feature maps of different scales by the edge thinning residual modules to obtain multiple edge thinning feature maps; Inputting a plurality of the edge refinement feature maps into an adaptive bidirectional feature fusion module, and fusing the edge refinement feature maps layer by layer from top to bottom and from bottom to top through the adaptive bidirectional feature fusion module to obtain a fused feature map; The fused feature map is upsampled and channel-adjusted to obtain a predicted binary image including change information between the front-phase remote sensing image and the back-phase remote sensing image.

7. The remote sensing image change detection method according to claim 6, characterized in that: The edge refinement residual module includes a first branch and a second branch, and the plurality of spatiotemporal difference enhancement feature maps of different scales are respectively input into a plurality of corresponding edge refinement residual modules, and the edges of the change areas in the spatiotemporal difference enhancement feature maps of different scales are enhanced by the edge refinement residual modules to obtain a plurality of edge refinement feature maps, including: Inputting the spatiotemporal difference enhancement feature maps of multiple different scales into the corresponding first branch, unifying the number of channels of the spatiotemporal difference enhancement feature maps at different scales through the first branch, and obtaining multiple standard channel feature maps; Inputting the plurality of standard channel feature maps into the corresponding second branch, and refining the spatiotemporal difference enhancement feature maps of different scales through the second branch to obtain a plurality of refined feature maps; The plurality of standard channel feature maps and the refined feature maps are summed up respectively to obtain a plurality of edge refined feature maps.

8. A remote sensing image change detection device, characterized in that: The device comprises: An acquisition module, used to acquire a front-phase remote sensing image and a back-phase remote sensing image, wherein the front-phase remote sensing image is used to represent a surface feature image of a target area at a first time point, and the back-phase remote sensing image is used to represent a surface feature image of the target area at a second time point; A feature extraction generation module, used for inputting the front-phase remote sensing image and the back-phase remote sensing image into an encoder to obtain a plurality of spatiotemporal difference enhanced feature maps of different scales, wherein the encoder includes a feature extraction module and a plurality of spatiotemporal difference enhancement modules, the feature extraction module is used for extracting features from the front-phase remote sensing image and the back-phase remote sensing image to generate a plurality of dual-phase feature maps of different scales, and the plurality of spatiotemporal difference enhancement modules are used for capturing remote context information and local cross-feature information of changing areas in the dual-phase feature maps of different scales to generate a plurality of spatiotemporal difference enhanced feature maps of different scales; The feature refinement and fusion module is used to input the spatiotemporal difference enhancement feature maps of multiple different scales into a decoder, and perform edge refinement and feature fusion on the spatiotemporal difference enhancement feature maps of multiple different scales through the decoder to obtain a predicted binary image including the change information between the front-phase remote sensing image and the back-phase remote sensing image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the remote sensing image change detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the remote sensing image change detection method according to any one of claims 1 to 7 is implemented.

Images