Remote sensing change detection method and system based on multi-modal space-time difference feature enhancement

Abstract

The invention discloses a remote sensing change detection method, system, equipment, medium and program based on multi-mode space-time difference feature enhancement, and belongs to the technical field of remote sensing image processing. The method comprises the following steps: constructing a data set required by change detection based on an earth observation satellite image; and inputting the test set of the data set required by change detection into the pre-trained change detection model, and processing to obtain a change detection result. Through an efficient multi-modal space-time difference feature enhancement strategy, a flexible network architecture design and a wide backbone network, the method has excellent change detection performance, and meanwhile, the simplicity and universality of a network structure are kept.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing change detection method, system, equipment, medium and program based on multimodal spatiotemporal differential feature enhancement. Background Art

[0002] With the rapid development of remote sensing technology, change detection, as an important means of monitoring dynamic changes on the earth's surface, is gradually shifting from single-modality to multi-modality. Single-modality methods mainly rely on a single type of satellite sensor data. Although this method can meet basic needs to a certain extent, its limitations are becoming increasingly prominent. For example, when satellite performance is limited or the natural environment is harsh, single-modality methods often face severe challenges such as poor data quality and missing data. These problems not only affect the accuracy of change detection, but also limit its breadth and depth in practical applications.

[0003] In order to overcome the limitations of single-modal methods, multimodal methods have emerged. By fusing data from different types of sensors, multimodal methods can fully utilize the information advantages of different modalities, thereby improving the performance of change detection. For example, in the field of remote sensing image change detection, the multi-source, multi-scale and multi-resolution characteristics of multi-temporal remote sensing images provide researchers with a rich source of information. These data from different sensors can work together to reveal the dynamic changes on the earth's surface through reasonable fusion methods.

[0004] However, although multimodal methods have broad application prospects in the field of remote sensing, their development also faces many technical challenges. First, the inherent pattern changes of different sensors and different spatial resolutions are different, which brings great difficulties to the fusion of multimodal data. How to effectively fuse these heterogeneous data and extract useful change information is a technical problem that needs to be solved urgently. Secondly, although the existing multimodal change detection technology has achieved certain results, it still has many shortcomings. For example, although some methods can realize land cover change detection of dual-time heterogeneous remote sensing images, the computational requirements are very high and it is difficult to promote them on a large scale in practical applications. Although other methods have proposed pixel-level data fusion solutions for multi-source data, the scalability of the model still needs to be strengthened and cannot adapt to data changes of different types and scales.

[0005] Therefore, how to reduce computational costs and improve model scalability while ensuring change detection performance has become a technical problem that needs to be urgently solved in the field of remote sensing image change detection. Summary of the invention

[0006] In response to the problem in the prior art of how to reduce computing costs and improve model scalability while ensuring change detection performance, the present invention provides a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement. Through an efficient multimodal spatiotemporal differential feature enhancement strategy, a flexible network architecture design and a wide backbone network, the method has excellent change detection performance while maintaining the simplicity and versatility of the network structure.

[0007] In order to achieve the above object, the present invention provides the following technical solutions.

[0008] In a first aspect, the present invention provides a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement, comprising: Build the dataset required for change detection based on earth observation satellite images; Input the test set of the data set required for change detection into the pre-trained change detection model and process it to obtain the change detection result; The training method of the pre-trained change detection model comprises: Based on the collection of earth observation satellite images, the dataset required for change detection is constructed; Preprocessing a training set in a data set required for change detection to obtain preprocessed training data; The preprocessed training data is input into the remote sensing change detection network to obtain multimodal spatiotemporal differential features; The change detection model is trained based on the multimodal spatiotemporal differential features to obtain a trained change detection model; The test set in the data set required for change detection is input into the trained change detection model to obtain the change detection result.

[0009] As a further improvement of the present invention, the step of constructing a data set required for change detection based on collecting earth observation satellite images includes: According to satellite acquisition, different synthetic aperture radar images and multispectral images covering the same area at a single time node; Perform image registration on synthetic aperture radar images and multispectral images, mark and generate change detection labels, and obtain the data set required for change detection; The dataset required for change detection is divided into training set, validation set and test set.

[0010] As a further improvement of the present invention, the preprocessing of the training set in the data set required for change detection to obtain the preprocessed training data includes: The synthetic aperture radar images and multispectral images in the training set of the data set required for change detection are resized by bilinear interpolation, and the label map of change detection is resized by nearest neighbor interpolation to obtain the resized image; The adjusted image is flipped left to right to generate new training samples and obtain preprocessed training data.

[0011] As a further improvement of the present invention, the pre-processed training data is input into the remote sensing change detection network enhanced with multi-modal spatiotemporal differential features to obtain multi-modal spatiotemporal differential features, including: The preprocessed training data is fed into a remote sensing change detection network enhanced with multimodal spatiotemporal differential features, and the absolute value difference between the multispectral image after the change and the synthetic aperture radar image before the change, and between the synthetic aperture radar image after the change and the multispectral image before the change, is calculated to obtain multimodal differential features; Calculate the absolute value difference between the multispectral image after the change and the multispectral image before the change, and the absolute value difference between the synthetic aperture radar image after the change and the synthetic aperture radar image before the change, and obtain the spatiotemporal differential features; The multimodal differential features and spatiotemporal differential features are integrated to obtain the multimodal spatiotemporal differential features.

[0012] As a further improvement of the present invention, the change detection model is trained based on the multimodal spatiotemporal differential features to obtain the trained change detection model, including: The remote sensing change detection network is trained based on multimodal spatiotemporal differential features, and the learning rate and weight decay parameters are adjusted to obtain a change detection model. Using the training set in the data set required for change detection, the change detection model is trained to obtain a training result; The training results are verified using the verification set in the data set required for change detection. If the verification is qualified, the trained change detection model is obtained.

[0013] As a further improvement of the present invention, the step of inputting a test set in a data set required for change detection into a trained change detection model to obtain a change detection result includes: Input the test set in the data set required for change detection into the trained change detection model and output the change detection graph; Determine whether the output change detection map has input of a change detection label; If there is no input of change detection label, the change detection result includes output change detection map; If there is an input of change detection label, the change detection results include change detection map, intersection-over-union ratio of each category, the average intersection-over-union ratio of all categories, F1 score of each category, the average F1 score of all categories, and overall accuracy.

[0014] In a second aspect, the present invention provides a remote sensing change detection system based on multimodal spatiotemporal differential feature enhancement, comprising: Building dataset module: used to build the dataset required for change detection based on earth observation satellite images; Detection result module: used to input the test set of the data set required for change detection into the pre-trained change detection model and process it to obtain the change detection result; The training system of the pre-trained change detection model comprises: Acquisition dataset module: used to build the dataset required for change detection based on the collected earth observation satellite images; Preprocessing data module: used to preprocess the training set in the data set required for change detection to obtain preprocessed training data; Feature acquisition module: used to input the preprocessed training data into the remote sensing change detection network to obtain multimodal spatiotemporal differential features; Detection model module: used to train the change detection model based on multimodal spatiotemporal differential features to obtain the trained change detection model; Obtain result module: used to input the test set in the data set required for change detection into the trained change detection model to obtain the change detection result.

[0015] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement when executing the computer program.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement are implemented.

[0017] In a fifth aspect, the present invention provides a computer program product, characterized in that it includes computer instructions, which, when executed by a processor, implement the steps of the remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention first provides a rich and accurate information source for change detection by constructing an earth observation satellite image dataset, thereby ensuring the effectiveness and reliability of model training and testing. In the model training stage, by introducing a preprocessing step, not only the quality and consistency of the data are improved, but also the model is helped to better capture the characteristics of changes. The core lies in that the present invention innovatively adopts a multimodal spatiotemporal differential feature enhancement strategy, which can make full use of multi-source information in remote sensing images and dynamic changes in time series to effectively extract richer and more discriminative feature representations. These features not only enhance the model's sensitivity to changing areas, but also improve the ability to recognize different types of changes, thereby significantly reducing the computational complexity while maintaining high performance, and avoiding the problem of excessive resource consumption caused by feature redundancy or improper processing in traditional methods.

[0019] Furthermore, the remote sensing change detection network architecture designed by the present invention is flexible and supports the combined use of multiple backbone networks and decoders, which not only gives the model strong adaptability and scalability, but also allows users to flexibly adjust the network structure to achieve optimal performance according to actual application scenarios and data characteristics. This modular and configurable design concept has greatly promoted the popularization and application of technology, lowered the technical threshold, and enabled even non-professional users to easily deploy and customize a change detection system that meets their needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In the drawings: Figure 1 It is a flow chart of a remote sensing change detection method based on multi-modal spatiotemporal differential feature enhancement of the present invention; Figure 2 A schematic diagram of a flow chart of a training method for a pre-trained change detection model of the present invention; Figure 3 It is a structural schematic diagram of a remote sensing change detection system based on multi-modal spatiotemporal differential feature enhancement of the present invention; Figure 4 A schematic diagram of the structure of a training system for a pre-trained change detection model of the present invention; Figure 5 Schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0023] In view of the problem in the prior art of how to reduce the computational cost and improve the scalability of the model while ensuring the change detection performance, the present invention provides a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement, such as Figure 1 As shown, the method includes: S100: Building a dataset for change detection based on Earth observation satellite images; S200: inputting a test set of a data set required for change detection into a pre-trained change detection model, and processing the model to obtain a change detection result; like Figure 2 As shown, the training method of the pre-trained change detection model includes: S201: constructing a data set required for change detection based on the collected earth observation satellite images; S202: preprocessing a training set in a data set required for change detection to obtain preprocessed training data; S203: inputting the preprocessed training data into the remote sensing change detection network to obtain multimodal spatiotemporal differential features; S204: training a change detection model based on the multimodal spatiotemporal differential features to obtain a trained change detection model; S205: Input the test set in the data set required for change detection into the trained change detection model to obtain the change detection result.

[0024] The present invention achieves excellent change detection performance while maintaining the simplicity and versatility of the network structure through an efficient multimodal spatiotemporal differential feature enhancement strategy, a flexible network architecture design, and a wide range of backbone networks.

[0025] The present invention is further explained below.

[0026] The present invention provides a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement, such as Figure 1 As shown, including: S1: We build a usable dataset by collecting Sentinel-1 and Sentinel-2 images covering the same area but at different time points, manually registering the images, and manually generating change detection labels.

[0027] Specifically, in this embodiment, the synthetic aperture radar (SAR) image obtained from the Sentinel-1 satellite includes two channels, VV and VH. When constructing the data set, a third channel VV / VH is added to form a three-channel image. The multispectral image (MS) obtained from the Sentinel-2 satellite contains 13 bands from visible light to short-wave infrared. In the present invention, RGB three bands are selected to form a three-channel image. The ENVI platform is used for image registration, and then the change detection label is generated by manual marking. Finally, the data set is divided into a training set, a validation set, and a test set. Each set of data in the data set includes SAR and MS images before the change, SAR and MS images after the change, and a change detection label map.

[0028] This embodiment uses the method of manual image registration and manual generation of change detection labels to construct a high-quality, accurate and reliable change detection dataset, thereby scientifically and rationally verifying the method.

[0029] S2: Send the training set to the training set preprocessing module for preprocessing; S21: Enhanced operation of adjusting image size.

[0030] Specifically, by scaling the images, the diversity of image sizes in the dataset is increased, thereby helping the model learn feature representations at different scales.

[0031] Specifically, for SAR and MS images, images are resized by bilinear interpolation, and for label maps for change detection, images are resized by nearest neighbor interpolation.

[0032] S22: Random cropping augmentation operation.

[0033] Specifically, the cropping operation simulates the local observation scenarios that may be encountered in practical applications by randomly selecting a sub-region on the image as the new input.

[0034] S23: Horizontal flip enhancement operation.

[0035] Specifically, horizontal flipping is a simple and effective data augmentation method that generates new training samples by flipping the image left and right. This step not only increases the diversity of the dataset, but also helps the model learn spatial invariance features.

[0036] S3: Constructing a remote sensing change detection network with enhanced multimodal spatiotemporal differential features.

[0037] S31: Build a backbone network. Select a series of advanced deep learning models as the backbone network Specifically, it aims to capture the multi-level and multi-dimensional features in remote sensing images. These backbone networks include: MiT (Mix Transformer): combines the advantages of convolutional neural networks (CNN) and Transformer, and can effectively capture global context information while maintaining the local receptive field; PVTv2 (Pyramid Vision Transformer v2): By introducing a pyramid structure, the Transformer model can process multi-scale features and is suitable for input images of different resolutions. LSKNet: A lightweight CNN architecture designed for efficient feature extraction in resource-constrained environments. Vision Transformer (ViT): directly applies the Transformer model in natural language processing to image recognition tasks, demonstrating powerful feature representation capabilities; Swin Transformer and Swin Transformerv2: By introducing window and shift mechanisms, the computational efficiency and feature extraction capabilities of ViT are further optimized; ConvNeXt: While maintaining the simplicity of CNN, it draws on the design concept of Transformer to improve the generalization performance of the model; FocalNet: Focuses on the attention mechanism of feature pyramid to enhance feature representation by focusing on key areas; InternImage: A novel Transformer architecture designed to enhance image recognition performance through internal representation learning. VMamba: A Transformer variant that combines multiple optimization techniques to improve the robustness and efficiency of the model. UniRepLKNet: A network that emphasizes unified representation learning and can process multiple types of data simultaneously, suitable for multimodal remote sensing image analysis.

[0038] S32: Constructing the neck network.

[0039] Specifically, the neck network acts as a bridge between the backbone network and the decoder, responsible for further processing and fusing features to enhance the model's recognition ability for complex changing scenes.

[0040] For PVTv2, we constructed FPN (Feature Pyramid Network), which effectively integrated features of different scales through the feature pyramid structure to improve the multi-scale perception ability of the model. For ViT, we introduced multilevelneck, which enables the model to make full use of the global context information of Transformer while maintaining sensitivity to local details through cross-level feature interaction and enhancement.

[0041] S33: Construct multimodal spatiotemporal differential features, and construct multimodal spatiotemporal differential features by using the four features obtained by passing the MS and SAR images before and after the input changes through the backbone network or the backbone network plus the neck network.

[0042] S331: Obtain multimodal differential features, calculate the absolute value difference between the features obtained from the MS image after the change and the features obtained from the SAR image before the change, and calculate the absolute value difference between the features obtained from the SAR image after the change and the features obtained from the MS image before the change. These two are the multimodal differential features.

[0043] S332: obtaining spatiotemporal differential features, calculating the absolute value difference between the features obtained from the MS image after the change and the features obtained from the MS image before the change, and calculating the absolute value difference between the features obtained from the SAR image after the change and the features obtained from the SAR image before the change. These two are the spatiotemporal differential features.

[0044] S333: Obtain multimodal spatiotemporal differential features, and take the average of the four features obtained in step S331 to obtain multimodal spatiotemporal differential features.

[0045] S34: Construct a decoder.

[0046] Specifically, the decoder is responsible for converting the feature map output by the neck network or the backbone network into the final change detection result map. In order to match the output characteristics of different backbone networks and meet the requirements of the change detection task, the present invention configures a corresponding decoder for each backbone network: For MiT, we chose SegFormer, a decoder that combines a lightweight Transformer with a progressive upsampling strategy to achieve efficient and accurate semantic segmentation. For PVTv2, Semantic FPN is used to improve the decoder's ability to understand complex scenes by further fusing multi-scale features; For LSKNet, UNetFormer is configured, which combines the structural advantages of UNet and the feature representation ability of Transformer, and is suitable for change detection in high-resolution images; For ViT, Swin Transformer, Swin Transformerv2, ConvNeXt, FocalNet, InternImage, VMamba and UniRepLKNet, UPerNet is uniformly adopted. The decoder improves the model's ability to handle boundaries and details by enhancing the feature representation in the upsampling path, and is suitable for tasks that require accurate change boundary detection.

[0047] Among them, S33: construct multimodal spatiotemporal differential features. Make full use of the multimodal information of MS and SAR images. The multimodal differential features make full use of the complementarity between different sensor data. MS images usually provide rich spectral information, while SAR images are not affected by lighting and weather conditions and can provide physical structure information of the surface. This cross-modal difference analysis helps to capture more comprehensive information on surface changes. At the same time, the spatiotemporal differential features enhance the spatiotemporal sensitivity and can capture subtle changes in the same area at different time points, which is crucial for detecting dynamic changes in the surface. The integration of multimodal differential features and spatiotemporal differential features retains the advantages of each feature, improves detection accuracy and reliability, and enhances the generalization ability of the model.

[0048] S31, S32, and S34 construct the model part. In order to further enhance the comprehensiveness and versatility of the present invention, the present invention adopts diversified design schemes in both the backbone network and the decoder part. At the backbone network level, a variety of advanced convolutional neural network architectures and emerging Transformer models are integrated. These models have their own characteristics and can extract feature representations at different levels to meet the needs of different application scenarios. By flexibly configuring these backbone networks, the present invention can cope with surface cover change detection tasks ranging from simple to complex and varied. In the decoder construction part, a variety of decoders are also introduced to achieve refined decoding of fused features and generate more accurate and detailed change detection result maps. This multi-mode parallel design concept not only improves the scope of application of the present invention, but also provides a broad space for its customized optimization in practical applications.

[0049] S4: Train the network.

[0050] In this series of experiments, in order to ensure that different network architectures can achieve optimal performance on their respective tasks, the widely used optimizer AdamW was uniformly adopted, but the specific learning rate and weight decay parameters of each network were carefully adjusted.

[0051] Specifically, for advanced Transformer-based architectures such as MiT, ViT, Swin Transformer, and VMamba, the present invention sets the learning rate to 0.00006 and the weight decay to 0.01; for PVTv2, a variant of Vision Transformer that combines a pyramid structure, a slightly higher learning rate of 0.0001 and a lower weight decay of 0.0001 are adopted; LSKNet, as a lightweight CNN architecture, has its learning rate set to 0.0004 and weight decay to 0.01; for ConvNeXt and FocalNet, two networks that focus on feature representation and focal area learning, the learning rate is set to 0.0001, and the weight decay is increased to 0.05; for different scale variants of InternImage, the parameter settings are further refined according to the size and complexity of the model. Specifically, for the three smaller versions, Tiny, Small, and Base, the learning rate and weight decay are set to 0.00006 and 0.05, respectively; for the three larger versions, Large, xLarge, and Huge, in order to maintain the stability of training and avoid overfitting, the learning rate is reduced to 0.00002, while the weight decay is kept at 0.05; for Swin Transformer v2, the learning rate and weight decay are set to 0.00006 and 0.05, respectively; for UniRepLKNet, a network designed to achieve unified representation learning, a learning rate of 0.0001 and a weight decay of 0.05 were selected.

[0052] S5: Test set preprocessing S51: Adjust the image size enhancement operation.

[0053] Specifically, for satellite images from Sentinel-1 and Sentinel-2, the image size is adjusted by bilinear interpolation. During the test process, the label map of change detection is used to generate the value of the evaluation index, which is not required to be input. If the label map of change detection is input, the image size is adjusted by nearest neighbor interpolation.

[0054] S6: Testing the neural network.

[0055] Input the processed test set into the trained network. If there is no input of change detection label, only the change detection map is output.

[0056] If the "model soup" technique is used in the inference phase, the following three steps are required: The first step is to load the weights, traverse the path list, and load the model weights under each path.

[0057] The second step is to merge the weights. For each key-value pair in the first weight file, accumulate the values ​​of the same key in other weight files and calculate the average (or calculate the average after converting the type).

[0058] The third step is to save the results and save the merged weights to a new file.

[0059] If the "model soup" technique is not used during the inference phase, only the weights of the model to be inferred need to be loaded.

[0060] If there is an input of change detection label, in addition to outputting the change detection map, it is also necessary to output evaluation indicators such as the intersection over union (IOU) of each category, the average of the intersection over union (mIOU) of all categories, the F1 score of each category, the average of the F1 scores of all categories (mF1), and the overall accuracy (OA).

[0061] The details are as follows: Before introducing the formula for each evaluation indicator, you need to understand the following concepts: TP: stands for correct prediction. Specifically, it means that the model predicts that it is a positive example, but it is actually a positive example. FP: stands for prediction error. Specifically, it means that the model predicts a positive example, but it is actually a negative example. FN: stands for prediction error. Specifically, it means that the model predicts a negative example, but it is actually a positive example. TN: represents a correct prediction. Specifically, it represents the case where the model predicts a counterexample, but it is actually a counterexample.

[0062] Precision: Precision, which indicates how many samples predicted to be positive are actually positive. The expression is:

[0063] Recall: Recall rate, which indicates how many positive examples are correctly predicted. The expression is:

[0064] According to Precision and Recall, we can get F1:

[0065] Since change detection is actually a two-classification problem, mF1 can be obtained based on the F1 of the two categories:

[0066] in, and Represents the F1 value of the two categories respectively.

[0067] According to TP, FN, FP, IoU can be obtained:

[0068] Similarly, according to the IoU of two categories, mIoU can be obtained:

[0069] in, Represents the IoU value of the two categories respectively.

[0070] According to TP, FN, FP, and TN, OA can be obtained:

[0071] Specifically, the remote sensing image change detection in this embodiment refers to the use of remote sensing images covering the same area but at different time nodes to detect and analyze the dynamic evolution of surface cover. Its application areas widely cover dynamic resource monitoring, disaster warning, urban planning, and flood detection. The core goal of the present invention is to propose an innovative solution to the performance bottlenecks, complex network structures, and lack of versatility in current multimodal remote sensing image change detection technology. This method aims to achieve a multimodal remote sensing image change detection method that has excellent change detection performance while maintaining simplicity and versatility of network structure through an efficient multimodal spatiotemporal differential feature enhancement strategy, a flexible network architecture design, and a wide range of backbone networks and decoder selections.

[0072] The "model soup" technology was introduced in S6. It can improve model stability: by merging the weights of multiple training checkpoints, the fluctuation of model performance can be reduced, making the final model more stable. At the same time, it can improve generalization ability: model fusion can reduce overfitting to a certain extent and improve the performance of the model on unseen data. And it can simplify model selection: there is no need to manually select the best single checkpoint, but use the information of all checkpoints to generate a comprehensive and more stable model. The "model soup" technology of the present invention is easy to implement: it can be implemented using PyTorch's state_dict and simple Python code, without the need for complex mathematics or programming skills. The second object of the present invention is to propose a remote sensing change detection system based on multimodal spatiotemporal differential feature enhancement, such as Figure 3 As shown, including: The dataset building module 100 is used to build a dataset required for change detection based on earth observation satellite images; Detection result module 200: used to input the test set of the data set required for change detection into the pre-trained change detection model, and process it to obtain the change detection result; like Figure 4As shown, the training system of the pre-trained change detection model includes: The module 201 for acquiring data sets is used to construct data sets required for change detection based on the collected earth observation satellite images; The data preprocessing module 202 is used to preprocess the training set in the data set required for change detection to obtain preprocessed training data; Feature acquisition module 203: used to input the pre-processed training data into the remote sensing change detection network to obtain multi-modal spatiotemporal differential features; Detection model module 204: used to train the change detection model based on the multimodal spatiotemporal differential features to obtain a trained change detection model; The result acquisition module 205 is used to input the test set in the data set required for change detection into the trained change detection model to obtain the change detection result.

[0073] like Figure 5 As shown, the third object of the present invention is to provide an electronic device, the electronic device comprising: a processor 601, a memory 602 and a display screen 603. Among them, the memory 602 and the display screen 603 are connected to the processor 601, such as through a bus 604. Optionally, the electronic device may also include a transceiver 605. It should be noted that in actual applications, the transceiver 605 is not limited to one, and the structure of the electronic device does not constitute a limitation on the embodiments of the present application.

[0074] Processor 601 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 601 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0075] The bus 604 may include a path to transmit information between the above components. The bus 604 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 604 may be divided into an address bus, a data bus, a control bus, etc.

[0076] The memory 602 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0077] The memory 602 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 601. The processor 601 is used to execute the application code stored in the memory 602 to implement the content shown in the above method embodiment.

[0078] Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] The fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is stored thereon, and when the program is executed by a processor, the aforementioned Figure 1 to Figure 2 The various processes of the method embodiment shown in the figure include, for example, a memory including instructions, and the instructions can be executed by a processor of an electronic device to complete the method.

[0080] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. A computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, a computer-readable storage medium may be a portable computer disk, a hard disk, a USB flash drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a lectern random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, an optical disk, a magnetic disk, a mechanical encoding device, and any combination thereof.

[0081] A fifth object of the present invention is to provide a computer program product, comprising computer instructions, which, when executed by a processor, implement the above Figure 1 to Figure 2 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0082] Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of the present teachings should not be determined with reference to the above description, but rather with reference to the preceding claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the preceding claims is not intended to be a disclaimer of such subject matter, nor should it be considered that the applicants did not consider such subject matter to be part of the disclosed inventive subject matter.

[0083] The above content is a further detailed description of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, they can also make several simple deductions or substitutions, which should be regarded as belonging to the protection scope of the present invention determined by the submitted claims.

Claims

1. A remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement, characterized in that: include: Build the dataset required for change detection based on earth observation satellite images; Input the test set of the data set required for change detection into the pre-trained change detection model and process it to obtain the change detection result; The training method of the pre-trained change detection model comprises: Based on the collection of earth observation satellite images, the dataset required for change detection is constructed; Preprocessing a training set in a data set required for change detection to obtain preprocessed training data; The preprocessed training data is input into the remote sensing change detection network to obtain multimodal spatiotemporal differential features; The change detection model is trained based on the multimodal spatiotemporal differential features to obtain a trained change detection model; The test set in the data set required for change detection is input into the trained change detection model to obtain the change detection result.

2. The remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement according to claim 1 is characterized in that: The method of constructing a data set required for change detection based on collecting earth observation satellite images includes: According to satellite acquisition, different synthetic aperture radar images and multispectral images covering the same area at a single time node; Perform image registration on synthetic aperture radar images and multispectral images, mark and generate change detection labels, and obtain the data set required for change detection; The dataset required for change detection is divided into training set, validation set and test set.

3. The remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement according to claim 1 is characterized in that: The preprocessing of the training set in the data set required for change detection to obtain preprocessed training data includes: The synthetic aperture radar images and multispectral images in the training set of the data set required for change detection are resized by bilinear interpolation, and the label map of change detection is resized by nearest neighbor interpolation to obtain the resized image; The adjusted image is flipped left to right to generate new training samples and obtain preprocessed training data.

4. The remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement according to claim 1 is characterized in that: The pre-processed training data is input into the remote sensing change detection network enhanced with multi-modal spatiotemporal differential features to obtain multi-modal spatiotemporal differential features, including: The preprocessed training data is fed into a remote sensing change detection network enhanced with multimodal spatiotemporal differential features, and the absolute value difference between the multispectral image after the change and the synthetic aperture radar image before the change, and between the synthetic aperture radar image after the change and the multispectral image before the change, is calculated to obtain multimodal differential features; Calculate the absolute value difference between the multispectral image after the change and the multispectral image before the change, and the absolute value difference between the synthetic aperture radar image after the change and the synthetic aperture radar image before the change, and obtain the spatiotemporal differential features; The multimodal differential features and spatiotemporal differential features are integrated to obtain the multimodal spatiotemporal differential features.

5. The remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement according to claim 1 is characterized in that: The change detection model is trained based on the multimodal spatiotemporal differential features to obtain a trained change detection model, including: The remote sensing change detection network is trained based on multimodal spatiotemporal differential features, and the learning rate and weight decay parameters are adjusted to obtain a change detection model. Using the training set in the data set required for change detection, the change detection model is trained to obtain a training result; The training results are verified using the verification set in the data set required for change detection. If the verification is qualified, the trained change detection model is obtained.

6. The remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement according to claim 1 is characterized in that: The step of inputting the test set in the data set required for change detection into the trained change detection model to obtain the change detection result includes: Input the test set in the data set required for change detection into the trained change detection model and output the change detection graph; Determine whether the output change detection map has input of a change detection label; If there is no input of change detection label, the change detection result includes output change detection map; If there is an input of change detection label, the change detection results include change detection map, intersection-over-union ratio of each category, the average intersection-over-union ratio of all categories, F1 score of each category, the average F1 score of all categories, and overall accuracy.

7. A remote sensing change detection system based on multimodal spatiotemporal differential feature enhancement, characterized in that: include: Building dataset module: used to build the dataset required for change detection based on earth observation satellite images; Detection result module: used to input the test set of the data set required for change detection into the pre-trained change detection model and process it to obtain the change detection result; The training system of the pre-trained change detection model comprises: Acquisition dataset module: used to build the dataset required for change detection based on the collected earth observation satellite images; Preprocessing data module: used to preprocess the training set in the data set required for change detection to obtain preprocessed training data; Feature acquisition module: used to input the preprocessed training data into the remote sensing change detection network to obtain multimodal spatiotemporal differential features; Detection model module: used to train the change detection model based on multimodal spatiotemporal differential features to obtain the trained change detection model; Obtain result module: used to input the test set in the data set required for change detection into the trained change detection model to obtain the change detection result.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of a remote sensing change detection method based on multimodal spatiotemporal differential feature enhancement as described in any one of claims 1 to 6.

Images