Remote sensing image time sequence interpolation method and device based on LSTM prediction

Abstract

The invention discloses a remote sensing image time sequence interpolation method and device based on LSTM prediction, and relates to the technical field of remote sensing image time sequence interpolation prediction.The method comprises the steps that historical time sequence remote sensing image data of a target area are obtained and preprocessed; dividing into a training set, a verification set and a test set; a Swin-LSTM model is constructed, and training verification is carried out; and generating time sequence interpolation image prediction by using the trained and verified Swindow-LSTM model. According to the method, the Swin Transform module and the LSTM model are combined to form the Swin-LSTM model, the model can more accurately predict and interpolate missing remote sensing image data through fusion of time and space features, and the continuity and precision of a prediction result are improved through a dynamically updated reasoning mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image time series interpolation prediction, and in particular to a remote sensing image time series interpolation method and device based on LSTM prediction. Background Art

[0002] With the rapid development of remote sensing technology, remote sensing images are playing an increasingly important role in environmental monitoring, resource management, and disaster assessment. Remote sensing images provide valuable data support for dynamic monitoring and trend analysis of regional ecological environment by periodically acquiring surface information. However, due to various factors, such as climate conditions, sensor failures, and satellite transit frequency restrictions, the time series data of remote sensing images are often incomplete and missing. This data loss seriously affects the continuity and accuracy of time series analysis, thereby limiting the potential of remote sensing images in practical applications.

[0003] The traditional solution is to use time series interpolation methods such as linear interpolation, spline interpolation and Kriging interpolation, which are effective in processing simple time series data, but their effectiveness decreases significantly when faced with complex spatiotemporal dependencies. This is because these methods usually cannot fully utilize the temporal and spatial characteristics of image data, making it difficult to generate high-precision interpolation results.

[0004] In recent years, deep learning models, especially convolutional neural network (CNN) and recurrent neural network (RNN) models, have been applied and have made significant progress in the field of image time series prediction. Long short-term memory networks (LSTM) are widely used in meteorological forecasting, financial time series analysis and other fields due to their advantages in capturing time dependencies. However, when processing remote sensing images, LSTM simply relies on feature extraction in the time dimension and often ignores the spatial structure information in the image.

[0005] The Vision Transformer (ViT) model effectively captures the global spatial dependencies in the image through the self-attention mechanism, demonstrating excellent image processing capabilities. Among them, Swin Transformer is an improved version of ViT. By introducing the window-based self-attention mechanism (W-MSA) and the shifted window-based self-attention mechanism (SW-MSA), it improves the modeling capability while reducing the computational complexity, making it more suitable for large-scale image processing tasks.

[0006] The invention application with application number 202410833210.2 discloses a method, device and electronic device for reconstructing missing areas of time series remote sensing images. The method obtains the time series remote sensing images to be reconstructed, preprocesses the time series remote sensing images, performs intrinsic orthogonal decomposition on the preprocessed time series remote sensing images, determines the target spatial modal set and the corresponding target temporal modal set, interpolates or predicts the one-dimensional target temporal modal set according to the position of the missing area of ​​the time series remote sensing images, combines the interpolation result set or the prediction result set with the target spatial modal set, and obtains the corresponding missing image. The method can effectively capture the changes of the objects, effectively utilize the information of the spatial and temporal modalities, has strong interpretability, avoids the direct calculation of the two-dimensional spatial data, and is fast and accurate in calculation. However, the solution has the problem of difficulty in accurately capturing the changes of complex time series. Summary of the invention

[0007] In view of the above-mentioned problems, the purpose of the present invention is to provide a remote sensing image time series interpolation method and device based on LSTM prediction, so as to solve the problem that when the existing technology processes the missing time series data of remote sensing images, the traditional interpolation method is difficult to accurately capture the changes in complex time series. By effectively learning the long-term dependencies in the time series data through the deep learning model, the missing image data can be more accurately predicted and interpolated, thereby improving the integrity and accuracy of the remote sensing image time series analysis.

[0008] The embodiments of the present invention provide a remote sensing image time series interpolation method and device based on LSTM prediction.

[0009] Aspect 1: A remote sensing image time series interpolation method based on LSTM prediction, including: S1. Obtain historical time series remote sensing image data of the target area and perform preprocessing; S2, dividing the preprocessed data into training set, validation set and test set; S3, build the Swin-LSTM model and perform training verification; S4. Use the trained and verified Swin-LSTM model to generate time series interpolation image predictions.

[0010] As a further solution of the present invention, the historical time-series remote sensing images of the region in S1 are Sentinel-2 remote sensing images of the target area, and L2A-level image products are used.

[0011] As a further solution of the present invention, the preprocessing in S1 includes resampling the acquired remote sensing image data, performing true color band combination visualization processing, and synthesizing and converting it into GeoTIFF format.

[0012] As a further solution of the present invention, the training set, validation set and test set are grayscale images obtained after the time series remote sensing images are cropped and histogram equalization processed.

[0013] As a further solution of the present invention, the key equation for constructing the Swin-LSTM model is: (1) (2) (3) Among them, F t is the filter gate, represents Hadamard product, STB represents Swin Transformer module, and LP represents linear projection.

[0014] As a further solution of the present invention, the key equation of the Swin Transformer module is: (4) (5) (6) (7) Among them, z l-1 and z l represents residual connection, W-MSA is window-based self-attention mechanism, SW-MSA is shift window-based self-attention mechanism, MLP is multi-layer perceptron, and LN is normalization.

[0015] As a further solution of the present invention, the Swin-LSTM model is used in S4 to generate a time series interpolation image prediction. When the first two images are used to predict the image at the next moment, the model output image is used as the input image for the next prediction of the model for dynamic iterative update.

[0016] A second aspect: A device applied to the above method, comprising: Input module, input the selected remote sensing image; The Swin-LSTM module uses the trained and verified Swin-LSTM model to perform time series interpolation image prediction based on the selected remote sensing image pixels; Output module, outputs the time series interpolation image prediction results.

[0017] A third aspect: An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect when executing the program.

[0018] A fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method provided in the first aspect.

[0019] Beneficial effects of the present invention: 1. The present invention combines the Swin Transformer module and the LSTM model to form a Swin-LSTM model. By fusing time and space features, the model can more accurately predict and interpolate missing remote sensing image data. Through the dynamically updated reasoning mechanism, the continuity and accuracy of the prediction results are improved.

[0020] 2. The present invention generates interpolated images based on dual-phase remote sensing images of Swin-LSTM. Aiming at the current problems of missing time series data and insufficient interpolation accuracy of remote sensing images in complex environments, it provides a new idea for time series interpolation of remote sensing images, enhances the potential of high-precision image interpolation in complex scenes, provides new solutions for tasks such as agricultural monitoring and environmental assessment, and improves the accuracy and reliability of time series remote sensing data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is the principle flow chart of the method of the present invention; Figure 3 This is a schematic diagram of the Swin-LSTM model structure of the present invention; Figure 4 It is a schematic diagram of the application effect of the present invention; Figure 5 It is a structural schematic diagram of the device of the present invention; Figure 6 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0022] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0023] When dealing with missing time series data in remote sensing images, existing interpolation methods are unable to accurately capture the changes in complex time series.

[0024] In view of the above problems, the present invention provides a remote sensing image time series interpolation method based on LSTM prediction. Figure 1 A schematic diagram of a flow chart of a remote sensing image time series interpolation method based on LSTM prediction provided by an embodiment of the present invention, Figure 2 The principle flow chart of the method of the present invention is as follows: S1. Obtain historical time series remote sensing image data of the target area and perform preprocessing.

[0025] For the target area that needs time series interpolation image prediction, the required time range of the historical time series remote sensing images of the target area is clarified, and the Sentinel-2 remote sensing images required for the target area are downloaded. The downloaded Sentinel-2 remote sensing image data is L2A-level product. L2A-level data mainly contains the atmospheric bottom reflectance data that has been calibrated by radiation and atmospheric correction, which is conducive to improving the prediction accuracy.

[0026] The acquired remote sensing image data is resampled, the true color band combination is visualized and synthesized and converted into GeoTIFF format for export.

[0027] S2. Divide the preprocessed data into training set, validation set and test set.

[0028] The time series remote sensing image data in GeoTIFF format of the target area are cropped, histogram equalized, processed into grayscale images, and divided into training set, validation set and test set.

[0029] For example: The target area for research is Taihu Lake, located on the southern edge of the Yangtze River Delta, with boundaries between 30°55'40"~31°32'58" north latitude and 119°52'32"~120°36'10" east longitude. It is one of the five largest freshwater lakes in China. Its ecological environment and water resources management are crucial to the sustainable development of the region. Accurate prediction will help to accurately monitor the dynamic environmental changes in the lake area.

[0030] The time range of remote sensing image processing is from 2021-05-03 to 2024-03-08. The constructed time series contains 14 phases. Based on the acquired sequence data of the 14 phases, the first 10 images are used as training data sets, 2 images are used as validation data sets, and the remaining 2 images are used as test data sets.

[0031] In order to reduce the computer operation load, the acquired time series images were cropped to a size of 512*512, histogram equalization was performed, and processed into grayscale images.

[0032] S3. Build the Swin-LSTM model and perform training and verification.

[0033] The Swin-LSTM model is a model that combines the Swin Transformer module and the simplified LSTM. It applies the Swin Transformer module on the basis of the LSTM model to build a prediction network model for spatiotemporal prediction tasks.

[0034] like Figure 3 As shown in (a), first, at time step t, the remote sensing image is segmented into non-overlapping image patches, each of which is of size P 2 , the characteristic dimension of the patch is C•P 2 (C is the number of channels).

[0035] The image patches are then flattened and fed into the patch embedding layer, which linearly maps the original features of the image patches to arbitrary dimensions.

[0036] Then, the SwinLSTM layer receives the transformed image patch, the hidden state H t-1 and cell status C t-1 , generate the hidden state H t and cell status C t , where H t is copied into two copies, one for rebuilding the SwinLSTM layer and the other for C t Together with the SwinLSTM layer for the next time step.

[0037] Among them, the key equation for constructing the Swin-LSTM model is: (1) (2) (3) Among them, F t is the filter gate, represents Hadamard product, STB represents Swin Transformer module, and LP represents linear projection.

[0038] like Figure 3 As shown in (b), F t As a filter gate, it is used to control the flow of information. The input X after linear projection (LP) is processed by the Swin Transformer module (STB). t and the hidden state H at the previous moment t-1 After processing, it is obtained through the sigmoid function, and its value is between 0 and 1.

[0039] In the SwinLSTM module, the cell state C t and the hidden state H t The information of is updated horizontally to capture long-term and short-term temporal dependencies, while the Swin Transformer module learns global spatial dependencies vertically.

[0040] Specifically, the key equation of the Swin Transformer module is: (4) (5) (6) (7) Among them, z l-1 and z l represents residual connection, W-MSA is window-based self-attention mechanism, SW-MSA is shift window-based self-attention mechanism, MLP is multi-layer perceptron, and LN is normalization.

[0041] The purpose of the two mechanisms of formula (4) and formula (6) is to solve the problem of high computational complexity of the global multi-head self-attention mechanism in the traditional Vision Transformer, especially when processing high-resolution images or large-scale data. By limiting the attention calculation to a local window, the amount of calculation can be greatly reduced, making it more friendly to intensive prediction tasks.

[0042] By alternately using W-MSA and SW-MSA in different modules, the interaction and fusion of information from different regions can be increased, the modeling ability can be improved, and richer global spatial dependencies can be captured. l-1 and z l Represents residual connection. Residual connection helps the flow and propagation of information, avoids the problem of gradient disappearance or explosion in deep networks, and makes the network easier to train. It allows the model to learn small changes between input and output more easily, improving the performance and generalization ability of the model.

[0043] The multi-layer perceptron (MLP) in formula (5) and formula (7) is used to perform further nonlinear transformation on the features after normalization LN (LayerNorm) and self-attention processing. MLP can learn more complex feature representations and enhance the expressiveness of the model. It can capture high-order relationships in input features and extract more abstract and discriminative features. The residual connection in the formula also helps in the training and optimization of the model, allowing information to flow more smoothly between different modules, improving the stability and performance of the model.

[0044] The training set is input into the Swin-LSTM model for training, in which four SwinLSTM layer units are used, the Swin Transformer module is (2,6,6,2), the patch size is 2, the number of training rounds is set to 1000, and the optimizer used is the adaptive learning rate Adam strategy. The initial learning rate is 1e-4, which decays every 100 rounds with a decay coefficient of 0.1.

[0045] The model parameters are updated in each round of training, and the best performing model parameters in multiple rounds of training are saved. The mean square error (MSE), mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as indicators to evaluate the prediction quality of the model to obtain the Swin-LSTM model after training and verification.

[0046] S4. Use the trained and verified Swin-LSTM model to generate time series interpolation image predictions.

[0047] The specific approach is to input the data in the test set into the trained Swin-LSTM model, use the first two remote sensing images to predict the image at the next moment, and dynamically update the input image, that is, the next inference uses at least one predicted image as input, and iterates to obtain the time-series interpolation image of the predicted output.

[0048] The trained and verified Swin-LSTM model is used to perform inference and prediction on the test set data of the Taihu Lake area, and an interpolation map of remote sensing images of the Taihu Lake area can be obtained.

[0049] First, the initial two remote sensing images in the test set are input into the trained and verified Swin-LSTM model. The Swin-LSTM model predicts the remote sensing image at the next moment by learning the temporal relationship and spatial features between the input images; in order to ensure the continuity and authenticity of the prediction results, a dynamic update strategy is adopted in the reasoning process, that is, after each prediction, the predicted image is used as one of the inputs for the next reasoning, and together with the real or predicted image of the previous moment, it is input into the model again for the prediction at the next moment.

[0050] Effects such as Figure 4 As shown in the figure, Input 1 and Input 2 are the remote sensing images input to the Swin-LSTM model, Target is the actual target image, and Output is the predicted time series interpolation image. It can be seen from the figure that the predicted time series interpolation image is relatively ideal and can meet the needs of use.

[0051] This iterative reasoning process can not only effectively capture the subtle differences in the time series changes in the Taihu Lake area, but also improve the continuity and accuracy of the prediction results through gradual iteration. Through this method, a complete time series remote sensing image interpolation map of the Taihu Lake area can be generated, providing high-quality time series data support for further environmental monitoring, trend analysis and resource management. This dynamically updated reasoning mechanism also enhances the model's adaptability in the face of complex environmental changes, enabling it to better handle the uncertainty and data missing problems in practical applications.

[0052] The present invention also discloses a device applied to the above method, such as Figure 5As shown, including: Input module, input the selected remote sensing image; The Swin-LSTM module uses the trained and verified Swin-LSTM model to perform time series interpolation image prediction based on the selected remote sensing image pixels; Output module, outputs the time series interpolation image prediction results.

[0053] When in use, the input module is used to input two remote sensing images of the target area within the required time range, and the Swin-LSTM model in the Swin-LSTM module is used to iteratively predict the time series interpolation images to obtain the prediction results of the time series interpolation images within the required time range, and the output module is used to output the obtained results.

[0054] The present invention also provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory, for example, to execute the following method: S1. Obtain historical time series remote sensing image data of the target area and perform preprocessing; S2, dividing the preprocessed data into training set, validation set and test set; S3, build the Swin-LSTM model and perform training verification; S4. Use the trained and verified Swin-LSTM model to generate time series interpolation image predictions.

[0055] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0056] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including: S1. Obtain historical time series remote sensing image data of the target area and perform preprocessing; S2, dividing the preprocessed data into training set, validation set and test set; S3, build the Swin-LSTM model and perform training verification; S4. Use the trained and verified Swin-LSTM model to generate time series interpolation image predictions.

[0057] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image time series interpolation method based on LSTM prediction, characterized in that: include: S1. Obtain historical time series remote sensing image data of the target area and perform preprocessing; S2, dividing the preprocessed data into training set, validation set and test set; S3, build the Swin-LSTM model and perform training verification; S4. Use the trained and verified Swin-LSTM model to generate time series interpolation image predictions.

2. The remote sensing image time series interpolation method based on LSTM prediction according to claim 1 is characterized in that: The historical time series remote sensing images of the region in S1 are Sentinel-2 remote sensing images of the target area, using L2A-level image products.

3. The remote sensing image time series interpolation method based on LSTM prediction according to claim 2 is characterized in that: The preprocessing in S1 includes resampling the acquired remote sensing image data, performing true color band combination visualization processing, and synthesizing and converting it into GeoTIFF format.

4. The remote sensing image time series interpolation method based on LSTM prediction according to claim 1, characterized in that: The training set, validation set and test set are grayscale images obtained after the time series remote sensing images are cropped and histogram equalization processed.

5. The remote sensing image time series interpolation method based on LSTM prediction according to claim 1, characterized in that: The key equation for constructing the Swin-LSTM model is: (1) (2) (3) Among them, F t is the filter gate, represents Hadamard product, STB represents Swin Transformer module, and LP represents linear projection.

6. The remote sensing image time series interpolation method based on LSTM prediction according to claim 5 is characterized in that: The key equation of the Swin Transformer module is: (4) (5) (6) (7) Among them, z l-1 and z l represents residual connection, W-MSA is window-based self-attention mechanism, SW-MSA is shift window-based self-attention mechanism, MLP is multi-layer perceptron, and LN is normalization.

7. The remote sensing image time series interpolation method based on LSTM prediction according to claim 1, characterized in that: In S4, the Swin-LSTM model is used to generate a time series interpolation image prediction. When the first two images are used to predict the image at the next moment, the model output image is used as the input image for the next prediction of the model for dynamic iterative update.

8. A device according to any one of claims 1 to 7, characterized in that: include: Input module, input the selected remote sensing image; The Swin-LSTM module uses the trained and verified Swin-LSTM model to perform time series interpolation image prediction based on the selected remote sensing image pixels; Output module, outputs the time series interpolation image prediction results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the remote sensing image time series interpolation method based on LSTM prediction as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the remote sensing image time series interpolation method based on LSTM prediction as described in any one of claims 1 to 7 are implemented.

Images