Space-time spectrum joint self-supervision feature extraction method and system for multi-temporal remote sensing image
By extracting spatiotemporal spectral joint features from remote sensing images through a multi-task learning network, this approach solves the problems of irregular temporal sequences, registration difficulties, and scarce annotations in remote sensing image processing. It achieves high-quality feature extraction and robust feature representation, making it suitable for various remote sensing interpretation tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2025-12-16
- Publication Date
- 2026-04-21
AI Technical Summary
Existing remote sensing image processing methods struggle to handle irregular time-series data, face challenges in high-resolution image registration, and suffer from a scarcity of high-quality annotations, resulting in poor feature extraction performance when applied across regions.
A multi-task learning network is adopted, which extracts spatiotemporal spectral joint features from unlabeled remote sensing images through temporal dynamic modeling, spectral semantic consistency learning and spatial feature alignment modules. Self-supervised learning methods are used to perform feature enhancement and contrastive learning in the frequency domain and temporal domain to achieve robust feature representation.
It achieves high-quality spatiotemporal spectral joint feature extraction from remote sensing time series of arbitrary length, reduces application costs, improves feature representation ability and generalization, and can effectively handle cloud occlusion, spectral perturbation and spatial registration error.
Smart Images
Figure CN121904492A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, and specifically relates to a representation learning method that can process sequences of arbitrary length and jointly model spatiotemporal spectral features. Specifically, it relates to a spatiotemporal spectral joint self-supervised feature extraction method and system for multi-temporal remote sensing images. Background Technology
[0002] Benefiting from its advantages in spectral, spatial, and temporal resolution, remote sensing interpretation is widely used in land management, environmental monitoring, disaster assessment, and urban planning. In recent years, deep learning has become the mainstream solution for remote sensing interpretation due to its ability to automatically extract features and model complex spatiotemporal dependencies. Multi-temporal remote sensing analysis, by mining the continuity of time series data, can not only locate changes but also reconstruct their evolutionary trajectories, thereby supporting complex applications such as vegetation phenology monitoring, urban expansion assessment, land degradation tracking, and ecological disturbance detection.
[0003] Despite the significant advantages of deep learning-based feature extraction methods, they still face numerous challenges when applied across large regions, which constitutes the technical problem that this invention aims to solve:
[0004] First, existing methods struggle to handle irregular time-series data. These methods typically assume consistent image sequence intervals and lengths, while real-world remote sensing time series are often highly irregular, making accurate modeling of temporal features difficult. The main reasons include: ① Cloud cover varies by region and season, often resulting in missing images and longer intervals; ② Satellite orbital characteristics lead to excessively high sampling frequencies in high latitudes or certain revisit areas, and sparse frequencies in other areas; ③ Commercial satellites are scheduled on demand, resulting in dense observations of economic hotspots and sparse observations of remote areas. Therefore, sequence irregularity is one of the core challenges restricting the cross-regional deployment of multi-temporal feature extraction methods.
[0005] Second, high-resolution imagery amplifies the difficulty of spatial registration. The emergence of small constellations has significantly improved the spatiotemporal resolution of remote sensing images, but high resolution also amplifies the difficulty of image registration: even after radiometric correction and geometric registration, differences in illumination can still cause pixel-level deviations, which in turn generate spurious change signals in multi-time series comparisons, reducing the accuracy and reliability of detection.
[0006] Third, deep learning-based methods rely on a large number of labeled sequences, but high-quality temporal labels are expensive and time-consuming, making it difficult to scale up to large-scale monitoring. We note that self-supervised learning can learn temporal feature sequences in a pre-task that does not require manual labeling, alleviating data scarcity. However, existing self-supervised learning methods focus on sequence rearrangement, mask reconstruction, or autoregressive short-term predictions, neglecting the gradual changes of land cover over long time series, and thus failing to capture the subtle evolution of land cover over time. Summary of the Invention
[0007] The purpose of this invention is to address the three major problems of irregular temporal sequence, spatial misalignment, and scarce annotation in remote sensing images. It provides a spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images. By simultaneously capturing temporal dynamics, spectral semantics, and spatial relationships through a multi-task learning network, the proposed multi-temporal remote sensing image feature extraction model learns robust spatiotemporal spectral joint feature representations end-to-end from unlabeled remote sensing time series of arbitrary length with noise and displacement. No manual annotation is required, and it is highly robust to data noise and spatial displacement.
[0008] According to one aspect of the present invention, a spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images is provided, comprising:
[0009] Acquire the time series of remote sensing images and preprocess each phase image in the time series of remote sensing images;
[0010] Each phase image from the preprocessed remote sensing image time series is input into a trained multi-temporal remote sensing image feature extraction model, which outputs spatiotemporal spectral joint features; wherein, the training of the multi-temporal remote sensing image feature extraction model includes:
[0011] A remote sensing image dataset is constructed based on the acquired time series of remote sensing images;
[0012] A multi-temporal remote sensing image feature extraction model is constructed, including: a feature extraction network, which encodes the input remote sensing image data to obtain a temporal feature sequence; and a multi-task self-supervised learning network, which simultaneously extracts temporal features, spectral features, and spatial features from the temporal feature sequence.
[0013] The multi-temporal remote sensing image feature extraction model is trained on the constructed remote sensing image dataset. The temporal dynamic modeling module, spectral semantic consistency learning module, and spatial feature alignment module in the multi-task self-supervised learning network are weighted by the multi-task loss function to output the trained multi-temporal remote sensing image feature extraction model.
[0014] Furthermore, the expression for the multi-task loss function is:
[0015] ,
[0016] in, , , For non-negative hyperparameters, For spatial feature alignment loss, To model the loss dynamically over time, This represents the loss for spectral semantic consistency.
[0017] Furthermore, temporal feature extraction includes:
[0018] Spatial global pooling is performed on the time series feature sequence to obtain a global feature vector. The global feature vector is then augmented in time by adding noise, amplitude scaling, and nonlinear deformation to obtain an augmented time series.
[0019] The acquisition date of the enhanced time series is time-coded, and the resulting time-coded vector is added to the image features at the corresponding time to obtain the enhanced feature vector;
[0020] The enhanced feature vector is input into the Dual-Mamba module to perform frequency-domain-based sequence decomposition, and then compared with the information-noise contrastive estimation loss function to obtain the temporal features.
[0021] Furthermore, frequency-domain-based sequence decomposition is performed, including:
[0022] The eigenvectors are transformed to the frequency domain using Fourier transform. Frequency components with amplitudes higher than the amplitude threshold are classified as seasonal components, and frequency components with amplitudes lower than or equal to the amplitude threshold are classified as trend components.
[0023] The frequency domain result is then converted to the time domain using the inverse Fourier transform.
[0024] Furthermore, the Dual-Mamba module includes a main branch and an auxiliary branch. The main branch includes a global state space submodule and a local state space submodule. The global state space submodule is used to capture long-range temporal dependencies. The local state space submodule is used to model the spectral-temporal interaction relationship within a local time period. The auxiliary branch performs feature fusion with the output of the main branch through a shortcut connection to enhance feature representation.
[0025] Further, spectral feature extraction includes:
[0026] Spatial global pooling is performed on the temporal feature sequence to obtain a global feature vector, and adaptive mean pooling is performed on the global feature vector to obtain global spectral features;
[0027] The global spectral features are transformed to the frequency domain and data augmentation is performed to obtain enhanced global spectral features.
[0028] Based on global spectral features and enhanced global spectral features, a comparative learning method is used with an information noise contrast estimation loss function to obtain spectral features.
[0029] Furthermore, spatial feature extraction includes:
[0030] Select image feature pairs from adjacent time points in the temporal feature sequence;
[0031] The image feature pairs are nonlinearly mapped using a projection layer to obtain the mapped features;
[0032] Based on the mapped features, the feature similarity matrix between adjacent time points is calculated and pixel-level matching is performed to obtain the matching results.
[0033] Based on the matching results, a spatial alignment loss function is constructed, and spatial features including feature similarity, spatial consistency, and feature uniformity are extracted using the spatial alignment loss function.
[0034] According to one aspect of the present invention, a spatiotemporal spectral joint self-supervised feature extraction system for multi-temporal remote sensing images is provided, comprising:
[0035] The data preprocessing module is used to acquire the time series of remote sensing images and preprocess each phase image in the time series of remote sensing images.
[0036] The spatiotemporal spectrum joint feature acquisition module is used to input each phase image from the preprocessed remote sensing image time series into a trained multi-temporal remote sensing image feature extraction model, and output spatiotemporal spectrum joint features; wherein, the training of the multi-temporal remote sensing image feature extraction model includes:
[0037] A remote sensing image dataset is constructed based on the acquired time series of remote sensing images;
[0038] A multi-temporal remote sensing image feature extraction model is constructed, including: a feature extraction network, which encodes the input remote sensing image data to obtain a temporal feature sequence; and a multi-task self-supervised learning network, which simultaneously extracts temporal features, spectral features, and spatial features from the temporal feature sequence.
[0039] The multi-temporal remote sensing image feature extraction model is trained on the constructed remote sensing image dataset. The temporal dynamic modeling module, spectral semantic consistency learning module, and spatial feature alignment module in the multi-task self-supervised learning network are weighted by the multi-task loss function to output the trained multi-temporal remote sensing image feature extraction model.
[0040] According to one aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the spatiotemporal spectrum joint self-supervised feature extraction method for multi-temporal remote sensing images.
[0041] According to one aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the spatiotemporal spectrum joint self-supervised feature extraction method for multi-temporal remote sensing images.
[0042] Compared with the prior art, the beneficial effects of the present invention are:
[0043] 1. This invention uses a multi-temporal remote sensing image feature extraction model to learn high-quality spatiotemporal spectral joint feature representations from remote sensing time series of arbitrary lengths.
[0044] 2. This invention learns feature representations from unlabeled remote sensing image data through self-supervised learning, eliminating the need for manual annotation and significantly reducing application costs.
[0045] 3. This invention captures temporal dynamics, spectral semantics, and spatial relationships simultaneously through a multi-task learning network, achieving joint feature learning and stronger feature representation capabilities. In addition, it is insensitive to factors such as cloud occlusion, spectral perturbation, and spatial registration errors, demonstrating strong robustness. The learned features can be widely applied to various downstream multi-temporal remote sensing interpretation tasks, improving generalization. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a schematic diagram of the multi-temporal remote sensing image feature extraction model framework provided by the present invention.
[0048] Figure 2 This is a schematic diagram of the time-series data enhancement sequence provided by the present invention.
[0049] Figure 3 A schematic diagram of the time dynamic modeling module provided by the present invention.
[0050] Figure 4 A schematic diagram of the spectral semantic consistency learning module provided by this invention.
[0051] Figure 5 A schematic diagram of the spatial feature alignment module provided by the present invention. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] This invention proposes a spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images, comprising: acquiring a time series of remote sensing images and preprocessing each temporal image in the time series; inputting each temporal image in the preprocessed time series into a trained multi-temporal remote sensing image feature extraction model, and outputting spatiotemporal spectral joint features. This invention solves three major problems: irregular image temporal sequence, spatial misalignment, and scarce annotations. It learns high-quality spatiotemporal spectral joint feature representations from remote sensing time series of arbitrary length, achieving joint feature learning and stronger feature representation capabilities. Furthermore, it is insensitive to factors such as cloud cover, spectral perturbation, and spatial registration errors, demonstrating strong robustness. The training of the multi-temporal remote sensing image feature extraction model, such as... Figure 1 As shown, the process includes: Step 1, acquiring the time series of remote sensing images and preprocessing each temporal image; Step 2, using a shared-weight feature extraction network to independently encode each temporal remote sensing image in the time series, obtaining a temporal feature sequence; Step 3, constructing a multi-task self-supervised learning network AnyTime, which jointly optimizes feature representation through three complementary modules: 1) Temporal dynamic modeling module: mining temporal patterns in the sequence and decoupling long-term trend components from seasonal components; 2) Spectral semantic consistency learning module: performing data augmentation in the frequency domain and learning feature representations robust to spectral perturbations; 3) Spatial feature alignment module: achieving pixel-level correspondence through feature matching and learning feature representations invariant to spatial displacement; Step 4, jointly optimizing the three modules through a multi-task loss function to train an optimized multi-temporal remote sensing image feature extraction model.
[0054] Specifically, the embodiments of the present invention provide the specific content of step 1:
[0055] Acquire time series of remote sensing images. Perform standard preprocessing operations such as radiometric correction and coarse geometric registration on each time-phase image. Subsequently, crop the original images and corresponding annotations (if any) into multiple 128-pixel × 128-pixel image patches. To evaluate the model's generalization ability and avoid data leakage, the dataset is divided into training, validation, and test sets based on independent geographic regions.
[0056] Specifically, the embodiments of the present invention provide the following details for step 2:
[0057] The preprocessed time-series images are input into a shared-weight feature extraction network. This network adopts an encoder-decoder architecture, and its overall process is as follows: Figure 1 As shown in the left-hand diagram.
[0058] The encoder, based on the Swin Transformer Tiny architecture, is responsible for independently encoding each single-temporal image into a high-dimensional feature map. Where F represents the temporal feature sequence tensor, R represents the real number field, T represents the length of the time series, H and W represent the spatial height and width of the feature map, respectively, and D represents the number of channels of the feature map; in this embodiment, H=8, W=8, and D=768 are set.
[0059] The decoder, a network based on stacked transposed convolutional layers (UPNet), upsamples the temporal feature sequence F to the spatial size of the input image (128×128) for subsequent self-supervised task computation. The temporal feature sequence F is the encoder's output. The decoder's role is to upsample (restore details) the downsampled temporal feature sequence F output by the encoder to obtain a high spatial resolution feature map or prediction map, used to calculate the loss of the self-supervised task (such as reconstruction loss). It does not itself constitute the final temporal feature sequence.
[0060] Specifically, the shared-weight feature extraction network in step 2 uses an encoder based on the Swin Transformer Tiny architecture, in conjunction with a decoder based on an upsampling convolutional layer (UPNet), to compress each single-temporal image into a compact feature representation while capturing multi-scale contextual information.
[0061] Specifically, the embodiments of the present invention provide the specific content of step 3:
[0062] By integrating feature learning across three dimensions—time, space, and spectral—into a unified self-supervised learning paradigm, and through the design of complementary modules, including temporal dynamic modeling, spectral semantic consistency learning, and spatial feature alignment, the model can achieve a comprehensive understanding and feature recognition of surface change processes.
[0063] Step 3.1, Time Dynamic Modeling, specifically includes the following steps:
[0064] Step 3.1.1, Time series data augmentation.
[0065] Spatial global pooling is performed on the temporal feature sequences to obtain a compact temporal representation. Each of them ∈ It is the feature map at the corresponding time. The global feature vector is obtained after pooling. Enhanced time series are generated using time augmentation methods such as adding noise (jitter), scaling, and nonlinear deformation (MagWarp). For example... Figure 2 As shown, although these enhanced time series have changed in form, they fundamentally retain the underlying structure and temporal information, thus enabling the network to focus on invariant temporal patterns.
[0066] Step 3.1.2, Time Encoding.
[0067] To enable the network to understand irregular time intervals, absolute time information is embedded for each moment in the sequence. This is achieved by using the acquisition date of each phase image. Encode the position of each feature in the time series, and the encoding vector. The formula is:
[0068] (1)
[0069] in, The dimension of the feature vector (i.e., the number of channels defined in step 2). It is a constant scaling factor, here set to r=1000. This serves as the index for the feature dimension. Subsequently, the encoded vector is added to the image features at the corresponding time point. = + ;in It is the enhanced feature vector at the t-th time step after embedding the time encoding.
[0070] Step 3.1.3: Time-based dynamic modeling.
[0071] This process is the core of this task, used to capture complex temporal dependencies and dynamic relationships in a sequence and decouple mixed components from the signal. This task integrates time coding and Dual-Mamba modules, with the specific structure as follows: Figure 3 As shown.
[0072] 1. Dual-Mamba Module: This module is a variant of the standard Mamba module, employing a dual-branch structure. The main branch contains two parallel sub-paths (global and local). The Global State Space Submodule (G-SSM) uses continuous-time state equations to propagate hidden states between semantic features at different time points in the sequence. This design enables the model to efficiently capture long-range temporal dependencies spanning the entire sequence. The Local State Space Submodule (L-SSM) employs a time-segmentation strategy to model spectral-temporal interactions within local time intervals. This design encourages the model to learn short-term dynamic features at a fine temporal granularity.
[0073] Auxiliary branches: such as Figure 3 As shown, the blue lines and arrows in the right-hand box are fused with the output of the main branch in a shortcut manner to enhance the feature representation of the main branch and stabilize the training process.
[0074] 2. Frequency Sequence Decomposition Module: Employs a frequency-domain-based sequence decomposition method to extract and decouple the trend and seasonal terms from the feature sequence. Fourier Transform (FFT) is used to transform the input sequence... The process involves converting to the frequency domain, then using high frequencies as the seasonal component and low frequencies as the trend component, and finally converting back to the time domain using the inverse Fourier transform (iFFT). This process can be represented as:
[0075] (2)
[0076] in, ; ;[ ] indicates that the absolute value is within the threshold. The above high-frequency components, and [ This indicates that low-frequency components related to long-term trends are retained. Quantity This represents the long-term progress of the input sequence, while Quantity This reflects the observation of periodic dynamic patterns within fixed time intervals. At this point, the seasonal component... Since the data contains a significant amount of noise, the seasonal component with the highest frequency, matching the number of trend components, is selected and retained for effective noise reduction. Finally, the seasonal component is input into the next Dual-Mamba module to further learn the temporal dynamic pattern.
[0077] Step 3.1.4: Comparative learning.
[0078] The time loss function adopts the InfoNCE loss function (Information Noise-Contrastive Estimation), which brings the original time series feature sequence closer to its enhanced time series and widens the distance from other samples, thereby learning a time dynamic representation that is robust to time series perturbations.
[0079] Step 3.2, Spectral semantic consistency learning. For example... Figure 4 As shown, the specific steps include:
[0080] Step 3.2.1, Time aggregation.
[0081] Adaptive mean pooling is applied to time-series feature sequences:
[0082] (3)
[0083] in, This represents the k-th time window of the adaptive partitioning. Transformed into stable global spectral features This reflects the spectral characteristics of land cover at most points in the time series.
[0084] Step 3.2.2, Frequency Domain Enhancement.
[0085] The aggregated spectral features are converted to the frequency domain. The algorithm employs frequency mixing and frequency masking (data augmentation). Frequency mixing involves randomly swapping low-frequency components of samples within the same batch; frequency masking randomly masks a certain proportion of frequency components. Both augmented spectra are reconstructed into new spectral features using inverse Fourier transform. Specifically, the frequency mixing strategy involves randomly selecting sample pairs within the same training batch and partially swapping their frequency domain representations. The low-frequency components in.
[0086] Step 3.2.3: Comparative learning.
[0087] The spectral loss function adopts the InfoNCE loss function, which brings the original spectral features closer to their enhanced spectral features and pushes them further away from other samples, thereby learning semantically invariant features that are robust to spectral perturbations.
[0088] Step 3.3, Spatial Feature Alignment. For example... Figure 5 As shown, the specific steps include:
[0089] Step 3.3.1, Feature mapping.
[0090] Select biphase feature pairs from the feature sequence. ]; where, two phases represent adjacent moments, The feature map representing the first time point. The feature map represents the second time point. Represents a real number tensor. H represents the height of the feature map (number of pixels). W represents the width of the feature map (number of pixels). D represents the number of channels in the feature map. A non-linear mapping is performed using a 1×1 convolution to obtain [ In this model, a simple "projection layer" (usually a 1×1 convolution) processes Q and K respectively, mapping them to new features q and k in a new feature space that is more suitable for fine comparison. S represents the spatial size of the projected feature map. Since a 1×1 convolution is used, the spatial size remains unchanged, S=H=W. d represents the new feature dimension after projection, which can be the same as or different from D.
[0091] Step 3.3.2, Feature matching.
[0092] Projected features and Flattening along the spatial dimension yields The cosine similarity is calculated pixel by pixel to obtain the similarity matrix. Each element M ij express The i-th eigenvector and The similarity of the j-th feature vector.
[0093] for The i-th eigenvector in the similarity matrix M is determined by finding the maximum value in the i-th row. The best matching (positive sample) position in :
[0094] (4)
[0095] in, Representing the similarity matrix The maximum value in the i-th row. i and j both represent index variables. i∈[1,S2], used to index the flattened features. The i-th feature vector in S1. j∈[1,S2], used to index the flattened features. The j-th eigenvector in the dataset. The i-th row of the similarity matrix M is a vector containing... and The similarity score of all vectors in the dataset.
[0096] Step 3.3.3: Construct the spatial alignment loss function as follows:
[0097] (5)
[0098] in, Features The Middle 1 eigenvector In feature map Zhongyu The positive pairs. The negative pairs come from the feature vectors after pooling in different view spaces. for The i-th eigenvector in the feature map Spatial location within. It can be viewed as a similarity matrix Maximum value in the i-th row , This represents the offset of the dual-temporal characteristics. This represents the cosine similarity between the i-th query feature vector and the j-th negative sample feature vector, used to push away mismatched sample pairs in the feature uniformity term.
[0099] Specifically, the time dynamic modeling in step 3 includes:
[0100] 1) Apply randomization enhancement to the time feature sequence after global pooling, including adding noise, amplitude modification, and nonlinear transformation;
[0101] 2) Semantic features at each time step are embedded into global temporal encoding;
[0102] 3) Design a Dual-Mamba module for global-local time modeling to capture complex temporal dependencies and dynamic relationships in sequences;
[0103] 4) Use frequency domain sequence decomposition to decouple long-term trends and seasonal cycle components;
[0104] 5) Use the contrastive learning loss function to optimize the temporal feature representation.
[0105] Specifically, step 3, spectral semantic consistency learning, includes:
[0106] 1) Perform adaptive temporal aggregation on the time feature sequences to obtain global spectral features;
[0107] 2) Two enhancement strategies are applied in the frequency domain: frequency mixing and frequency masking;
[0108] 3) By using a contrastive learning loss function, the model learns spectral semantically invariant features.
[0109] Specifically, the spatial feature alignment task in step 3 includes:
[0110] 1) Select image feature pairs from adjacent time points in the feature sequence;
[0111] 2) Nonlinear mapping is performed through dense projection layers;
[0112] 3) Calculate the feature similarity matrix and establish pixel-level matching;
[0113] 4) Construct a spatial alignment loss function based on the matching results, which includes three components: feature similarity, spatial consistency, and feature uniformity.
[0114] Specifically, the embodiments of the present invention provide the specific content of step 4:
[0115] Joint training. Using weighted and multi-task loss functions. Perform joint optimization. Weights of the multi-task loss function. , , These are non-negative hyperparameters used to balance the spatial feature alignment loss. Time-dynamic modeling loss and spectral semantic consistency loss The multi-task loss function is a weighted sum of the losses from the three modules.
[0116] Specifically, embodiments of the present invention utilize a pre-trained multi-temporal remote sensing image feature extraction model to extract high-quality spatiotemporal spectral joint feature representations from new remote sensing time series. These features can be directly used for downstream tasks such as change detection and land cover classification, or fine-tuned by adding a lightweight task header.
[0117] The implementation of the various embodiments of this invention is based on programmed processing through a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of this invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of this invention provide a spatiotemporal spectral joint self-supervised feature extraction system for multi-temporal remote sensing images. This system is used to execute a spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images from the above method embodiments.
[0118] The system includes: a data preprocessing module for acquiring remote sensing image time series and preprocessing each temporal image in the remote sensing image time series; and a spatiotemporal spectral joint feature acquisition module for inputting each temporal image in the preprocessed remote sensing image time series into a trained multi-temporal remote sensing image feature extraction model and outputting spatiotemporal spectral joint features. The training of the multi-temporal remote sensing image feature extraction model includes: constructing a remote sensing image dataset based on the acquired remote sensing image time series; constructing the multi-temporal remote sensing image feature extraction model, including: a feature extraction network for encoding the input remote sensing image data to obtain a temporal feature sequence; and a multi-task self-supervised learning network for simultaneously extracting temporal features, spectral features, and spatial features from the temporal feature sequence; training the multi-temporal remote sensing image feature extraction model on the constructed remote sensing image dataset, and weighting the temporal dynamic modeling module, spectral semantic consistency learning module, and spatial feature alignment module in the multi-task self-supervised learning network using a multi-task loss function to output the trained multi-temporal remote sensing image feature extraction model.
[0119] The spatiotemporal spectral joint self-supervised feature extraction system for multi-temporal remote sensing images provided in this invention addresses three major problems: irregular temporal sequence, spatial misalignment, and scarce annotations in remote sensing images. It employs several modules to acquire time series of remote sensing images of arbitrary length; utilizes a shared-weight feature extraction network to extract depth features for each temporal image; and jointly optimizes the spatiotemporal spectral feature representation through a multi-task self-supervised learning network. Specifically, the temporal dynamic modeling task decouples long-term trends from seasonal cycles, the spectral semantic consistency learning task enhances robustness in the frequency domain, and the spatial feature alignment task achieves pixel-level correspondence through feature matching. Finally, a high-quality spatiotemporal spectral joint feature representation is obtained.
[0120] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention also provides an electronic device, including a memory and a processor. The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize a spatiotemporal spectrum joint self-supervised feature extraction method for multi-temporal remote sensing images as proposed in the above embodiments.
[0121] This invention also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, this program can learn high-quality spatiotemporal spectral joint feature representations from remote sensing time series of arbitrary lengths, without manual annotation, and exhibits strong robustness to data noise and spatial displacement.
[0122] In summary, the present invention provides a workflow for a spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images, comprising: acquiring a remote sensing image time series of arbitrary length and preprocessing each temporal image in the series; and using a shared-weight encoder-decoder network to independently encode each temporal image in the preprocessed time series to obtain a temporal feature sequence. ,in Indicates spatial dimension, The length of the time series. For the channel dimension, a multi-task self-supervised learning network is constructed. This network jointly optimizes the temporal feature sequence through three complementary modules: temporal dynamic modeling, spectral semantic consistency learning, and spatial feature alignment. A multi-task loss function, which is a weighted sum of the loss functions of each module, is used to jointly optimize the three modules. The optimized feature extraction network is trained; using the trained feature extraction network, spatiotemporal spectral joint feature representations that can be directly used for downstream tasks are extracted from new remote sensing time series.
[0123] Finally, it should be noted that the above specific embodiments are merely representative examples of the present invention. Obviously, the present invention is not limited to the above specific embodiments and many variations are possible. Any simple modifications, equivalent changes, and alterations made to the above specific embodiments based on the technical essence of the present invention should be considered within the protection scope of the present invention.
Claims
1. A spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images, characterized in that, include: Acquire the time series of remote sensing images and preprocess each phase image in the time series of remote sensing images; Each phase image from the preprocessed remote sensing image time series is input into a trained multi-temporal remote sensing image feature extraction model, which outputs spatiotemporal spectral joint features; wherein, the training of the multi-temporal remote sensing image feature extraction model includes: A remote sensing image dataset is constructed based on the acquired time series of remote sensing images; A multi-temporal remote sensing image feature extraction model is constructed, including: a feature extraction network, which encodes the input remote sensing image data to obtain a temporal feature sequence; and a multi-task self-supervised learning network, which simultaneously extracts temporal features, spectral features, and spatial features from the temporal feature sequence. The multi-temporal remote sensing image feature extraction model is trained on the constructed remote sensing image dataset. The temporal dynamic modeling module, spectral semantic consistency learning module, and spatial feature alignment module in the multi-task self-supervised learning network are weighted by the multi-task loss function to output the trained multi-temporal remote sensing image feature extraction model.
2. The spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images according to claim 1, characterized in that, The expression for the multi-task loss function is: , in, , , For non-negative hyperparameters, For spatial feature alignment loss, To model the loss dynamically over time, This represents the loss for spectral semantic consistency.
3. The spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images according to claim 1, characterized in that, Temporal feature extraction, including: Spatial global pooling is performed on the time series feature sequence to obtain a global feature vector. Then, the global feature vector is augmented in time by adding noise, amplitude scaling, and nonlinear deformation to obtain an augmented time series. The acquisition date of the enhanced time series is time-coded, and the resulting time-coded vector is added to the image features at the corresponding time to obtain the enhanced feature vector; The enhanced feature vector is input into the Dual-Mamba module to perform frequency-domain-based sequence decomposition, and then compared with the information-noise contrastive estimation loss function to obtain the temporal features.
4. The spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images according to claim 3, characterized in that, Frequency-domain based sequence decomposition includes: The eigenvectors are transformed to the frequency domain using Fourier transform. Frequency components with amplitudes higher than the amplitude threshold are classified as seasonal components, and frequency components with amplitudes lower than or equal to the amplitude threshold are classified as trend components. The frequency domain result is then converted to the time domain using the inverse Fourier transform.
5. The spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images according to claim 3, characterized in that, The Dual-Mamba module includes a main branch and an auxiliary branch. The main branch includes a global state space submodule and a local state space submodule. The global state space submodule is used to capture long-range temporal dependencies. The local state space submodule is used to model the spectral-temporal interaction relationship within a local time period; The auxiliary branch is connected to the output of the main branch via a quick connection to perform feature fusion, thereby enhancing the feature representation.
6. The spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images according to claim 1, characterized in that, Spectral feature extraction, including: Spatial global pooling is performed on the temporal feature sequence to obtain a global feature vector, and adaptive mean pooling is performed on the global feature vector to obtain global spectral features; The global spectral features are transformed to the frequency domain and data augmentation is performed to obtain enhanced global spectral features. Based on global spectral features and enhanced global spectral features, a comparative learning method is used with an information noise contrast estimation loss function to obtain spectral features.
7. The spatiotemporal spectral joint self-supervised feature extraction method for multi-temporal remote sensing images according to claim 1, characterized in that, Spatial feature extraction, including: Select image feature pairs from adjacent time points in the temporal feature sequence; The image feature pairs are nonlinearly mapped using a projection layer to obtain the mapped features; Based on the mapped features, the feature similarity matrix between adjacent time points is calculated and pixel-level matching is performed to obtain the matching results. Based on the matching results, a spatial alignment loss function is constructed, and spatial features including feature similarity, spatial consistency, and feature uniformity are extracted using the spatial alignment loss function.
8. A spatiotemporal spectral joint self-supervised feature extraction system for multi-temporal remote sensing images, characterized in that, include: The data preprocessing module is used to acquire the time series of remote sensing images and preprocess each phase image in the time series of remote sensing images. The spatiotemporal spectrum joint feature acquisition module is used to input each phase image from the preprocessed remote sensing image time series into a trained multi-temporal remote sensing image feature extraction model, and output spatiotemporal spectrum joint features; wherein, the training of the multi-temporal remote sensing image feature extraction model includes: A remote sensing image dataset is constructed based on the acquired time series of remote sensing images; A multi-temporal remote sensing image feature extraction model is constructed, including: a feature extraction network, which encodes the input remote sensing image data to obtain a temporal feature sequence; and a multi-task self-supervised learning network, which simultaneously extracts temporal features, spectral features, and spatial features from the temporal feature sequence. The multi-temporal remote sensing image feature extraction model is trained on the constructed remote sensing image dataset. The temporal dynamic modeling module, spectral semantic consistency learning module, and spatial feature alignment module in the multi-task self-supervised learning network are weighted by the multi-task loss function to output the trained multi-temporal remote sensing image feature extraction model.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the spatiotemporal spectrum joint self-supervised feature extraction method for multi-temporal remote sensing images as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the spatiotemporal spectrum joint self-supervised feature extraction method for multi-temporal remote sensing images as described in any one of claims 1 to 7.