Remote sensing enhancement time sequence prediction method based on physically-assisted multi-modal Transform

Abstract

The invention relates to the technical field of remote sensing data fusion, and discloses a remote sensing enhanced time sequence prediction method based on a physically-assisted multi-modal Transform, which utilizes the macroscopic coverage advantage of remote sensing data and the high time resolution characteristic of time-space sequence data. Complementary information of the two is fully utilized to avoid information loss and reconstruction errors caused by up-sampling or down-sampling, and the downstream task effect is remarkably improved. According to the method, time sequence features are efficiently processed through cross Transformer, meanwhile, the efficiency of remote sensing feature processing is also very high, in order to effectively maintain feature distribution characteristics of different modal data, a special parameter configuration scheme specially arranged for all modals is adopted, directional information exchange is achieved through multi-head cross attention inside, and the method has the advantages of being simple in structure and convenient to operate. One mode is allowed to selectively pay attention to the key features of the other mode, the fusion efficiency is remarkably improved, and deep fusion of the features of the two modes is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing data fusion, in particular to a remote sensing enhanced time series prediction method based on physical auxiliary multi-modal Transformer. BACKGROUND

[0002] With the rapid development of remote sensing technology and sensor technology, remote sensing data and spatio-temporal sequence data have been widely used in fields such as agriculture, meteorology, and environmental monitoring. Therefore, how to efficiently fuse remote sensing data and spatio-temporal sequence data to meet actual application has become a difficult problem. Remote sensing data has a wide geographical coverage and presents a macroscopic perspective, which can provide environmental information over a large area. However, it has a coarse time granularity, low absolute accuracy, and low sampling frequency. Spatio-temporal sequence data has a high sampling frequency and can capture fine-grained local differences. However, due to the high cost of sensor deployment, the spatial distribution is sparse. How to effectively fuse data of these two modalities to fully utilize their complementary advantages is a key challenge in spatio-temporal data mining.

[0003] However, existing methods face many problems when fusing remote sensing data and spatio-temporal sequence data. Traditional data fusion methods usually rely on strict spatio-temporal alignment. For example, one method uses contrastive learning to enhance the remote sensing encoder, but the actual pre-training is very complex, and forcibly downsampling to align the time granularity will introduce serious reconstruction loss. Another method uses cross-attention fusion, but requires the number of tokens of the two modalities to be strictly equal, which cannot handle cases where the time granularity is inconsistent. These methods often amplify the errors caused by reconstruction and cause information loss, which affects the final prediction results of the model. In addition, existing methods are prone to overfitting when dealing with large-scale data. For example, some methods directly add the vectors of remote sensing data and time series data. This simple fusion method is very easy to introduce noise, which will interfere with the model's learning process, causing the model to overfit local features on the training data and affecting the model's generalization ability. One method aligns ECG and CT in the medical field, but only relies on vector concatenation, ignoring semantic alignment. Since there is no semantic alignment, the model cannot effectively distinguish important features and noise in different modal data, causing the model to easily learn noise features in the training data during the training process, resulting in poor performance on test data. Although contrastive learning has shown effectiveness in image-text alignment, the pairing rules such as "image-text pairs" or "same timestamp" will fail in the remote sensing and time series hybrid scenario: different time points in the same area are often mislabeled as negative samples, while different areas may be mistakenly judged as positive due to the same time. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a remote sensing enhanced time series prediction method based on a physically assisted multi-modal Transformer, which has the advantages of efficient and lossless fusion of remote sensing images and spatio-temporal sequence data, and solves the above technical problems.

[0005] To achieve the above object, the present application provides the following technical scheme: a remote sensing enhanced time series prediction method based on a physically assisted multi-modal Transformer, comprising the following steps: S1: data acquisition, acquiring spatio-temporal sequence data and remote sensing image data, and constructing a graph structure according to the latitude and longitude coordinates of the sensor and the wind field data; S2: data preprocessing: preprocessing the data collected in S1; S3: constructing a physical information differential equation and solving it through a neural ODE solver, and encoding the solving result to obtain , and processing the spatio-temporal sequence data based on multi-scale position encoding and multi-head attention to obtain , and fusing the two to obtain a fusion result , and based on the fusion result obtaining a time series latent feature ; S4: extracting the remote sensing image data based on a remote sensing encoder and outputting a remote sensing latent feature ; S5: using a cross-Transformer to perform multi-modal fusion on the time series latent feature and the remote sensing latent feature ; S6: constructing a contrastive loss function and a total loss function based on the contrastive loss function; S7: outputting a prediction result.

[0006] As a preferred technical scheme of the present application, the preprocessing in S2 specifically comprises: Patch cutting: cutting the spatio-temporal sequence data into TS patches at a fixed time interval T , and cutting the remote sensing image data into time sequence patches at a fixed spatial size HxW ; wherein, R is a real space, is the number of time steps, is the number of single time step samples, is the sample attribute dimension, is the number of remote sensing image bands, is the number of patch vertical pixels, is the number of patch horizontal pixels; Feature standardization: Z-score standardization is performed on numerical features in the TS and RS patches.

[0007] As a preferred embodiment of the present invention, the physical information differential equation constructed in S3 specifically comprises: a discrete diffusion-advection differential equation. The specific expression is as follows: in, This represents the element-wise multiplication operator, used for gating coefficients. Dynamic weighting of physical terms, The state tensor representing spatiotemporal sequence data. Indicates the gating coefficient. It is a continuous-time independent variable that describes the dynamic evolution of a physical process. Represents the Laplace operator for the diffusion graph. This represents the Laplace operator for the advection graph. Represents the diffusion constant. Represents the coefficient of the residual term. express right The differential; The above physical information differential equations are solved using a neural ODE solver. The solver generates the objective variable from time... arrive Future trajectory: in, This represents a prediction of the future state under physical constraints, integrating the evolution results of the diffusion-advection process. This represents the ODE neural solver function. Indicates historical time T The status input at any given time, Representing the discrete diffusion-adsorption differential equation, It is the historical deadline step. It is the number of steps to predict the future; Will Encoding into the latent space : in, Indicates parameters The encoder maps the physical prediction results to the latent space. Represents the latent features of physical encoding, used for multimodal fusion; The process in S3, which involves processing spatiotemporal sequence data based on multi-scale positional encoding and multi-head attention, yields the results. The steps are as follows: S3.1: For any discrete time position and given scale First, the position is scaled, and then mapped to a sine-cosine space. The specific formula is as follows: in, Indicates low-frequency long-period components. For a predefined scale, Represents the index value in the time series. Represented by the embedding dimension Determined frequency components, , They represent sine and cosine respectively. Representing scale Location encoding, embedding dimension refers to the encoding of discrete-time locations. The target dimension when mapped to a high-dimensional feature space; S3.2: Multi-scale aggregation, the specific expression is as follows: in, Indicates to of Summation, This represents multi-scale aggregated positional encoding. express The number of elements in the middle; S3.3: Will and sequence features output by the time encoder The elements are added together and then fed into a standard multi-head attention system, as shown in the following expression: in, This represents an augmented representation of time-series data after multi-scale location encoding and multi-head attention processing. Indicates standard multi-head attention. This represents the sequence features output by the time encoder. This represents multi-scale aggregated location encoding; The fusion process in S3 is implemented based on the fusion module, and the specific expression is as follows: in, This represents the potential characteristics of physical coding. It includes vector concatenation and gating units. An augmented representation of time-series data after multi-scale location encoding and multi-head attention processing; The fusion module will and The vector splicing is performed to obtain a spliced feature vector, and a gating unit is matched to adjust the feature vector processed by the gating unit, which is directly used as a time sequence latent feature Output.

[0008] As a preferred technical solution of the present application, the S4 extracts remote sensing image data based on a remote sensing encoder and outputs a remote sensing latent feature Specifically, Among them, represents the remote sensing data at time t, represents a remote sensing encoder, represents a spatial dimension parameter of the remote sensing data, represents a state tensor of the spatio-temporal sequence data, represents an original pyramid visual Transformer model, represents a diffusion constant.

[0009] As a preferred technical solution of the present application, the fusion process formula of the first l+ layer in the cross Transformer is: Among them, and respectively represent the time sequence latent feature and the remote sensing latent feature of the first l layer, and respectively represent the time sequence feature and the remote sensing feature of the first l+1 layer, represents a remote sensing side cross fusion parameter, represents a time sequence side cross fusion parameter, represents a cross Transformer.

[0010] As a preferred technical solution of the present application, the S6 constructs a contrast loss function expression as follows: Among them, represents a contrast loss function, represents the total number of regions, represents a cosine similarity function, is a temperature scaling factor, represents the remote sensing latent feature of the i-th region, represents the time sequence latent feature of the i-th region, represents the time sequence latent feature of the j-th region, denotes an exponential function with a natural constant as a base number, denotes a logarithmic function; The total loss function expression is constructed based on the contrast loss function as follows: wherein, is a mean square error loss, denotes a weight parameter, denotes a total loss function Compared with the prior art, the present application provides a remote sensing enhanced time series prediction method based on a physical auxiliary multi-modal Transformer, which has the following beneficial effects: 1. The present application efficiently processes time series features through cross-Transformer, and the efficiency of processing remote sensing features is also high. In order to effectively maintain the feature distribution characteristics of different modal data, the present application adopts a special parameter configuration scheme for each modal, realizes directional information exchange through multi-head cross attention inside, allows one modal to selectively focus on the key features of another modal, significantly improves the fusion efficiency, and realizes the deep fusion of two modal features.

[0011] 2. The present application defines the remote sensing features and time series features of the same geographical area as positive sample pairs, and the features of different areas as negative sample pairs. Through the contrast loss function, the same area cross-modal features are forced to be closer in the semantic space, while the distance of the different area features is enlarged. The contrast loss and the mean square error loss are combined to form a total loss function, which optimizes the prediction accuracy while ensuring the numerical and semantic consistency of the cross-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0012] Fig. 1 is a flowchart of the present application; Fig. 2 is a schematic diagram of the experimental results of the present application. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0014] The application breaks the shackles of space-time granularity by patch division, thereby avoiding the errors caused by upsampling or downsampling, and then disperses the diffusion-advection equation into a graph Laplacian operator and adds it as a physical bias to the attention weight of the Transformer, so that the model can follow the physical law and retain long-range dependence, redefine the contrast samples according to the standard of 'positive in the same region and negative in different regions', and realize lossless coupling of remote sensing images and ground time series and long-term stable prediction without accurate alignment.

[0015] Neural ODE Solver: Neural ODE Solver is an innovative technology in the cross field of machine learning and numerical solution of differential equations, which gives the model the ability to efficiently approximate and solve complex ordinary differential equations and partial differential equations. The technology system involves various aspects of numerical solution, including discrete time step optimization, continuous dynamics modeling, and gradient flow calculation. With these basic conditions, Neural ODE Solver achieves core tasks such as dynamic system prediction and physical process simulation. With the continuous development of deep learning architecture and adaptive step algorithm, the accuracy of Neural ODE Solver has been significantly improved, and it has been widely used in weather forecasting, engineering simulation and other fields, greatly improving the efficiency of solving complex differential equations and promoting the intelligentization of scientific computing; Contrastive Loss Function: Contrastive Loss Function is a crucial loss function technology in the field of machine learning and deep learning, which gives the model the ability to effectively learn the similarity and difference between data and optimize feature representation. The technology system involves various aspects of loss calculation, including positive and negative sample pair definition, similarity measurement, and temperature parameter adjustment. With these basic conditions, Contrastive Loss Function achieves core tasks such as cross-modal feature alignment and similar sample clustering. With the continuous development of self-supervised learning and multi-modal fusion technology, the performance of Contrastive Loss Function has been significantly improved, and it has been widely used in image retrieval, text semantic matching and other fields, greatly improving the effectiveness of feature learning; Multi-Head Attention: Multi-Head Attention (MHA) is a core component of time series encoder, which realizes deep mining of time series features by fusing multi-scale position encoding. This mechanism first maps time series features to multiple subspaces, and each attention head focuses on capturing different time scale dependencies, such as short-term high-frequency fluctuations and long-term periodic trends. Then, through parallel computing, it integrates multi-dimensional correlation information. Combined with the output of the physically driven dynamic model, MHA generates a representation that combines data correlation and physical reasonableness, effectively alleviating the information confusion problem of traditional attention when dealing with non-stationary spatio-temporal sequences, and laying a precise time series feature foundation for subsequent cross-modal fusion. Position encoding: Position encoding is a key technology in sequence modeling, which provides the position information of elements to the model, making up for the lack of attention mechanism's insensitivity to order, and is widely used in natural language processing, video analysis and other fields. In this model, multi-scale position encoding introduces multiple frequency scale sinusoidal functions, combined with a logarithmic scaling factor, so that the encoding contains information of different time resolutions at the same time, effectively alleviating the periodic aliasing problem, providing rich time sequence clues for the multi-head attention mechanism, and enhancing the model's ability to model complex spatio-temporal dynamics.

[0016] Please refer to Figs. 1-2 , the remote sensing enhanced time series prediction method based on physically assisted multi-modal Transformer, including the following steps: S1: Data acquisition, acquire spatio-temporal sequence data and remote sensing image data, and construct a graph structure according to the latitude and longitude coordinates of the sensor and the wind field data; In the data acquisition stage, the data comes from professional channels such as sensor networks, remote sensing satellites, and meteorological monitoring stations. Specifically, through the deployment of sensors in the monitoring area, environmental and target variable data such as temperature, humidity, and pollutant concentration are collected to form spatio-temporal sequence information; through meteorological stations, wind profile radars, or reanalysis datasets, wind speed and direction data synchronized with the spatio-temporal sequence are obtained for advection process modeling; satellite remote sensing, unmanned aerial vehicle remote sensing, and other methods are used to collect multispectral / hyperspectral remote sensing images containing visible light, near-infrared, and other spectral channels.

[0017] At the same time, according to the latitude and longitude coordinates of the sensor and the wind field data, a graph structure containing nodes, edges, and weights is constructed - the nodes correspond to the sensor positions, and the weights of the edges are calculated through spatial distance or wind field. These data provide an information base that covers time series dynamics, spatial semantics, and physical driving relationships for the model, and after collection, preliminary verification is required to ensure the integrity and usability of the data; S2: Data preprocessing: preprocess the data collected in S1; The collected data is pre-processed to convert the original data into a format suitable for training and input into each module. Through patch cutting, missing value processing, standardization, and a series of operations, the data quality and generalization ability of the model are improved.

[0018] Patch cutting: For spatio-temporal sequence data, cut into (TS patches) at fixed time intervals T; for remote sensing image data, cut into (time series patches) at fixed spatial size HxW, realizing the encapsulation of local information in time series and space; Missing value processing: If there are missing values in the spatio-temporal sequence data collected by the sensor, for continuous physiological indicators, use time series interpolation method to fill in; for local missing of remote sensing image, use neighborhood pixel replacement or interpolation method based on graph structure to process; Feature standardization: For numerical features in TS patches and RS patches, perform Z-score standardization, formula is ( is the mean, is the standard deviation, calculated based on the training set), eliminate the influence of dimension, and make the features in comparable scale; Graph structure adaptation: Check the integrity and rationality of the adjacency matrix of the constructed graph structure data. If the edge weight is abnormal due to changes in sensor position, etc., recalculate and adjust based on the latest position and wind field data to ensure that the graph structure can accurately assist the feature encoding of the physical dynamics module; S3: Construct the physical information differential equation and solve it through the neural ODE solver, encode the solution to get , and process the spatio-temporal sequence data based on multi-scale position encoding and multi-head attention to get , and fuse the two to get the fusion result , and based on the fusion result get the time series latent feature ; Discrete diffusion-advection equation in closed system: linear method (MOL) discretizes partial differential equations into ordinary differential equations at specific positions Grid, get: where, represents the Laplacian of the diffusion graph, used to simulate the diffusion process of data in space. The Laplacian is , where is the degree matrix (the elements on the diagonal are the degrees of the nodes), is the adjacency matrix. Laplace operator for advection, used to model the advection (transport) process of data in space. Approximated by a Chebyshev graph neural network (GNN). Different graph structures are constructed, respectively, for modeling spatial diffusion and wind-driven advection dynamics.

[0019] Diffusion graph construction: The diffusion graph captures spatial-level correlations by fusing spatial distance and semantic similarity from remote sensing images. Traditional studies can only model the proximity at the station level, which largely limits large-scale diffusion modeling. To break this limitation, the invention utilizes the regional feature correlations extracted from remote sensing images to construct a graph structure that can reflect more extensive spatial interactions. An undirected weighted graph is constructed, where the weight of each edge reflects the joint influence of spatial adjacency and feature similarity based on remote sensing images: where, denotes the Haversine distance (earth surface distance) between locations i and j is a scaling factor to adjust the spatial decay rate. To incorporate the semantic similarity extracted from remote sensing images, is dynamically defined as the cosine similarity between remote sensing image embeddings: where, and denote the remote sensing spectral feature embeddings at locations i and j, respectively. This design ensures that edges are not only enhanced due to spatial proximity but also due to spectral similarity, thus achieving more physically consistent graph-based diffusion modeling.

[0020] Advection graph construction: An undirected weighted graph is constructed to model the advection process driven by wind dynamics. Instead of directly relying on raw wind speed and direction, the invention employs neural networks such as temporal convolutional networks (TCNs) or multi-layer perceptrons (MLPs) to convert historical wind features into latent representations that encode the local flow characteristics of each node.

[0021] The edge weight between nodes i and j is defined by the distance between their transformed representations: where, and denote the neural network-based embeddings of wind features at nodes i and j , respectively. denotes the existence of a connection between nodes (i.e., the element in the adjacency matrix of the graph is 1). ​is an activation function that computes the difference in wind speed between nodes and .

[0022] Discrete advection-diffusion differential equation: by integrating the diffusion and advection processes in previous studies, the present invention defines the control dynamics as: where, denotes the learnable parameters for approximating the graph Laplacian. The gating coefficient is obtained by a linear mapping that mimics the relative contribution of diffusion and advection at each spatiotemporal location. The diffusion constant k and the residual term coefficient β are estimated by a recurrent neural network (RNN) called coefficient estimator.

[0023] The present invention employs a neural ODE solver to solve the above physical information differential equation ; Given the initial condition at time T , the solver generates the future trajectory of the target variable from time to : which is subsequently encoded into the latent space: where, captures the latent spatiotemporal dynamics of the target variable and is enhanced by the physics-based structure, which improves the model's generalization ability and interpretability by combining domain knowledge with data-driven learning; To enhance the model's sensitivity to heterogeneous temporal changes, the present invention employs a multi-scale positional encoding (PE) scheme, which turns the "temporal resolution" into a learnable hyperparameter, allowing the same set of attention mechanisms to naturally possess multi-focal perception. This encoding, based on multiple frequency bands and resolution levels of sinusoidal functions, enables the model to detect short-term changes observed in physical transmission processes.

[0024] Single-scale encoding: for any discrete temporal position p and a given scale , first scale the position and then map it to the sine-cosine space, with the specific formula as follows: where, is the predefined scale, p denotes the index value in the time series, represents the low-frequency long-period component, which is amplified with the growth of time, while the high-frequency short-period component is compressed, preventing the early position from being erased due to too small values, is the frequency component determined by the embedding dimension, which is distributed in proportion to the wide frequency band and can fully cover the wide frequency band, ensuring that each scale can cover the complete phase space.

[0025] Multi-scale aggregation: To form a unified time encoding, the invention takes a simple arithmetic mean on three predefined scales without manual weighting: where, represents the time diffusion level. This produces a rich representation that hierarchically embeds the time structure, and the mean operation hierarchically embeds short-term fluctuations, medium-term rhythms, and long-term trends into the same channel, becoming the "time prior" of subsequent attention.

[0026] In order to let different attention heads focus on the most relevant time scale in their respective subspace, rather than manually segmenting the frequency band. The invention adds the sequence features output by the time encoder (element-wise addition, not concatenation, maintains the original dimension and reduces computational complexity) to the standard multi-head attention (MHA): where, represents the enhanced representation of the time series data after multi-scale position encoding and multi-head attention processing. Here, represents the time window from time to . represents the original time series data, also covering the time window from time to . represents the multi-scale aggregated position encoding, which provides the model with position information for each time point in the time series. In this formula, the position encoding is generated by multi-scale position encoding, aiming to enhance the model's sensitivity to changes in different time scales; The time encoder is a core module specifically designed to extract dynamic features of time series. Its function is to generate sequence features that reflect future trends based on the input historical time patch, providing key support for time series on the multi-modal fusion side, and forming a dual feature extraction pillar with the remote sensing encoder. The input of this encoder is the historical time patch, i.e. the subsequence fragment obtained by sliding window segmentation from the original time series observation data (such as sensor records), denoted as where, is the number of time steps, is the number of single time step samples, is the sample attribute dimension, covering continuous time steps from to .

[0027] The process of outputting sequence features is divided into three steps: first, multi-scale position encoding is performed to inject multi-scale time information such as day and week for each time step of the time sequence patch, and the position encoding of the integrated time information is obtained by averaging the single-scale encoding generated by the sine-cosine function ; then multi-head attention aggregation is performed, the time sequence patch and the position encoding are spliced in the feature dimension to form enhanced features, and the dynamic dependence between time steps is learned through multi-head attention to output attention weighted features; finally, through future step mapping, the historical features after attention aggregation are projected to the future time window through a linear layer to obtain sequence features of future steps.

[0028] In order to integrate the representation of physical guidance and the representation based on attention, the present application introduces a fusion module to align the two potential dynamics: wherein, represents the spatiotemporal dynamics constrained by physics, represents the time pattern captured by the attention mechanism; The gating unit includes a set of learnable gating parameters that learn the contribution weight of the physical constraint features and the time sequence enhanced features in different scenarios through model training. Specifically, the gating unit first performs linear transformation on the spliced features and generates gating coefficients consistent with the spliced feature dimension through an activation function (such as Sigmoid), and then element-wise multiplies the gating coefficients and the spliced features to achieve selective enhancement of the physical constraint information and the time dynamic information - that is, to give high weight to key features and low weight to redundant or noisy features. The feature vector processed by the gating unit is directly used as the time sequence latent feature output.

[0029] This step inputs the physical dynamics encoding and the time pattern captured by the attention mechanism to the fusion module at the same time. The fusion module then performs vector splicing to better integrate the two representations, and a gating unit is used to adjust the dynamic weight, and finally better output the time sequence latent feature .

[0030] In this way, the model can utilize the prior knowledge of the physical process, and also fully play the role of the data-driven attention mechanism, thereby improving the performance of the model, enabling the model to achieve efficient and lossless multi-modal data fusion without performing spatio-temporal resampling operations, the fusion mechanism can capture short-term fluctuations and long-term trends, and the model can still maintain high prediction accuracy under strong noise interference and sparse observation conditions, improving the generalization ability and interpretability of the model; S4: extracting the remote sensing image data based on the remote sensing encoder and outputting remote sensing latent features ; By the pyramid ViT, the present application defines a remote sensing (RS) encoder with physical perception as follows: wherein, is the input of RS, which refers to remote sensing data at time t, PVT represents the original pyramid vision transformer (Pyramid Vision Transformer) model, which functions to process and extract features from the remote sensing data, represents the remote sensing encoder, which encapsulates the PVT model to realize multi-scale spatial feature extraction. Specifically, the PVT learns the spatial dependency between patches through a self-attention mechanism, and gradually extracts high-level semantic features through a multi-layer pyramid vision transformer structure, and finally outputs remote sensing latent features ; S5: fusing the time series latent features and the remote sensing latent features using a cross-Transformer; The multi-modal bottleneck Transformer integrates shared tokens from remote sensing (RS) and time series (TS) representations to perform modality-specific attention. It identifies common distribution patterns of the two modalities and promotes alignment by integrating time series tokens, remote sensing tokens, and shared tokens in a unified attention mechanism.

[0031] (1) Shared tokens and modality-specific attention: Introduce a small number of shared tokens as a "bottleneck" for the interaction of RS and TS representations, which filters the core correlation patterns of the two modalities. At the same time, perform modality-specific attention mechanism - TS tokens preferentially focus on shared tokens related to their own region, and RS tokens preferentially focus on shared tokens related to their own region, ensuring that the unique characteristics of each modality are preserved.

[0032] (2) Fusion of cross-Transformer: In order to realize the fusion of time series and remote sensing input To effectively exchange information between the derived features, the application adopts a multi-modal Transformer fusion framework. Specifically, the application adopts a cross-Transformer mechanism that allows asymmetric attention across modalities and preserves modality-specific parameters.

[0033] and respectively represent the time series features and remote sensing features of the first l layer. The fusion process of the first l+1 layer is formulaed as: Each cross-Transformer applies modality-specific attention with shared context information. The parameters and are different and learnable, reflecting modality-specific adaptability.

[0034] Internally, the cross-Transformer follows a generalized multi-head cross-attention formula: wherein, and represent the query and key-value modalities, denotes layer normalization. This cross-attention framework allows one modality to selectively focus on the other, facilitating bidirectional context exchange.

[0035] If parameter sharing is enforced (i.e. = ), the cross-Transformer simplifies to a symmetric fusion architecture, functionally equivalent to a vanilla Transformer layer with joint token sequences, but this framework better adapts to the essential differences between TS and RS in temporal / spatial characteristics through asymmetric parameter design.

[0036] After iterative optimization by the cross-Transformer, the updated TS tokens, RS tokens, and shared tokens associated with the condensed cross-modal core are incorporated into a unified attention mechanism, wherein the TS tokens reference associated RS tokens and shared tokens, the RS tokens reference focus on matching TS tokens and shared tokens, and the shared tokens act as a "bottleneck" to filter redundant noise, ultimately generating region-level fusion features . Both the fine-grained temporal dynamics of TS and the macroscopic spatial distribution of RS are preserved, and the cross-modal semantic consistency and complementarity are strengthened through the synergy of shared tokens and modality-specific parameters, providing a reliable foundation for subsequent region-aware contrastive learning; S6: build a contrast loss function, and build a total loss function based on the contrast loss function; The region-aware contrastive learning module integrates the remote sensing latent features of multiple regions and the time sequence input, and performs region-level alignment by minimizing the distribution difference of the remote sensing and time sequence data in each region.

[0037] In order to effectively align the heterogeneous features between remote sensing (RS) images and time sequence (TS) signal modalities, the present application introduces a region-aware contrastive learning mechanism on the basis of region-level fusion features, from which the remote sensing latent features and the time sequence latent features of each region are extracted. These features have preliminarily integrated the cross-modal context information (such as the RS features integrated into the dynamic of the key nodes of the crop grain filling period, and the TS features carrying the spatial region attribute of the plain / mountainous area) through the fusion process of the multi-modal bottleneck Transformer. By defining the remote sensing latent features and the time sequence latent features of the same geographical region as a positive sample pair, the features of different regions (such as the remote sensing latent features of region and the time sequence latent features of region , ≠ ) as a negative sample pair, and using a contrastive loss function to minimize the distribution difference of the two modal features within the region and to strengthen the feature discrimination between regions, the region-level alignment fusion features are finally upgraded to have both the fusion advantages and the region-level semantic alignment, while preserving the original spatial global information (such as the terrain correlation between regions) and the time dynamic mode (such as the trend of the crop growth cycle).

[0038] Contrastive loss function: following the SimCLR (Simple Contrastive Learning of Visual Representations) paradigm, in order to promote the model to learn the cross-modal pairs with semantic alignment to be closer, and the unrelated pairs to be farther apart, the present application uses the contrastive loss function as shown below: wherein, represents the contrastive loss function, represents the total number of regions, represents the cosine similarity function, is a temperature scaling factor, represents the remote sensing latent features of the i-th region, represents the time sequence latent features of the i-th region, represents the time sequence latent features of the j-th region, represents the exponential function with the natural constant as the base number, represents the logarithmic function. Thus, the modal alignment within the same region is forced, and the feature representations of different geographical regions are distinguished.

[0039] Total loss function: The present application combines this pair of contrastive objectives with the main task loss (e.g. mean square error) by weighted sum: where, is the mean square error loss, measuring the difference between model prediction and true value. represents the weight parameter, used to balance the influence of contrastive loss and mean square error loss.

[0040] The key of the total loss function is to form a closed-loop optimization of fusion-alignment-prediction, through backpropagation, the gradient of which will simultaneously act on the modality-specific parameters (WRS, WTS) and shared token of the multi-modal bottleneck Transformer. For example, if the alignment error of a certain regional feature is large, the loss gradient will prompt to enhance the spatial-related components in the time-series feature of that region, to enhance the time-series-related information in the spatial feature, so that the subsequently generated features are more likely to satisfy the regional-level alignment constraints; S7: Output the prediction results; In units of regions, map the alignment fusion features of each region to the specific crop yield prediction results of the corresponding region. This process, through the macro-spatial distribution of the retained remote sensing (RS) latent features in the features (such as regional vegetation coverage differences, terrain patterns, and soil fertility spatial distribution) and the time-series (TS) input fine-grained temporal dynamics (such as the temporal fluctuations of crop growth rate, precipitation distribution, and accumulated temperature changes during the crop growing season), combined with the cross-modal context association formed during the multi-modal bottleneck Transformer fusion process (such as the association of the vegetation index change during the grain filling period with the accumulated temperature and precipitation during that period in the TS feature, and the RS feature carrying the RS spatial region attributes such as the differences in crop growth cycle in different terrain regions), realizes the accurate mapping of features to yield prediction values. The mapping of features to yield prediction values follows the biological logic of crop growth. For the alignment features of the same region with "high vegetation coverage and fast growth period of crops", a high yield prediction will be obtained. For example, when the remote sensing latent features show that the crops in a certain region have high vegetation coverage and good leaf growth during the grain filling period, and the time-series input shows that the accumulated temperature is sufficient and the precipitation is suitable during that period, which matches the time-series dynamics of crop rapid growth, then the yield prediction value of that region will be relatively high. For different regions, due to feature differences (such as fertile plain areas and barren mountain areas), the yield prediction values will also differ, matching the regional geographical attributes and time-series dynamics. Fertile plain areas usually show high vegetation growth in remote sensing features and suitable growing season weather in time-series features, so the yield prediction is relatively high. Barren mountain areas usually show low vegetation coverage in remote sensing features and factors such as drought during the growing season in time-series features, so the yield prediction is relatively low.

[0041] Finally, the mapping results are presented in regions by structured output, including yield values of each region, and the results retain the time dynamic trend (such as the yield change trajectory in the whole growth period) and the spatial distribution rule (such as the yield correlation of adjacent regions due to environmental similarity), ensuring that the spatial correlation, time dynamics and cross-modal semantic consistency of the region-level alignment fusion features are matched; To evaluate the RS4TS proposed in the present application, it is compared with a series of baseline models grouped by input modalities: remote sensing only (RS-only), time series only (TS-only), and fusion-based models.

[0042] Remote sensing only models: ResNet, ViT, PyViT, ConvLSTM, UNet-ConvLSTM, which extract spatio-temporal patterns only from remote sensing images.

[0043] Time series only models: LSTM, STAEformer, STID, STGCN, GraphWaveNet, BigST, GNNRNN, which extract time dynamics only from continuous sensor data.

[0044] Fusion-based models: CrossViViT, MMST-ViT, BLIP, CLIP, which combine remote sensing and time series modalities through joint attention or representation learning to improve prediction performance.

[0045] Fig. 2 Crop yield prediction results. The best results are shown in bold, and the second best results are underlined. The innovative content of the present application is as follows: (1) Non-aligned multi-modal patch fusion Non-aligned multi-modal patch fusion is the core means to solve the inconsistency of spatio-temporal sampling granularity between remote sensing data and spatio-temporal sequence data. This technology cuts the spatio-temporal sequence into TS patches at fixed time intervals and cuts the remote sensing image into RS patches at fixed spatial size, realizing the encapsulation of local spatio-temporal information; at the same time, a small amount of shared tokens are introduced as the "bottleneck" of cross-modal information interaction, controlling the information exchange strength of the two modal features, avoiding the interference of intra-modal redundant noise, without the need for traditional up-sampling or down-sampling operations, reducing information loss, and thus realizing the effective fusion of non-strictly aligned data.

[0046] (2) Physically driven spatio-temporal dynamics modeling The physical-driven spatiotemporal dynamics modeling aims to embed the diffusion-advection physical process into the neural network to realize the fusion of physical priori and data-driven learning. The technology discretizes the partial differential equation into a system of ordinary differential equations on a graph structure based on the line method, approximates the diffusion graph Laplacian and the advection graph Laplacian through Chebyshev GNN respectively, simulates the spatial diffusion and the advection process driven by the wind field; constructs a diffusion graph combining the spatial distance and the semantic similarity of remote sensing, and an advection graph based on the neural network embedding of the wind field features; and solves the physical information differential equation by using a neural ODE solver to generate the future trajectory with physical constraints.

[0047] (3) Multi-scale position encoding enhances attention The multi-scale position encoding enhances attention is used to improve the sensitivity of the model to heterogeneous temporal changes. The technology balances short-term fluctuations and long-term cycles by combining logarithmic scaling and sine-cosine functions through single-scale encoding; and realizes multi-scale aggregation by arithmetic averaging of predefined scales, hierarchically embeds short-term, medium-term and long-term time features, provides "time priori" for multi-head attention, and enables different attention heads to automatically focus on the most relevant time scale, enhancing the model's ability to capture information of different time resolutions.

[0048] (4) Multi-modal bottleneck Transformer fusion The multi-modal bottleneck Transformer fusion realizes deep interaction of remote sensing and time series features through asymmetric cross-attention. In the l+1 layer fusion, the technology efficiently processes the time series features while also efficiently processing the remote sensing features through cross-Transformer. In order to effectively maintain the feature distribution characteristics of different modal data, the invention adopts a special parameter configuration scheme for each modality, realizes directional information exchange through multi-head cross-attention, allows one modality to selectively focus on the key features of another modality, significantly improves the fusion efficiency, and realizes deep fusion of the two modal features.

[0049] (5) Region-aware contrastive learning The region-aware contrastive learning is used to realize semantic alignment of remote sensing and spatiotemporal sequence features. The technology defines remote sensing features and time series features in the same geographical region as positive sample pairs, and features in different regions as negative sample pairs; through a contrastive loss function, it forces the cross-modal features in the same region to be closer in the semantic space while widening the distance between the features in different regions; combines the contrastive loss with the mean square error loss to form a total loss function, which optimizes the prediction accuracy while ensuring the numerical and semantic consistency of the cross-modal features.

[0050] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A remote sensing-enhanced time series prediction method based on physical-assisted multimodal Transformer, characterized in that: Includes the following steps: S1: Data acquisition, obtaining spatiotemporal sequence data and remote sensing image data, and constructing a graph structure based on the latitude and longitude coordinates of the sensor and wind field data; S2: Data preprocessing: Preprocessing the data collected in S1; S3: Construct physical information differential equations and solve them using a neural ODE solver, then encode the solution results. Furthermore, it obtains spatiotemporal sequence data based on multi-scale location encoding and multi-head attention processing. The two are then merged to obtain the fusion result. And based on the fusion results Obtain latent features of time series ; S4: Extract and output remote sensing latent features from remote sensing image data based on the remote sensing encoder. ; S5: Utilizing latent features of time series and remote sensing latent features Multimodal fusion is performed using a cross-transformer. S6: Construct the contrastive loss function, and construct the total loss function based on the contrastive loss function; S7: Output the prediction results.

2. The remote sensing-enhanced time series prediction method based on physical-assisted multimodal Transformer according to claim 1, characterized in that: The preprocessing in S2 specifically includes: Patch cutting: Spatiotemporal sequence data is cut into TS patches at fixed time intervals T. Remote sensing image data is cut into time-series patches according to a fixed spatial size H×W. ; in, R For the real number space, It is the number of time steps. This is the number of samples per time step. It is the sample attribute dimension. It is the number of bands in the remote sensing image. It is the vertical pixel count of the patch. It is the number of horizontal pixels in the patch; Feature standardization: Z-score standardization is performed on numerical features in the TS and RS patches.

3. The remote sensing-enhanced time series prediction method based on physical-assisted multimodal Transformer according to claim 1, characterized in that: The physical information differential equation constructed in S3 is specifically: Discrete diffusion-advection differential equation. The specific expression is as follows: in, This represents the element-wise multiplication operator, used for gating coefficients. Dynamic weighting of physical terms, The state tensor representing spatiotemporal sequence data. Indicates the gating coefficient. It is a continuous-time independent variable that describes the dynamic evolution of a physical process. Represents the Laplace operator for the diffusion graph. This represents the Laplace operator for the advection graph. Represents the diffusion constant. Represents the coefficient of the residual term. express right The differential; The above physical information differential equations are solved using a neural ODE solver. The solver generates the objective variable from time... arrive Future trajectory: in, This represents a prediction of the future state under physical constraints, integrating the evolution results of the diffusion-advection process. This represents the ODE neural solver function. Indicates historical time T The status input at any given time, Representing the discrete diffusion-adsorption differential equation, It is the historical deadline step. It is the number of steps to predict the future; Will Encoding into the latent space : in, Indicates parameters The encoder maps the physical prediction results to the latent space. Represents the latent features of physical encoding, used for multimodal fusion; The process in S3, which involves processing spatiotemporal sequence data based on multi-scale positional encoding and multi-head attention, yields the results. The steps are as follows: S3.1: For any discrete time position and given scale First, the position is scaled, and then mapped to a sine-cosine space. The specific formula is as follows: in, Indicates low-frequency long-period components. For a predefined scale, Represents the index value in the time series. Represented by the embedding dimension Determined frequency components, , They represent sine and cosine respectively. Representing scale Location encoding, embedding dimension refers to the encoding of discrete-time locations. The target dimension when mapped to a high-dimensional feature space; S3.2: Multi-scale aggregation, the specific expression is as follows: in, Indicates to of Summation, This represents multi-scale aggregated positional encoding. express The number of elements in the middle; S3.3: Will and sequence features output by the time encoder The elements are added together and then fed into a standard multi-head attention system, as shown in the following expression: in, This represents an augmented representation of time-series data after multi-scale location encoding and multi-head attention processing. Indicates standard multi-head attention. This represents the sequence features output by the time encoder. This represents multi-scale aggregated location encoding; The fusion process in S3 is implemented based on the fusion module, and the specific expression is as follows: in, This represents the potential characteristics of physical coding. It includes vector concatenation and gating units. An augmented representation of time-series data after multi-scale location encoding and multi-head attention processing; The fusion module will and Vector concatenation is performed to obtain the concatenated feature vector, which is then adjusted using a gating unit. The feature vector processed by the gating unit is directly used as the latent feature of the time series. Output.

4. The remote sensing-enhanced time series prediction method based on physical-assisted multimodal Transformer according to claim 1, characterized in that: In step S4, remote sensing latent features are extracted from remote sensing image data based on the remote sensing encoder and output. Specifically: in, Remote sensing data representing time t, Indicates a remote sensing encoder. The spatial dimension parameter representing remote sensing data, The state tensor representing spatiotemporal sequence data. This represents the original pyramid visual Transformer model. This represents the diffusion constant.

5. The remote sensing-enhanced time series prediction method based on physical-assisted multimodal Transformer according to claim 1, characterized in that: The first in the cross Transformer l+ The formula for the fusion process of layer 1 is: in, and They represent the first l Time-series latent features and remote sensing latent features of the layer, and They represent the first l+1 Time series features and remote sensing features of the layer Indicates the cross-fusion parameters of the remote sensing side. Indicates the cross-fusion parameters on the time series side. This indicates a cross Transformer.

6. The remote sensing-enhanced time series prediction method based on physical-assisted multimodal Transformer according to claim 1, characterized in that: The S6 constructs the contrastive loss function expression as follows: in, This represents the contrastive loss function. Indicates the total number of regions. Represents the cosine similarity function. It is a temperature scaling factor. Represents the remote sensing latent features of the i-th region. Represents the temporal latent features of the i-th region. Represents the temporal latent features of the j-th region. This represents an exponential function with the natural constant as the base. Represents a logarithmic function; The expression for the total loss function constructed based on the contrastive loss function is as follows: in, It is the mean squared error loss. This represents the weight parameters. This represents the total loss function.

Images