Marine heterogeneous time series data adaptive prediction method based on sparse expert mixing

Abstract

The invention relates to the technical field of marine environment, and particularly provides an adaptive prediction method for marine heterogeneous time series data based on sparse expert mixing. The method comprises the following steps: preprocessing an input original ocean observation time sequence to obtain a token sequence; modeling according to the token sequence to form a token embedding sequence; preprocessing the external covariable time sequence to obtain a context vector; fusing the token embedding sequence and the context vector through cross attention to obtain a final token sequence guided by the covariable; according to the token sequence finally guided by the covariable, parameters describing probability distribution of future data points are output through the sparse gating expert layer so as to obtain a prediction result, the method achieves fine decoupling and accurate prediction of the complex ocean process, and the adaptive capacity of the model is enhanced.

Description

Technical Field

[0001] This invention relates to the field of marine environmental technology, and in particular to an adaptive prediction method for heterogeneous marine time-series data based on sparse expert mixing. Background Technology

[0002] With the increasing importance of global climate change and marine resources, accurate monitoring and forecasting of the marine environment has become crucial. Traditional ocean observation methods mainly rely on fixed-point buoys, ship-based surveys, and limited satellite remote sensing data, which have limitations in coverage and temporal resolution. To overcome these issues, the EU-led Copernicus Programme was established, aiming to provide continuous and comprehensive global environmental monitoring services through the integrated use of satellite and ground facilities. In particular, the Copernicus Marine Environment Monitoring Service (CMEMS) focuses on the marine domain, utilizing advanced satellite remote sensing technology and model simulations to generate various high-quality time-series datasets, including ocean temperature, salinity, sea level height, ocean currents, and sea ice cover.

[0003] The data provided by CMEMS not only boasts extensive geographical coverage, but its diverse observational parameters also offer indispensable support for studying ocean circulation patterns, the impacts of climate change, and biodiversity conservation. However, significant differences between various observational devices and technologies lead to the so-called heterogeneity problem: First, there is heterogeneity in sampling frequency: the data covers multiple timescales, from second-level (e.g., turbulence observations) to daily and monthly (e.g., climate change indicators); second, there is heterogeneity in observed variables: it includes a variety of physical and biochemical variables such as temperature, salinity, depth, current velocity, current direction, and chlorophyll concentration; third, different types of observation platforms, or different deployment missions of the same platform, carry different combinations of sensors. This means that the number and specific types of variables contained in each observation site (or each time series file) are different. For example, a basic CTD (Conductivity, Temperature, Depth) profile contains only three elements, while a comprehensive water quality monitoring buoy may simultaneously observe more than a dozen variables. Finally, there is model heterogeneity: ocean dynamics are complex, and even data with the same frequency and variables will exhibit drastically different underlying patterns depending on the sea area, season, or weather system (such as typhoons or eddies). In other words, the heterogeneity, characterized by diverse data sources, varying observation frequencies, and diverse variable types, increases the complexity of data analysis and modeling.

[0004] Existing ocean prediction models, including some early pre-trained models, typically employ predefined, coarse-grained mechanisms to address this heterogeneity, such as designing dedicated data processing chains or model mapping layers for different observation frequencies. The drawback of this approach is that it relies on frequency—an artificially defined, superficial characteristic—while ignoring the inherent dynamic patterns of ocean data. For example, hourly temperature data from different buoys may exhibit drastically different patterns due to geographical location and seasonal variations; conversely, daily salinity data and hourly current data may share similar tidal or seasonal periodic characteristics. Therefore, this rigid division based on prior knowledge limits the model's accuracy and generalization ability, making it difficult to construct a universal ocean foundation model. Summary of the Invention

[0005] In view of this, the present invention provides an adaptive prediction method for heterogeneous marine time series data based on sparse expert hybridization, which can achieve fine decoupling and accurate prediction of complex marine processes and enhance the adaptive capability of the model.

[0006] In a first aspect, the present invention provides an adaptive prediction method for heterogeneous marine time-series data based on sparse expert mixing, the method comprising: Step 1: Preprocess the input raw ocean observation time series to obtain the token sequence; Step 2: Model the token sequence to form a token embedding sequence; Step 3: Preprocess the time series of external covariates to obtain the context vector; Step 4: Fuse the token embedding sequence and the context vector through cross attention to obtain the final token sequence guided by covariates; Step 5: Based on the final token sequence guided by covariates, output parameters describing the probability distribution of future data points through a sparse gating expert layer to obtain the prediction results.

[0007] Optionally, step 1 includes: Given a time series of one or more variables, the data is first processed by block partitioning, which cuts it along the time dimension into multiple fixed-length, non-overlapping data blocks. The expression for the number of data blocks N is: ; in, This represents the floor function; S represents the total length of the input time series; P represents the preset fixed length of each data block; Locally continuous temporal information is aggregated into an independent unit; then, causal normalization is applied to each data block, that is, the normalized statistics of each data block are calculated only from itself and historical data blocks; after causal normalization is completed, the data block is flattened into a one-dimensional vector to obtain the token sequence.

[0008] Optionally, step 2 includes: A single, shared input projection module, consisting of a multilayer perceptron (MLP), is used. The token sequence is mapped through this module to a unified D-dimensional feature space, forming a D-dimensional token embedding x_emb, whose expression is: ; Where f_proj represents the nonlinear mapping function; x_patch represents the input data block; After the input projection operation is completed, the data is normalized and then sent to the cross-attention module for unified processing.

[0009] Optionally, step 3 includes: First, the external covariate time series is preprocessed, including block processing and projection, to convert the external covariate time series into a high-dimensional feature representation consistent with the token sequence dimension, i.e., to obtain the covariate token sequence. Subsequently, the covariate token sequence is input into the covariate adapter, which is used to perform deep encoding and information extraction on the covariate tokens to generate a context vector.

[0010] Optionally, step 4 includes: In each Transformer layer, a cross-attention module is introduced to inject information; the token embedding sequence is used as the query Q, and the context vector is used as the key K and value V. By computing cross-attention, the data of external covariates related to ocean state in the current time step are evaluated, and the data of external covariates related to ocean state are incorporated into the token sequence through residual connection. The token sequence after incorporating relevant data is first normalized; then, it is input into the causal self-attention module to perform deep modeling of the internal time dependencies of the token sequence after incorporating relevant data, resulting in the final token sequence guided by covariates.

[0011] Optionally, the sparse gating expert layer in step 5 includes: The token sequence guided by covariates is input into the sparse gated expert layer for the final nonlinear feature transformation. The sparse gated expert layer consists of multiple stacked Transformer layers, each of which contains an arbitrary variable self-attention module and a sparse expert hybrid module. First, a binary attention bias for variable indices is introduced. To distinguish between interactions within and between variables in the binary attention bias, two learnable scalar bias terms are introduced. , The attention score is calculated using the following expression:

[0012] in, , Let represent the query vector and the key vector, respectively, which correspond to the linear transformations of the m-th variable at time point i and the n-th variable at time point j. This represents the rotation matrix, used to encode relative position information, i.e., the distance between time points i and j; This represents the unnormalized attention score; Using the Softmax function Normalization is performed to obtain the final attention score. Its expression is: ; Secondly, in each layer, a sparse expert hybrid MoE layer is used to replace the fully connected feedforward network FFN. For the token embedding x_emb input to the MoE layer, the processing procedure is as follows: Expert selection in a gated network: A gated network G is used to activate experts for the input token embedding x_emb; first, the affinity score vector s between x_emb and all M experts is calculated through a linear transformation. ; Where x_emb represents the D-dimensional token embedding; W_g represents the trainable gating weight matrix with dimensions D×M; s represents the score vector of dimension M, and the affinity of x_emb with the i-th expert is... ; To achieve sparse activation, the gated network selects only the K experts with the highest scores; the TopK function returns these K highest scores and their corresponding indices; then, the Softmax function transforms the normalized K highest scores into a sparse weight vector g: ; Here, TopK(s, K) represents an operation that finds the K largest values ​​in the score vector s, i.e., the K highest scores, and sets the remaining MK values ​​to... The Softmax function transforms the normalized K highest scores into a probability distribution, ultimately yielding a sparse weight vector g, where only K positions have non-zero weight values, and the weights at the remaining positions are all 0; the sparse weight vector g is the final expert selection and weighted decision. Finally, the weighted combination of expert outputs: After determining the sparse weight vector g, the outputs of all M expert networks for the input token embedding x_emb are computed in parallel. The final output y of the MoE layer is expressed as follows: (The expression is based on the weighted summation of the sparse weight vector g.) ; in, This represents the i-th element in the sparse weight vector g, i.e., the weight assigned to the i-th expert; This represents the result of the i-th expert network processing the input token x.

[0013] Optionally, the expression for the gate function G(x) is: ; Where C represents an M×D matrix, each row of which... It is the i-th cluster center obtained by performing K-means clustering on the token sequence generated by the pre-trained model; d(x, C) represents the calculation of the input token x and all M cluster centers. The function is the Euclidean distance between them; the minus sign - is used to convert the distance into a score so that the TopK function selects the K closest experts; the steps are as follows: First, a pre-trained model is used to process the raw ocean observation data and extract the token sequences generated within it. Then, K-means clustering is applied to cluster the token sequences, resulting in M ​​cluster centers representing typical ocean data patterns. These cluster centers are then combined into a matrix C. Finally, in the MoE layer, the gating weight matrix W_g of the gating network is set as the cluster center matrix C. The process of calculating the affinity score through the gating network is as follows: .

[0014] Optionally, it includes: The system predicts the next token sequence based on historical token sequences, following an autoregressive prediction paradigm. After deep processing through all MoE layers, the final token sequence passes through a single, shared output projection layer, which outputs parameters describing the probability distribution of future data points. The system minimizes the negative log-likelihood L_pred between the predicted distribution and the actual observations. For predictions at time step t, the loss expression is: ; ; Where x_{t+1} represents the actual observed value at time step t+1; Indicates by parameters The probability distribution is determined; h_t represents the final output representation at time step t; f_out represents the unified output projection layer, which maps h_t to the probability distribution parameters. ; The total loss L_total is the weighted sum of the prediction loss and the load balancing loss, and its expression is: ; in, Indicates the predicted loss; This indicates the load balancing loss.

[0015] In a second aspect, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to execute the adaptive prediction method for ocean heterogeneous time-series data based on sparse expert hybridization, as described in the first aspect or any possible implementation thereof.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the adaptive prediction method for sparse expert hybrid ocean heterogeneous time-series data in the first aspect or any possible implementation of the first aspect.

[0017] The technical solution provided by this invention includes preprocessing the input raw ocean observation time series to obtain a token sequence; modeling based on the token sequence to form a token embedding sequence; preprocessing the external covariate time series to obtain a context vector; fusing the token embedding sequence and the context vector through cross-attention to obtain a final covariate-guided token sequence; and outputting parameters describing the probability distribution of future data points through a sparse gating expert layer based on the final covariate-guided token sequence to obtain prediction results. This method achieves fine decoupling and accurate prediction of complex ocean processes, enhancing the model's adaptive capability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A framework diagram of the adaptive prediction method for heterogeneous marine time-series data based on sparse expert hybridization provided in the embodiments of the present invention; Figure 2 The training loss of the model provided in this embodiment of the invention on moored buoy data; Figure 3 The validation loss of the model provided in the embodiments of the present invention on moored buoy data; Figure 4 This is a prediction result diagram of chlorophyll a concentration data of a certain Copernicus site based on the proposed model provided in an embodiment of the present invention; Figure 5 This is a diagram showing the prediction effect of the proposed model on the salinity data of a certain Copernicus station, provided in an embodiment of the present invention. Figure 6 This is a diagram showing the prediction effect of the proposed model on dissolved oxygen data at a Copernicus site, provided in an embodiment of the present invention. Figure 7 This is a graph showing the prediction effect of the proposed model on temperature data of a certain Copernicus station, provided in an embodiment of the present invention. Figure 8 This is a diagram showing the prediction effect of the proposed model on turbidity data at a Copernicus site, provided in an embodiment of the present invention. Figure 9 This is a diagram illustrating the prediction results of horizontal wind speed data at a Copernicus station based on the proposed model, provided in an embodiment of the present invention. Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0025] Figure 1 This is a framework diagram of the adaptive prediction method for heterogeneous marine time-series data based on sparse expert hybridization provided in the embodiments of the present invention, as shown below. Figure 1 As shown, the method includes: The overall architecture of this invention is a unified, decoder-only Transformer model optimized for processing heterogeneous time-series data. Its core design philosophy lies in achieving adaptive processing of diverse data patterns through a single input-output path and an internally dynamic expert network. In this architecture, the data flow is unidirectional and autoregressive. First, the token sequence is encoded into a high-dimensional embedding through a unified front-end, and then passed layer by layer through the MoE transformer stack. At each layer, information is first exchanged with historical tokens through a self-attention mechanism to aggregate contextual information. Then, a gating network dynamically assigns the tokens to the most suitable expert for deep feature extraction. Finally, the fully processed token representation is decoded into an accurate prediction of the future through a unified output. This end-to-end unified architecture, combined with internal dynamic sparse specialization, forms the technical foundation for the high efficiency and high generalization capability of this invention.

[0026] Step 1: Preprocess the input raw ocean observation time series to obtain the token sequence.

[0027] In this embodiment of the invention, step 1 includes: Given a time series of one or more variables (such as sea surface temperature and salinity), the data is first processed using patching, which divides it along the time dimension into multiple fixed-length, non-overlapping data patches. The expression for the number of data patches, N, is as follows: ; in, This represents the floor function; S represents the total length of the input time series; P represents the preset fixed length of each data block; Locally continuous time-series information is aggregated into a single unit. Then, to eliminate distributional differences between data from different sources and with different dimensions, and to adapt to the autoregressive prediction framework, causal normalization is applied to each data block. This means that the normalized statistics for each data block are calculated only from itself and historical data blocks, preventing future information leakage. After causal normalization, the data blocks are flattened into one-dimensional vectors to obtain token sequences. This step transforms the original ocean observation time series into a series of standardized token sequences, laying the foundation for subsequent unified processing.

[0028] Step 2: Model the token sequence to form a token embedding sequence.

[0029] In this embodiment of the invention, step 2 includes: To achieve unified modeling of all heterogeneous ocean token sequences, a single and shared input projection module is adopted, which consists of a multilayer perceptron (MLP). The token sequence is mapped to a unified D-dimensional feature space through the input projection module, forming a D-dimensional token embedding x_emb, the expression of which is: ; Where f_proj represents a nonlinear mapping function with shared parameters implemented by a multilayer perceptron (MLP); x_patch represents the input data block; After the input projection operation is completed, the data is normalized and then sent to the cross-attention module for unified processing.

[0030] In this embodiment of the invention, the above design fundamentally abandons the traditional approach of setting different entry points for different data types, which is the key to achieving model uniformity and generalization ability.

[0031] Step 3: Preprocess the time series of external covariates to obtain the context vector.

[0032] In this embodiment of the invention, step 3 includes: After constructing a robust general oceanographic foundation model, this invention proposes a fine-tuning strategy based on multi-type covariate inputs to further improve its prediction accuracy in specific sea areas or tasks. This strategy fully utilizes the unique processing capabilities of the model's infrastructure for different covariates. First, the external covariate time series is preprocessed, including block processing and projection, to convert the external covariate time series into a high-dimensional feature representation consistent with the token sequence dimension, i.e., to obtain the covariate token sequence. Subsequently, the covariate tokens are not directly mixed with the main sequence data. Instead, the covariate token sequence is fed into a specially designed, lightweight covariate adapter, which is a small neural network independent of the main model. This adapter is used to deeply encode and extract information from the covariate tokens, generating context vectors that are specifically used as keys and values ​​in subsequent attention calculations.

[0033] Step 4: Fuse the token embedding sequence and the context vector through cross attention to obtain the final token sequence guided by covariates.

[0034] In this embodiment of the invention, step 4 includes: In each Transformer layer, a Cross-Attention module is introduced to inject information; the token embedding sequence is used as the query (Query, Q) and the context vector is used as the key (Key, K) and value (Value, V). By computing the Cross-Attention, the data of external covariates related to the ocean state in the current time step are evaluated, and the data of external covariates related to the ocean state are incorporated into the token sequence through residual connections. The token sequence, after incorporating relevant data, does not directly enter the next module, but continues to follow the inherent data processing flow within the main model. First, it undergoes normalization; then, it is input into the causal self-attention module to perform deep modeling of the internal time dependencies of the token sequence after incorporating relevant data, resulting in the final token sequence guided by covariates.

[0035] In this embodiment of the invention, the advantage of this fine-tuning strategy lies in its parameter efficiency. Only a small number of parameters of the covariate projection layer and the adapter module need to be trained, which can enable the large pre-trained model to quickly adapt to new tasks. It retains the general knowledge learned by the model on massive amounts of data, and injects real-time environmental information of specific scenarios, thereby achieving a further leap in performance on the basis of zero-shot prediction.

[0036] Step 5: Based on the final token sequence guided by covariates, output parameters describing the probability distribution of future data points through a sparse gating expert layer to obtain the prediction results.

[0037] In this embodiment of the invention, the sparse gating expert layer in step 5 includes: The token sequence guided by covariates is input into the Sparse Gating Expert Layer for final nonlinear feature transformation. The Sparse Gating Expert Layer is used to perform deep temporal feature extraction and specialization on the token embedding sequence. It consists of multiple stacked Transformer layers, each of which contains an arbitrary variable self-attention module and a sparse expert hybrid module. After flattening the multivariate sequence, the model must be able to distinguish the original variable to which each data point belongs and flexibly handle the interactions between them. This invention achieves this through a specially designed arbitrary variable self-attention mechanism, ensuring that the model has permutation equivariance and flexibility when facing different combinations of variables. Its core implementation is as follows: First, binary attention biases for variable indices are introduced. To distinguish between interactions within and between variables in binary attention biases, two learnable scalar bias terms are introduced. , The attention score is calculated using the following expression:

[0038] in, , Let these represent the query vector and key vector, respectively. The query vector and key vector correspond to the linear transformations of the m-th variable at time point i and the n-th variable at time point j, respectively. The query vector and key vector are transformed by their respective linear transformation matrices. and Applied to the original input; The rotation matrix is ​​used to encode relative position information, i.e., the distance between time points i and j. It helps to capture long-term dependencies in time series and can alleviate the limitations of traditional position coding to some extent. This represents the unnormalized attention score. The final attention score is determined by the above elements and reflects the strength of the association between different time steps at a given time point and variables. Using the Softmax function Normalization is performed to obtain the final attention score. Its expression is: ; By employing the above methods, the arbitrary variable self-attention mechanism not only fundamentally solves the problem that traditional methods cannot handle an arbitrary number of variables (channels), but also ensures, through learnable bias terms, that the model can dynamically and effectively learn the complex correlations within multivariate time series and make accurate predictions without knowing the specific combination of variables in advance.

[0039] Secondly, in each layer, a sparse expert hybrid MoE layer is used instead of a fully connected feed-forward network (FFN). The processing procedure for the token embedding x_emb input to the MoE layer is as follows: Expert selection in gating networks: A gating network (G) is used to activate experts for the input token embedding x_emb; first, the affinity score vector s between x_emb and all M experts is computed through a linear transformation.

[0040] Where x_emb represents the D-dimensional token embedding; W_g represents the trainable gating weight matrix with dimensions D×M; s represents the score vector of dimension M, and the affinity of x_emb with the i-th expert is... ; To achieve sparse activation, the gated network selects only the K experts with the highest scores; the TopK function returns these K highest scores and their corresponding indices; then, the Softmax function transforms the normalized K highest scores into a sparse weight vector g: ; Here, TopK(s, K) represents an operation that finds the K largest values ​​in the score vector s, i.e., the K highest scores, and sets the remaining MK values ​​to... The Softmax function transforms the normalized K highest scores into a probability distribution, ultimately yielding a sparse weight vector g, where only K positions have non-zero weight values, and the weights at the remaining positions are all 0; the sparse weight vector g is the final expert selection and weighted decision. Finally, the weighted combination of expert outputs: After determining the sparse weight vector g, the outputs of all M expert networks for the input token embedding x_emb are computed in parallel. The final output y of the MoE layer is expressed as follows: (The expression is based on the weighted summation of the sparse weight vector g.) ; in, This represents the i-th element in the sparse weight vector g, i.e., the weight assigned to the i-th expert; This represents the processing result of the i-th expert network (an independent feedforward network) on the input token x.

[0041] Since g is a sparse weight vector with only K non-zero values, the summation operation only involves K experts in actual computation. Although the model has M experts, only K experts are activated to participate in the computation when processing any token, thus achieving extremely high computational efficiency while maintaining a huge model capacity.

[0042] To further improve the accuracy of expert assignment by gating networks, this invention proposes an optimal gating mechanism based on pre-trained knowledge. This mechanism no longer requires the gating network to learn how to assign tasks from scratch, but instead uses the knowledge contained in a pre-trained ocean basic model (e.g., a dense Transformer model without MoE) to guide the assignment.

[0043] In this embodiment of the invention, the expression for the gate function G(x) is: ; Where C represents an M×D matrix, each row of which... It is the i-th cluster center obtained by performing K-means clustering on the token sequence generated by the pre-trained model; d(x, C) represents the calculation of the input token x and all M cluster centers. The function is the Euclidean distance between them; the negative sign (-) is used to convert the distance (smaller is better) into a score (larger is better) so that the TopK function can select the K closest experts; the steps are as follows: First, a pre-trained model is used to process the raw ocean observation data and extract the token sequences generated within it. Then, K-Means clustering is applied to cluster the token sequences, resulting in M ​​cluster centers representing typical ocean data patterns. These cluster centers are then combined into a matrix C. Finally, in the MoE layer, the gating weight matrix W_g of the gating network is set as the cluster center matrix C. The process of calculating the affinity score through the gating network is as follows: This approach allows experts to be selected from the outset based on a deep understanding of the distribution of real ocean data, thereby accelerating model convergence and significantly improving its final performance.

[0044] In this embodiment of the invention, it includes: The next token sequence is predicted based on historical token sequences, following an autoregressive prediction paradigm. After deep processing through all MoE layers, the final token sequence will pass through a single and shared output projection layer, which outputs parameters describing the probability distribution of future data points. Minimizing the negative log-likelihood L_pred between the predicted distribution and the actual observations, for the prediction at time step t, the loss expression is: ; ; Where x_{t+1} represents the actual observed value at time step t+1; Indicates by parameters The probability distribution is determined; h_t represents the final output representation at time step t; f_out represents the unified output projection layer, which maps h_t to the probability distribution parameters. ; The total loss L_total is the weighted sum of the prediction loss and the load balancing loss, and its expression is: ; in, Indicates the predicted loss; This indicates the load balancing loss.

[0045] Verification of the effectiveness of the method of this invention: (1) Ocean observation datasets for the construction and validation of pre-trained models for heterogeneous ocean data; To ensure that the model can learn comprehensive and representative ocean dynamics and to objectively and fairly evaluate its performance, the following two types of datasets were constructed: Large-scale pre-trained datasets; The model pre-training of this invention is based on an authoritative, globally covered ocean in-situ observation dataset, the Copernicus Global Ocean In-Situ Near-Real-Time Observations dataset. This dataset integrates real-time and near-real-time data from various ocean observation platforms worldwide, exhibiting high heterogeneity and complexity.

[0046] In the experiments of this invention, the moored buoy (MO) subset was primarily utilized. This subset covers over 2,800 ocean monitoring stations globally, containing over 3 billion observation records. Its wide geographical distribution provides diverse learning samples for the model, thereby greatly enhancing the model's generalization ability. During the data preprocessing stage, the completeness of the time series was rigorously screened. Because the model uses a specific length (e.g., 100 bytes) of preceding sequence features, any observation record segment whose length is insufficient to extract a complete preceding sequence was discarded. After cleaning and reconstruction, the final time series dataset used for model pre-training contains continuous, high-quality observation records from 2,827 observation stations globally from 1860 to 2024, providing a solid data foundation for model construction.

[0047] Zero-sample capability and few-sample capability validation test sets; To rigorously evaluate the model's zero-shot prediction capability when faced with unseen data and its few-shot capability after fine-tuning with a small amount of data, an independent test dataset with clearly defined geographical and environmental characteristics was constructed. This dataset also originates from the observation data of the moored buoy network in the aforementioned Copernicus dataset, but the observation stations and time periods included in it were strictly excluded from the large-scale pre-training dataset.

[0048] The test set, named the Copernicus Moored Buoy Dataset (CMOD), primarily originates from specific observation stations in the Baltic Sea and the Mediterranean Sea. These two sea areas possess unique ocean dynamics and ecosystem characteristics that differ significantly from the open ocean, thus effectively validating the model's environmental adaptability and generalization performance. Throughout the validation process, the CMOD dataset was used exclusively for the final model performance evaluation and was never used for any training or fine-tuning phases, ensuring the objectivity and impartiality of the evaluation results.

[0049] The validation work of this invention mainly focuses on the following key oceanographic parameters: sea surface temperature (TEMP), salinity (PSAL), dissolved oxygen concentration (DOX1), turbidity (TUR4), chlorophyll a concentration (CPHL), and horizontal wind speed (WSPD). These parameters cover multiple aspects of physical, chemical, and biological oceanography, and their time-series characteristics and the diversity of data distribution can fully reflect the comprehensive performance of the model in real-world applications.

[0050] (2) Experimental setup; The parameters that need to be set for this model include: block size, context length, mask rate, model hidden layer dimension, number of network layers, number of attention heads, total number of experts, number of activated experts, expert network feedforward dimension, load balancing loss coefficient, optimizer, learning rate, learning rate scheduler, warm-up steps, weight decay, batch size, total number of training steps, and computational accuracy.

[0051] (3) Model training, evaluation, and testing; To verify the effectiveness and accuracy of the technical solution of this invention, a standard experimental procedure was designed, including model pre-training and generalization ability testing. The entire process was executed on a unified high-performance computing platform (a computing cluster composed of NVIDIA A100 GPUs) to ensure the reliability of the experimental results.

[0052] The model proposed in this invention is trained on the aforementioned large-scale pre-training dataset. The training process employs the AdamW optimizer, combined with a learning rate scheduling strategy of linear warm-up followed by cosine annealing, aiming to guide the model to converge stably and efficiently to the optimal state. To improve training efficiency and save computational resources while ensuring numerical accuracy, the entire training process uses bfloat16 mixed precision.

[0053] In embodiments of the present invention, such as Figure 2 and Figure 3 As shown, the training objective of the model is to minimize the negative log-likelihood between the predicted probability distribution and the true observations. Furthermore, to ensure the effectiveness of the sparse expert hybrid (MoE) module, a load balancing loss is introduced as an auxiliary objective during training. This loss term encourages the gating network to evenly distribute computational tasks among all expert networks, preventing some experts from being undertrained, thereby maximizing the overall knowledge capacity and learning efficiency of the model.

[0054] The following two widely used evaluation metrics are used to quantify the model's point prediction accuracy: Mean Absolute Error (MAE): This indicator calculates the average of the absolute values ​​of the differences between the predicted and actual values. MAE provides a clear picture of the actual magnitude of the prediction error, and its calculation formula is as follows: ; Where n represents the total number of test samples; This represents the true observation value of the i-th sample; This represents the predicted value for the i-th sample; Root Mean Square Error (RMSE): This metric is calculated as the square root of the mean of the squares of the differences between the predicted and actual values. RMSE is more sensitive to larger prediction errors (i.e., outliers) and reflects the predictive stability of the model. Its calculation formula is as follows: ; Of the two indicators mentioned above, the lower the value, the smaller the prediction error of the model and the better its performance. Figures 4 to 9 As can be seen, the model exhibits good predictive performance and strong generalization ability. The MAE and RMSE values ​​calculated on the CMOD test set objectively and comprehensively verify the predictive accuracy and practical application value of this invention in processing complex, heterogeneous marine data.

[0055] This invention abandons the approach of relying on prior knowledge such as observation frequency to divide the model structure, and instead adopts a single, unified input-output structure. The modeling task of diverse ocean time series patterns is delegated to the Sparse Mixture of Experts (MoE) transformation layer inside the model.

[0056] The technical solution proposed in this invention is as follows: (1) To construct a unified model architecture driven by the inherent patterns of data, replacing the heterogeneous processing framework that relies on hard-coded prior rules such as observation frequency. The primary problem this invention aims to solve is the fundamental limitation of current model design—namely, pre-setting different processing links for data of different frequencies or types. This invention projects all heterogeneous ocean data into a unified representation space by adopting a single, shared input / output architecture. Its core contribution lies in shifting the specialized modeling task of diverse ocean dynamic patterns from a fixed model front-end to a sparse expert hybrid (MoE) module within the model, thereby achieving efficient and flexible processing of ocean data from any source within a unified framework.

[0057] (2) A token-level dynamic assignment mechanism based on intelligent gating is established to achieve refined and adaptive modeling of the non-stationary characteristics of ocean time-series data. A significant feature of the ocean environment is its dynamic variability, meaning that data patterns change drastically over time. To address this issue, this invention introduces an intelligent gating network, enabling adaptive computation at the extremely fine-grained level of tokens (i.e., data fragments). This network can analyze the pattern information contained in each token in real time and dynamically match it with the most suitable expert subnetwork for processing. This refined dynamic scheduling capability enables the model to accurately capture and respond to instantaneous changes in ocean processes, thereby fundamentally improving the model's predictive robustness under complex and variable conditions.

[0058] (3) This paper proposes an efficient computational paradigm based on sparse activation, aiming to decouple the huge knowledge capacity of the model from the actual computational overhead during forward inference. To construct a general ocean foundation model, the model needs a large parameter capacity to memorize rich ocean patterns. However, this usually leads to huge computational costs. This invention perfectly solves this contradiction through a sparse activation mechanism. The model can contain a massive expert network, but only a few selected experts participate in the computation each time data is processed. This paradigm successfully decouples the total parameter size of the model (representing knowledge capacity) from the computational cost of a single prediction (representing operational efficiency), ultimately contributing a practical ocean prediction model that combines high performance and high efficiency.

[0059] This invention constructs a highly generalizable general oceanographic model through large-scale pre-training on massive amounts of multi-source heterogeneous observation data (such as buoy and underwater vehicle data). Unlike many traditional models that can only process time-series data with consistent sampling frequencies, the framework proposed in this invention has no strict restrictions on the input frequency, thus possessing greater versatility. The core innovation of this invention lies in the introduction of a Sparse Mixture of Experts (MoE) architecture, which enables dynamic adaptive identification and specialized processing of ocean data patterns. The model contains multiple expert sub-networks, each of which, through training, autonomously evolves into an expert adept at interpreting specific ocean dynamic patterns. For example, some are skilled at capturing tidal cycles, while others are more sensitive to mesoscale eddy signals. An intelligent gating network acts as a scheduling center, analyzing each small segment of input ocean observation data (i.e., tokens) in real time and dynamically matching it with the most suitable expert for processing. This means that the model no longer relies on a rough division of data into hourly or daily data, but rather on a refined and intelligent allocation of computational resources based on the specific marine phenomenon reflected by the data. Due to this token-level adaptive adjustment capability, the model deeply understands the non-stationary nature of the marine environment—even data from the same observation station can exhibit drastic changes in intrinsic patterns at different times (such as before and after a typhoon), and the model can respond flexibly. This method employs an efficient decoder-only training objective, endowing the model with powerful zero-shot forecasting capabilities, meaning it can reliably predict new sea areas or new observed variables without additional training. Combined with techniques such as rotational position embedding, the model effectively enhances its ability to capture long-term dependencies in marine phenomena. It also possesses few-shot forecasting capabilities. Existing techniques typically involve end-to-end training of a dedicated model or fine-tuning the entire pre-trained model, resulting in high computational costs and the potential for forgetting general knowledge. The fine-tuning strategy of this invention perfectly resolves this contradiction.

[0060] Compared with the prior art, the present invention has the following advantages: (1) It has achieved fine decoupling and accurate prediction of complex ocean processes.

[0061] Existing models typically employ a single, fixed pattern to process all data, making it difficult to distinguish between multiple ocean dynamic processes (such as tides, eddies, and seasonal variations) superimposed within the same time series. This invention, through an internal network of multiple expert networks, enables the model to autonomously evolve specialized modules each adept at analyzing different physical processes. The intelligent gating network acts as a command center, dynamically combining the expertise of these specialists for each segment of data, thus effectively decoupling mixed signals. This refined processing mechanism allows the model to gain a deeper understanding of the intrinsic patterns in ocean data, significantly improving the prediction accuracy of key elements such as sea surface temperature and salinity.

[0062] (2) It significantly enhances the model’s ability to adapt to dynamic changes in the marine environment.

[0063] The marine environment is highly non-stationary, and its data patterns can change drastically due to sudden events (such as typhoons and cold waves) or periodic phenomena (such as El Niño). Traditional static models struggle to adapt to such dynamic changes. This invention endows the model with unprecedented adaptive capabilities by dynamically assigning experts at an extremely fine granular level—the token. When data patterns change, the gating network can instantly sense this and switch to a more suitable expert combination for processing. This real-time adjustment capability ensures that the model maintains predictive robustness and reliability under various complex and even extreme marine conditions.

[0064] (3) It provides a highly unified and generalized marine data processing framework.

[0065] Ocean observation data comes from diverse sources (buoys, moorings, shore-based stations, etc.), with varying variables and frequencies. Traditional methods typically require building different dedicated models for different data sources, resulting in complex processes and difficulties in data integration. This invention employs a single, unified input-output architecture, capable of processing heterogeneous time-series data from any source without discrimination. This design, where one model serves all scenarios, greatly simplifies the ocean data modeling process. Furthermore, leveraging its general knowledge learned from large-scale heterogeneous data, it exhibits excellent zero-shot generalization capabilities, easily applicable to new sea areas or new observation variables without retraining.

[0066] (4) It takes into account both the knowledge capacity and computational efficiency of the model and has the potential for business application.

[0067] To accurately predict complex ocean systems, models require a massive parameter capacity to store rich ocean model knowledge. However, this typically translates to high computational costs, limiting their application in real-time operational forecasting. This invention perfectly resolves this contradiction through a sparse activation mechanism. The model can possess a massive expert network to ensure its powerful knowledge capacity, but when processing any data point, only a small subset is activated and computed. This design enables the model to operate at a computational cost far lower than its total parameter size, achieving the performance of a large model with the energy consumption of a small model, thus providing technical feasibility for deployment in practical operational ocean forecasting systems.

[0068] (5) It provides a smooth evolution path from general prediction to precise customization, enhancing the practical value of the model.

[0069] This invention not only constructs a powerful, general-purpose oceanographic foundation model but also proposes a covariate-guided fine-tuning method. This expands the model's application mode from a single zero-shot prediction to a dual mode of zero-shot prediction plus supervised fine-tuning. Users can flexibly choose according to their actual needs: when local data is lacking, zero-shot prediction capabilities can be used directly; when observational data and relevant covariates for a specific sea area are available, efficient fine-tuning can quickly yield an expert-level prediction model for that sea area and task. This design breaks down the barriers between basic research and operational applications, providing a plug-and-play, high-performance technical solution for practical scenarios such as marine environmental forecasting, disaster early warning, and resource management.

[0070] The technical solution provided by this invention includes preprocessing the input raw ocean observation time series to obtain a token sequence; modeling based on the token sequence to form a token embedding sequence; preprocessing the external covariate time series to obtain a context vector; fusing the token embedding sequence and the context vector through cross-attention to obtain a final covariate-guided token sequence; and outputting parameters describing the probability distribution of future data points through a sparse gating expert layer based on the final covariate-guided token sequence to obtain prediction results. This method achieves fine decoupling and accurate prediction of complex ocean processes, enhancing the model's adaptive capability.

[0071] The various steps in the embodiments of the present invention can be performed by an electronic device. This electronic device includes, but is not limited to, tablet computers, portable PCs, and desktop computers.

[0072] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the electronic device containing the computer-readable storage medium to execute the above-described embodiment of the adaptive prediction method for ocean heterogeneous time-series data based on sparse expert hybridization.

[0073] Figure 10 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 10 As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the adaptive prediction method for ocean heterogeneous time series data based on sparse expert mixing in the embodiment. To avoid repetition, it will not be described in detail here.

[0074] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0075] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0076] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive prediction method for heterogeneous marine time-series data based on sparse expert hybridization, characterized in that, The method includes: Step 1: Preprocess the input raw ocean observation time series to obtain the token sequence; Step 2: Model the token sequence to form a token embedding sequence; Step 3: Preprocess the time series of external covariates to obtain the context vector; Step 4: Fuse the token embedding sequence and the context vector through cross attention to obtain the final token sequence guided by covariates; Step 5: Based on the final token sequence guided by covariates, output parameters describing the probability distribution of future data points through a sparse gating expert layer to obtain the prediction results.

2. The method according to claim 1, characterized in that, Step 1 includes: Given a time series of one or more variables, the data is first processed by block partitioning, which cuts it along the time dimension into multiple fixed-length, non-overlapping data blocks. The expression for the number of data blocks N is: ； in, This represents the floor function; S represents the total length of the input time series; P represents the preset fixed length of each data block; Locally continuous temporal information is aggregated into an independent unit; then, causal normalization is applied to each data block, that is, the normalized statistics of each data block are calculated only from itself and historical data blocks; after causal normalization is completed, the data block is flattened into a one-dimensional vector to obtain the token sequence.

3. The method according to claim 2, characterized in that, Step 2 includes: A single, shared input projection module, consisting of a multilayer perceptron (MLP), is used. The token sequence is mapped through this module to a unified D-dimensional feature space, forming a D-dimensional token embedding x_emb, whose expression is: ； Where f_proj represents the nonlinear mapping function; x_patch represents the input data block; After the input projection operation is completed, the data is normalized and then sent to the cross-attention module for unified processing.

4. The method according to claim 3, characterized in that, Step 3 includes: First, the external covariate time series is preprocessed, including block processing and projection, to convert the external covariate time series into a high-dimensional feature representation consistent with the token sequence dimension, i.e., to obtain the covariate token sequence. Subsequently, the covariate token sequence is input into the covariate adapter, which is used to perform deep encoding and information extraction on the covariate tokens to generate a context vector.

5. The method according to claim 4, characterized in that, Step 4 includes: In each Transformer layer, a cross-attention module is introduced to inject information; the token embedding sequence is used as the query Q, and the context vector is used as the key K and value V. By computing cross-attention, the data of external covariates related to ocean state in the current time step are evaluated, and the data of external covariates related to ocean state are incorporated into the token sequence through residual connection. The token sequence after incorporating relevant data is first normalized; then, it is input into the causal self-attention module to perform deep modeling of the internal time dependencies of the token sequence after incorporating relevant data, resulting in the final token sequence guided by covariates.

6. The method according to claim 5, characterized in that, The sparse gating expert layer in step 5 includes: The token sequence guided by covariates is input into the sparse gated expert layer for the final nonlinear feature transformation. The sparse gated expert layer consists of multiple stacked Transformer layers, each of which contains an arbitrary variable self-attention module and a sparse expert hybrid module. First, a binary attention bias for variable indices is introduced. To distinguish between interactions within and between variables in the binary attention bias, two learnable scalar bias terms are introduced. , The attention score is calculated using the following expression: in, , Let represent the query vector and the key vector, respectively, which correspond to the linear transformations of the m-th variable at time point i and the n-th variable at time point j. This represents the rotation matrix, used to encode relative position information, i.e., the distance between time points i and j; This represents the unnormalized attention score; Using the Softmax function Normalization is performed to obtain the final attention score. Its expression is: ； Secondly, in each layer, a sparse expert hybrid MoE layer is used to replace the fully connected feedforward network FFN. For the token embedding x_emb input to the MoE layer, the processing procedure is as follows: Expert selection in a gated network: A gated network G is used to activate experts for the input token embedding x_emb; first, the affinity score vector s between x_emb and all M experts is calculated through a linear transformation. ； Where x_emb represents the D-dimensional token embedding; W_g represents the trainable gating weight matrix with dimensions D×M; s represents the score vector of dimension M, and the affinity of x_emb with the i-th expert is... ; To achieve sparse activation, the gated network selects only the K experts with the highest scores; the TopK function returns these K highest scores and their corresponding indices; then, the Softmax function transforms the normalized K highest scores into a sparse weight vector g: ； Here, TopK(s, K) represents an operation that finds the K largest values ​​in the score vector s, i.e., the K highest scores, and sets the remaining MK values ​​to... The Softmax function transforms the normalized K highest scores into a probability distribution, ultimately yielding a sparse weight vector g, where only K positions have non-zero weight values, and the weights at the remaining positions are all 0; the sparse weight vector g is the final expert selection and weighted decision. Finally, the weighted combination of expert outputs: After determining the sparse weight vector g, the outputs of all M expert networks for the input token embedding x_emb are computed in parallel. The final output y of the MoE layer is expressed as follows: (The expression is based on the weighted summation of the sparse weight vector g.) ； in, This represents the i-th element in the sparse weight vector g, i.e., the weight assigned to the i-th expert; This represents the result of the i-th expert network processing the input token x.

7. The method according to claim 6, characterized in that, The expression for the gate function G(x) is: ； Where C represents an M×D matrix, each row of which... It is the i-th cluster center obtained by performing K-means clustering on the token sequence generated by the pre-trained model; d(x, C) represents the calculation of the input token x and all M cluster centers. The function is the Euclidean distance between them; the minus sign - is used to convert the distance into a score so that the TopK function selects the K closest experts; the steps are as follows: First, a pre-trained model is used to process the raw ocean observation data and extract the token sequences generated within it. Then, K-means clustering is applied to cluster the token sequences, resulting in M ​​cluster centers representing typical ocean data patterns. These cluster centers are then combined into a matrix C. Finally, in the MoE layer, the gating weight matrix W_g of the gating network is set as the cluster center matrix C. The process of calculating the affinity score through the gating network is as follows: .

8. The method according to claim 7, characterized in that, include: The next token sequence is predicted based on historical token sequences, following an autoregressive prediction paradigm. After deep processing through all MoE layers, the final token sequence will pass through a single and shared output projection layer, which outputs parameters describing the probability distribution of future data points. Minimizing the negative log-likelihood L_pred between the predicted distribution and the actual observations, for the prediction at time step t, the loss expression is: ； ； Where x_{t+1} represents the actual observed value at time step t+1; Indicates by parameters The probability distribution is determined; h_t represents the final output representation at time step t; f_out represents the unified output projection layer, which maps h_t to the probability distribution parameters. ; The total loss L_total is the weighted sum of the prediction loss and the load balancing loss, and its expression is: ； in, Indicates the predicted loss; This indicates the load balancing loss.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the adaptive prediction method for ocean heterogeneous time-series data based on sparse expert mixing as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the adaptive prediction method for ocean heterogeneous time-series data based on sparse expert mixing as described in any one of claims 1 to 8.

Images