An ENSO prediction method based on an attention mechanism-based MSTKAN model

Abstract

The present application belongs to the technical field of ENSO prediction method, and particularly relates to an ENSO prediction method based on an MSTKAN model of an attention mechanism, comprising the following steps: obtaining monthly data of three important marine and atmospheric variables of global upper ocean temperature anomaly and zonal wind stress and meridional wind stress; performing preliminary processing on the data mainly including: constructing a prediction model; performing network training optimization by using an Adam algorithm; training and verifying the model. The MSTKAN can capture fine-grained features in different directions by its multi-axis spatiotemporal attention mechanism, and fully considers the inherent directional dependence in ENSO data. And by introducing a Kolmogorov-Arnold network to replace the traditional linear layer, a learnable nonlinear function is used to more flexibly and accurately approximate the output features, thereby enhancing the ability of the model in capturing complex nonlinear relationships. Through the improved architecture, the MSTKAN surpasses the existing mainstream models in the long-term and short-term prediction accuracy of ENSO, and provides more reliable prediction results.

Description

Technical Field

[0001] The present invention belongs to the technical field of ENSO prediction methods, and specifically relates to an ENSO prediction method based on the MSTKAN model of the attention mechanism. Background Art

[0002] Accurately predicting real-time ocean-atmosphere conditions remains a long-standing challenge in climate research and has important scientific and economic significance. Take the El Niño-Southern Oscillation (ENSO) as an example. This is the most significant ocean-atmosphere phenomenon in the tropical Pacific, occurring on an interannual scale, and is mainly manifested as basin-wide sea surface temperature (SST) anomalies and the associated atmospheric circulation anomalies. Studies have shown that ENSO has a profound impact on the climate system and society through remote interactions between the atmosphere and the ocean. For example, the occurrence of ENSO events often triggers extreme weather on a global scale and even affects global crop yields in the following year. Therefore, over the past few decades, a large amount of research has been devoted to understanding and predicting ENSO.

[0003] With the deepening of observations and understanding of the El Niño-Southern Oscillation (ENSO), significant progress has been made in its simulation and prediction. Physics-based dynamical models have been a core tool for understanding and predicting ENSO. However, deficiencies in these models in representing key processes have led to biases in climate system simulations and made long-term ENSO predictions challenging. Currently, predicting ENSO for periods exceeding one year using traditional physics-based models remains extremely challenging.

[0004] However, the rapid development of deep learning (DL) algorithms and their innovative applications in earth sciences have provided new perspectives for improving the modeling of weather and climate phenomena. Unlike models based on physical theories, data-driven DL models automatically capture the intrinsic relationship between input predictors and output predictors through neural networks without relying on explicit physical laws, thereby significantly improving the modeling accuracy of nonlinear systems, including ENSO predictions. For example, DL models have successfully achieved The sea surface temperature (SST) index has been predicted more than 15 months in advance and has effectively alleviated the spring predictability barrier (SPB). In addition, the DL model can more comprehensively predict ENSO-related SST and atmospheric anomalies, generating more reasonable and robust results by considering the spatiotemporal dependencies of multiple anomaly fields.

[0005] In recent years, deep learning, particularly neural networks (CNN, RNN, LSTM), has made significant progress in ENSO prediction, especially in capturing spatiotemporal dependencies and improving prediction accuracy. However, these models face challenges in balancing long-range dependencies and extracting local features. To this end, Transformer-based models utilize self-attention mechanisms to address the problem of modeling long-range dependencies, but still face difficulties in extracting fine-grained local features. Most spatiotemporal models focus on global dependencies and ignore the east-west and north-south spatial dependencies in ENSO data, which are crucial for accurate ENSO prediction. In addition, traditional linear prediction layers cannot effectively capture the complex nonlinear relationships in ENSO evolution, especially in long-term climate trend predictions, where model performance generally declines. MSTKAN replaces traditional linear layers by introducing a Kolmogorov-Arnold network (KAN). This uses learnable nonlinear functions to more flexibly and accurately approximate output features, thereby enhancing the model's ability to capture complex nonlinear relationships. Summary of the Invention

[0006] In response to the technical problems existing in the prediction lead time of the above-mentioned deep learning network, the present invention provides an ENSO prediction method based on the MSTKAN model of the attention mechanism.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] An ENSO prediction method based on the MSTKAN model of attention mechanism includes the following steps:

[0009] S1. Obtain monthly data on three important ocean and atmospheric variables: global upper ocean temperature anomaly, zonal wind stress, and meridional wind stress;

[0010] S2. Preliminary data processing mainly includes: replacing missing values ​​and invalid values; setting the required latitude and longitude range; interpolating to a regular grid and normalizing; constructing input suitable for deep learning model processing; dividing the processed data into training set, validation set, and test set;

[0011] S3. Build a prediction model: The MSTKAN model consists of three parts: a multi-layer convolutional encoder, a multi-axis spatiotemporal attention module, and a KAN prediction network. The multi-axis spatiotemporal attention module is divided into a spatial attention module, Intra-SA, and a temporal attention module, Inter-TA. It is responsible for extracting features in both time and space and capturing complex dependencies in the data. The KAN prediction network replaces the traditional linear layer with KAN, and two KAN layers are used here.

[0012] S4. Use the Adam algorithm to optimize network training, including: setting the learning rate to 1e-5, using a custom learning rate change strategy, and adopting a warm-up strategy to gradually increase the learning rate at the beginning of training and then gradually reduce the learning rate as training progresses; improving the stability and efficiency of model training;

[0013] S5. Training and validating the model: Use the model constructed in S3 and the training and validation sets obtained in S2 to train and validate the model. Propose a joint loss function based on the characteristics of El Niño and ocean and atmospheric variables. Save the best model in the validation process based on the evaluation indicators. Use the test set to experimentally test the effectiveness of the proposed model.

[0014] The method for preliminary processing of input data in S2 is:

[0015] First, long-term trends and seasonal cycles are removed to calculate monthly anomalies. The data are then interpolated to a regular grid with a latitude of 1 degree and a longitude of 2 degrees, a latitude of 0.5 degrees from 5°S to 5°N, and a latitude of 1 degree in other areas. Land grids and missing values ​​are assigned to zero. Wind stress and temperature anomalies are then normalized by the spatial mean standard deviation. Finally, nine layers of data are spliced ​​along the layer axis and input into the model.

[0016] The network structure of the MSTKAN model in S3 is:

[0017] The first module, Multi-layer Convolutional Encoder, consists of three layers of convolution;

[0018] The second module is Multi-Axis Spatiotemporal Attention Structure, which stacks several layers. Each layer contains a spatial attention module Intra-SA and a temporal attention module Inter-SA.

[0019] The third module, KAN Prediction Head, consists of global average pooling and two KAN layers.

[0020] The construction method of the multi-layer convolutional encoder in S3 is as follows: a multi-layer convolutional encoder module is used to further process the input data, including three convolutional layers Conv, using a Leaky ReLU activation function, and applying Dropout to prevent overfitting.

[0021] The construction method of the multi-axis spatiotemporal attention module in S3 is as follows: let X represent a set of input features with size Where T, C, H and W are time, number of channels, height and width respectively.

[0022] The construction method of the spatial attention module Intra-SA is as follows:

[0023] The Intra-SA block contains two parallel branches: horizontal attention (H-SA) and vertical attention (V-SA); in H-SA, the query, key and value are first generated, which are represented as and For multi-head attention, the number of heads of multi-head attention is set to M=8; next, ,and Reshape into a 2D tensor with dimensions express H horizontal blocks; the attention features of H-SA are After that, Reshape Back Then concatenate all the attention features to obtain Symmetrically, V-SA calculates vertical attention in the vertical direction to generate attention features Similarly, all V-SA features are reshaped and concatenated as The features of horizontal attention and vertical attention are concatenated to form the output of the spatial attention module.

[0024] The construction method of the temporal attention module Inter-TA is as follows:

[0025] The Inter-TA block contains two parallel branches: horizontal attention (H-TA) and vertical attention (V-TA); in H-TA, X h First, it is segmented into H non-overlapping and co-localized horizontal regions i∈{1,2,...,H}; Next, according to Generate queries, keys, and values Next, we will Reshape into a 2D tensor with dimensions Then, horizontal attention features are generated in the time dimension Afterwards, Reshape into And connect the features of all heads along the channel dimension to generate Then, the H attention outputs Fold into represents the weighted features on the horizontal plane; symmetrically, vertical attention is applied to X v To obtain the weighted features of V-TA The features of horizontal attention and vertical attention are concatenated to form the output of the temporal attention module.

[0026] The method of the joint loss function used in S5 is:

[0027] Loss_var: This loss function is used to calculate the root mean square error (RMSE) between the model's predicted variables and the true values; and average and sum over multiple dimensions to get the overall variable loss; the formula is as follows:

[0028]

[0029] Where: y pred represents the predicted value; y true represents the true value; B represents the batch size; T represents the time step; H and W represent the height and width respectively;

[0030] Loss_nino: This loss function is used to calculate the root mean square error (RMSE) between the variables predicted by the model and the true value; the formula is as follows:

[0031]

[0032] Where: n pred represents the predicted value; n true represents the true value; N represents the number of samples; T represents the number of time steps;

[0033] Combine_loss: The joint loss function combines the two loss functions to comprehensively consider the error; the formula is as follows:

[0034] combine_loss=loss_var+loss_nino.

[0035] The evaluation indicators in S5 are:

[0036] Correlation Coefficient (Corr); The correlation coefficient is used to measure the linear correlation between the predicted value and the true value; the formula is as follows:

[0037]

[0038] Where: y true is the true value; y pred is the predicted value; and They are y true and y pred The mean of ; n is the number of samples;

[0039] Root Mean Squared Error (RMSE) RMSE is used to measure the average error between the predicted value and the true value; the formula is as follows:

[0040]

[0041] Where: y true is the true value; y pred is the predicted value; n is the number of samples;

[0042] Mean Absolute Error (MAE) MAE is used to measure the average absolute error between the predicted value and the true value; the formula is as follows:

[0043]

[0044] Where: y true is the true value; y pred is the predicted value; n is the number of samples.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The proposed MSTKAN, through its multi-axis spatiotemporal attention mechanism, is able to capture fine-grained features in different directions, fully accounting for the inherent directional dependencies in ENSO data. Furthermore, by introducing a Kolmogorov-Arnold network (KAN), which replaces traditional linear layers with learnable nonlinear functions, it more flexibly and accurately approximates output features, thereby enhancing the model's ability to capture complex nonlinear relationships. Through this improved architecture, MSTKAN surpasses existing mainstream models in both long-term and short-term ENSO forecast accuracy, providing more reliable predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0048] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0049] Figure 1 This is the framework diagram of the MSTKAN model of the present invention

[0050] Figure 2 For the MSTKAN model Plots evaluating the index's predictive skill;

[0051] Figure 3 A seasonal assessment graph of the skills associated with the present invention;

[0052] Figure 4 This is a comparison chart of the prediction and actual ENSO events of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0055] This embodiment provides an ENSO prediction method based on the MSTKAN model of the attention mechanism. Figure 1-4 As shown, the following steps are included:

[0056] 1. Collect monthly data on ocean and atmospheric variables at different time points

[0057] The study area covers 92°E to 330°E and 20°S to 20°N. Three key variables were selected: meridional and zonal wind stress, and temperature anomalies at seven depths (5m, 20m, 40m, 60m, 90m, 120m and 150m) in the upper ocean.

[0058] 2. Process the collected raw flux meteorological data

[0059] Monthly anomalies are calculated by removing long-term trends and seasonal variations in the climate state. Anomaly data within the selected latitude and longitude regions are interpolated onto a regular grid. The longitudinal resolution is 2 degrees, and the latitudinal resolution is 0.5 degrees within the range of 5 degrees south and 5 degrees north latitude, and 1 degree in other areas. All land grids and missing values ​​are assigned zero. Normalization is performed by normalizing wind stress and temperature anomalies by the spatial mean standard deviation (SD) to eliminate the influence of magnitude differences during training. Finally, all normalized data are concatenated along the layer axis to construct a dataset containing nine layers, which is provided as input to the model.

[0060] 3. Build the model

[0061] An ENSO prediction method based on the MSTKAN attention mechanism is established. The basic architecture of this model consists of the following layers: a multi-layer convolutional encoder, a multi-axis spatiotemporal attention network, and a KAN prediction module. The multi-axis spatiotemporal attention module is further divided into a spatial attention module, Intra-SA, and a temporal attention module, Inter-TA. The KAN prediction network replaces traditional linear layers with KAN layers, using two KAN layers.

[0062] The multi-layer convolutional encoder module further processes the input data, including three convolutional layers Conv, which gradually extract input features, use a 3×3 convolution kernel with a step size of 1 and set padding = 1; reduce the spatial resolution, increase the number of channels, and finally adjust the input variables to a shape suitable for processing by the multi-axis spatiotemporal attention module. Leaky ReLU activation function is used, and Dropout is applied to prevent overfitting.

[0063] The multi-axis spatiotemporal attention network consists of Intra-SA and Inter-TA modules. The Inter-SA block contains two parallel branches: horizontal attention (H-SA) and vertical attention (V-SA). Let X represent a set of input features with size Where T, C, H and W are time, number of channels, height and width respectively. For each time point t∈{1,2,...,T}, first normalized using layer normalization and split along the channel to obtain the input features of H-SA and V-SA and The formula is as follows:

[0064]

[0065] Where C′=C / 2.

[0066] In H-SA, we first generate query, key and value, which are represented as and For multi-head attention, the formula is as follows:

[0067]

[0068] in, and m∈1,2,...,M, respectively represent the linear transformation matrices of query, key and value, and M=8 is the number of heads of multi-head attention. and Reshape into a 2D tensor with dimensions Attentional characteristics of H-SA The calculation is as follows: Afterwards, Reshape Back Then concatenate all the attention features to obtain Symmetrically, V-SA computes vertical attention in the vertical direction to generate attention features Similarly, all V-SA features are reshaped and concatenated as Final spatial attention module SA output Generates the following:

[0069]

[0070] For the temporal attention module Intra-TA, spatiotemporal attention is applied to the co-localized regions of a set of input sequences to extract temporal information. It has two parallel branches: horizontal attention (H-TA) and vertical attention (V-TA). We will Split into X h and Layer normalization is performed to generate the input features of H-TA and V-TA, and the formula is as follows:

[0071] (X h ,X v )=Split(LN(X)).

[0072] In H-TA, X h First, it is segmented into H non-overlapping and co-localized horizontal regions i∈{1,2,...,H}. Next, according to Generate queries, keys, and values Next, Reshape into a 2D tensor with dimensions Then, attention features are generated in the time dimension The formula is as follows:

[0073]

[0074] Afterwards, Reshape into And connect the features of all heads along the channel dimension to generate Then, these H attention outputs Fold into Represents the weighted features in the horizontal direction. Symmetrically, attention is applied to X v To obtain the weighted features of V-TA The same process as that used to generate the final spatial attention module Intra-SA output is used to generate the final temporal attention module TA output. By interweaving the multi-head spatial attention module SA and the temporal attention module TA in the MSTKAN network, it can better capture the long-distance dependencies between longitude and latitude regions. The network can learn richer features and increase the credibility of the prediction results.

[0075] The KAN prediction network consists of a global average pooling layer and two KAN layers. It transforms the encoded and attention-processed features into the final output prediction. It first reshapes the input into a suitable shape, aggregates the information through global average pooling, then generates the predicted output through KAN, which is finally reshaped into a shape that meets the output requirements.

[0076] 4. Model Training

[0077] In this example, the input data dimensions are [12, 9, 51, 120]. The target data dimensions are [20, 9, 51, 120]. To achieve optimal training results, the training batch size is set to 2, the number of training iterations is set to 50, and an early stopping strategy is used. Finally, the trained model is saved and used to obtain prediction results on the test set.

[0078] 5. Model Evaluation

[0079] Correlation Coefficient (Corr). The correlation coefficient is used to measure the linear correlation between the predicted value and the true value. The formula is as follows:

[0080]

[0081] Where: y true is the true value. pred is the predicted value. and They are y true and y pred The mean of . n is the number of samples.

[0082] Root Mean Squared Error (RMSE) is used to measure the average error between the predicted value and the true value. The formula is as follows:

[0083]

[0084] Where: y true is the true value, y pred is the predicted value and n is the number of samples.

[0085] Mean Absolute Error (MAE) MAE is used to measure the average absolute error between the predicted value and the true value. The formula is as follows:

[0086]

[0087] Where: y true is the true value, y pred is the predicted value and n is the number of samples.

[0088] The above only describes in detail the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the scope of protection of the present invention.

Claims

1. An ENSO prediction method based on the MSTKAN model of attention mechanism, characterized by: The following steps are involved: S1. Obtain monthly data on three important ocean and atmospheric variables: global upper ocean temperature anomaly, zonal wind stress, and meridional wind stress; S2. Preliminary data processing mainly includes: replacing missing values ​​and invalid values; setting the required latitude and longitude range; interpolating to a regular grid and normalizing; constructing input suitable for deep learning model processing; dividing the processed data into training set, validation set, and test set; S3. Build a prediction model: The MSTKAN model consists of three parts: a multi-layer convolutional encoder, a multi-axis spatiotemporal attention module, and a KAN prediction network. The multi-axis spatiotemporal attention module is divided into a spatial attention module, Intra-SA, and a temporal attention module, Inter-TA. It is responsible for extracting features in both time and space and capturing complex dependencies in the data. The KAN prediction network replaces the traditional linear layer with KAN, and two KAN layers are used here. Let X represent a set of input features with size , where T, C, H and W are time, number of channels, height and width respectively; The Intra-SA block contains two parallel branches: horizontal attention (H-SA) and vertical attention (V-SA); in H-SA, the query, key and value are first generated, which are represented as , ,and , used for multi-head attention, the number of heads of multi-head attention is set to M=8; next, Reshape into a 2D tensor with dimensions ,express H horizontal blocks; the attention features of H-SA are After that, Reshape Back , and then concatenate all the attention features to obtain ; Symmetrically, V-SA calculates vertical attention in the vertical direction to generate attention features , similarly, all V-SA features are reshaped and concatenated as ;Concatenate the features of horizontal attention and vertical attention to form the output of the spatial attention module; The Inter-TA block contains two parallel branches: horizontal attention (H-TA) and vertical attention (V-TA); in H-TA, First, it is segmented into H non-overlapping and co-localized horizontal regions ∈ , Next, according to Generate queries, keys, and values Next, we will Reshape into a 2D tensor with dimensions ; Then, generate horizontal attention features in the time dimension Afterwards, Reshape into , and connect the features of all heads along the channel dimension to generate ; Then, H attention outputs Fold into , representing the weighted features on the horizontal plane; symmetrically, vertical attention is applied to To obtain the weighted features of V-TA ;Concatenate the features of horizontal attention and vertical attention to form the output of the temporal attention module; S4. Use the Adam algorithm to optimize network training, including: setting the learning rate to 1e-5, using a custom learning rate change strategy, and adopting a warm-up strategy to gradually increase the learning rate at the beginning of training and then gradually reduce the learning rate as training progresses; improving the stability and efficiency of model training; S5. Training and validating the model: Use the model constructed in S3 and the training and validation sets obtained in S2 to train and validate the model. Propose a joint loss function based on the characteristics of El Niño and ocean and atmospheric variables. Save the best model in the validation process based on the evaluation indicators. Use the test set to experimentally test the effectiveness of the proposed model.

2. The ENSO prediction method based on the MSTKAN model of the attention mechanism according to claim 1 is characterized in that: The method for preliminary processing of input data in S2 is: First, the long-term trend and seasonal cycle are removed to calculate the monthly anomalies. The data are then interpolated to a regular grid with a latitude of 1 degree and a longitude of 2 degrees, and a latitude of 0.5 degrees from 5°S to 5°N, and other regions are 1 degree. The land grid and missing values ​​are assigned to zero. The wind stress and temperature anomalies are then normalized by the spatial mean standard deviation. Finally, the nine layers of data are spliced ​​along the layer axis and input into the model.

3. The ENSO prediction method based on the MSTKAN model of the attention mechanism according to claim 1 is characterized in that: The network structure of the MSTKAN model in S3 is: The first module, Multi-layer Convolutional Encoder, consists of three layers of convolution; The second module is Multi-Axis Spatiotemporal Attention Structure, which stacks several layers. Each layer contains a spatial attention module Intra-SA and a temporal attention module Inter-TA. The third module, KAN Prediction Head, consists of global average pooling and two KAN layers.

4. The ENSO prediction method based on the MSTKAN model of the attention mechanism according to claim 1 is characterized in that: The construction method of the multi-layer convolutional encoder in S3 is as follows: a multi-layer convolutional encoder module is used to further process the input data, including three convolutional layers Conv, using a Leaky ReLU activation function, and applying Dropout to prevent overfitting.

5. The ENSO prediction method based on the MSTKAN model of the attention mechanism according to claim 1 is characterized in that: The method of the joint loss function used in S5 is: : This loss function is used to calculate the root mean square error (RMSE) between the model's predicted variables and the true values; and average and sum over multiple dimensions to get the overall variable loss; the formula is as follows: in: represents the predicted value; represents the true value; B represents the batch size; T represents the time step; H and W represent the height and width respectively; : This loss function is used to calculate the root mean square error (RMSE) between the variables predicted by the model and the true value; the formula is as follows: in: represents the predicted value; represents the true value; N represents the number of samples; T represents the number of time steps; : The joint loss function combines the two loss functions to comprehensively consider the error; the formula is as follows: 。 6. The ENSO prediction method based on the MSTKAN model of the attention mechanism according to claim 1 is characterized in that: The evaluation indicators in S5 are: Correlation Coefficient (Corr); The correlation coefficient is used to measure the linear correlation between the predicted value and the true value; the formula is as follows: in: is the true value; is the predicted value; They are and The mean of ; n is the number of samples; Root Mean Squared Error (RMSE) RMSE is used to measure the average error between the predicted value and the true value; the formula is as follows: in: is the true value; is the predicted value; n is the number of samples; Mean Absolute Error (MAE) MAE is used to measure the average absolute error between the predicted value and the true value; the formula is as follows: in: is the true value; is the predicted value; n is the number of samples.

Images