A cloud sea probability prediction method based on transformer

Abstract

The application provides a cloud sea probability prediction method based on a Transformer. The cloud sea probability prediction method based on the Transformer comprises the following steps: S1, data input; S2, data preprocessing; S3, feature extraction; S4, model training; and S5, probability prediction. The cloud sea probability prediction method based on the Transformer has the advantages of strong universality, wide adaptation range, high calculation efficiency, the ability to meet short-term and nowcasting prediction requirements, and high prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of meteorological forecasting technology, and in particular to a cloud and sea probability forecasting method based on Transformer. Background Technology

[0002] Sea of ​​clouds, a highly scenic natural meteorological landscape in mountainous areas, relies on the coupled effects of multiple factors, including a stable inversion layer, abundant water vapor, and suitable wind fields. Accurate probability forecasts of sea of ​​clouds are crucial for tourism meteorological services and scenic area operation and management. Currently, sea of ​​clouds forecasting techniques are mainly divided into three categories: traditional empirical forecasting methods, statistical learning forecasting methods, and numerical forecasting methods.

[0003] Traditional empirical forecasting methods rely on forecasters' experience and combine it with weather patterns to determine the probability of cloud formation. While the physical meaning of the forecast indicators is clear, they are heavily influenced by human experience, exhibiting strong subjectivity and insufficient accuracy and versatility. Statistical learning forecasting methods (such as logistic regression, support vector machines, and decision trees) build forecasting models by fitting historical meteorological data and cloud observation data. Although this reduces subjectivity to some extent, they suffer from insufficient capture of dependencies in long-term meteorological data and difficulty in characterizing the coupling effects of multi-scale meteorological factors. Numerical forecasting methods simulate atmospheric motion processes by constructing meteorological numerical models, but they have high computational complexity, demanding high hardware computing power, and are not precise enough in characterizing the formation and dissipation mechanisms of inversion layers and the effects of topographic disturbances. They suffer from insufficient forecast lead time and difficulty in achieving refined probabilistic forecasts.

[0004] In existing technologies, various cloud sea forecasting methods have been disclosed. For example, the study "Research on Cloud Sea Forecasting in Beijing's Alpine Regions Based on Multi-Source Stereoscopic Detection and Fuzzy Logic Algorithm" (Li Chen et al., Climate and Environmental Research, 2025) discloses a method that uses relative humidity as the core forecasting factor and combines it with a fuzzy logic algorithm to establish a probabilistic cloud sea forecasting model. This method suffers from problems such as insufficient capture of the correlation between forecasting factors, limited model generalization ability, and the need to improve the accuracy of probabilistic forecasts. Another example is the study "Research on Cloud Sea Landscape Forecasting in Yuanjiang Hani, Yunnan Based on Multiple Models" (Ding Sheng et al., Journal of Meteorology and Environment, 2020), which discloses a scheme using multiple methods such as empirical forecasting, Logistic regression, support vector machines, and decision trees for binary classification forecasting. This scheme suffers from defects such as insufficient ability to capture temporal features, lack of probabilistic forecasting ability, and poor model generalization ability. For example, a generalized forecasting method for mountain cloud sea landscapes based on vertically stratified meteorological factors forecasts by acquiring meteorological profiles, predicting the dynamic evolution of inversion layers, and combining multi-scale process coupling models. However, it suffers from problems such as high model complexity, low computational efficiency, lack of probabilistic forecasts, and insufficient capture of temporal dependencies.

[0005] Therefore, it is necessary to provide a new Transformer-based probabilistic cloud and sea forecasting method to solve the above-mentioned technical problems. Summary of the Invention

[0006] The technical problem solved by this invention is to provide a Transformer-based cloud and sea probability forecasting method that is highly versatile, widely adaptable, computationally efficient, meets the requirements of short-term forecasting, and has high forecasting accuracy.

[0007] To address the aforementioned technical problems, the cloud sea probability prediction method based on Transformer provided by this invention includes the following steps:

[0008] S1. Data Input: Collect multi-source input data, including meteorological data, topographic data, and historical cloud sea observation data; wherein, the meteorological data includes numerical weather prediction model output data, multi-source stereo sounding data, and ERA5 reanalysis data, the topographic data includes digital elevation model data, and the historical cloud sea observation data includes cloud sea appearance time, duration, and observation point location;

[0009] S2. Data Preprocessing: The multi-source input data is preprocessed to generate standardized model input data. Specifically, this includes: vertical interpolation, spatiotemporal alignment, and fusion of meteorological observation data to generate standardized meteorological profiles; filling missing data using linear interpolation; removing outliers using the 3σ criterion; and normalizing to the [0,1] interval. Based on the digital elevation model data, the topographic relief is calculated using the sliding window method, and the topographic disturbance coefficient field is calculated and standardized using the aspect factor. Historical cloud sea observation data is converted into probability labels, where the probability label for the time period in which cloud sea is observed is set to 0.8 to 1.0, and the probability label for the time period in which cloud sea is not observed is set to 0 to 0.2. The preprocessed meteorological profile data, topographic disturbance coefficient field, and mesoscale weather system elements are spliced ​​together according to the time series to construct the input sequence of the Transformer model. The sequence length T is set according to the temporal resolution of the historical data, and the prediction step size is set to 1.

[0010] S3. Feature Extraction: An improved Transformer model is used for feature extraction. The improved Transformer model adds a feature fusion module to the encoder and a probability prediction head to the decoder. The encoder consists of multiple self-attention layers and a feedforward neural network to capture long-range temporal dependencies of the input sequence. The feature fusion module is used to nonlinearly fuse temporal features with topographic perturbation coefficient fields and mesoscale weather system elements to generate a comprehensive feature vector. The probability prediction head uses a sigmoid activation function to output a cloud and sea probability value between 0 and 1.

[0011] S4. Model Training Step: The improved Transformer model is trained based on the preprocessed training data. Specifically, this includes: dividing historical data into training, validation, and test sets in a 7:2:1 ratio; using a combination of cross-entropy and mean squared error loss functions as the total loss function; training with the Adam optimizer, setting the initial learning rate to 1e-4 and employing a learning rate decay strategy; using an early stopping strategy, stopping training when the validation set loss function does not decrease for 5 consecutive epochs; introducing L2 regularization; and evaluating performance using the test set, adjusting hyperparameters and retraining if the preset requirements are not met.

[0012] S5. Probability Forecast: Input the preprocessed data of the target time period and target area into the trained improved Transformer model. The model outputs the probability and confidence level of cloud sea occurrence. Then, combined with the fuzzy logic algorithm, the probability threshold is adjusted according to the regional characteristics of the target area to correct the output probability results. Finally, the corrected cloud sea probability forecast result and confidence level are output.

[0013] Preferably, in step S1, the numerical forecasting model is a WRF or MESO high-resolution model, the multi-source stereo detection data includes microwave radiometer, wind profiler radar and gradient densification automatic weather station data, and the ERA5 reanalysis data is used to supplement mesoscale weather system elements, which include horizontal circulation field elements, thermal forcing elements, water vapor transport elements and mesoscale disturbance system parameters.

[0014] Preferably, in step S2, the terrain disturbance coefficient field is calculated as follows: the terrain undulation is calculated using the sliding window method based on digital elevation model data, the slope aspect factor is obtained by calculating the cosine of the surface slope aspect angle and the prevailing wind direction angle, and the terrain disturbance coefficient field is calculated by combining the undulation and the slope aspect factor.

[0015] Preferably, in step S3, the temporal feature extraction specifically involves: inputting the input sequence into a Transformer encoder, calculating the attention weights between each time step and each feature through a self-attention layer, and focusing on capturing the temporal variation patterns of key meteorological factors such as inversion layer height, wet layer thickness, and wind speed, as well as the impact of topographic disturbances on these factors.

[0016] Preferably, in step S4, the preset requirements are an accuracy of ≥90%, an F1 score of ≥0.85, and a TS score of ≥55; the hyperparameters include the number of self-attention layers and the feature fusion weights.

[0017] Preferably, in step S5, the input data preparation involves preprocessing the multi-source data of the target time period and target area according to the method in step S2 and constructing the input sequence.

[0018] Preferably, in step S5, the fuzzy logic algorithm is replaced by Bayesian estimation to correct the probability results.

[0019] Compared with related technologies, the cloud sea probability prediction method based on Transformer provided by this invention has the following beneficial effects:

[0020] This invention provides a Transformer-based probabilistic cloud and sea forecasting method. Utilizing the self-attention mechanism of the Transformer model, it accurately captures long-term temporal dependencies in meteorological data, effectively characterizing the dynamic evolution of meteorological factors such as inversion layers, humidity layers, and wind fields. Combined with a multi-source data fusion strategy, it comprehensively considers the impact of topographic disturbances and mesoscale weather systems. Compared to existing statistical learning models and fuzzy logic algorithms, the forecast accuracy is improved by more than 10%, with a TS score ≥ 55, an F1 score ≥ 0.85, and significantly reduced false alarm and missed alarm rates. Employing a standardized data preprocessing workflow and fine-tunable model parameters, it eliminates the need to rebuild models for different regions; only a few parameters need to be adjusted to adapt to the cloud and sea forecasting needs of different mountainous areas. This solves the problems of strong regionality and difficulty in widespread application of existing technologies, and can be widely used throughout the country. This invention provides a mountain-like cloud sea forecast. Through an improved Transformer model and probabilistic prediction head, it accurately outputs the probability and confidence level of cloud sea occurrence. Compared to existing binary and qualitative forecasts, it offers a more scientific basis for tourism meteorological services and scenic area operation management, preventing tourists from arriving disappointed and enhancing their travel experience. Compared to existing multi-scale process coupling models, the Transformer model of this invention has a simpler structure, higher feature extraction and fusion efficiency, and lower computational complexity, enabling rapid forecast output and increasing the forecast lead time to over 24 hours, meeting the practical needs of short-term forecasts. Through early termination strategies, regularization mechanisms, and model fine-tuning, it effectively avoids model overfitting, improving model stability and generalization ability. It maintains high forecast accuracy and strong adaptability in tests across different regions and seasons. Attached Figure Description

[0021] Figure 1 The flowchart shows the cloud sea probability prediction method based on Transformer provided by this invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Please refer to the following: Figure 1 ,in, Figure 1 The flowchart illustrates the cloud and sea probabilistic forecasting method based on Transformer provided by this invention. The Transformer-based cloud and sea probabilistic forecasting method includes the following steps:

[0024] S1. Data Input:

[0025] Collect multi-source input data, including three core data types, to ensure the comprehensiveness and accuracy of the data, as detailed below:

[0026] S1.1 Meteorological Data: Select a numerical weather prediction model suitable for the target mountainous area (such as WRF or MESO high-resolution model) and extract meteorological observation data from the model output, including meteorological elements such as temperature, humidity, wind speed, and wind direction; at the same time, collect multi-source stereo data (microwave radiometer, wind profiler radar, and gradient densification automatic weather station data) to supplement the vertical stratified meteorological data at the near-surface and upper atmosphere; in addition, collect ERA5 reanalysis data to supplement mesoscale weather system elements (horizontal circulation field elements, thermal forcing elements, water vapor transport elements, and mesoscale disturbance system parameters).

[0027] S1.2 Topographic Data: Obtain digital elevation model (DEM) data of the target mountain, extract topographic feature parameters such as topographic relief and slope aspect distribution, and use them to generate a topographic disturbance coefficient field to characterize the disturbance effect of topography on the formation of cloud sea.

[0028] S1.3 Historical observation data: Collect historical cloud sea observation data (including cloud sea appearance time, duration, observation point location, etc.) of the target area and similar areas, as well as corresponding historical meteorological data and topographic data, for model training and validation.

[0029] S2, Data Preprocessing:

[0030] The multi-source data collected in S1 is preprocessed to ensure data quality and generate standardized model input data. The specific processing procedure is as follows:

[0031] S2.1 Meteorological data preprocessing: Vertical interpolation is performed on meteorological observation data to generate meteorological element data distributed sequentially according to altitude layers; spatiotemporal alignment and fusion of meteorological data from different sources are performed to generate standardized meteorological profiles covering the entire elevation range of the target mountain; missing data are filled using linear interpolation and outliers are removed using the 3σ criterion; meteorological element data are standardized (normalized to the [0,1] interval) to eliminate the influence of dimensions.

[0032] S2.2 Topographic data preprocessing: Based on digital elevation model (DEM) data, the sliding window method is used to calculate the topographic relief, and the cosine of the surface slope angle and the prevailing wind direction angle is calculated to obtain the slope aspect factor. The topographic disturbance coefficient field is calculated by combining the relief and the slope aspect factor. The topographic feature parameters are standardized to be consistent with the dimensions of the meteorological data.

[0033] S2.3, Label Data Preprocessing: Convert historical cloud sea observation data into label data. If cloud sea appears in a certain time period or a certain area, the label is set to 1; otherwise, it is set to 0. At the same time, combine the confidence level of historical cloud sea observation data to generate cloud sea probability labels (e.g., the probability label is set to 0.8-1.0 for the time period when cloud sea is observed, and 0-0.2 for the time period when it is not observed), which are used for probability training of the model.

[0034] S2.4 Input Sequence Construction: The preprocessed meteorological profile data, topographic disturbance coefficient field, and mesoscale weather system elements are spliced ​​together according to the time series to construct the input sequence of the Transformer model. The sequence length is set to T (based on the time resolution of historical data, such as 6 hours or 12 hours), and the prediction step size is set to 1 (i.e., predicting the probability of cloud cover in the next period).

[0035] S3, Feature Extraction:

[0036] An improved Transformer model is used for feature extraction to fully capture the temporal dependencies and coupling features of meteorological and topographic factors, as detailed below:

[0037] S3.1, Transformer Model Structure Improvement: A feature fusion module is added to the encoder part of the traditional Transformer model, and a probability prediction head is added to the decoder part to adapt to the needs of cloud and sea probability forecasting. The encoder consists of multiple self-attention layers and a feedforward neural network to capture long-distance temporal dependencies of the input sequence. The feature fusion module is used to fuse meteorological features, topographic features and mesoscale weather system features to improve feature representation ability. The probability prediction head adopts the sigmoid activation function and outputs cloud and sea probability values ​​between 0 and 1.

[0038] S3.2 Temporal Feature Extraction: The input sequence constructed in S2 is input into the Transformer encoder. The attention weights between each time step and each feature are calculated through the self-attention layer. The focus is on capturing the temporal variation patterns of key meteorological factors such as inversion layer height, wet layer thickness, and wind speed, as well as the impact of topographic disturbances on these factors, and extracting discriminative temporal features.

[0039] S3.3 Multi-feature fusion: Through the feature fusion module, the temporal features are nonlinearly fused with the topographic disturbance coefficient field and mesoscale weather system elements to generate a comprehensive feature vector. This vector contains the dynamic evolution characteristics of meteorological factors, the topographic disturbance effect and the influence of mesoscale weather systems, providing support for subsequent probabilistic forecasts.

[0040] S4. Model Training:

[0041] Based on the preprocessed training data (input sequence + probability labels), the improved Transformer model is trained and the model parameters are optimized. The specific process is as follows:

[0042] S4.1 Dataset Partitioning: Divide the historical data into training, validation, and test sets in a 7:2:1 ratio. The training set is used for training model parameters, the validation set is used for adjusting model hyperparameters (such as learning rate, batch size, number of encoder layers, etc.), and the test set is used for evaluating model performance.

[0043] S4.2 Loss Function Selection: The cross-entropy loss function combined with the mean squared error loss function is used as the model's loss function. The cross-entropy loss function is used to optimize the model's classification ability (determining whether a sea of ​​clouds appears), while the mean squared error loss function is used to optimize the model's probability prediction accuracy, ensuring that the model can accurately determine whether a sea of ​​clouds appears and accurately output probability values.

[0044] S4.3 Model Training and Optimization: The Adam optimizer is used to train the model, with an appropriate learning rate set (initial learning rate is set to 1e-4, and a learning rate decay strategy is adopted). The model parameters are updated through backpropagation. During the training process, an early stopping strategy is adopted. When the loss function of the validation set does not decrease for 5 consecutive epochs, training is stopped to avoid model overfitting. At the same time, a regularization mechanism (L2 regularization) is introduced to further improve the generalization ability of the model.

[0045] S4.4 Model Fine-tuning: Use the test set to evaluate the performance of the trained model. If the model performance does not meet the preset requirements (such as accuracy ≥90%, F1 score ≥0.85, TS score ≥55), adjust the model hyperparameters (such as the number of self-attention layers, feature fusion weights, etc.) and retrain the model until the model performance meets the requirements.

[0046] S5. Probability Forecasting:

[0047] The preprocessed data (meteorological profiles, topographic disturbance coefficient fields, and mesoscale weather system elements) for the target time period and target area are input into the trained Transformer model. The model extracts features, fuses features, and predicts probabilities to output the probability and confidence level of cloud cover occurrence in the target area within the target time period, as detailed below:

[0048] S5.1 Input Data Preparation: Following the preprocessing method in S2, preprocess the multi-source data of the target time period and target area to construct the input sequence.

[0049] S5.2 Probability Prediction: Input the input sequence into the trained Transformer model, and the model outputs the probability of cloud cover occurrence during the target time period (a value between 0 and 1). The closer the probability value is to 1, the greater the likelihood of cloud cover occurrence. At the same time, the confidence level of the output probability is used to characterize the reliability of the forecast results.

[0050] S5.3 Result Correction: Combining fuzzy logic algorithms, the probability results output by the model are corrected. Based on the regional characteristics of the target area (such as terrain and climate characteristics), the probability threshold is adjusted to further improve the forecast accuracy. Finally, the corrected cloud sea probability forecast results and confidence levels are output to provide users with accurate probability guidance.

[0051] Compared with related technologies, the cloud sea probability prediction method based on Transformer provided by this invention has the following beneficial effects:

[0052] This invention provides a Transformer-based probabilistic cloud and sea forecasting method. Utilizing the self-attention mechanism of the Transformer model, it accurately captures long-term temporal dependencies in meteorological data, effectively characterizing the dynamic evolution of meteorological factors such as inversion layers, humidity layers, and wind fields. Combined with a multi-source data fusion strategy, it comprehensively considers the impact of topographic disturbances and mesoscale weather systems. Compared to existing statistical learning models and fuzzy logic algorithms, the forecast accuracy is improved by more than 10%, with a TS score ≥ 55, an F1 score ≥ 0.85, and significantly reduced false alarm and missed alarm rates. Employing a standardized data preprocessing workflow and fine-tunable model parameters, it eliminates the need to rebuild models for different regions; only a few parameters need to be adjusted to adapt to the cloud and sea forecasting needs of different mountainous areas. This solves the problems of strong regionality and difficulty in widespread application of existing technologies, and can be widely used throughout the country. This invention provides a mountain-like cloud sea forecast. Through an improved Transformer model and probabilistic prediction head, it accurately outputs the probability and confidence level of cloud sea occurrence. Compared to existing binary and qualitative forecasts, it offers a more scientific basis for tourism meteorological services and scenic area operation management, preventing tourists from arriving disappointed and enhancing their travel experience. Compared to existing multi-scale process coupling models, the Transformer model of this invention has a simpler structure, higher feature extraction and fusion efficiency, and lower computational complexity, enabling rapid forecast output and increasing the forecast lead time to over 24 hours, meeting the practical needs of short-term forecasts. Through early termination strategies, regularization mechanisms, and model fine-tuning, it effectively avoids model overfitting, improving model stability and generalization ability. It maintains high forecast accuracy and strong adaptability in tests across different regions and seasons.

[0053] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A cloud sea probability forecasting method based on Transformer, characterized in that, Includes the following steps: S1. Data Input: Collect multi-source input data, including meteorological data, topographic data, and historical cloud sea observation data; wherein, the meteorological data includes numerical weather prediction model output data, multi-source stereo sounding data, and ERA5 reanalysis data, the topographic data includes digital elevation model data, and the historical cloud sea observation data includes cloud sea appearance time, duration, and observation point location; S2. Data Preprocessing: The multi-source input data is preprocessed to generate standardized model input data. Specifically, this includes: vertical interpolation, spatiotemporal alignment, and fusion of meteorological observation data to generate standardized meteorological profiles; filling missing data using linear interpolation; removing outliers using the 3σ criterion; and normalizing to the [0,1] interval. Based on the digital elevation model data, the topographic relief is calculated using the sliding window method, and the topographic disturbance coefficient field is calculated and standardized using the aspect factor. Historical cloud sea observation data is converted into probability labels, where the probability label for the time period in which cloud sea is observed is set to 0.8 to 1.0, and the probability label for the time period in which cloud sea is not observed is set to 0 to 0.

2. The preprocessed meteorological profile data, topographic disturbance coefficient field, and mesoscale weather system elements are spliced ​​together according to the time series to construct the input sequence of the Transformer model. The sequence length T is set according to the temporal resolution of the historical data, and the prediction step size is set to 1. S3. Feature Extraction: An improved Transformer model is used for feature extraction. The improved Transformer model adds a feature fusion module to the encoder and a probability prediction head to the decoder. The encoder consists of multiple self-attention layers and a feedforward neural network to capture long-range temporal dependencies of the input sequence. The feature fusion module is used to nonlinearly fuse temporal features with topographic perturbation coefficient fields and mesoscale weather system elements to generate a comprehensive feature vector. The probability prediction head uses a sigmoid activation function to output a cloud and sea probability value between 0 and 1. S4. Model Training Step: The improved Transformer model is trained based on the preprocessed training data. Specifically, this includes: dividing historical data into training, validation, and test sets in a 7:2:1 ratio; using a combination of cross-entropy and mean squared error loss functions as the total loss function; training with the Adam optimizer, setting the initial learning rate to 1e-4 and employing a learning rate decay strategy; using an early stopping strategy, stopping training when the validation set loss function does not decrease for 5 consecutive epochs; introducing L2 regularization; and evaluating performance using the test set, adjusting hyperparameters and retraining if the preset requirements are not met. S5. Probability Forecast: Input the preprocessed data of the target time period and target area into the trained improved Transformer model. The model outputs the probability and confidence level of cloud sea occurrence. Then, combined with the fuzzy logic algorithm, the probability threshold is adjusted according to the regional characteristics of the target area to correct the output probability results. Finally, the corrected cloud sea probability forecast result and confidence level are output.

2. The cloud sea probability forecasting method based on Transformer according to claim 1, characterized in that, In S1, the numerical forecasting model is the WRF or MESO high-resolution model, the multi-source stereo detection data includes microwave radiometer, wind profiler radar and gradient densification automatic weather station data, and the ERA5 reanalysis data is used to supplement mesoscale weather system elements, which include horizontal circulation field elements, thermal forcing elements, water vapor transport elements and mesoscale disturbance system parameters.

3. The cloud sea probability forecasting method based on Transformer according to claim 1, characterized in that, In S2, the terrain disturbance coefficient field is calculated as follows: the terrain undulation is calculated using the sliding window method based on digital elevation model data, the slope aspect factor is obtained by calculating the cosine of the surface slope aspect angle and the prevailing wind direction angle, and the terrain disturbance coefficient field is calculated by combining the undulation and the slope aspect factor.

4. The cloud and sea probability forecasting method based on Transformer according to claim 1, characterized in that, In S3, the temporal feature extraction specifically involves: inputting the input sequence into a Transformer encoder, calculating the attention weights between each time step and each feature through a self-attention layer, and focusing on capturing the temporal variation patterns of key meteorological factors such as inversion layer height, wet layer thickness, and wind speed, as well as the impact of topographic disturbances on these factors.

5. The cloud sea probability forecasting method based on Transformer according to claim 1, characterized in that, In S4, the preset requirements are accuracy ≥ 90%, F1 score ≥ 0.85, and TS score ≥ 55; the hyperparameters include the number of self-attention layers and feature fusion weights.

6. The cloud sea probability forecasting method based on Transformer according to claim 1, characterized in that, In step S5, the input data preparation involves preprocessing the multi-source data of the target time period and target area according to the method in step S2 and constructing the input sequence.

7. The cloud and sea probability forecasting method based on Transformer according to claim 1, characterized in that, In step S5, the fuzzy logic algorithm is replaced by Bayesian estimation to correct the probability results.

Images