A lightweight modeling method, device, system and storage medium for high-dimensional data weather prediction tasks
By using an improved variational autoencoder strategy and the XGBoost algorithm to screen out key predictive variables with clear physical meaning, the problem of balancing computational efficiency and model accuracy in high-dimensional data meteorological forecasting tasks is solved, and efficient and reliable extreme weather forecasting is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF INFORMATION SCI & TECH
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-26
AI Technical Summary
In existing high-dimensional data weather forecasting tasks, existing models struggle to balance computational efficiency, model accuracy, and physical constraints, resulting in limited reliability and interpretability of forecast results, particularly in extreme weather event forecasting.
By using an improved variational autoencoder strategy, potential predictive variables that are strongly correlated with the target variable and have clear physical meaning are screened from high-dimensional data. A lightweight prediction model is built by combining XGBoost and LightGBM algorithms. Extreme value labeling and regularization terms are used to optimize the reconstruction error. Correlation analysis and redundancy detection are performed, and an adaptive early stopping mechanism is used to optimize the hyperparameters.
It significantly reduces computational resource requirements, improves prediction performance and stability, enhances model interpretability and credibility, shortens training and inference time, and is suitable for environments with limited computational performance.
Smart Images
Figure CN122286293A_ABST
Abstract
Description
Technical Field
[0001] This invention discloses a lightweight modeling method, device, system, and storage medium for high-dimensional data meteorological forecasting tasks, belonging to the technical field of calculation, estimation, or counting. Background Technology
[0002] In recent years, with the accumulation of data in various fields such as meteorology, remote sensing, medicine, and finance, large-model technology, represented by deep neural networks and large-scale pre-trained models, has rapidly developed in the field of artificial intelligence and demonstrated excellent performance in various high-dimensional data prediction tasks. In the field of meteorological forecasting, traditional numerical forecasting methods and prediction methods based on statistical or empirical formulas have long been the main technical means. For example, freezing rain prediction usually relies on physical models and empirical formulas. Although these methods have certain applicability in some situations, their prediction accuracy is often limited because they ignore the complex nonlinear relationships between multiple variables. In contrast, large models can automatically extract features and complex nonlinear relationships between data based on high-dimensional input data. Especially for the prediction of extreme weather such as freezing rain, by learning latent patterns from a large amount of historical meteorological data, they can effectively capture complex spatiotemporal features and improve the accuracy and reliability of predictions. However, this kind of method of improving the accuracy and reliability of predictions through large models usually requires a lot of computing resources, including powerful hardware support and long training time, resulting in huge overhead in the training and inference process, which may be difficult to deploy, especially in environments with limited computing power. Secondly, due to their high complexity and lack of physical constraints, large models often lack sufficient interpretability and transparency, which casts doubt on their credibility and reliability in practical applications. Furthermore, redundant features and noise in high-dimensional data inputs can also affect the generalization ability of large models and impact pattern stability.
[0003] To alleviate this problem, various improvement methods have been proposed in existing technologies. For example, model pruning and parameter quantization techniques reduce computational overhead by decreasing model size, but often lead to a decrease in accuracy and it is difficult to achieve a balance between compression ratio and performance. Knowledge distillation methods improve efficiency by having a small model learn the knowledge of a large model, but the distillation effect is limited in high-dimensional and complex data tasks, and the training process is relatively complex. On the other hand, dimensionality reduction techniques such as Principal Component Analysis (PCA) or traditional Variational Autoencoder (VAE) methods can reduce the number of input variables to some extent, thereby reducing computational complexity. Among them, VAE can capture more complex nonlinear relationships, but the new variables generated by these dimensionality reduction methods lack clear physical meaning and physical constraints, requiring improvements to existing dimensionality reduction methods. This is particularly critical in applications such as weather forecasting, because variables lacking physical constraints may cause large models to fail to accurately reflect the physical mechanisms of extreme weather events, thus affecting the credibility and interpretability of the prediction results. Therefore, existing technologies have not yet effectively reconciled the contradiction between large model performance and computational efficiency in high-dimensional data prediction tasks. Meanwhile, existing dimensionality reduction methods generally neglect the combination of physical credibility and domain knowledge, which limits the effectiveness and reliability of large models in practical applications of weather forecasting.
[0004] Therefore, there is an urgent need to propose an efficient prediction method that can balance computational efficiency, model accuracy, and physical constraints in order to address the various challenges in high-dimensional data meteorological prediction tasks, especially the prediction of extreme weather events. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of the aforementioned background technology by providing a lightweight modeling method, device, system, and storage medium for high-dimensional data meteorological forecasting tasks. This solves the technical problem that existing lightweight modeling techniques are insufficient to meet the requirements of accurately obtaining reliable forecast results for high-dimensional data meteorological forecasting tasks. The invention aims to enhance forecasting performance while achieving a lightweight high-dimensional data meteorological forecasting model by constructing and screening key predictive variables that have clear physical meaning to the target variable.
[0006] To achieve the above-mentioned objectives, the present invention employs the following technical solution:
[0007] A lightweight modeling method for high-dimensional data weather forecasting tasks includes:
[0008] S1, Obtain the original high-dimensional data for the target weather forecasting task, and construct the forecast vector and target variable;
[0009] S2, using an improved variational autoencoder strategy, reconstructs potential predictor variables that affect or potentially affect the target variable from the prediction vector, filters out potential predictor variables that are strongly correlated with the predictor variable from the potential predictor variables, and constructs a candidate predictor variable dataset. The improved variational autoencoder strategy optimizes and reconstructs the error term and regularization term by labeling extreme values in the prediction vector.
[0010] S3, perform correlation analysis and redundancy detection on the candidate predictor variable dataset, screen out the candidate predictor variable dataset that is strongly correlated with the target variable and has clear physical meaning, and select the key predictor variable dataset from the screened candidate predictor variable dataset.
[0011] S4. Build a lightweight prediction model based on the dataset of key predictor variables selected in S3.
[0012] S5 evaluates the performance of the lightweight prediction model established in S4.
[0013] As a further optimization of a lightweight modeling method for high-dimensional data weather forecasting tasks, the improved variational autoencoder strategy in S2 optimizes and reconstructs the error term by marking extreme values in the prediction vector, specifically: ,in, To reconstruct the error term, To predict the total number of vectors, For the first The extreme values of each prediction vector. To predict the extreme value weights of the vector, The number of predictors in the prediction vector. Weighting coefficients for extreme values of predictor variables. For the first The nth prediction vector One predictor variable, For the reconstruction of the first The th prediction vector in the th prediction vector One potential predictor variable, , , For the first In the predicted vector, the 1st, 2nd, ..., ... The results of determining the extreme values of each predictor variable. For the first The mean of each predictor variable, For the first The standard deviation of each predictor variable;
[0014] The improved variational autoencoder strategy in S2 reconstructs the regularization term by optimizing the prediction vector through extreme value labeling, specifically: , For L1 regularization, Internal parameters computed for the improved variational autoencoder strategy. The number of internal parameters, For the first The nth prediction vector L1 regularization coefficients of the predictor variables, To improve the variational autoencoder strategy, the training process learns the first... The nth prediction vector The first predictor variable is the... One internal variable, , The L1 regularization coefficient is... This is the penalty decay factor.
[0015] As a further optimization of a lightweight modeling method for high-dimensional data weather forecasting tasks, S2 selects potential predictor variables that are strongly correlated with the predicted variables from the potential predictor variables. Specifically, it performs a correlation test on each predictor variable and potential predictor variable contained in the prediction vector and retains the potential variables that meet the correlation threshold requirements.
[0016] As a further optimization of a lightweight modeling method for high-dimensional data weather forecasting tasks, S3 performs correlation analysis and redundancy detection on candidate predictor variable datasets to select candidate predictor variable datasets that are strongly correlated with the target variable and have clear physical meaning. Specifically: ,in, For the first One candidate predictor variable and target variable The correlation coefficient, For the first One candidate predictor variable and target variable The correlation coefficient, for Redundancy correction item For the first One candidate predictor variable and the One candidate predictor variable The correlation coefficient, The relevance threshold for candidate predictor variables. and Through expression calculate, For the first One candidate predictor variable and target variable The correlation coefficient, For the first One candidate predictor variable and target variable The weights of the Pearson correlation coefficients between them. For the first One candidate predictor variable and target variable Pearson correlation coefficient between them For the first One candidate predictor variable and target variable Spearman correlation coefficient between them , This is the weight difference control parameter. for or .
[0017] As a further optimization of a lightweight modeling method for high-dimensional data weather forecasting tasks, S3 selects key predictor variable datasets from the selected candidate predictor variable datasets. Specifically, the selected candidate predictor variable datasets are divided into training and validation sets. The XGBoost model is trained using the training set and cross-validation is performed. The comprehensive index score of each candidate predictor variable is calculated and the validation set TS score obtained in the cross-validation evaluation is recorded. The candidate predictor variable with the lowest comprehensive index score is removed, and the model is retrained using the remaining candidate predictor variables. The comprehensive index score of each predictor variable is calculated again and the validation set TS score obtained in the cross-validation evaluation is recorded. The above removal process is repeated until all candidate predictor variables are removed. The candidate predictor variables that reach the set validation set TS score when training the XGBoost model are extracted as key predictor variables.
[0018] As a further optimization of the lightweight modeling method for high-dimensional data weather forecasting tasks, S4 establishes a lightweight forecasting model based on the key predictive variable dataset selected by S3. Specifically, it selects the XGBoost model or the LightGBM model and optimizes the hyperparameters of the selected model by combining a Bayesian optimization algorithm with an adaptive early stopping mechanism.
[0019] As a further optimization of the lightweight modeling method for high-dimensional data meteorological forecasting tasks, S5 evaluates the performance of the lightweight forecasting model established in S4, specifically by using TS score, false alarm rate, and false miss rate to evaluate the forecasting accuracy of the lightweight forecasting model.
[0020] A lightweight modeling device for high-dimensional data meteorological forecasting tasks includes: a high-dimensional data acquisition module, a latent predictor variable reconstruction module, a key predictor variable screening module, a model building module, and a model evaluation module. The high-dimensional data acquisition module acquires the original high-dimensional data for the target meteorological forecasting task and constructs a prediction vector and a target variable. The latent predictor variable reconstruction module reconstructs latent predictor variables that influence or potentially influence the target variable from the prediction vector using an improved variational autoencoder strategy, selects latent predictor variables strongly correlated with the target variable from the latent predictor variables, and constructs a candidate predictor variable dataset. The improved variational autoencoder strategy optimizes the reconstruction of error terms and regularization terms by labeling extreme values in the prediction vector. The key predictor variable screening module performs correlation analysis and redundancy detection on the candidate predictor variable dataset, selects candidate predictor variable datasets that are strongly correlated with the target variable and have clear physical meaning, and selects a key predictor variable dataset from the selected candidate predictor variable datasets. The model building module builds a lightweight prediction model based on the selected key predictor variable dataset. The model evaluation module evaluates the performance of the lightweight prediction model.
[0021] A computer system includes a memory and a processor, wherein the memory stores a computer program that runs on the processor, and the processor executes the steps of the lightweight modeling method described above when running the computer program.
[0022] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the lightweight modeling method described above.
[0023] The present invention, by adopting the above technical solution, has the following beneficial effects:
[0024] 1. This invention effectively reduces the dimensionality of input data by constructing potential predictive variables closely related to the target variable and further screening key predictive variables. This reduces the computational burden caused by high-dimensional data, significantly reduces the dependence of meteorological forecasting tasks on computing and storage resources, and reduces the CPU and GPU resources required by the constructed lightweight model during the training and prediction inference stages. This enables the lightweight model to be directly deployed in scenarios with limited computing performance.
[0025] 2. To address the problem of lacking empirical formulas for constructing new meteorological variables, this invention reconstructs the error term and regularization term of the variational autoencoder by labeling extreme values, effectively uncovering potential predictive variables with clear physical meaning without the need for manual pre-setting of physical parameters.
[0026] 3. By screening out key predictive variables that have clear physical meaning to the target variable, this invention enables the lightweight model to achieve prediction accuracy and stability no less than that of large models that directly use high-dimensional inputs, and to far outperform large model methods in terms of model interpretability and result credibility.
[0027] 4. The model construction method proposed in this invention can significantly shorten the training and inference time while maintaining high prediction and fitting performance, reduce the cost of data processing and model updates, and improve the real-time performance and operability in practical applications. This feature gives it a significant advantage in meteorological forecasting tasks with high timeliness requirements.
[0028] 5. This invention demonstrates superior efficiency and scalability in actual prediction cases. It can be applied not only to weather forecasting but also to other high-dimensional data prediction tasks, and has a wide range of application scenarios. Attached Figure Description
[0029] Figure 1 This is a flowchart of the lightweight modeling method proposed in this invention.
[0030] Figure 2 This is a schematic diagram of the XGBoost-RFE cross-validation TS scoring results provided in one embodiment of the present invention.
[0031] Figure 3 This is a flowchart of an adaptive early stopping strategy for model training provided in one embodiment of the present invention.
[0032] Figure 4 This is a schematic diagram showing the comparison of model consumption time and performance between the original high-dimensional data model and the lightweight model provided in one embodiment of the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only some, not all, of the embodiments of this invention.
[0034] like Figure 1 As shown, the present invention provides a lightweight modeling method for high-dimensional data meteorological forecasting tasks, including the following S1~S5.
[0035] S1: Obtain the raw high-dimensional data required for the target weather forecasting task and construct the prediction vector. and target variable Raw high-dimensional data can come from observations, measurements, simulations, forecasts, reanalysis, operational databases, or other data sources.
[0036] In one embodiment of the present invention, taking the freezing rain prediction task in meteorological forecasting as an example, the original high-dimensional data includes ERA5 reanalysis data and weather phenomenon data from ground observation stations, wherein the prediction vector Hourly data in a 0.25° × 0.25° latitude and longitude grid, containing 234 predictor variables in ERA5 reanalysis; target variable This refers to weather phenomenon data from ground observation stations, specifically weather phenomenon data observed at 02:00, 05:00, 08:00, 11:00, 14:00, 17:00, 20:00, and 23:00 during the winter half-year from 1995 to 2019 by 778 meteorological stations in southern China.
[0037] S2: Based on domain knowledge and physical rules of the prediction task, from the prediction vector Construct target variables that have an impact or potential impact. The latent predictor variables should be constructed according to the principle that newly constructed latent predictor variables have clear physical meaning, avoiding a decrease in model interpretability and credibility due to ambiguity in the meaning of the dependent variable. Therefore, it is preferred to use prediction vectors based on empirical formulas or domain research results. Construct potential predictor variables with clear physical meaning.
[0038] In one embodiment of the present invention, based on the freezing rain formation mechanism, predictive variables such as air temperature, relative humidity, and air pressure are extracted from ERA5 data to construct potential predictive variables such as wet-bulb temperature, atmospheric profile type, and warm layer thickness. These potential predictive variables are then compared with the prediction vector constructed in step one. It directly serves to characterize the conditions for freezing rain formation.
[0039] However, faced with massive and high-dimensional datasets, the specific calculation formulas for some latent predictor variables are not readily known. Therefore, for prediction vectors that may possess potential physical meaning, an improved variational autoencoder is used to mine latent predictor variables. The loss function of a traditional VAE consists of reconstruction error and KL divergence, where the default reconstruction error is calculated using mean squared error to measure the difference between the original predictor variables and the latent predictor variables reconstructed by the decoder. However, meteorological data distributions are complex and unstable, and for prediction tasks involving low-probability extreme weather events such as freezing rain, the calculation of mean squared error can easily lead to insufficient learning of extreme weather information by the model, thus affecting the physical validity of the latent predictor variables. Therefore, this problem needs to be addressed by optimizing the reconstruction error term, firstly by determining the extreme values of the input predictor variables. Mark extreme samples Furthermore, extreme value weights are set to enhance the fitting of extreme samples and variables. The specific formula for modifying the reconstruction error is as follows:
[0040] (1)
[0041] (2)
[0042] (3)
[0043] In equations (1) to (3), For the first The nth prediction vector One predictor variable; For the first The mean of each predictor variable; For the first The standard deviation of each predictor variable; For the first The nth prediction vector The results of determining the extreme values of each predictor variable; For the first The extreme values of each prediction vector; This refers to the optimized reconstruction error term; The total number of prediction vectors, i.e., the number of samples. The number of predictors in the prediction vector; The extreme value weights for the prediction vector can be set from 0.01 to 10. The first reconstructed by the decoder The th prediction vector in the th prediction vector One potential predictor variable; The extreme value weighting coefficient for the predicted variable is a hyperparameter that can be set to 0.2~0.6.
[0044] Furthermore, the more predictor variables included in the input prediction vector, the more difficult it is to clearly understand the physical meaning of the unearthed latent predictor variables. Therefore, to protect key predictor variables, reduce the influence of weakly affected variables, and avoid the problem of the original L1 parameter penalizing all variables equally, an improved adaptive L1 regularization coefficient is used. Introducing a penalty decay factor This ensures that key variable information is not lost, thereby preserving the physical meaning of key predictive variables.
[0045] (4)
[0046] (5)
[0047] In equations (4) and (5), For L1 regularization terms; For the first The nth prediction vector L1 regularization coefficients of the predictor variables; The L1 regularization coefficient; The penalty decay factor is a hyperparameter that can be set to 0.3~0.5; Internal parameters calculated for the variational autoencoder The number of internal parameters; To learn the first step during the training of the variational autoencoder The nth prediction vector The first predictor variable is the... One internal variable.
[0048] The improved loss function of the variational autoencoder proposed in this invention for:
[0049] = (6)
[0050] In equation (6), This is the KL divergence term.
[0051] The improved variational autoencoder is trained by Equation (6) to construct a batch of potential predictors that affect or potentially affect the target variable from the original predictors.
[0052] Finally, a correlation test is introduced to find the potential physical meaning represented by the new predictor variables. Simultaneously, to determine the optimal number of potential predictor variables, an automated screening process is performed based on a preset correlation threshold and the number of potential predictor variables. For example, the improved variational autoencoder reconstructs K-1 potential predictor variables from the original input prediction vector containing K predictor variables. A correlation test is performed on the original and potential predictor variables, retaining those that meet the correlation threshold. Finally, potential predictor variables strongly correlated with the original predictor variables are selected from the potential predictor variables, which, together with the prediction vector constructed in step one, constitute the candidate predictor variable dataset. This method avoids the problem of unclear physical meaning of the constructed potential predictor variables due to a large number of input variables used for feature compression or new feature construction. It can also efficiently discover and extract core data with physical or statistical significance from large-scale complex data.
[0053] S3: Based on the candidate predictor variable dataset that influences or potentially influences the target variable constructed in step two, correlation analysis and redundancy detection are performed on the candidate predictor variables. Candidate predictor variables that are weakly correlated with or redundant to the target variable are eliminated, while key predictor variables that influence the model's predictions are selected, thereby reducing the dimensionality of the input data. However, different correlation analysis methods are suitable for different types of data relationships. To ensure the accuracy and efficiency of the analysis, a strategy of comprehensive correlation analysis is adopted in this specific embodiment. The comprehensive correlation calculation method between variables is as follows:
[0054] (7)
[0055] In equation (7), Indicates the first One candidate predictor variable and target variable The weights of the Pearson correlation coefficients between them; Indicates the first One candidate predictor variable and target variable The correlation coefficient; Indicates the first One candidate predictor variable and target variable Pearson correlation coefficient between them; Indicates the first One candidate predictor variable and target variable Spearman correlation coefficient between them; This is the weight difference control parameter, set to 10 here.
[0056] For the calculated number One candidate predictor variable and the One candidate predictor variable correlation coefficient Candidate predictors that are highly linearly correlated with each other are selected; for example, candidates that satisfy the following criteria are selected. Candidate predictor variables. Candidate predictor variables that are more strongly correlated with the target variable and have a clear physical mechanism for freezing rain formation are selected. The selection formula for candidate predictor variables is as follows:
[0057] (8)
[0058] In equation (8), Indicates the first One candidate predictor variable and target variable The correlation coefficient; Indicates the first One candidate predictor variable and target variable The correlation coefficient; express Redundancy correction items; This represents the relevance threshold of the candidate predictor variable; when the inequality holds, the candidate predictor is selected. One candidate predictor variable And remove the first One candidate predictor variable Conversely, choose the first option. One candidate predictor variable And remove the first One candidate predictor variable .
[0059] Based on the correlation analysis results above, strongly correlated candidate predictor variables and candidate predictor variables significantly unrelated to the target variable are removed. The candidate predictor variable dataset after removal is divided into training and test sets. The extreme gradient boosting (XGBoost) algorithm or recursive feature elimination (RFE) is used to select key predictor variables. The XGBoost algorithm can output importance indicators for each candidate predictor variable to the model prediction, including gain, coverage, and frequency. To optimize the problem of losing some feature importance information caused by selecting a single importance index, a comprehensive importance index is selected. Key predictor variables are selected, where the value of the important indicator represents the overall contribution of the candidate predictor variables to the model's predictions. The selected candidate predictor variable dataset is then processed by calculating the important indicator, and those whose overall important indicator values meet the threshold requirements are chosen as key predictor variables.
[0060] (9)
[0061] In equation (9), Indicates the first training set The comprehensive key index value of the candidate predictor variables, Indicates the first training set Gains of candidate predictors Indicates the first training set Coverage of candidate predictor variables, Indicates the first training set The frequency of each candidate predictor variable This indicates the number of candidate predictor variables included in the training set.
[0062] First, an initial XGBoost model is trained using the training dataset and cross-validated for evaluation. Then, the performance of each candidate predictor variable is calculated. The XGBoost model is retrained using the TS score obtained from cross-validation and the overall importance index. Based on the ranking of the overall importance index, the candidate predictor variable ranked last in the overall importance index is removed. Then, the XGBoost model is retrained using the remaining candidate predictor variables and cross-validated. Simultaneously, the overall importance index score and the TS score are calculated, and the candidate predictor variable ranked last in the overall importance index is removed. If all candidate predictor variables have been removed, the algorithm stops; otherwise, the above removal process is repeated. Finally, the candidate predictor variables corresponding to the XGBoost model trained when the set TS score is reached are extracted to construct a key predictor variable dataset, which is used as input to the lightweight model. Figure 2 The XGBoost-RFE cross-validation TS score results are shown in the example.
[0063] S4: Based on the selected dataset of key predictor variables, a multi-level, integrated, lightweight prediction model was constructed. The prediction model can choose not only classic machine learning models such as decision trees, gradient boosting trees, and random forests, but also various deep learning architectures such as convolutional neural networks and long short-term memory networks. It also employs ensemble learning frameworks such as stacking and boosting to further improve the model's generalization ability and prediction performance. In this embodiment, two efficient gradient boosting decision tree (GBDT) algorithms, XGBoost and Light Gradient Boosting Machine (LightGBM), were selected as the core prediction models, and their performance in freezing rain prediction was evaluated through horizontal comparison. To enhance the model's accuracy and stability, a Bayesian optimization algorithm combined with an adaptive early stopping mechanism was used to precisely optimize the model's hyperparameters, preventing overfitting or underfitting.
[0064] Furthermore, since freezing rain events are typically low-probability events, the dataset suffers from class imbalance, with a severe imbalance in the ratio of positive to negative samples in the predictions, posing a challenge to the model's learning and prediction accuracy. To address this, during training, the influence of positive and negative samples is balanced by weighting the sample weights. The weights are set using the hyperparameter `scale_pos_weight` to increase the model's attention to the minority class. Additionally, the loss functions of the XGBoost and LightGBM algorithms are optimized. The optimized loss function not only considers the prediction error but also incorporates a penalty coefficient for minority class predictions. , and This improves the model's sensitivity to freezing rain prediction. Its loss function is as follows:
[0065] (10)
[0066] In equation (10), , These are the loss functions for the XGBoost model and the LightGBM model, respectively. For the first The weights of the key predictor variables are defined by the hyperparameter scale_pos_weight; For the first True labels of key predictor variables; For the first Predicted values of key predictor variables; The total number of trees; This represents the number of leaf nodes; This is the penalty coefficient; and The regularization coefficient is used. For the first The weight of each leaf node.
[0067] During model training, to prevent overfitting or underfitting during hyperparameter optimization, a Bayesian optimization algorithm combined with adaptive learning curve analysis and convergence difference monitoring mechanisms is used to intelligently search for the optimal hyperparameter combination. The dynamic changes in learning error between the training and validation sets are tracked in real time to improve model performance and stability. Specifically, such as... Figure 3 As shown, during the Bayesian optimization process, the model dynamically monitors the changing trends of training and validation errors under different hyperparameter combinations through adaptive learning curve analysis. When the errors in the training and validation sets no longer change significantly, the system evaluates based on the convergence difference. If the convergence difference and the test set score do not change significantly within multiple iteration cycles, an adaptive early stopping mechanism is triggered to automatically terminate the training process, thereby avoiding overtraining and invalid computation. At the same time, it ensures that the model can achieve optimal learning and fitting effects under optimized hyperparameters.
[0068] S5: Model performance evaluation. The model is assessed in terms of prediction accuracy, computational resource consumption, and prediction inference speed, and compared with a model trained directly using the original high-dimensional variables. In this embodiment, the Threat Score (Threat Score) is used to evaluate the model's prediction accuracy. Its calculation formula is as follows:
[0069] (11)
[0070] In equation (11), This represents the number of times freezing rain occurred, both in actual events and in forecasts. This refers to the number of times freezing rain was predicted but no actual freezing rain occurred. This represents the number of times freezing rain actually occurred but was not forecasted. Indicates the false alarm rate; This indicates the underreporting rate.
[0071] The lightweight model established in step four was used to test its predictive performance on the training set and the independent test set. The results are shown in Table 1. It can be seen that the model's TS score on the training set is 0.7 or higher, indicating a good fit to the target variable. On the independent test set, the model's true performance and generalization ability on unseen data are reflected; both sets have TS scores exceeding 0.4, and the LightGBM model exhibits a lower false alarm rate, resulting in the best overall predictive performance.
[0072] Table 1. Prediction performance of the lightweight model on the training and test sets.
[0073]
[0074] To verify the advantages of the lightweight model in prediction performance, model size, and inference speed, the original high-dimensional dataset was modeled using the best-performing LightGBM algorithm. The optimal hyperparameters were found using Bayesian optimization to build the model. Model size, time consumption, and the test set TS score were recorded. The lightweight model and the original model were run in the same environment: 24 cores, unlimited memory, no GPU acceleration, and Python 3.9. To reduce random errors, each model was run 10 times, and the maximum and minimum values were removed, with the average value used for statistical analysis. The results are as follows: Figure 4 As shown, during model training, the lightweight model reduced training time by 78.44% compared to the model trained on the original high-dimensional data, with average memory usage reduced by 83.1% and peak memory usage reduced by 84.1%. Similarly, when calling the model for prediction, the runtime was reduced by 56.11%, and both peak and average memory usage were reduced by approximately 80%. Using the lightweight model under full CPU load, the reduction in model runtime and memory usage contributes to resource conservation. Furthermore, it is worth noting that in model validation on the same test set, the lightweight model reduced the false alarm rate and missed alarm rate by 7.11% and 8.22% respectively compared to the original data model, while its TS score improved by 8.48%. In other words, under the same hardware resources, the lightweight model proposed in this invention significantly reduces the number of predictive variables driving the model by selecting key predictive variables without sacrificing accuracy, thus reducing model consumption. Moreover, the newly constructed variables with clear physical meaning to the target variable help increase the model's recognition of the target variable, enhancing the model's predictive performance.
[0075] In one embodiment of the present invention, a lightweight modeling device for high-dimensional data meteorological forecasting tasks is provided. The device includes: a high-dimensional data acquisition module, a latent predictor variable reconstruction module, a key predictor variable screening module, a model building module, and a model evaluation module. The high-dimensional data acquisition module acquires the original high-dimensional data for the target meteorological forecasting task and constructs a prediction vector and a target variable. The latent predictor variable reconstruction module reconstructs latent predictor variables that influence or potentially influence the target variable from the prediction vector using an improved variational autoencoder strategy, selects latent predictor variables strongly correlated with the target variable from the latent predictor variables, and constructs a candidate predictor variable dataset. The improved variational autoencoder strategy optimizes the reconstruction of error terms and regularization terms by labeling extreme values in the prediction vector. The key predictor variable screening module performs correlation analysis and redundancy detection on the candidate predictor variable dataset, selects candidate predictor variable datasets that are strongly correlated with the target variable and have clear physical meaning, and selects a key predictor variable dataset from the selected candidate predictor variable datasets. The model building module builds a lightweight prediction model based on the selected key predictor variable dataset. The model evaluation module evaluates the performance of the lightweight prediction model.
[0076] In one embodiment of the present invention, a computer system is provided, the computer system including a memory and a processor, the memory storing a computer program that runs on the processor, and the processor executing the steps of the lightweight modeling method described above when running the computer program.
[0077] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the lightweight modeling method described above.
[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles and preparation effects of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the claims and their equivalents.
Claims
1. A lightweight modeling method for high-dimensional data weather forecasting tasks, characterized in that, include: S1, Obtain the original high-dimensional data for the target weather forecasting task, and construct the forecast vector and target variable; S2, using an improved variational autoencoder strategy, reconstructs potential predictor variables that affect or potentially affect the target variable from the prediction vector, filters out potential predictor variables that are strongly correlated with the target variable from the potential predictor variables, and constructs a candidate predictor variable dataset. The improved variational autoencoder strategy optimizes and reconstructs error terms and regularization terms by labeling extreme values in the prediction vector. S3, perform correlation analysis and redundancy detection on the candidate predictor variable dataset, screen out the candidate predictor variable dataset that is strongly correlated with the target variable and has clear physical meaning, and screen out the key predictor variable dataset from the screened candidate predictor variable dataset. S4. Establish a lightweight prediction model based on the key predictor variable dataset selected in S3. S5, Evaluate the performance of the lightweight prediction model established in S4.
2. The lightweight modeling method for high-dimensional data meteorological forecasting tasks according to claim 1, characterized in that, The improved variational autoencoder strategy in S2 reconstructs the error term by marking extreme values in the prediction vector, specifically as follows: ,in, To reconstruct the error term, To predict the total number of vectors, For the first The extreme values of each prediction vector. To predict the extreme value weights of the vector, The number of predictors in the prediction vector. Weighting coefficients for extreme values of predictor variables. For the first The nth prediction vector One predictor variable, For the reconstruction of the first The th prediction vector in the th prediction vector One potential predictor variable, , , For the first In the predicted vector, the 1st, 2nd, ..., ... The results of determining the extreme values of each predictor variable. For the first The mean of each predictor variable, For the first The standard deviation of each predictor variable; The improved variational autoencoder strategy in S2 reconstructs the regularization term by optimizing the prediction vector through extreme value labeling, specifically as follows: , For L1 regularization, Internal parameters computed for the improved variational autoencoder strategy. The number of internal parameters, For the first The nth prediction vector L1 regularization coefficients of the predictor variables, To improve the variational autoencoder strategy, the training process learns the first... The nth prediction vector The first predictor variable is the... One internal variable, , The L1 regularization coefficient is... This is the penalty decay factor.
3. The lightweight modeling method for high-dimensional data meteorological forecasting tasks according to claim 2, characterized in that, S2 selects potential predictor variables that are strongly correlated with the potential predictor variables from the potential predictor variables. Specifically, it performs a correlation test on each predictor variable and potential predictor variable contained in the prediction vector and retains the potential variables that meet the correlation threshold requirements.
4. The lightweight modeling method for high-dimensional data meteorological forecasting tasks according to claim 3, characterized in that, S3 performs correlation analysis and redundancy detection on the candidate predictor variable dataset to select candidate predictor variable datasets that are strongly correlated with the target variable and have clear physical meaning, specifically: ,in, For the first Candidate predictor variables and target variable The correlation coefficient, For the first Candidate predictor variables and target variable The correlation coefficient, for Redundancy correction item For the first Candidate predictor variables and the One candidate predictor variable The correlation coefficient, The relevance threshold for candidate predictor variables. and Through expression calculate, For the first Candidate predictor variables and target variable The correlation coefficient, For the first One candidate predictor variable and target variable The weights of the Pearson correlation coefficients between them. For the first One candidate predictor variable and target variable Pearson correlation coefficient between them For the first One candidate predictor variable and target variable Spearman correlation coefficient between them , This is the weight difference control parameter. for or .
5. The lightweight modeling method for high-dimensional data meteorological forecasting tasks according to claim 4, characterized in that, S3 involves selecting key predictor variable datasets from the selected candidate predictor variable datasets. Specifically, the selected candidate predictor variable datasets are divided into training and validation sets. The XGBoost model is trained using the training set and cross-validation is performed. The comprehensive index score of each candidate predictor variable is calculated and the validation set TS score obtained in the cross-validation evaluation is recorded. The candidate predictor variable with the lowest comprehensive index score is removed, and the model is retrained using the remaining candidate predictor variables. The comprehensive index score of each predictor variable is calculated again and the validation set TS score obtained in the cross-validation evaluation is recorded. The above removal process is repeated until all candidate predictor variables are removed. The candidate predictor variables that reach the set validation set TS score when training the XGBoost model are extracted as key predictor variables.
6. The lightweight modeling method for high-dimensional data meteorological forecasting tasks according to claim 5, characterized in that, S4 establishes a lightweight prediction model based on the key predictor variable dataset selected in S3. Specifically, it selects the XGBoost model or the LightGBM model and optimizes the hyperparameters of the selected model using a Bayesian optimization algorithm with an adaptive early stopping mechanism.
7. The lightweight modeling method for high-dimensional data meteorological forecasting tasks according to claim 6, characterized in that, The performance of the lightweight prediction model established in S4 is evaluated by S5, specifically by using the TS score, false alarm rate, and false negative rate to evaluate the prediction accuracy of the lightweight prediction model.
8. A lightweight modeling device for high-dimensional data meteorological forecasting tasks, characterized in that, include: The high-dimensional data acquisition module is used to acquire the raw high-dimensional data for the target weather forecasting task and construct the forecast vector and target variable; The latent predictor variable reconstruction module is used to reconstruct latent predictor variables that affect or potentially affect the target variable from the prediction vector through an improved variational autoencoder strategy, select latent predictor variables that are strongly correlated with the target variable from the latent predictor variables, and construct a candidate predictor variable dataset. The improved variational autoencoder strategy optimizes the reconstruction of error terms and regularization terms by labeling extreme values in the prediction vector. The key predictor variable screening module is used to perform correlation analysis and redundancy detection on the candidate predictor variable dataset, screen out the candidate predictor variable dataset that is strongly correlated with the target variable and has clear physical meaning, and screen the key predictor variable dataset from the screened candidate predictor variable dataset. The model building module builds a lightweight predictive model based on the selected key predictor variable dataset; and, The model evaluation module is used to evaluate the performance of the lightweight prediction model.
9. A computer system comprising a memory and a processor, wherein the memory stores a computer program that runs on the processor, characterized in that, When the processor runs a computer program, it executes the steps of the lightweight modeling method of claim 1.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the lightweight modeling method of claim 1.