Expert model selection method and device for exogenous variable long-term time series prediction

Abstract

The application relates to the technical field of machine learning model selection, and specifically discloses an expert model selection method and device for exogenous variable long-term time series prediction, which comprises the following steps: obtaining historical time series data of a target prediction task; constructing a current prediction scene according to a data set to which the historical time series data belongs and a preset prediction length; constructing a candidate expert model pool according to the current prediction scene; dividing the historical time series data into a training set and a verification set, training each candidate prediction expert model according to the training set, and obtaining multi-dimensional utility information of each trained candidate prediction expert model according to the verification set; comparing each candidate prediction expert model with each other by using a pre-trained preference decision model according to the multi-dimensional utility information, so as to obtain a comprehensive recommendation score of each candidate prediction expert model; and determining a recommended prediction expert model of the target prediction task from the candidate expert model pool according to the comprehensive recommendation score.

Description

Technical Field

[0001] This application relates to the field of machine learning model selection technology, and in particular to an expert model selection method and apparatus for long-term time series prediction of exogenous variables. Background Technology

[0002] Long-term time series forecasting has wide applications in scenarios such as power system dispatching, traffic flow control, meteorological monitoring and early warning, and industrial equipment condition monitoring. Actual forecasting tasks typically include not only the target variable to be predicted but also multiple auxiliary or exogenous variables, such as temperature, humidity, and holiday information in power load forecasting, and weather conditions and time-period characteristics in traffic flow forecasting. In recent years, various forecasting models have emerged in the field of long-term time series forecasting, including linear forecasting models, decomposition-based forecasting models, Transformer-based forecasting models, patch-based forecasting models, variable-dimensional modeling models, and exogenous variable fusion models. Different models have different inductive biases, computational costs, and input variable utilization capabilities. How to select the most suitable model for the current forecasting task from multiple candidate expert models has become a key issue affecting the actual deployment effectiveness of forecasting systems.

[0003] In existing technologies, the selection of prediction models is typically based on validation set error. This involves training each candidate model, calculating the prediction error on the validation set, and selecting the model with the lowest error as the final deployment model. However, this selection method has significant limitations. First, while validation set error reflects a portion of a model's predictive ability, the optimal model on the validation set does not always perform best in subsequent testing or deployment phases. Relying solely on validation set error can lead to instability in the selection process. Second, the effectiveness of exogenous variables depends on the specific prediction scenario. Introducing exogenous variables may improve prediction performance, or it may introduce redundant information or noise, resulting in a decline in prediction performance. Existing model selection methods typically do not explicitly measure the contribution of exogenous variables to the prediction. Summary of the Invention

[0004] In view of this, this application provides an expert model selection method, apparatus, storage medium, and computer equipment for long-term time series prediction of exogenous variables. By constructing a pool of candidate expert models consisting of individual prediction models for the target variable and multivariate fusion prediction models, the method obtains the prediction error information of each candidate prediction expert model during the validation phase and the contribution information of exogenous variables used to characterize whether exogenous variables bring positive benefits. The method then uses a preference decision model to perform pairwise comparisons and winner aggregation of each candidate prediction expert model to determine the recommended prediction expert model for the current prediction scenario. This solves the technical problems in the prior art where it is difficult to determine the actual contribution of exogenous variables due to the lack of explicit evaluation of the effectiveness of exogenous variables, and the instability of predictions during the deployment phase caused by selecting models solely based on validation set errors, which ultimately affect the accuracy of the selection of recommended prediction expert models.

[0005] According to one aspect of this application, an expert model selection method for long-term time series forecasting of exogenous variables is provided, comprising: Acquire historical time series data for the target prediction task; the historical time series data includes target time series data corresponding to the target variable to be predicted, and exogenous time series data corresponding to at least one exogenous variable; Based on the dataset to which the historical time series data belongs and the preset prediction length, construct the current prediction scenario; Based on the current prediction scenario, a pool of candidate expert models is constructed; the pool of candidate expert models includes multiple candidate prediction expert models, which include at least a target variable prediction model that uses only the target time series data for prediction, and a multivariate fusion prediction model that uses the target time series data and the exogenous time series data for prediction. The historical time series data is divided into a training set and a validation set. Each candidate prediction expert model is trained based on the training set, and the multidimensional utility information of each candidate prediction expert model after training is obtained based on the validation set. The multidimensional utility information includes prediction error information and exogenous variable contribution information. Based on the multidimensional utility information, a pre-trained preference decision model is used to compare each candidate prediction expert model pairwise to obtain the comprehensive recommendation score of each candidate prediction expert model. Based on the comprehensive recommendation score, the recommended prediction expert model for the target prediction task is determined from the pool of candidate expert models.

[0006] According to another aspect of this application, an expert model selection device for long-term time series forecasting of exogenous variables is provided, comprising: The data acquisition module is used to acquire historical time series data of the target prediction task; the historical time series data includes target time series data corresponding to the target variable to be predicted, and exogenous time series data corresponding to at least one exogenous variable. The prediction scenario construction module is used to construct the current prediction scenario based on the dataset to which the historical time series data belongs and the preset prediction length; The model pool construction module is used to construct a candidate expert model pool based on the current prediction scenario. The candidate expert model pool includes multiple candidate prediction expert models, which include at least a target variable standalone prediction model that uses only the target time series data for prediction, and a multivariate fusion prediction model that uses both the target time series data and the exogenous time series data for prediction. The training and validation module is used to divide the historical time series data into a training set and a validation set, train each candidate prediction expert model based on the training set, and obtain multidimensional utility information of each candidate prediction expert model after training based on the validation set; the multidimensional utility information includes prediction error information and exogenous variable contribution information. The scoring calculation module is used to compare each candidate prediction expert model pairwise based on the multidimensional utility information and a pre-trained preference decision model to obtain the comprehensive recommendation score of each candidate prediction expert model. The expert model determination module is used to determine the recommended prediction expert model for the target prediction task from the candidate expert model pool based on the comprehensive recommendation score.

[0007] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described expert model selection method for long-term time series forecasting of exogenous variables.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described expert model selection method for long-term time series forecasting of exogenous variables.

[0009] By employing the aforementioned technical solutions, this application provides an expert model selection method, apparatus, storage medium, and computer equipment for long-term time series forecasting of exogenous variables. By acquiring multi-dimensional utility information of each candidate prediction expert model during the validation phase and using a preference decision model for pairwise comparisons, it changes the traditional method of selecting models solely based on the error of a single validation set, effectively avoiding the problem of unstable performance of the optimal model on the validation set during the deployment phase. Simultaneously, by incorporating the contribution information of exogenous variables into the comparison framework, it can explicitly quantify whether exogenous variables truly bring positive benefits in the current prediction scenario, enabling both models adept at utilizing effective exogenous variables and models that avoid ineffective exogenous variables to be reasonably recommended. This achieves automatic judgment of the effectiveness of exogenous variables and dynamic adaptation of model selection. Furthermore, the pairwise preference judgment and winning mechanism does not require manual weighting or normalization of each utility dimension. The preference decision model automatically learns the comprehensive impact of each dimension from historical data, ultimately recommending the expert model with better prediction performance during the deployment phase based on the current dataset and prediction length, thereby improving the prediction accuracy and selection stability of the long-term time series forecasting system in real deployment scenarios.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The diagram illustrates a flowchart of an expert model selection method for long-term time series forecasting of exogenous variables provided in an embodiment of this application. Figure 2 This illustration shows a schematic diagram of the average isomorphic target-only comparison gain of different models provided in an embodiment of this application; Figure 3 This illustration shows a schematic diagram of the average isomorphic target-only control gain across different datasets, as provided in an embodiment of this application. Figure 4 This illustration shows a schematic diagram of an expert model selection device for long-term time series prediction of exogenous variables provided in an embodiment of this application. Figure 5 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides an expert model selection method for long-term time series forecasting of exogenous variables, such as... Figure 1 As shown, the method includes: Step 101: Obtain historical time series data for the target prediction task; the historical time series data includes target time series data corresponding to the target variable to be predicted, and exogenous time series data corresponding to at least one exogenous variable.

[0014] Step 102: Construct the current prediction scenario based on the dataset to which the historical time series data belongs and the preset prediction length.

[0015] Step 103: Based on the current prediction scenario, construct a pool of candidate expert models; the pool of candidate expert models includes multiple candidate prediction expert models, which include at least a target variable prediction model that uses only the target time series data for prediction, and a multivariate fusion prediction model that uses the target time series data and the exogenous time series data for prediction.

[0016] Step 104: Divide the historical time series data into a training set and a validation set, train each candidate prediction expert model based on the training set, and obtain the multidimensional utility information of each candidate prediction expert model after training based on the validation set; the multidimensional utility information includes prediction error information and exogenous variable contribution information.

[0017] Step 105: Based on the multidimensional utility information, use the pre-trained preference decision model to perform pairwise comparisons of each candidate prediction expert model to obtain the comprehensive recommendation score of each candidate prediction expert model.

[0018] Step 106: Based on the comprehensive recommendation score, determine the recommended prediction expert model for the target prediction task from the candidate expert model pool.

[0019] This application provides an expert model selection method for long-term time series forecasting of exogenous variables. First, historical time series data of the target forecasting task is acquired. This data contains two types of information: one is the value of the target variable to be predicted at different historical moments, and the other is the value of related auxiliary variables (i.e., exogenous variables) at different historical moments. Here, exogenous variables can be auxiliary variables such as weather temperature, humidity, and holiday markers that can affect the target variable but are not themselves used as forecast outputs. The purpose of acquiring these two types of data is to allow the expert model to either make predictions based solely on the historical patterns of the target variable itself, or to use information provided by exogenous variables to assist in the prediction. By comparing the effects of the two methods, it can be determined whether the exogenous variables have played a positive role in the current task.

[0020] After acquiring historical time-series data, a current prediction scenario can be constructed based on the dataset to which the historical time-series data belongs and the user-preset prediction length. The dataset reflects the source and statistical characteristics of the data itself; for example, electricity load data and traffic flow data have different periodicity, trends, and noise distributions. The prediction length determines the difficulty of the task; for example, predicting 96 time steps ahead and predicting 720 time steps ahead place completely different demands on the model. Combining these two elements into the current prediction scenario aims to bind subsequent expert model selection and evaluation to this specific scenario. This is because the performance of the same expert model may vary significantly across different datasets or prediction lengths. Only by evaluating and selecting models for a specific scenario can the reliability of the recommendation results be guaranteed.

[0021] For the constructed current prediction scenario, a candidate expert model pool can be established, containing multiple candidate prediction expert models of different types. These candidate prediction expert models include at least two categories: one is a target variable standalone prediction model that uses only the historical data of the target variable itself for prediction, and the other is a multivariate fusion prediction model that uses historical data of both the target variable and exogenous variables for prediction. Both types of models are included because different models may perform drastically differently in different prediction scenarios. Some models have simple structures and good generalization performance, while others have complex structures but strong fitting capabilities. Some models are good at utilizing exogenous variable information, while others experience a decline in performance due to noise interference after introducing exogenous variables. Constructing a candidate expert model pool containing multiple types of models provides ample options for subsequently selecting the most suitable expert model for the current prediction scenario. In a specific embodiment, candidate prediction expert models may include: target variable standalone prediction models, linear prediction models, deep long-term time series prediction models, patch-based prediction models, variable dimensional modeling models, multivariate fusion prediction models, exogenous variable fusion prediction models, etc. Examples include: naive_last, ridge_target, DLinear, TimeXer, CrossLinear, XLinear, PatchTST, and iTransformer.

[0022] Subsequently, the historical time series data was divided into two parts: a training set and a validation set. The training set was used to train each candidate prediction expert model in the pool. Specifically, training data from the training set was input into the candidate prediction expert models, allowing them to learn the mapping relationship between the target variable and the input variables, and adjusting the model's internal parameters to fit the patterns in the training data as accurately as possible. The validation set was used to evaluate the prediction performance of each model after training: input data from the validation set was fed into the trained model, allowing the model to output predicted values. These predicted values ​​were then compared with the actual values ​​of the target variable in the validation set to calculate the prediction error information for each model. Furthermore, based on this prediction error information, the exogenous variable contribution information for each model could be calculated, i.e., the degree of error improvement obtained by the model compared to prediction using only the target variable, thereby determining whether the exogenous variables truly brought positive benefits under this model structure.

[0023] After extracting the multidimensional utility information of each model, a pre-trained preference decision model can be used to compare all candidate prediction expert models pairwise to obtain the comprehensive recommendation score of each candidate prediction expert model. Then, the candidate prediction expert model with the highest comprehensive recommendation score is used as the recommended prediction expert model for the target prediction task.

[0024] By applying the technical solution of this embodiment, multi-dimensional utility information of each candidate prediction expert model during the validation phase is obtained, and pairwise comparisons are performed using a preference decision model. This changes the traditional method of selecting a model based solely on the error of a single validation set, effectively avoiding the problem of unstable performance of the optimal model on the validation set during the deployment phase. Simultaneously, by incorporating the contribution information of exogenous variables into the comparison framework, it is possible to explicitly quantify whether exogenous variables truly bring positive benefits in the current prediction scenario. This allows models that effectively utilize exogenous variables and models that avoid ineffective exogenous variables to be reasonably recommended, achieving automatic identification of the effectiveness of exogenous variables and dynamic adaptation of model selection. Furthermore, the pairwise preference judgment and winning mechanism does not require manual weighting or normalization of each utility dimension. The preference decision model automatically learns the comprehensive impact of each dimension from historical data, ultimately recommending the expert model with better prediction performance during the deployment phase based on the current dataset and prediction length. This improves the prediction accuracy and selection stability of the long-term time series prediction system in real deployment scenarios.

[0025] In this embodiment of the application, optionally, the prediction error information includes validation set prediction error, ranking results, and error interval; step 104, "obtaining multi-dimensional utility information of each candidate prediction expert model after training based on the validation set," includes: calculating the validation set prediction error of each candidate prediction expert model based on the difference between the prediction output of each candidate prediction expert model on the validation set and the true value; ranking each candidate prediction expert model according to the magnitude of the validation set prediction error of each candidate prediction expert model to obtain the ranking results of each candidate prediction expert model in the current prediction scenario; and obtaining the multi-dimensional utility information of each candidate prediction expert model from the validation set prediction error of each candidate prediction expert model. The minimum value is determined from the differences, and the error interval of each candidate prediction expert model is determined based on the difference between the validation set prediction error of each candidate prediction expert model and the minimum value. For each candidate prediction expert model, the error difference information between the validation set prediction error of the candidate prediction expert model and the reference prediction error is calculated as the exogenous variable contribution information of the candidate prediction expert model. The reference prediction error is the prediction error corresponding to the prediction of the target time series data based on the target variable alone. The exogenous variable contribution information is used to help determine whether the candidate prediction expert model brings positive prediction benefits in the current prediction scenario by utilizing exogenous variables.

[0026] In this embodiment, firstly, the prediction performance of each candidate prediction expert model on the validation set is quantitatively evaluated. Specifically, for each candidate prediction expert model, its prediction output on the validation set is compared with the actual target variable value in the validation set at each time step, and the degree of difference between the two is calculated. For example, the greater the difference, the lower the prediction accuracy of the model, and vice versa. This quantitative result is the validation set prediction error of the model.

[0027] Based on this, all candidate prediction expert models are ranked from best to worst according to the magnitude of their validation set prediction errors. The model with the smallest validation set prediction error is ranked best, and the model with the largest error is ranked last. The ranking results intuitively reflect the relative prediction accuracy of each model in the current prediction scenario.

[0028] Simultaneously, the error interval for each candidate predictive expert model is calculated, which is the difference between the current model's validation set prediction error and the smallest validation set prediction error among all candidate predictive expert models. This difference reflects the accuracy gap between the current model and the optimal model on the validation set in the current prediction scenario: the larger the difference, the further the model is from the optimal model; a difference of zero indicates that the model itself is the optimal model on the validation set. As supplementary information to the validation set ranking, the error interval not only tells a model its ranking but also quantitatively characterizes its specific distance from the top-ranked model, providing a more refined numerical basis for subsequent comparisons of preference decision models.

[0029] Furthermore, the exogenous variable contribution information of each candidate prediction expert model is calculated. The core logic is to calculate the difference between the validation set prediction error and a reference prediction error for each multivariate fusion prediction model. Here, the reference prediction error is the prediction error obtained using only the historical data of the target variable itself. For each multivariate fusion prediction model, its validation set prediction error is compared with the reference prediction error. If its validation set prediction error is significantly lower than the reference prediction error, it indicates that the model has indeed achieved positive prediction benefits in the current prediction scenario. This information, along with the validation set prediction error, ranking results, and error intervals, constitutes the model's multidimensional utility information and participates in pairwise comparisons and the calculation of the comprehensive recommendation score. It is important to note that for each target variable individual prediction model, its exogenous variable contribution information can be directly set to zero.

[0030] This application extracts supplementary features from three dimensions: ranking results, error interval, and exogenous variable contribution information. These three features, together with the validation set prediction error itself, constitute a more comprehensive characterization of each candidate prediction expert model. This allows subsequent pairwise comparisons to no longer rely on a single error value, but to make a comprehensive judgment based on richer multidimensional utility information. This effectively makes up for the core defect of traditional methods that the model with the lowest validation set prediction error may not be optimal during the deployment stage, and improves the stability and reliability of model recommendations in real prediction scenarios.

[0031] In this embodiment of the application, optionally, the exogenous variable contribution information is obtained by: determining a set of target variable individual prediction models from the candidate expert model pool; determining the minimum value of the validation set prediction error of each model in the target variable individual prediction model set as the reference prediction error; for each multivariate fusion prediction model, dividing the difference between the reference prediction error and the validation set prediction error of the multivariate fusion prediction model by the reference prediction error, and using the resulting ratio as the exogenous variable contribution information of the multivariate fusion prediction model; and setting the exogenous variable contribution information of each target variable individual prediction model to zero.

[0032] In this embodiment, firstly, all models that use only historical data of the target variable itself for prediction are selected from the pool of candidate expert models. These models do not rely on information from any exogenous variables, but only on the inherent laws such as the periodicity, trend, and autocorrelation of the target variable itself for prediction. These target variable-only prediction models can include simple baseline models (such as persistent models that directly replicate the most recent values ​​or regression models based on historical means) and univariate deep models with a certain learning ability. The purpose of grouping these models into an independent set is that they represent a type of prediction benchmark that does not rely on any auxiliary information, and their prediction performance can reflect the predictability inherent in the historical information of the target variable itself, thus providing an objective reference for subsequently measuring whether the exogenous variable is truly effective.

[0033] Next, within the set of target variable individual prediction models, the model with the smallest prediction error on the validation set is identified, and its error value is determined as the reference prediction error. This minimum value represents the best performance achievable in the current prediction scenario, relying solely on the historical information of the target variable itself. Using the error of this optimal target variable individual prediction model as a benchmark, all subsequent candidate prediction expert models (including other target variable individual prediction models and multivariate fusion prediction models) can be compared with this target variable individual prediction model to determine whether they are truly superior to the best solution in the current prediction scenario without the aid of exogenous variables.

[0034] For each multivariate fusion prediction model in the pool, the degree to which its validation set prediction error deviates from the aforementioned reference prediction error is calculated. Specifically, the model's validation set prediction error is subtracted from the reference prediction error, and then the difference is divided by the reference prediction error to obtain a ratio. This ratio represents the "relative error improvement rate of the model relative to the univariate optimal benchmark": if the ratio is positive, for example, 0.1, it means that the model's error is 10% lower than the univariate optimal benchmark, meaning it has achieved a positive gain based on the univariate optimal benchmark. This gain may come from a better model structure or the effective use of exogenous variables; if the ratio is zero, it means that the model is on par with the univariate optimal benchmark; if the ratio is negative, for example, -0.05, it means that the model's error is actually 5% higher than the univariate optimal benchmark, implying that introducing a more complex model structure or using exogenous variables has failed to provide positive assistance in the current prediction scenario. Ultimately, this ratio is directly used as the exogenous variable contribution information for the multivariate fusion prediction model; the larger the value, the greater the positive gain of the model relative to the univariate optimal benchmark. In addition, the exogenous variable contribution information of each target variable prediction model can be set to zero, since these models do not contain exogenous variables and therefore will not generate exogenous variable contribution information.

[0035] Compared to directly comparing the absolute values ​​of prediction errors, this embodiment normalizes the errors of all multivariate fusion prediction models by referring to the prediction errors, making the contribution information of exogenous variables a dimensionless relative ratio. This ratio not only identifies which models are better than the optimal single-variable benchmark and which models are worse than the benchmark, but also provides the specific degree of superiority or inferiority, making the contribution information of each model comparable in different prediction scenarios. This mechanism effectively avoids the problem of cross-scenario incomparability caused by the different prediction difficulties in different scenarios, and provides standardized input features for the unified judgment of subsequent preference decision models in different prediction scenarios.

[0036] Optionally, in this embodiment, the exogenous variable contribution information is obtained by: determining a set of multivariate fusion prediction models from the candidate expert model pool; for each model in the multivariate fusion prediction model set, constructing a corresponding first version and a second version; the first version uses the target time series data of the target variable and the exogenous time series data of the exogenous variable as input; the second version uses only the target time series data of the target variable as input, and the first version and the second version adopt the same model structure; obtaining the validation set prediction errors of the first version and the second version respectively; dividing the difference between the validation set prediction error of the second version and the validation set prediction error of the first version by the validation set prediction error of the second version, and using the resulting ratio as the corresponding exogenous variable contribution information; setting the exogenous variable contribution information of each target variable individual prediction model to zero.

[0037] In this embodiment, firstly, all multivariate fusion prediction models that support multivariate input are selected from the candidate expert model pool; that is, models whose model structure can simultaneously accept both target variables and exogenous variables as input for prediction. These models can include various Transformer-based fusion prediction models, Patch-based fusion prediction models, and variable-dimensional modeling models, etc. These models are designed to utilize information provided by exogenous variables to improve prediction accuracy. These models are grouped into a separate set because they support two input modes: one mode uses both target and exogenous variables for prediction, and the other mode uses only the historical information of the target variable itself for prediction. By switching between the two input modes on the same model structure, it is possible to accurately measure whether the exogenous variables truly play a positive role under this model structure.

[0038] For each selected multivariate fusion prediction model, two versions were constructed: the first version used the complete input, meaning it simultaneously incorporated the historical time-series data of the target variable and all exogenous variables as input, allowing the model to fully explore the correlation between the exogenous and target variables during training; the second version used only the historical time-series data of the target variable as input, without providing any exogenous variable information. Both versions employed the exact same model structure, number of layers, number of hidden units, and training hyperparameters; the only difference was whether the input included exogenous variables. This "controlled variable method" design ensured that subsequent error differences could be entirely attributed to the difference in input information, rather than differences in model structure or other factors, thus accurately assessing the net contribution of exogenous variables under this model structure.

[0039] Next, the prediction errors of the two versions of the same model on the validation set were obtained. Since the two versions were identical in all aspects except for the input variables—training set partitioning, validation set partitioning, prediction length, and model structure—the only source of the difference in prediction error calculated on the validation set was the input difference of whether exogenous variable information was introduced. If the validation set prediction error of the first version (including exogenous variables) was significantly lower than that of the second version (target variable only), it indicates that the information contained in the exogenous variables was effectively utilized by the model, thereby improving prediction accuracy. Conversely, if the validation set prediction error of the first version was not lower than that of the second version, it indicates that the information in the exogenous variables did not match the fitting method of the model structure and failed to bring positive benefits.

[0040] For each multivariate fusion prediction model, the difference between the prediction error of the second version of the validation set and the prediction error of the first version is calculated. This difference is then divided by the prediction error of the second version of the validation set to obtain the exogenous variable contribution information of the model. This ratio reflects the degree of relative error improvement gained by the model due to the introduction of exogenous variables: if the ratio is positive, for example, 0.08, it means that the prediction error of the model decreased by 8% after introducing exogenous variables, indicating that the exogenous variables brought positive benefits under this model structure; if the ratio is zero, it means that the prediction effect before and after introduction is comparable, and the exogenous variables did not provide additional information gain; if the ratio is negative, for example, -0.03, it means that the error actually increased by 3% after introducing exogenous variables, indicating that the exogenous variables introduced noise or redundant information under this model structure. Ultimately, this ratio serves as the exogenous variable contribution information of the multivariate fusion prediction model; a larger value indicates a more significant positive benefit of the exogenous variables under this model structure.

[0041] The embodiments of this application directly compare two input versions of the same model, which has the advantage of eliminating the interference of differences in model structure. At the same time, this approach can also provide comparable benefit measures for different types of multivariate fusion prediction models.

[0042] Optionally, in this embodiment, step 105 includes: for any two candidate prediction expert models in the candidate expert model pool, obtaining first multidimensional utility information of the first candidate prediction expert model and second multidimensional utility information of the second candidate prediction expert model; constructing a differential utility feature between the first candidate prediction expert model and the second candidate prediction expert model based on the difference between the first multidimensional utility information and the second multidimensional utility information; inputting the differential utility feature into the preference decision model to obtain a preference judgment result that the first candidate prediction expert model is superior to the second candidate prediction expert model; traversing all pairwise combinations in the candidate expert model pool to obtain the preference judgment result of each candidate prediction expert model relative to all other candidate prediction expert models; and counting the number of wins for each candidate prediction expert model based on its winning performance in the preference judgment result, using the number of wins as the comprehensive recommendation score of the candidate prediction expert model.

[0043] In this embodiment, firstly, two candidate prediction expert models are randomly selected from the candidate expert model pool, and their multidimensional utility information extracted during the validation phase is obtained. The multidimensional utility information includes at least the model's validation set prediction error, ranking results, error interval, and exogenous variable contribution information, among other metrics. The purpose of obtaining the multidimensional utility information of these two models is to compare their relative performance in the current prediction scenario. This comparison is not based on a single metric but on the comprehensive utility across multiple dimensions; therefore, the complete utility information of both models is extracted first as the basis for comparison.

[0044] After obtaining the multidimensional utility information of two candidate predictive expert models (the first candidate predictive expert model and the second candidate predictive expert model), the second multidimensional utility information of the second candidate predictive expert model is subtracted from the first multidimensional utility information of the first candidate predictive expert model item by item, resulting in a set of difference vectors. This set of difference vectors represents the differential utility features between the two models. Specifically, if the first candidate predictive expert model has a lower validation set prediction error, a higher ranking, a smaller error interval, and a higher contribution of exogenous variables than the second candidate predictive expert model, then these differences can be reflected as positive or negative values ​​in the corresponding dimensions of the differential utility features, fully characterizing the advantages and disadvantages of the first candidate predictive expert model relative to the second candidate predictive expert model in each utility dimension. Compared to directly concatenating the original utility information of the two models and inputting it into the preference decision model, the differential utility features can more directly reflect the relative differences between the two models, transforming absolute evaluation into relative comparison and reducing the learning complexity required by the preference decision model.

[0045] Furthermore, the constructed differential utility features are input into a pre-trained preference decision model. The preference decision model is essentially a trained binary classifier that has learned the ability to predict which model will perform better in actual deployment, given the differential utility features of any two candidate models, through training data from a large amount of historical prediction scenarios. This model accepts the differential utility features as input and outputs a preference judgment result, which can be a preference probability (e.g., a value between 0 and 1, representing the probability that the first candidate prediction expert model is superior to the second candidate prediction expert model). This preference judgment result is not calculated based on manually set rules or weights, but is an automatic comprehensive judgment given by the preference decision model based on the patterns it has learned from historical data, enabling the model selection process to automatically adapt to the distribution of utility features under different prediction scenarios. In a specific embodiment, the preference decision model can be implemented using logistic regression, decision trees, random forests, support vector machines, gradient boosting trees, multilayer perceptrons, or other classifiers. In one embodiment, the training labels of the preference decision model can be constructed based on the prediction errors of different candidate prediction expert models during the testing phase in historical prediction scenarios.

[0046] Since the pool of candidate expert models contains multiple candidate predictive expert models, comparing only one pair of models is insufficient to determine the globally optimal recommendation. Therefore, this embodiment of the application traverses all possible pairwise combinations in the pool, that is, for each pair of models, it performs the complete operation described above: obtaining utility information, constructing differential features, and inputting the model into the preference decision to obtain the preference judgment result. This traversal process ensures that each candidate predictive expert model is directly compared with all other models, providing comprehensive and unbiased comparative data for subsequent winner statistics.

[0047] After all pairwise comparisons are completed, a comprehensive recommendation score is calculated based on the performance of each model in the preference judgment results. Specifically, for each candidate prediction expert model, the number of times it wins in all pairwise comparisons is counted, that is, in how many comparisons the preference decision model judges it to be superior to the other model. This number of wins is directly used as the comprehensive recommendation score of the candidate prediction expert model. The more wins, the more outstanding the model's performance and the more obvious its comprehensive advantage in comparison with all other candidate prediction expert models. The model with the highest final recommendation score can be determined as the optimal recommendation model for the current prediction scenario.

[0048] Compared to traditional methods such as directly sorting by validation set error and selecting the model with the smallest error, or sorting by weighted summation of multi-dimensional indicators, this application's embodiment achieves the organic integration of multi-dimensional utility information through pairwise comparisons and a winner aggregation mechanism. In each pair of comparisons, the preference decision model comprehensively judges the relative merits of the two models based on differential utility characteristics, avoiding subjective biases caused by manually setting weights. In the aggregation stage, the statistics of the number of wins ensure that the final scores of each candidate prediction expert model have good interpretability. This application's embodiment effectively eliminates the one-sidedness of single-indicator ranking and the subjectivity of human weighting, making the final recommendation results more robust, reliable, and adaptable to different scenarios.

[0049] In this embodiment of the application, optionally, the preference judgment result includes the preference probability that the first candidate prediction expert model is better than the second candidate prediction expert model; the step of "statistically counting the number of times each candidate prediction expert model wins based on the winning situation of each candidate prediction expert model in the preference judgment result" includes: when the preference probability is greater than a preset threshold, determining that the first candidate prediction expert model wins over the second candidate prediction expert model; otherwise, determining that the first candidate prediction expert model does not win; and counting the number of times each candidate prediction expert model wins in all pairwise combinations.

[0050] In this embodiment, the preference judgment result is further defined. Specifically, the output of the preference decision model is a preference probability, which can be a value between 0 and 1, representing the likelihood that the first candidate prediction expert model is better than the second candidate prediction expert model. For example, a preference probability of 0.85 means that the first candidate prediction expert model is better than the second candidate prediction expert model by 85%, while a preference probability of 0.30 means that the first candidate prediction expert model is better than the second candidate prediction expert model by only 30%. Simultaneously, a preset threshold is introduced to convert the continuous value of the preference probability into a discrete winning result, thereby enabling the statistical counting of the number of wins.

[0051] After obtaining the preference probability between any two candidate predictive expert models, this probability is compared with a preset threshold to determine the winner. In a specific embodiment, the preset threshold can be 0.5, because in binary classification comparisons, 0.5 represents a natural boundary of "unbiasedness." When the preference probability is greater than 0.5, it means that the preference decision model believes the probability of the first candidate predictive expert model being better than the second candidate predictive expert model exceeds the level of random guessing, thus determining that the first candidate predictive expert model wins. Conversely, when the preference probability is not greater than 0.5, it means that the first candidate predictive expert model has not shown sufficient evidence to be better than the second candidate predictive expert model, and in this case, the first candidate predictive expert model does not win. The advantage of using a threshold mechanism is that it can filter out comparison results with insufficient confidence, thereby improving the reliability of the winning statistical results.

[0052] After determining the preference probabilities for all pairwise combinations, the total number of wins for each candidate predictive expert model in all pairwise combinations is counted. For example, if there are 8 models in the pool, each model participates in 7 pairwise comparisons. In each comparison, the model is judged to win based on the comparison result of the preference probability and a preset threshold. Finally, the total number of wins in the 7 comparisons is counted, and this number of wins is directly used as the model's comprehensive recommendation score. The more wins a model has, the more outstanding its performance and the stronger its comprehensive competitiveness in comparison with all other candidate predictive expert models.

[0053] The embodiments of this application convert the preference probability into a discrete decision of winning / not winning by setting a preset threshold and then count the number of winnings, which has better robustness and noise resistance.

[0054] Optionally, in this embodiment of the application, the multidimensional utility information may further include deployment cost information of each candidate prediction expert model; the deployment cost information includes at least one of model parameter count, training time, inference time, memory usage, and computing resource consumption; the deployment cost information is obtained by calculating or reading the complexity and deployment-side features of each candidate prediction expert model.

[0055] In this embodiment, the multidimensional utility information, in addition to prediction error information and exogenous variable contribution information, may also include deployment cost information for each candidate prediction expert model. Deployment cost information refers to a comprehensive measure of the various computing resources and time costs required to support the normal operation of the model after it is deployed in a real production environment. Specifically, the number of model parameters refers to the total number of trainable parameters in the model network. A larger number of parameters usually means a more complex model with stronger expressive power, but it also requires more storage space and computing resources. Training time refers to the time required for the model to complete a full training cycle on the training set, reflecting the time overhead of the model during offline training. Inference time refers to the time required for the model to make a prediction on a new input sample after deployment, directly affecting the response speed of the prediction system, and is particularly important for scenarios with high real-time requirements. Memory usage refers to the memory capacity required on the GPU or other accelerators during model operation, determining whether the model can be deployed in a specific hardware environment. Computational resource consumption covers a broader range of computational overhead, such as the number of floating-point operations during model operation. The reason for unifying these different types of resource consumption into deployment cost information is that in actual prediction system deployment, prediction accuracy is not the only consideration. Even if a model has the lowest prediction error, if its inference time is too long to meet real-time requirements, or if its memory usage is too high to run on existing hardware, the model is still not feasible for deployment in real-world scenarios.

[0056] Deployment cost information can be obtained by calculating or reading the complexity and deployment features of each candidate predictive expert model. Here, calculation refers to quantitative indicators obtained after static analysis of the model structure or actual test runs. For example, the number of model parameters can be directly calculated by counting the number of parameters in each layer of the model, and inference time can be obtained by running the model multiple times on a validation set or representative samples and measuring the average time. Reading refers to directly obtaining this information from existing records. For example, the model has already recorded and saved its parameter count, training time, inference time, and other metadata during previous training. In the subsequent model selection stage, these pre-stored calculation results can be directly read without repeated measurements. In a specific embodiment, deployment cost information can be obtained after each candidate predictive expert model has been trained but before pairwise comparisons and recommendation decisions. This deployment cost information, along with prediction error information and exogenous variable contribution information, is then incorporated into the multidimensional utility information of each candidate predictive expert model, making deployment cost a formal input dimension for subsequent pairwise comparisons and comprehensive recommendation score calculations.

[0057] This application's embodiments incorporate deployment cost information as a component of multidimensional utility information into a pairwise comparison and winner aggregation framework, achieving a unified consideration of the third dimension of utility beyond prediction accuracy and exogenous variable gains. Deployment cost information is given equal importance to other utility information, participating in the entire process of pairwise difference feature construction, preference decision model judgment, and winner count statistics. This allows the preference decision model to automatically learn the trade-off between accuracy improvement and cost increase under different prediction scenarios based on historical data. Without sacrificing recommendation stability, it considers the model's practical deployment feasibility, ensuring that the final recommended model not only performs excellently in prediction accuracy but also runs stably in real hardware environments with acceptable resource overhead, thereby improving the practical value of the technical solution in industrial deployment scenarios.

[0058] In one specific embodiment, multiple publicly available multivariate long-term time series prediction datasets are selected to construct the main experiment prediction scenarios. Each prediction scenario is determined by the dataset and the prediction length. The candidate expert model pool includes target variable standalone prediction models, linear prediction models, deep long-term time series prediction models, patch-based prediction models, variable dimensionality modeling models, and multivariate fusion prediction models. The main experiment settings are shown in Table 1.

[0059] Table 1 Main Experiment Setup Table

[0060] To evaluate the recommendation capability of the expert model selector in scenarios without prior prediction, this embodiment employs two offline verification methods. It should be noted that the expert model selector encapsulates the expert model selection method for long-term time series prediction of exogenous variables proposed in this application.

[0061] The first method is leave-one-out-of-scene validation, where one "dataset-prediction length" combination scenario is left out as the evaluation scenario each time. The expert model selector is trained using the remaining prediction scenarios, and then its recommendation results are evaluated on the left-out prediction scenario. This method is used to assess the expert model selector's adaptability to unseen prediction scenarios.

[0062] The second approach is leave-one-out validation, where a complete dataset is left out each time, the expert model selector is trained using prediction scenarios on the remaining datasets, and then its recommendation results are evaluated on all prediction lengths of the left-out dataset. This method is more rigorous and is used to evaluate the expert model selector's ability to generalize to unseen datasets.

[0063] In both of the above verification methods, the test set error of the held-out scenario or the held-out dataset is not used as a recommended input, but only for the final effect evaluation.

[0064] To verify the effectiveness of this method, it is compared with the validation-best strategy, which selects the model solely based on validation set error. Furthermore, an oracle is set as a post-hoc theoretical upper limit, i.e., selecting the candidate model with the lowest error on the test set. It should be noted that the oracle is only used for offline performance evaluation and is not used as a practical recommendation input.

[0065] The experimental results are shown in Table 2. The average MSE of the recommended model using this method is 0.255880, which is lower than the average MSE of 0.266151 of the validation-best strategy. This indicates that compared to selecting a model solely based on the validation set error, this method can improve the stability of expert model selection by fusing multiple utility features.

[0066] Table 2 Comparison of average MSE for different model selection strategies

[0067] To further verify whether exogenous variables bring benefits under the same model structure, an isomorphic target-only control experiment was set up based on the main experiment. This experiment did not change the prediction scenario in the main experiment, but instead constructed full-variable input versions and target-only input versions for some candidate prediction expert models that support multivariate inputs. Both used the same or corresponding model structure, prediction length, training set partitioning, validation set partitioning, and evaluation metrics, differing only in the input variables. The experimental settings are shown in Table 3.

[0068] Table 3. Isomorphic target-only control experiment setup table

[0069] The results of the isomorphic target-only control experiments are shown in Table 4. All 180 isomorphic control experiments were successfully completed. The average SameModelGain_val was 0.013537, and the median was 0.000459. The full-variable input version won 52.22% of the tests on the validation set and 51.67% of the tests. This indicates that the full-variable input does not outperform the target-only input in all prediction scenarios, and the effectiveness of exogenous variables is significantly dependent on the dataset, prediction length, and model structure.

[0070] Table 4 Overall Results of the Isomorphic Target-Only Control Experiment

[0071] Figure 2The results show that there are differences in the average isomorphic target-only control returns (i.e., MeanSameModelGain_val) among different candidate predictive expert models. Figure 2 The horizontal axis represents the candidate predictive expert models, and the vertical axis represents the average isomorphic target-only control return. Model 1 is a CrossLinear model, Model 2 is an iTransformer model, Model 3 is a PatchTST model, Model 4 is a TimeXer model, and Model 5 is an XLinear model. Figure 2 The average isomorphic target-only gain of CrossLinear (Model 1), iTransformer (Model 2), XLinear (Model 5) and PatchTST (Model 3) is positive, indicating that their full-variable input versions have an advantage over the target-only versions in some scenarios; the Mean SameModelGain_val of TimeXer (Model 4) is negative, indicating that full-variable input is not beneficial to all models.

[0072] Figure 3 The results show that the average isomorphic target-only control gains differ significantly across different datasets. Figure 3The horizontal axis represents the name of each dataset, and the vertical axis represents the average benefit of isomorphic target-only controls. Among them, dataset 1 is the ETT1 dataset, dataset 2 is the ETT2 dataset, dataset 3 is the ETTm1 dataset, dataset 4 is the ETTm2 dataset, dataset 5 is the Electricity dataset, dataset 6 is the Exchange dataset, dataset 7 is the Illness dataset, dataset 8 is the Traffic dataset, and dataset 9 is the Weather dataset. Specifically, the ETTh1 and ETTh2 datasets are power transformer datasets, collected from two different power transformers, with a sampling frequency of 1 hour and a sequence length of 17,420 time points. Each dataset contains 7 variables. The ETTm1 and ETTm2 datasets are also power transformer datasets, corresponding to the same transformer as ETTh1 and ETTh2, but with a sampling frequency increased to 15 minutes, resulting in a significantly increased sequence length of 69,680 time points, capable of capturing finer-grained details of load changes. The Electricity dataset is an electricity load dataset, with a sampling frequency of 1 hour, containing 321 variables and a sequence length of 26,304 time points. This dataset is characterized by a large number of variables and obvious periodic patterns. The Exchange dataset is a currency exchange rate dataset. The first dataset has a sampling frequency of 1 day, includes 8 variables, and a sequence length of 7588 time points. This data is characterized by being close to a random walk process with almost no obvious periodicity. The second dataset, the Illness dataset, is a public health dataset with a sampling frequency of 1 week, includes 7 variables, and a sequence length of 966 time points. This data is characterized by its extremely short sequence length, therefore the prediction length can only be 24, 36, 48, or 60 time steps. The third dataset, the Traffic dataset, is a traffic flow dataset with a sampling frequency of 1 hour, includes 862 variables, and a sequence length of 17544 time points. This data is characterized by the largest number of variables and obvious daily and weekly periodic patterns. The fourth dataset, the Weather dataset, is a meteorological dataset with a sampling frequency of 10 minutes, includes 21 variables, and a sequence length of 52696 time points. This data is characterized by the highest sampling density and a clear daily periodic pattern. Figure 3 The benefits are most pronounced on the Traffic dataset, while the average benefits are negative on some ETT and Electricity datasets, indicating that the effectiveness of exogenous variables is closely related to the specific dataset.

[0073] Furthermore, the isomorphic target-only comparison benefit was used as a supplementary input feature of the expert model selector to verify its impact on the model's recommendation performance. The experimental results are shown in Table 5.

[0074] Among them, "leave-one-out-of-scene validation" is used to evaluate the ability of the expert model selector to adapt to unseen prediction scenarios, and "leave-one-out-of-dataset validation" is used to evaluate the ability of the expert model selector to generalize to unseen complete datasets.

[0075] As shown in Table 5, the experimental results indicate that adding SameModelGain_val as a supplementary utility feature resulted in slight improvements in both leave-one-out scenario validation and leave-one-out dataset validation, suggesting that the isomorphic target-only comparison benefit can provide supplementary information for expert model selection.

[0076] Table 5 Comparison of selector effects before and after adding SameModelGain_val

[0077] In summary, the main experiment in this embodiment shows that, on an experimental matrix consisting of 9 datasets, 36 prediction scenarios, and 8 candidate prediction expert models, the proposed method achieves a lower average prediction error compared to the validation-best strategy which selects the model solely based on the validation set error.

[0078] Furthermore, isomorphic target-only control experiments show that full-variable input does not outperform target-only input on all models and datasets, and the gains of exogenous variables exhibit significant model and dataset dependencies. Therefore, using validation set performance, validation-side variable gains, model complexity, deployment-side features, and model type information together for expert model selection can improve the stability and interpretability of long-term time series prediction model recommendations.

[0079] Furthermore, as Figure 1 In its specific implementation, this application provides an expert model selection device for long-term time series forecasting of exogenous variables, such as... Figure 4 As shown, the device includes: The data acquisition module is used to acquire historical time series data of the target prediction task; the historical time series data includes target time series data corresponding to the target variable to be predicted, and exogenous time series data corresponding to at least one exogenous variable. The prediction scenario construction module is used to construct the current prediction scenario based on the dataset to which the historical time series data belongs and the preset prediction length; The model pool construction module is used to construct a candidate expert model pool based on the current prediction scenario. The candidate expert model pool includes multiple candidate prediction expert models, which include at least a target variable standalone prediction model that uses only the target time series data for prediction, and a multivariate fusion prediction model that uses both the target time series data and the exogenous time series data for prediction. The training and validation module is used to divide the historical time series data into a training set and a validation set, train each candidate prediction expert model based on the training set, and obtain multidimensional utility information of each candidate prediction expert model after training based on the validation set; the multidimensional utility information includes prediction error information and exogenous variable contribution information. The scoring calculation module is used to compare each candidate prediction expert model pairwise based on the multidimensional utility information and a pre-trained preference decision model to obtain the comprehensive recommendation score of each candidate prediction expert model. The expert model determination module is used to determine the recommended prediction expert model for the target prediction task from the candidate expert model pool based on the comprehensive recommendation score.

[0080] Optionally, the prediction error information includes the validation set prediction error, ranking results, and error interval; the training and validation module is used for: The validation set prediction error of each candidate prediction expert model is calculated based on the difference between the predicted output of each candidate prediction expert model on the validation set and the true value. Based on the magnitude of the prediction error on the validation set of each candidate prediction expert model, the candidate prediction expert models are ranked to obtain the ranking result of each candidate prediction expert model in the current prediction scenario. The minimum value is determined from the validation set prediction errors of each candidate prediction expert model, and the error interval of each candidate prediction expert model is determined based on the difference between the validation set prediction errors of each candidate prediction expert model and the minimum value. For each candidate prediction expert model, the error difference information between the validation set prediction error of the candidate prediction expert model and the reference prediction error is calculated, which is used as the exogenous variable contribution information of the candidate prediction expert model; the reference prediction error is the prediction error corresponding to the prediction of the target time series data based on the target variable alone; the exogenous variable contribution information is used to help determine whether the candidate prediction expert model brings positive prediction benefits in the current prediction scenario by utilizing exogenous variables.

[0081] Optionally, the training and validation module is further configured to: A set of individual prediction models for the target variable is determined from the pool of candidate expert models; The minimum value of the validation set prediction error of each model in the set of individual prediction models for the target variable is determined as the reference prediction error; For each multivariate fusion prediction model, the difference between the reference prediction error and the validation set prediction error of the multivariate fusion prediction model is divided by the reference prediction error, and the resulting ratio is used as the exogenous variable contribution information of the multivariate fusion prediction model. Set the exogenous variable contribution information of each target variable to zero for the individual prediction model.

[0082] Optionally, the training and validation module is further configured to: A set of multivariate fusion prediction models is determined from the pool of candidate expert models; For each model in the multivariate fusion prediction model set, a corresponding first version and a second version are constructed; the first version takes the target time series data of the target variable and the exogenous time series data of the exogenous variable as input; the second version takes only the target time series data of the target variable as input, and the first version and the second version adopt the same model structure. Obtain the prediction errors of the validation sets for the first and second versions respectively; The difference between the second version's validation set prediction error and the first version's validation set prediction error is divided by the second version's validation set prediction error, and the resulting ratio is used as the corresponding exogenous variable contribution information. Set the exogenous variable contribution information of each target variable to zero for the individual prediction model.

[0083] Optionally, the score calculation module is used for: For any two candidate prediction expert models in the candidate expert model pool, obtain the first multidimensional utility information of the first candidate prediction expert model and the second multidimensional utility information of the second candidate prediction expert model; Based on the difference between the first multidimensional utility information and the second multidimensional utility information, construct the differential utility feature between the first candidate prediction expert model and the second candidate prediction expert model; The differential utility features are input into the preference decision model to obtain the preference judgment result that the first candidate prediction expert model is superior to the second candidate prediction expert model; By traversing all pairwise combinations in the pool of candidate expert models, the preference judgment results of each candidate predictive expert model relative to all other candidate predictive expert models are obtained. Based on the success of each candidate prediction expert model in the preference judgment result, the number of times each candidate prediction expert model wins is counted, and the number of wins is used as the comprehensive recommendation score of the candidate prediction expert model.

[0084] Optionally, the preference judgment result includes the preference probability that the first candidate prediction expert model is superior to the second candidate prediction expert model; the score calculation module is further used for: When the preference probability is greater than a preset threshold, the first candidate prediction expert model is determined to win over the second candidate prediction expert model; otherwise, the first candidate prediction expert model is determined not to win. Count the number of times each candidate prediction expert model wins in all pairwise combinations.

[0085] Optionally, the multidimensional utility information also includes deployment cost information for each candidate prediction expert model; the deployment cost information includes at least one of the following: number of model parameters, training time, inference time, memory usage, and computing resource consumption. The deployment cost information is obtained by calculating or reading the complexity and deployment-side features of each candidate prediction expert model.

[0086] It should be noted that other corresponding descriptions of the functional units involved in the expert model selection device for long-term time series prediction of exogenous variables provided in this application embodiment can be found in the following references. Figures 1 to 3 The corresponding descriptions in the method will not be repeated here.

[0087] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 5 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0088] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0089] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0090] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. An expert model selection method for exogenous variable long-term time series prediction, characterized in that, include: Acquire historical time series data for the target prediction task; the historical time series data includes target time series data corresponding to the target variable to be predicted, and exogenous time series data corresponding to at least one exogenous variable; Based on the dataset to which the historical time series data belongs and the preset prediction length, construct the current prediction scenario; Based on the current prediction scenario, a pool of candidate expert models is constructed; The candidate expert model pool includes multiple candidate prediction expert models, which include at least a target variable prediction model that uses only the target time series data for prediction, and a multivariate fusion prediction model that uses both the target time series data and the exogenous time series data for prediction. The historical time series data is divided into a training set and a validation set. Each candidate prediction expert model is trained based on the training set, and the multidimensional utility information of each candidate prediction expert model after training is obtained based on the validation set. The multidimensional utility information includes prediction error information and exogenous variable contribution information; Based on the multidimensional utility information, a pre-trained preference decision model is used to compare each candidate prediction expert model pairwise to obtain the comprehensive recommendation score of each candidate prediction expert model. Based on the comprehensive recommendation score, the recommended prediction expert model for the target prediction task is determined from the pool of candidate expert models.

2. The method of claim 1, wherein, The prediction error information includes the validation set prediction error, ranking results, and error interval; the step of obtaining multi-dimensional utility information of each candidate prediction expert model after training based on the validation set includes: The validation set prediction error of each candidate prediction expert model is calculated based on the difference between the predicted output of each candidate prediction expert model on the validation set and the true value. Based on the magnitude of the prediction error on the validation set of each candidate prediction expert model, the candidate prediction expert models are ranked to obtain the ranking result of each candidate prediction expert model in the current prediction scenario. The minimum value is determined from the validation set prediction errors of each candidate prediction expert model, and the error interval of each candidate prediction expert model is determined based on the difference between the validation set prediction errors of each candidate prediction expert model and the minimum value. For each candidate prediction expert model, the error difference information between the validation set prediction error of the candidate prediction expert model and the reference prediction error is calculated, which is used as the exogenous variable contribution information of the candidate prediction expert model; the reference prediction error is the prediction error corresponding to the prediction of the target time series data based on the target variable alone; the exogenous variable contribution information is used to help determine whether the candidate prediction expert model brings positive prediction benefits in the current prediction scenario by utilizing exogenous variables.

3. The method of claim 2, wherein, The exogenous variable contribution information is obtained through the following methods: A set of individual prediction models for the target variable is determined from the pool of candidate expert models; The minimum value of the validation set prediction error of each model in the set of individual prediction models for the target variable is determined as the reference prediction error; For each multivariate fusion prediction model, the difference between the reference prediction error and the validation set prediction error of the multivariate fusion prediction model is divided by the reference prediction error, and the resulting ratio is used as the exogenous variable contribution information of the multivariate fusion prediction model. Set the exogenous variable contribution information of each target variable to zero for the individual prediction model.

4. The method according to claim 2, characterized in that, The exogenous variable contribution information is obtained through the following methods: A set of multivariate fusion prediction models is determined from the pool of candidate expert models; For each model in the multivariate fusion prediction model set, a corresponding first version and a second version are constructed; the first version takes the target time series data of the target variable and the exogenous time series data of the exogenous variable as input; the second version takes only the target time series data of the target variable as input, and the first version and the second version adopt the same model structure. Obtain the prediction errors of the validation sets for the first and second versions respectively; The difference between the second version's validation set prediction error and the first version's validation set prediction error is divided by the second version's validation set prediction error, and the resulting ratio is used as the corresponding exogenous variable contribution information. Set the exogenous variable contribution information of each target variable to zero for the individual prediction model.

5. The method according to claim 1, characterized in that, The step of comparing each candidate prediction expert model pairwise using a pre-trained preference decision model based on the multidimensional utility information to obtain a comprehensive recommendation score for each candidate prediction expert model includes: For any two candidate prediction expert models in the candidate expert model pool, obtain the first multidimensional utility information of the first candidate prediction expert model and the second multidimensional utility information of the second candidate prediction expert model; Based on the difference between the first multidimensional utility information and the second multidimensional utility information, construct the differential utility feature between the first candidate prediction expert model and the second candidate prediction expert model; The differential utility features are input into the preference decision model to obtain the preference judgment result that the first candidate prediction expert model is superior to the second candidate prediction expert model; By traversing all pairwise combinations in the pool of candidate expert models, the preference judgment results of each candidate predictive expert model relative to all other candidate predictive expert models are obtained. Based on the success of each candidate prediction expert model in the preference judgment result, the number of times each candidate prediction expert model wins is counted, and the number of wins is used as the comprehensive recommendation score of the candidate prediction expert model.

6. The method according to claim 5, characterized in that, The preference judgment result includes the preference probability that the first candidate prediction expert model is superior to the second candidate prediction expert model; the step of counting the number of wins for each candidate prediction expert model based on its win rate in the preference judgment result includes: When the preference probability is greater than a preset threshold, the first candidate prediction expert model is determined to win over the second candidate prediction expert model; otherwise, the first candidate prediction expert model is determined not to win. Count the number of times each candidate prediction expert model wins in all pairwise combinations.

7. The method according to claim 1, characterized in that, The multidimensional utility information also includes deployment cost information for each candidate prediction expert model; the deployment cost information includes at least one of the following: number of model parameters, training time, inference time, memory usage, and computing resource consumption. The deployment cost information is obtained by calculating or reading the complexity and deployment-side features of each candidate prediction expert model.

8. An expert model selection device for long-term time series forecasting of exogenous variables, characterized in that, include: The data acquisition module is used to acquire historical time series data of the target prediction task; the historical time series data includes target time series data corresponding to the target variable to be predicted, and exogenous time series data corresponding to at least one exogenous variable. The prediction scenario construction module is used to construct the current prediction scenario based on the dataset to which the historical time series data belongs and the preset prediction length; The model pool construction module is used to construct a candidate expert model pool based on the current prediction scenario. The candidate expert model pool includes multiple candidate prediction expert models, which include at least a target variable standalone prediction model that uses only the target time series data for prediction, and a multivariate fusion prediction model that uses both the target time series data and the exogenous time series data for prediction. The training and validation module is used to divide the historical time series data into a training set and a validation set, train each candidate prediction expert model according to the training set, and obtain the multi-dimensional utility information of each candidate prediction expert model after training according to the validation set. The multidimensional utility information includes prediction error information and exogenous variable contribution information; The scoring calculation module is used to compare each candidate prediction expert model pairwise based on the multidimensional utility information and a pre-trained preference decision model to obtain the comprehensive recommendation score of each candidate prediction expert model. The expert model determination module is used to determine the recommended prediction expert model for the target prediction task from the candidate expert model pool based on the comprehensive recommendation score.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images