Hybrid expert model and search-enhanced time series data prediction method and application

Abstract

A time series data prediction method based on a hybrid expert model and retrieval enhancement and an application thereof, comprising the following steps: S1. processing input time series data using a self-supervised pre-training network to extract high-level features with rich semantic information; S2. using the extracted high-level features to retrieve features with high correlation with the to-be-predicted task from a feature pool through a retrieval module, wherein the feature pool is a pre-constructed representative data set; S3. using the extracted high-level features and the retrieval results to select the optimal expert network through an expert selection module; and S4. using the extracted high-level features and the retrieval results to obtain a prediction result through the optimal expert network. The method proposed in the application improves the accuracy of time series data prediction and provides better support for device operation, resource management and scheduling decisions of related applications in its application.

Description

Technical Field

[0001] This invention relates to artificial intelligence and its application technologies, and in particular to a time-series data prediction method based on a hybrid expert model and retrieval enhancement, and its application. Background Technology

[0002] Time series forecasting is a critical task in many fields. For example, in power dispatching, more accurate load forecasting allows operators to adjust power plant output more precisely, improving grid stability while reducing dispatching costs. However, the dynamics and complexity of the real world present significant challenges to researchers. To address these challenges and improve forecast accuracy, machine learning techniques have been introduced. Traditional models (such as decision tree-based models) and support vector machines (SVMs) have been widely used in the past. Unlike traditional statistical methods, deep learning models are better able to capture the complex nonlinear relationships between various variables that are difficult to model using traditional techniques.

[0003] While these existing forecasting methods have been successful, they rely on the assumption that all data follows the same distribution, employing a single model to model time series data. However, in reality, the characteristics of time series data are not static, and similar patterns can exist even in heterogeneous data. Therefore, when designing time series forecasting methods, it is necessary to consider the existence of multiple patterns to pursue higher predictive performance.

[0004] Although time series prediction models with different neural network structures have been developed, existing methods still have some limitations, mainly: 1. They do not make full use of data, especially heterogeneous data, which limits the feature extraction capabilities of the models; 2. Using a single model to predict data cannot effectively tap into the inherent diversity of the data.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings of the aforementioned background technology and provide a time-series data prediction method based on hybrid expert models and retrieval enhancement, as well as its application.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect of the present invention, a method for predicting time-series data based on a hybrid expert model and retrieval enhancement is provided, comprising the following steps:

[0009] S1. Use a self-supervised pre-trained network to process the input time series data and extract high-level features with rich semantic information;

[0010] S2. Using the extracted high-level features, the retrieval module retrieves features that are highly correlated with the task to be predicted from the feature pool, wherein the feature pool is a pre-constructed representative dataset;

[0011] S3. Using the extracted high-level features and retrieval results, select the optimal expert network through the expert selection module;

[0012] S4. Using the extracted high-level features and retrieval results, obtain prediction results through an optimal expert network.

[0013] In a second aspect of the present invention, a power load forecasting method using the aforementioned time-series data forecasting method is provided, comprising the following steps:

[0014] S1. Use a self-supervised pre-trained network to process historical load data of the power system and extract high-level features that reflect power consumption patterns;

[0015] S2. Using the extracted high-level features reflecting power consumption patterns, the retrieval module retrieves features that are highly correlated with the power load prediction task from the feature pool, wherein the feature pool is pre-built and contains a representative dataset of historical power load data.

[0016] S3. Using the extracted advanced features and retrieval results, the optimal expert network best suited to the characteristics of the power load is selected through the expert selection module;

[0017] S4. Using the extracted high-level features and retrieval results, the selected optimal expert network is used to accurately predict the load of the power system to assist in grid operation and dispatch decisions.

[0018] In a third aspect of the present invention, a traffic flow prediction method using the aforementioned time-series data prediction method is provided, comprising the following steps:

[0019] S1. Use a self-supervised pre-trained network to process historical traffic data and extract high-level features that reflect changes and patterns in traffic flow;

[0020] S2. Using the extracted high-level features reflecting traffic flow changes and patterns, the retrieval module retrieves features that are highly relevant to a specific traffic flow prediction task from the feature pool, wherein the feature pool is a pre-built dataset containing representative data of traffic flow data.

[0021] S3. Using the extracted high-level features and retrieval results, the optimal expert network best suited to traffic flow characteristics is selected through the expert selection module;

[0022] S4. Using the extracted high-level features and retrieval results, the selected optimal expert network is used to accurately predict traffic flow in a specific area or transportation network to support traffic management and planning decisions.

[0023] In a fourth aspect of the present invention, an internet user traffic prediction method using the aforementioned time-series data prediction method is provided, comprising the following steps:

[0024] S1. Use a self-supervised pre-trained network to process historical traffic data of Internet users and extract high-level features that reflect user access patterns and traffic trends;

[0025] S2. Using the extracted high-level features that reflect user access patterns and traffic trends, the retrieval module retrieves features that are highly correlated with the Internet user traffic prediction task from the feature pool, wherein the feature pool is a pre-built dataset containing representative data of Internet user traffic data.

[0026] S3. Using the extracted advanced features and search results, the optimal expert network that best suits the Internet traffic pattern is selected through the expert selection module;

[0027] S4. Using the extracted high-level features and retrieval results, accurately predict internet user traffic through the selected optimal expert network to support network resource management and traffic control decisions.

[0028] The present invention has the following beneficial effects:

[0029] This invention proposes a time-series data prediction method and its application based on self-supervised pre-training and hybrid expert model retrieval enhancement. By introducing and synergistically utilizing self-supervised pre-training and a hybrid expert model, it effectively overcomes the limitations of traditional time-series prediction models in fully capturing the temporal dimension and diversity of data. This method leverages self-supervised pre-training to allow the model to learn high-level representations containing rich semantic information from diverse and heterogeneous data. Based on the retrieval module of the hybrid expert model, it selects the high-level features most relevant to the prediction task and the expert network, working together with the input features to achieve more accurate prediction results. This invention designs a novel neural network structure to improve the performance of time-series prediction, enhances the model's ability to handle diverse data through a hybrid expert model, and fully utilizes pre-constructed representative datasets through retrieval enhancement techniques to further improve model capabilities. The proposed method has improved prediction accuracy in various time-series prediction applications such as power load prediction, traffic flow prediction, and internet user traffic prediction, providing better support for equipment operation, resource management, and scheduling decisions in related applications.

[0030] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the time-series data prediction method based on hybrid expert models and retrieval enhancement according to an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of a self-supervised pre-trained network according to an embodiment of the present invention.

[0033] Figure 3 This is a schematic diagram of the retrieval enhancement and hybrid expert prediction module in an embodiment of the present invention. Detailed Implementation

[0034] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0035] Time series forecasting technology has significant application value in production and daily life, such as electricity load forecasting and traffic flow forecasting. Existing methods mostly focus on improving the model itself, adjusting the model structure for specific data types and tasks, and conducting targeted training. They fail to utilize heterogeneous data to discover common patterns and struggle to handle drifting data. To address these issues, this invention proposes a time series data forecasting method based on self-supervised pre-training and hybrid expert model retrieval enhancement. Self-supervised pre-training allows the model to learn high-level representations containing rich semantic information from diverse heterogeneous data. The retrieval module based on the hybrid expert model can select the high-level features and expert network most relevant to the task to be predicted, achieving more accurate prediction results together with the input features.

[0036] Example 1

[0037] See Figure 1 This invention provides a time-series data prediction method based on a hybrid expert model and retrieval enhancement, comprising the following steps:

[0038] S1. Use a self-supervised pre-trained network to process the input time series data and extract high-level features with rich semantic information;

[0039] S2. Using the extracted high-level features, the retrieval module retrieves features highly relevant to the task to be predicted from the feature pool, where the feature pool is a pre-constructed representative dataset; whether a feature is highly relevant is determined by a predefined relevance standard or rule for judging relevance. For example, the similarity between the dataset in the feature pool and the target sample can be calculated to determine or rank them.

[0040] S3. Using the extracted high-level features and retrieval results, select the optimal expert network through the expert selection module;

[0041] S4. Using the extracted high-level features and retrieval results, obtain prediction results through an optimal expert network;

[0042] The algorithm framework of the preferred embodiment is described in the following reference. Figure 3 The prediction module, consisting of the pre-trained network, the expert selection module, and the expert network, works in conjunction with the retrieval module to achieve efficient prediction of time series data.

[0043] like Figure 2 As shown, in a preferred embodiment, the self-supervised pre-trained network includes: an encoder for encoding the input time-series data into high-dimensional hidden layer features; a decoder for decoding the hidden layer features and reconstructing the input time-series data; a masking module for randomly selecting a certain proportion of the input data for masking operations to generate a masked portion and a visible portion; and an embedding layer for embedding the visible portion and the masked portion after the masking operation respectively to obtain embedded features for subsequent processing. In a more preferred embodiment, the self-supervised pre-trained network uses the mean squared error (MSE) loss function to train the model, and the loss function is defined as:

[0044] L = MSE(S recon ,S vis )

[0045] Where S recon It is the output of the decoder reconstruction, S vis It is the actual value of the visible portion of the data that is not masked.

[0046] See Figure 2 and Figure 3 In a preferred embodiment, the retrieval module constructs a feature pool using high-dimensional representation information from the training set; step S2 specifically includes: extracting feature embeddings of the target sample using a pre-trained encoder, such as a transformer-based encoder; using the extracted feature embeddings as queries to retrieve a set of samples highly correlated with the target sample from the feature pool; performing a target sample-aware order-invariant aggregation operation on the retrieved relevant sample set to generate aggregated feature embeddings and aggregated label embeddings; step S3 specifically includes: inputting the aggregated label embeddings into the gating network of the expert module to adaptively determine precise routing information, dynamically determining which expert modules should be activated based on the aggregated label embeddings, and the contribution of each module to the final prediction result; step S4 specifically includes: inputting the aggregated feature embeddings and target feature embeddings into the activated expert modules for more fine-grained analysis.

[0047] In a preferred embodiment, step S2 specifically includes: the retrieval module divides the original continuous labels into multiple classes based on numerical size to discretize them, and stores them together with the features in the feature pool. The discretized labels are used to estimate feature patterns, determine the activated expert modules, and learn the mapping layer. For the target sample, the search engine retrieves a set of similar samples in the feature pool based on the cosine similarity of the features.

[0048] In a preferred embodiment, in step S2, for a predicted target s t The search engine retrieves similar samples from the feature pool based on the cosine similarity of the features, and inputs them into the aggregation layer for the following object-aware, order-invariant aggregation operation:

[0049]

[0050] Where R(x) t ) is for the target sample s t Relevant samples retrieved, x k It is sample s k The feature part, its weight α k The computation is performed using an attention mechanism, defined as follows:

[0051]

[0052] The label information of related samples is aggregated using the same attention weights:

[0053]

[0054] The obtained aggregated features r and aggregated labels l are used for subsequent predictions.

[0055] See Figure 3 In a preferred embodiment, a sparse hybrid expert system (SMoE) architecture is employed, comprising a set of parallel expert modules, each modeling different temporal feature patterns. A gating network determines the feature pattern to which the target sample belongs and decides which expert module should process the sample. Preferably, the system outputs a weighted sum of the outputs of multiple expert modules, with the weights adaptively determined by the gating network based on the aggregated label embedding. Preferably, the prediction head employs a multilayer perceptron (MLP), which integrates the weighted sum outputs to generate the final prediction result. The Softmax activation function can be used to process the weighted result to obtain the relative importance of each expert module output. The K highest-scoring expert module outputs are selected from the Softmax-processed weighted result to concentrate the most effective prediction information. The output result is input into the MLP layer for further nonlinear transformation and integration. The MLP layer outputs the final prediction result, which integrates the prediction information from multiple expert modules, serving as the output of the entire temporal data prediction method.

[0056] This invention presents a time-series data prediction method based on self-supervised pre-training and hybrid expert model retrieval enhancement. Self-supervised pre-training allows the model to learn high-level representations containing rich semantic information from diverse and heterogeneous data. The retrieval module of the hybrid expert model selects the high-level features most relevant to the prediction task and the expert network, working together with the input features to achieve more accurate prediction results. By introducing and synergistically combining self-supervised pre-training and the hybrid expert system, the limitations of traditional time-series prediction models in fully capturing the temporal dimension and diversity of data are effectively overcome. This embodiment designs a novel neural network structure to improve the performance of time-series prediction, enhances the model's ability to handle diverse data through a hybrid expert model, and fully utilizes pre-constructed representative datasets through retrieval enhancement techniques to further improve model capabilities. This invention can be applied to various time-series prediction applications such as power load prediction, traffic flow prediction, and internet user traffic prediction, improving prediction accuracy and providing better support for equipment operation, resource management, and scheduling decisions in related applications. Examples of these applications are further described below.

[0057] Example 2

[0058] A power load forecasting method using any of the time-series data forecasting methods in Embodiment 1 includes the following steps:

[0059] S1. Use a self-supervised pre-trained network to process historical load data of the power system and extract high-level features that reflect power consumption patterns;

[0060] S2. Using the extracted high-level features reflecting power consumption patterns, the retrieval module retrieves features that are highly correlated with the power load prediction task from the feature pool, wherein the feature pool is pre-built and contains a representative dataset of historical power load data.

[0061] S3. Using the extracted advanced features and retrieval results, the optimal expert network best suited to the characteristics of the power load is selected through the expert selection module;

[0062] S4. Using the extracted high-level features and retrieval results, the selected optimal expert network is used to accurately predict the load of the power system to assist in grid operation and dispatch decisions.

[0063] For example, historical load data for power systems can include hourly or 15-minute electricity consumption recorded over the past few years, covering electricity usage during different seasons, weekdays and weekends, holidays, and special events. High-level features reflecting electricity consumption patterns can include seasonal patterns (such as increased electricity consumption due to increased air conditioning use in summer), daily patterns (such as morning and evening peak hours on weekdays), and the impact of special events (such as changes in electricity consumption due to major sporting events or public holidays). High-level features can also include meteorological conditions such as temperature and humidity, as well as economic activity indicators. Features highly relevant to electricity load forecasting tasks can include various factors extracted from historical load data that significantly influence electricity consumption, such as historical electricity consumption for the same period, temperature change rates, and differences in electricity consumption between weekdays and weekends. These features help forecasting models more accurately predict future electricity load.

[0064] The construction of an expert network for power load forecasting is based on the analysis of historical power system load data to identify patterns and regularities in power consumption. This data may include power usage across different time periods (e.g., seasons, weekdays / weekends, holidays), regions, and economic activities. The expert network comprises multiple models, each trained and optimized for a specific power consumption pattern. For example, some experts may focus on analyzing the impact of temperature changes on power demand, while others may focus on predicting power load changes during special events or holidays. In this way, the constructed expert network can cover various factors that may affect power load and dynamically select the most suitable models for forecasting based on the characteristics of the input data.

[0065] Example 3

[0066] A traffic flow prediction method using any of the time-series data prediction methods in Embodiment 1 includes the following steps:

[0067] S1. Use a self-supervised pre-trained network to process historical traffic data and extract high-level features that reflect changes and patterns in traffic flow;

[0068] S2. Using the extracted high-level features reflecting traffic flow changes and patterns, the retrieval module retrieves features that are highly relevant to a specific traffic flow prediction task from the feature pool, wherein the feature pool is a pre-built dataset containing representative data of traffic flow data.

[0069] S3. Using the extracted high-level features and retrieval results, the optimal expert network best suited to traffic flow characteristics is selected through the expert selection module;

[0070] S4. Using the extracted high-level features and retrieval results, the selected optimal expert network is used to accurately predict traffic flow in a specific area or transportation network to support traffic management and planning decisions.

[0071] For example, historical traffic data can include vehicle flow counts, vehicle type distribution, average speed, and accident records from past instances at specific road sections or intersections. High-level features reflecting traffic flow changes and patterns can include traffic flow during morning and evening rush hours, holiday traffic patterns, the impact of special events (such as concerts or sporting events) on traffic flow, and the impact of weather conditions (such as rain, snow, and fog) on ​​traffic flow. Features highly correlated with a specific traffic flow prediction task can include historical traffic flow data from the same period, weather changes, large-scale event arrangements, and traffic pattern changes caused by road construction or accidents; these factors can all significantly impact traffic flow.

[0072] Traffic flow prediction expert networks are built upon the analysis of historical traffic data, which may include traffic volume counts, vehicle types, accident records, and weather conditions. The expert network comprises a series of specially designed expert models, each trained and optimized for specific traffic data and patterns. For example, some experts may focus on analyzing traffic flow during peak hours, while others may optimize for traffic patterns during special events or holidays. Furthermore, the expert network may include models capable of handling the impact of real-time traffic events such as accidents or road construction. By dynamically selecting the most suitable model based on the characteristics of the input data, the network can accurately predict traffic flow and adapt to constantly changing traffic conditions.

[0073] Example 4

[0074] An internet user traffic prediction method using any of the time-series data prediction methods in Embodiment 1 includes the following steps:

[0075] S1. Use a self-supervised pre-trained network to process historical traffic data of Internet users and extract high-level features that reflect user access patterns and traffic trends;

[0076] S2. Using the extracted high-level features that reflect user access patterns and traffic trends, the retrieval module retrieves features that are highly correlated with the Internet user traffic prediction task from the feature pool, wherein the feature pool is a pre-built dataset containing representative data of Internet user traffic data.

[0077] S3. Using the extracted advanced features and search results, the optimal expert network that best suits the Internet traffic pattern is selected through the expert selection module;

[0078] S4. Using the extracted high-level features and retrieval results, accurately predict internet user traffic through the selected optimal expert network to support network resource management and traffic control decisions.

[0079] For example, historical internet user traffic data can include user visits, page views, downloads, and uploads recorded at different times in the past. Advanced features reflecting user access patterns and traffic trends can include user activity time distribution, peak access periods, user behavior patterns (such as search keywords and click-through rates), and device type distribution (such as mobile phones, tablets, and computers). Features highly correlated with internet user traffic prediction tasks can include historical user visits for the same period, the impact of specific events (such as promotional activities or new product launches) on traffic, traffic changes caused by network maintenance or outages, and changes in user access patterns on holidays or special dates. These features help predictive models more accurately predict future internet user traffic.

[0080] Expert networks for internet user traffic prediction are built upon the analysis of historical user traffic data, which may include user visits, page views, downloads, and uploads. The expert network comprises multiple expert models, each trained and optimized for different types of user behavior and traffic patterns. For example, some experts may focus on analyzing user access patterns during specific time periods (such as weekdays, nighttimes, or weekends), while others may focus on predicting the impact of promotional activities or new product launches on traffic. Furthermore, the expert network can include models capable of analyzing the impact of different device types (such as mobile phones, tablets, and computers) on traffic. By dynamically selecting the most suitable models based on the characteristics of the input data, accurate internet user traffic predictions can be provided.

[0081] The following describes specific embodiments of the present invention.

[0082] Self-supervised pre-trained network

[0083] To retrieve similar samples, a pre-training task was designed to obtain representations with high-dimensional semantic information, such as... Figure 2 As shown. A masked autoencoder consists of an encoder and a decoder. The encoder encodes the input into high-dimensional hidden features, and the decoder decodes the hidden features to reconstruct the input, making the reconstructed data as similar as possible to the original input. A certain proportion of the input is randomly selected and its values ​​are set to 0 to perform the masking operation. Assume the input is S∈R. l*c The mask ratio is m, S gt ∈R ml*c As the truth value of the reconstructed part, the visible part S vis ∈R (1-m)l*cIt is used to learn contextual information and reconstruct missing information. The model is trained using MSE loss.

[0084] L = MSE(S recon S vis )

[0085] Where S recon It is the output of the decoder reconstruction, S vis It is the actual value of the visible portion of the data that is not masked.

[0086] The pre-trained model provides a high-dimensional representation with semantic information, which can be used to enhance retrieval in subsequent retrieval tasks.

[0087] Figure 3 The overall framework of the retrieval enhancement paradigm and hybrid expert prediction module is illustrated. First, a retrieval pool P is constructed based on the high-dimensional representation of the training set. For a target sample s... t First, a pre-trained Transformer is used to extract its feature embeddings, which are then used as a query to retrieve relevant samples from the feature pool. The aggregation layer performs a target-sample-aware, order-invariant aggregation operation on the retrieved relevant samples, aggregating not only the features but also the labels, which contain equally important information. The aggregated labels are input into the gated network GateNet of the prediction module to ensure accurate routing. The aggregated features and target features are then fed together into the expert module to generate more fine-grained analysis.

[0088] Search module

[0089] The retrieval module is responsible for retrieving data samples related to the target through nearest neighbor search. A retrieval pool is constructed using pre-trained high-dimensional representations from the training set. The original continuous labels are divided into β classes based on their numerical values ​​and stored in the retrieval pool along with the features. The discretization of continuous labels is based on the following considerations: (1) the labels used in the retrieval module are only used to estimate their feature patterns and determine the expert modules that need to be activated; precise numerical values ​​are not required. (2) the features and labels of neighboring samples are strong signals for predicting the target. Preferably, the labels are mapped to the same dimension as the features, and discretizing the labels is more conducive to the learning of the mapping layer.

[0090] For a prediction target s t The search engine retrieves similar samples from the feature pool based on the cosine similarity of the features, and inputs them into the aggregation layer for the following target-aware order-invariant aggregation operation:

[0091]

[0092] Where R(x) t) is for the target sample s t Relevant samples retrieved, x k It is sample s k The feature part, its weight α k The computation is performed using an attention mechanism, defined as follows:

[0093]

[0094] The label information of related samples is aggregated using the same attention weights:

[0095]

[0096] The obtained aggregated features T and aggregated labels l will be used in subsequent prediction modules to improve prediction performance.

[0097] Prediction module

[0098] like Figure 1 and Figure 3 As shown, the prediction module consists of the pre-trained network, the expert selection module, and the expert network. The prediction module works in conjunction with the retrieval module to achieve efficient prediction of time series data.

[0099] To capture diverse feature patterns, the prediction module incorporates a sparse hybrid expert system (SMoE) for differential modeling. The prediction module adopts a Smooth Hybrid Expert System (SMoE) architecture, which primarily consists of a set of parallel expert modules E1, E2, ..., E... N The system uses a gated network, GateNet. Expert modules are used to model different temporal feature patterns, and the gated network is used to determine the feature pattern to which a target sample belongs, i.e., which expert should process the sample. The system output is a weighted sum of the outputs of multiple expert modules, with weights G1(x), G2(x), ..., G... N (x) is determined by the gated network GateNet. The aggregated label embedding, a strong signal indicating the target sample pattern, is input into GateNet for adaptive and accurate routing. The aggregated feature embedding and the target feature embedding are input into the activated expert module for finer-grained analysis. Finally, the prediction head uses an MLP layer to output the final prediction result:

[0100]

[0101] in

[0102] Each activated expert module E k embedding of aggregated features and target feature embedding Perform analysis and generate output results. The gating network weights the output of each expert module. The gating network dynamically determines the feature patterns of the target samples obtained based on the aggregated label embedding. The weighted results are processed using a Softmax activation function to obtain the relative importance of each expert module's output. From the Softmax-processed weighted results, the top K expert module outputs with the highest scores are selected to concentrate the most effective prediction information. The outputs of the top K expert modules are then input into a Multilayer Perceptron (MLP) layer for further nonlinear transformation and integration. The MLP layer outputs the final prediction result, which integrates the prediction information from multiple expert modules, serving as the output of the entire time-series data prediction method.

[0103] Traditional time series forecasting models often model all data using a single model, which frequently fails to fully capture the temporal dimension and diversity of the data. Taking traffic flow forecasting as an example, heterogeneous data may originate from various sources, such as different traffic surveillance cameras, GPS tracking systems, and mobile applications. Heterogeneous data may include different types of data, such as numerical data (traffic flow counts), categorical data (vehicle types), temporal data (timestamps), and textual data (weather descriptions). The data structure may also differ; some data may be time series data, some event logs, and some images or video frames. Even within the same type of data, the characteristics may vary. For example, traffic flow at different times may be influenced by different factors, such as weekdays versus weekends, holidays, and special events.

[0104] The time series prediction method of this invention is based on self-supervised pre-training and hybrid expert model retrieval enhancement techniques. A core advantage lies in its ability to handle and model data with inherent diversity, thereby significantly improving prediction performance. First, the self-supervised pre-trained network can extract high-level features containing rich semantic information from various heterogeneous data. This capability allows the model to not only understand the surface features of the data but also to delve deeper into the semantic content behind the data. Second, this invention introduces hybrid expert models and retrieval enhancement techniques. The hybrid expert model allows the system to dynamically select the most suitable expert network based on the features of the input data. This allows for differentiated modeling strategies for different data patterns. The retrieval enhancement technique further enhances the model's ability to model diverse data by retrieving the most relevant data from a pre-built representative database. Furthermore, this invention specifically designs a self-supervised pre-training method, optimized specifically for the model structure to meet the needs of time series prediction.

[0105] This invention effectively overcomes the limitations of traditional time series forecasting models in fully capturing the temporal dimension and diversity of data by introducing and collaborating with self-supervised pre-training and a hybrid expert system. Based on the power, transportation, and internet forecasting applications proposed in this invention, the accuracy of time series forecasting can be significantly improved, yielding significant benefits. For example, more accurate power load forecasting can help achieve better power dispatching and reduce operating costs; more accurate traffic flow forecasting can optimize traffic management and planning.

[0106] In summary, this invention enhances the model's adaptability and flexibility to diverse and heterogeneous data, improves prediction accuracy, and has high practical value for time series prediction tasks.

[0107] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0108] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A power load forecasting method, employing a time-series data forecasting method based on a hybrid expert model and retrieval enhancement, characterized in that, Includes the following steps: S1. Use a self-supervised pre-trained network to process historical load data of the power system and extract high-level features that reflect power consumption patterns; S2. Using the extracted high-level features reflecting power consumption patterns, the retrieval module retrieves features that are highly correlated with the power load prediction task from the feature pool, wherein the feature pool is pre-built and contains a representative dataset of historical power load data. S3. Using the extracted advanced features and retrieval results, the optimal expert network best suited to the characteristics of the power load is selected through the expert selection module; S4. Using the extracted high-level features and retrieval results, the selected optimal expert network is used to accurately predict the load of the power system to assist in power grid operation and dispatch decisions; The retrieval module constructs a feature pool using high-dimensional representation information from the training set. Step S2 specifically includes: extracting feature embeddings from the target sample using a pre-trained encoder; using the extracted feature embeddings as queries to retrieve a set of samples highly correlated with the target sample from the feature pool; performing a target sample-aware order-invariant aggregation operation on the retrieved relevant sample set to generate aggregated feature embeddings and aggregated label embeddings. Step S3 specifically includes: inputting the aggregated label embeddings into the gating network of the expert module to adaptively determine precise routing information, dynamically deciding which expert modules should be activated based on the aggregated label embeddings, and the contribution of each module to the final prediction result. Step S4 specifically includes: inputting the aggregated feature embeddings and target feature embeddings into the activated expert modules for more fine-grained analysis.

2. A traffic flow prediction method, employing a time-series data prediction method based on a hybrid expert model and retrieval enhancement, characterized in that, Includes the following steps: S1. Use a self-supervised pre-trained network to process historical traffic data and extract high-level features that reflect changes and patterns in traffic flow; S2. Using the extracted high-level features reflecting traffic flow changes and patterns, the retrieval module retrieves features that are highly relevant to a specific traffic flow prediction task from the feature pool, wherein the feature pool is a pre-built dataset containing representative data of traffic flow data. S3. Using the extracted high-level features and retrieval results, the optimal expert network best suited to traffic flow characteristics is selected through the expert selection module; S4. Using the extracted high-level features and retrieval results, the selected optimal expert network is used to accurately predict traffic flow in a specific area or transportation network to support traffic management and planning decisions. The retrieval module constructs a feature pool using high-dimensional representation information from the training set. Step S2 specifically includes: extracting feature embeddings from the target sample using a pre-trained encoder; using the extracted feature embeddings as queries to retrieve a set of samples highly correlated with the target sample from the feature pool; performing a target sample-aware order-invariant aggregation operation on the retrieved relevant sample set to generate aggregated feature embeddings and aggregated label embeddings. Step S3 specifically includes: inputting the aggregated label embeddings into the gating network of the expert module to adaptively determine precise routing information, dynamically deciding which expert modules should be activated based on the aggregated label embeddings, and the contribution of each module to the final prediction result. Step S4 specifically includes: inputting the aggregated feature embeddings and target feature embeddings into the activated expert modules for more fine-grained analysis.

3. An internet user traffic prediction method, employing a time-series data prediction method based on a hybrid expert model and retrieval enhancement, characterized in that... Includes the following steps: S1. Use a self-supervised pre-trained network to process historical traffic data of Internet users and extract high-level features that reflect user access patterns and traffic trends; S2. Using the extracted high-level features that reflect user access patterns and traffic trends, the retrieval module retrieves features that are highly correlated with the Internet user traffic prediction task from the feature pool, wherein the feature pool is a pre-built dataset containing representative data sets of Internet user traffic data. S3. Using the extracted advanced features and search results, the optimal expert network best suited to the Internet traffic pattern is selected through the expert selection module; S4. Using the extracted high-level features and retrieval results, accurately predict internet user traffic through the selected optimal expert network to support network resource management and traffic control decisions; The retrieval module constructs a feature pool using high-dimensional representation information from the training set. Step S2 specifically includes: extracting feature embeddings from the target sample using a pre-trained encoder; using the extracted feature embeddings as queries to retrieve a set of samples highly correlated with the target sample from the feature pool; performing a target sample-aware order-invariant aggregation operation on the retrieved relevant sample set to generate aggregated feature embeddings and aggregated label embeddings. Step S3 specifically includes: inputting the aggregated label embeddings into the gating network of the expert module to adaptively determine precise routing information, dynamically deciding which expert modules should be activated based on the aggregated label embeddings, and the contribution of each module to the final prediction result. Step S4 specifically includes: inputting the aggregated feature embeddings and target feature embeddings into the activated expert modules for more fine-grained analysis.

4. The method according to any one of claims 1 to 3, characterized in that, The pre-trained network includes: An encoder is used to encode input time-series data into high-dimensional hidden layer features; The decoder is used to decode hidden layer features and reconstruct the input time series data. The masking module is used to randomly select a certain proportion of the input data for masking operations, generating the masked part and the visible part; The embedding layer is used to embed the visible part and the masked part after the masking operation respectively to obtain the embedding features for subsequent processing. The model is trained using the mean squared error (MSE) loss function, which is defined as follows: ； in It is the output of the decoder reconstruction. It is the actual value of the visible portion of the data that is not masked.

5. The method according to any one of claims 1 to 3, characterized in that, The pre-trained encoder is a transformer encoder.

6. The method according to any one of claims 1 to 3, characterized in that, Step S2 specifically includes: the retrieval module divides the original continuous labels into multiple classes based on numerical size to discretize them, and stores them together with the features in the feature pool. The discretized labels are used to estimate feature patterns, determine the activated expert modules, and learn the mapping layer. For the target sample, the search engine retrieves a set of similar samples in the feature pool based on the cosine similarity of the features.

7. The method as described in claim 6, characterized in that, In step S2, for a predicted target The search engine retrieves similar samples from the feature pool based on the cosine similarity of the features, and inputs them into the aggregation layer for the following object-aware, order-invariant aggregation operation: ； in, It is for the target sample Relevant samples retrieved, It is a sample The feature part, its weight The computation is performed using an attention mechanism, defined as follows: ； The label information of related samples is aggregated using the same attention weights: ； The obtained aggregated features 𝒓 and aggregated labels 𝒍 are used for subsequent predictions.

8. The method according to any one of claims 1 to 3, characterized in that, The Sparse Hybrid Expert System (SMoE) architecture is adopted, which includes a set of parallel expert modules, each of which is used to model different temporal feature patterns. The gating network determines the feature pattern to which the target sample belongs and decides which expert module should process the sample. The system outputs a weighted sum of the outputs of multiple expert modules, and the weights are adaptively determined by the gating network based on the aggregation label embedding. The prediction head adopts a multilayer perceptron (MLP) and combines the output of the weighted sum to generate the final prediction result.

9. The method as described in claim 8, characterized in that, The prediction results are calculated using the following formula: ； in, ; Among them, each activated expert module E k embedding of aggregated features and target feature embedding Perform analysis and generate output results. The gating network weights the output of each expert module, with the weights... The gating network dynamically determines the feature patterns of the target samples based on the aggregated label embedding. Specifically, the Softmax activation function is used to process the weighted results to obtain the relative importance of each expert module output; the K highest-scoring expert module outputs (TopK) are selected from the Softmax-processed weighted results to concentrate the most effective prediction information. The outputs of the K highest-scoring expert modules are input into the multilayer perceptron (MLP) layer for further nonlinear transformation and integration. The MLP layer outputs the final prediction result that integrates the prediction information of multiple expert modules, which serves as the output of the entire time series data prediction method.

Images