Multi-scale time series prediction method based on adaptive sparse expert selection strategy and closed continuous time neural network

Abstract

The invention discloses a multi-scale time sequence prediction method based on an adaptive sparse expert selection strategy and a closed continuous time neural network. The method comprises the following steps: carrying out normalization and low-dimensional feature mapping based on RevIN; performing trend-seasonal structure enhancement processing on the feature sequence after linear mapping; constructing a multi-scale expert model based on the feature sequence after trend-season enhancement; self-adaptive sparse expert selection and load balancing loss calculation are carried out; carrying out weighted aggregation and residual fusion on multi-scale expert output; global modeling of a closed continuous time neural network based on channel weighting is carried out; and finally performing prediction generation and reverse normalization. The multi-scale time series prediction method has the multi-time-scale adaptive modeling capability, the sparse expert efficient selection mechanism and the global continuous time modeling capability, and can be applied to various multivariable time series prediction scenes such as power load prediction, weather prediction, industrial production monitoring, traffic flow prediction and financial price prediction.

Description

Technical Field

[0001] This invention relates to the field of time series forecasting and intelligent information processing technology, specifically to a multi-scale time series forecasting method based on an adaptive sparse expert selection strategy and a closed continuous-time neural network, which can be applied to various multivariate time series forecasting scenarios such as power load forecasting, weather forecasting, industrial production monitoring, traffic flow forecasting, and financial quantity and price forecasting. Background Technology

[0002] Multivariate time series forecasting (MTSF) aims to uncover potential patterns of change from historical observations of multiple related variables to achieve accurate estimates of future values. Currently, MTSF is widely used in key areas such as traffic scheduling, financial analysis, and medical monitoring, providing crucial technical support for enterprise production scheduling optimization and government macroeconomic decision-making. Multivariate time series (MTS) typically exhibit significant volatility and multi-period structures. Their dynamic evolution is usually driven by both short-term disturbances and long-term regularities, displaying characteristics of nonlinear superposition and intertwined coupling. This complex dynamic pattern significantly increases the difficulty of MTSF.

[0003] Current mainstream methods mostly build time series prediction models based on a single time scale, attempting to enhance the expressive power of time series dependencies by introducing complex network structures. However, such methods are usually accompanied by high computational complexity and are more prone to overfitting, limiting their generalization performance in practical applications. Furthermore, single-time-scale modeling strategies struggle to fully capture the temporal characteristics inherent in multi-period structures, weakening the model's adaptability in complex dynamic systems. While some studies have attempted to employ multi-scale modeling strategies to capture temporal features under different periodic patterns, these methods typically rely on fixed combinations of time scales and lack the ability to adaptively model based on the dynamic changes in the data itself. Therefore, constructing an efficient MTSF model that can effectively capture multi-period structures remains a core challenge in current research.

[0004] To effectively address the aforementioned challenges, an ideal MTSF model should possess the following two key capabilities: First, it should have adaptive time-scale modeling capabilities, able to analyze and characterize intertwined and coupled temporal patterns at appropriate time scales based on the dynamic characteristics of the input sequence, thereby enhancing the model's ability to model complex temporal structures. Second, it should have an effective feature extraction mechanism, capable of mining key temporal information from short-term non-stationary fluctuations and long-term periodic changes to comprehensively depict the changing patterns of MTS.

[0005] The Sparse Mixture of Experts (MoE) architecture significantly reduces computational costs while maintaining model expressiveness by dynamically selecting sub-networks to participate in computation based on the features of the input samples. This structural design makes MoE well-suited for handling complex and variable input data. However, when applied to the MTSF task, the MoE architecture faces additional challenges. Time series data inherently possess temporal dependencies and contextual continuity, but the dynamic routing mechanism in MoE means that each expert only accesses a portion of the training samples, resulting in learned temporal patterns that often lack continuity and struggle to capture long-term dependencies.

[0006] Closed-Form Continuous Networks (CFCs), proposed in recent years, offer a novel modeling paradigm. CFCs model the dynamic evolution of neuron states in the continuous time domain using analytical forms, avoiding reliance on numerical differential solvers and possessing efficient and stable modeling capabilities. They exhibit good temporal consistency and contextual coherence when handling non-uniform sampling and fragmented input sequences. This characteristic makes CFCs promising to alleviate the problems of discontinuous temporal representations and missing contextual information in individual expert models within the MoE architecture, thereby enabling cross-expert collaborative modeling and enhancing the modeling ability for long-term dependencies.

[0007] Current technologies lack adaptive time-scale modeling capabilities, failing to automatically select appropriate expert paths at different time scales based on the dynamic characteristics of input data. Sparse routing leads to discontinuous time representations, and there is a lack of mechanisms for globally consistent modeling of time series across experts. Furthermore, the lack of collaborative modeling between local features and global time structure makes it difficult for models to simultaneously account for short-term local fluctuations and long-term dependency structures. Therefore, designing a multivariate time series forecasting method that combines multi-time-scale adaptive modeling capabilities, an efficient sparse expert selection mechanism, and global continuous time modeling capabilities has become a pressing technical problem in this field. Summary of the Invention

[0008] This invention aims to overcome the problems of existing multivariate time series prediction methods, such as the single processing time scale, the impact of sparse expert routing mechanism on the continuity of time series modeling, and the difficulty in unifying the modeling of local and global features. It provides a multi-scale time series prediction method based on an adaptive sparse expert selection strategy and a closed continuous-time neural network to achieve efficient and stable time series prediction.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A multi-scale time series prediction method based on an adaptive sparse expert selection strategy and a closed continuous-time neural network includes the following steps:

[0011] S1. RevIN Normalization and Low-Dimensional Feature Mapping: The original multivariate time series is reversibly normalized to obtain the normalized input sequence. Linear projection is used to map all channels in the input sequence to a uniform feature dimension, resulting in low-dimensional features after feature compression. ;

[0012] S2. Multi-scale Trend-Seasonal Structure Enhancement: First, the trend term of the input sequence is extracted using a multi-scale smoothing operator. Then, the seasonal term of the input sequence is extracted from the frequency domain based on a combination of Fourier basis functions. Finally, the trend term and seasonal term are fused with the input sequence in a residual manner to obtain the enhanced feature sequence. ;

[0013] S3. Local temporal feature modeling of multi-scale expert models: According to multiple preset slice sizes, the feature sequence is... The dataset is divided into multiple time slices of varying lengths along the time dimension, with each expert model corresponding to data at a specific scale. Each expert model employs a depthwise separable convolutional structure to perform local temporal modeling within its assigned patch and extract local dynamic features at different scales. Each expert model outputs a local feature representation specific to its chosen time scale. ;

[0014] S4. Adaptive Sparse Expert Selection and Load Balancing Loss Calculation: An adaptive sparse routing module is used with low-dimensional features. The scores and uncertainty estimates of each expert model are calculated as the criteria for discrimination. During the training phase, Gaussian noise based on learnable variance is injected into the expert model scores to enhance the exploratory nature. Sparse gating weights are obtained by combining Softmax and Top-K selection strategies to determine the activated expert models. The load balancing loss is further calculated based on the distribution of gating weights and the frequency of expert model usage to ensure the balance of expert activation during training.

[0015] S5. Weighted aggregation and residual fusion of multi-scale expert outputs: First, represent the local features of all expert models. Projecting the data onto a consistent feature space, and then performing a weighted summation according to the gated weights, yields the fused multi-scale comprehensive features. Finally, the comprehensive features are combined with the enhanced feature sequence. Perform residual connections to obtain the output features of the expert model. ;

[0016] S6. Global Modeling of Closed Continuous-Time Neural Networks Based on Channel Weighting: A channel feature weighting mechanism is introduced based on the output features of the expert model to obtain the feature representation after channel recalibration. Subsequently, this feature is input into a closed-loop continuous-time neural network. Utilizing an analytical structure consisting of a backbone network and two head networks, the evolution of the feature over time is modeled explicitly in a closed-loop manner in the continuous-time domain. Finally, a globally enhanced feature is output. ;

[0017] S7. Prediction Generation and Inverse Normalization: First, enhance global features. Output features of expert models The latent space representation is then fused to obtain the final latent space representation; the output projection layer is then mapped to the prediction time step, and a reversible normalization operation is performed to obtain the final prediction result.

[0018] Further, in step S2, low-frequency trend components of the input sequence are extracted using a multi-scale smoothing operator. This multi-scale smoothing operator consists of multiple sliding convolution kernels of different lengths, each of which can generate a trend candidate sequence at a different smoothing scale, thereby capturing the slow-changing structure of the input sequence at multiple time granularities. The specific steps are as follows:

[0019] First, let the preset set of convolution kernels be... They correspond to trend candidates respectively. The input sequence is then projected using a learnable linear mapping, and a fusion coefficient that dynamically changes over time is generated using the Softmax function. This allows the contribution ratio of each trend candidate to the final trend term to be adaptively adjusted; where the expression for the final trend term T is: (1).

[0020] Further, in step S2, the specific steps for extracting the seasonal term based on the Fourier basis function are as follows: first, perform a real-number fast Fourier transform on the input sequence in the time dimension to obtain a frequency domain representation, where each frequency component contains the amplitude and phase information of the corresponding periodic pattern; then, select the component with the largest amplitude from the entire frequency band. The main frequencies are identified, and three sets of basic Fourier components are generated accordingly. These components are then used to reconstruct the seasonality term in the time domain. Its expression is: (2);

[0021] In equation (2), Indicates the first The frequency corresponding to each dominant frequency component. and These represent the amplitude and phase parameters, respectively. Indicates a time index.

[0022] Furthermore, in step S2, after the trend and seasonality terms are extracted, the trend and seasonality terms are residually concatenated with the input sequence to form a structurally enhanced feature sequence. : (3).

[0023] Furthermore, in step S3, a multi-scale expert model is used to extract local temporal features at multiple temporal resolutions, including:

[0024] S3.1, Temporal patch partitioning: Enhanced feature sequence After entering the expert model, the original sequence is divided according to the patch time granularity corresponding to the expert model; where, let the size of the current patch be S and the sequence length be H, then the set of partitioned patches is... ;

[0025] S3.2, Depthwise Convolution: Within each patch, depthwise convolution is used to independently perform one-dimensional convolution operations on each channel to capture local temporal patterns within the channel; where, for the , The first layer Input features of each patch The depthwise convolution process is shown in equation (4): (4);

[0026] In equation (4), Indicates the first The first in the layer Input features of each patch This represents a depthwise convolution operation, with a corresponding kernel size of . , It is the GELU activation function, and BN represents the batch normalization operation;

[0027] S3.3, Pointwise Convolution: using a 1×1... The convolution process involves performing pointwise convolution on the output of all channels, as shown in equation (5): (5);

[0028] S3.4. Stack the output of step S3.3 in multiple layers and perform residual connection.

[0029] Further, in step S4, the adaptive sparse routing module consists of a gated expert selection strategy and a load balancing loss. The gated expert selection strategy consists of two sets of parallel parameterized networks, which are used to estimate the score and uncertainty of each sample on different expert models, thereby determining the expert path that should be activated for the current input. The load balancing loss is used to prevent some experts from being inactive for a long time, which would limit modeling ability and reduce generalization performance. It includes:

[0030] The low-dimensional features obtained in step S1 after feature compression Based on this, the gating expert selection strategy generates expert scoring vectors through two fully connected networks. and noise intensity estimation ,in For a learnable parameter matrix, the Softplus activation function is used to ensure that the estimated noise intensity is non-negative; and Gaussian noise is introduced into the expert model score L. The expert model score with noise was obtained. As shown in equation (6): (6);

[0031] Subsequently, the expert model was scored. The weights for each expert model are obtained by applying Softmax normalization. As shown in equation (7): (7);

[0032] By retaining the top M largest weights and resetting the remaining weights to 0, we obtain sparse gating weights. This serves as the basis for expert pathways.

[0033] Furthermore, in step S4, a load balancing regularization term is introduced. As an auxiliary loss, the load balancing regularization term is based on the sum of squared gated scores. Coefficient of variation of actual load squared The construction is as shown in equation (8): (8);

[0034] Finally, based on the sparse gating weight scores, the input features are distributed to the corresponding expert models to achieve sparse and efficient multi-path modeling.

[0035] Furthermore, the specific steps of step S5 are as follows:

[0036] S5.1 First, introduce the dimension transformation function. The outputs of each expert model are processed to obtain a feature representation with a unified dimension.

[0037] S5.2 Introducing the GESS module to calculate the weights of the expert model. The expert output is weighted and aggregated; where an indicator function is defined. ,when When the i-th expert model is activated, then set Conversely, it indicates that it is not activated. (Settings) The fused feature representation is shown in equation (9): (9);

[0038] In equation (9), and Let represent the output and transformation function of the i-th expert model, respectively;

[0039] S5.3, Integrate features With enhanced feature sequences The final fusion output of the expert models is obtained by adding them together using residual connections. As shown in equation (10): (10).

[0040] Furthermore, in step S6, the closed-loop continuous-time neural network enhancement module includes a channel weighting unit and a CFC module; the channel weighting unit introduces a set of learnable channel weights. ,right The importance of each channel is modeled to guide the model to focus on feature regions with high information density and suppress redundancy or noise interference. The weighting process can be expressed as shown in Equation (11): (11);

[0041] The network structure of the CFC module includes a backbone network. Head network as well as Gating mechanism The working mechanism of the CFC module is shown in equation (12): (12);

[0042] in, This represents the state representation generated by the backbone network at time step t; Hyperparameters of a network; backbone network The output determines the activation level of the Sigmoid gate, while the head network... and The output of the network is then gradually integrated with the output of the backbone network, and selective weighting is performed using the Sigmoid function to enhance the network's nonlinear expressive power and long-term time-series dependency modeling ability; the CFC module is used to process the complete time series. Global augmentation features of CFC are obtained after modeling. As shown in equation (13): (13).

[0043] Furthermore, the specific steps of step S7 are as follows:

[0044] S7.1 First, apply the global enhancement features obtained in step S6. Output features of the expert model in step S5 The residuals are summed to obtain a unified latent space representation that includes local multi-scale structure and global continuous dynamic information, as shown in Equation (14): (14);

[0045] S7.2 Then, according to the prediction layer structure, the unified latent space representation is rearranged and flattened in the time dimension and feature dimension, and mapped to the future prediction step size through a linear projection layer to obtain the intermediate prediction result as shown in Equation (15). (15);

[0046] S7.3 RevIN Inverse Normalization Output Final Prediction Result: Performing inverse normalization restores the model output to the true dimensions of the original data, yielding the future... Step-by-step multivariate time series forecast values The expression is as follows: (16).

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention discloses a multi-scale time series prediction method based on an adaptive sparse expert selection strategy and a closed continuous-time neural network, comprising the following steps: RevIN-based normalization and low-dimensional feature mapping; performing trend-seasonal structure enhancement processing on the linearly mapped feature sequence; constructing a multi-scale expert model based on the trend-seasonal enhanced feature sequence; adaptive sparse expert selection and load balancing loss calculation; weighted aggregation and residual fusion of multi-scale expert outputs; global modeling based on a channel-weighted closed continuous-time neural network; and final prediction generation and denormalization. This invention's multi-scale time series prediction method combines multi-timescale adaptive modeling capabilities, an efficient sparse expert selection mechanism, and global continuous-time modeling capabilities, enabling efficient and stable time series prediction.

[0049] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0050] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0051] Figure 1 This is a flowchart illustrating a multi-scale time series prediction method based on an adaptive sparse expert selection strategy and a closed continuous-time neural network according to the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, it should be noted that these embodiments are not intended to limit the present invention. Equivalent transformations or substitutions in function, method, or structure made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0053] Please see Figure 1 This embodiment provides a multi-scale time series prediction method based on an adaptive sparse expert selection strategy and a closed continuous-time neural network, including the following steps:

[0054] S1. RevIN-based normalization and low-dimensional feature mapping. The original multivariate time series is normalized channel-by-channel using Reversible Normalization (RevIN) to eliminate dimensional differences and scale biases between different variables, resulting in a normalized input sequence. Then the input sequence By employing linear projection, each channel is mapped to a unified feature dimension, resulting in a low-dimensional representation after feature compression. This low-dimensional feature It is used as the discriminant input of the adaptive sparse routing module to generate activation scores for each expert, calculate expert uncertainty, and determine the Top-K sparse selection path, thereby scheduling the subsequent multi-scale expert model operation.

[0055] S2. Multi-scale Trend-Seasonal Structure Enhancement. To improve the model's ability to represent long-term trends and periodic structures, trend-seasonal structure enhancement is performed on the linearly mapped feature sequence before it enters the multi-scale expert model. First, a multi-scale smoothing operator composed of multiple sets of convolutional kernels with different sizes is used to perform a multi-scale moving average on the input sequence to extract low-frequency trend components at different time scales. Then, based on a learnable combination of Fourier basis functions, the main frequency components with larger amplitudes are selected from the frequency domain to obtain the recurring periodic patterns of the time-series data. Finally, the trend term, seasonal term, and input sequence are fused with residuals to obtain the enhanced feature sequence. , as input to the multi-scale expert model.

[0056] The time series augmentation method employed in this step mainly comprises three parts: trend term extraction, seasonality term extraction, and structured sequence reconstruction. This augmentation method performs explicit structural decomposition on the original multivariate time series, making the periodic behavior and long-term transformation trends in the input sequence explicit, thereby improving the stability of subsequent adaptive sparse routing and multi-scale expert modeling. The specific process is as follows:

[0057] 1. Trend Item Extraction

[0058] For input multivariate time series This invention extracts low-frequency trend components from time series data using a multi-scale convolutional smoothing operator. This operator consists of multiple sliding convolutional kernels of varying lengths, each generating a trend candidate sequence at a different smoothing scale, thereby capturing the slow-changing structure of time series data across multiple temporal granularities. Let the preset set of convolutional kernels be... , respectively corresponding to trend candidates To avoid scale bias caused by manually fixed weights, this invention further projects the input sequence using a learnable linear mapping and generates dynamically changing fusion coefficients over time using the Softmax function. This allows the contribution ratio of each trend candidate in the final trend term to be adaptively adjusted. The expression for the final trend term T is shown in Equation (1). This trend extraction mechanism, through the combination of multi-scale smoothing and learnable weighting, can effectively suppress high-frequency noise, improve the recognizability of long-term structural patterns in the sequence, and provide a more stable input basis for subsequent multi-scale expert models. (1).

[0059] 2. Extraction of seasonal items

[0060] To characterize periodic patterns in time series, this invention employs a seasonal modeling method based on Fourier basis functions to extract the main periodic components of the sequence from a frequency domain perspective. First, the input sequence... A real-number fast Fourier transform is performed in the time dimension to obtain a frequency domain representation, where each frequency component contains amplitude and phase information of the corresponding periodic mode. To avoid noise amplification caused by all frequencies participating in the reconstruction simultaneously, this invention selects the three main frequencies (dominant frequency components) with the largest amplitudes from the entire frequency band and generates three sets of basic Fourier components accordingly. These are then used to reconstruct the seasonality term in the time domain. Its mathematical form is: (2);

[0061] In equation (2), Indicates the first The frequency corresponding to each dominant frequency component. and These represent the amplitude and phase parameters, respectively. Indicates a time index.

[0062] The seasonality term based on Fourier construction can effectively describe the periodic repetition structure, rhythmic pattern and long-term cyclical behavior in the sequence, thereby improving the model's ability to characterize periodic components and complementing the trend term, providing a structured periodic prior for subsequent multi-scale modeling.

[0063] 3. Sequence Structure Reconstruction

[0064] After the trend and seasonality terms are extracted, they are residually joined with the original sequence to form a structure-enhanced time series input. As shown in Equation (3). This enhancement process retains all the information of the original sequence and explicitly introduces periodic and trend structures to improve the richness of time series information and facilitate subsequent feature extraction. (3).

[0065] S3. Local temporal feature modeling using multi-scale expert models. Based on trend-seasonally enhanced feature sequences. A multi-scale expert model (ASRM) is constructed to achieve multi-timescale modeling of local temporal structures. The feature sequences are processed according to multiple preset patch sizes. The dataset is divided into time slices of varying lengths, with each expert model corresponding to a specific scale of data. Each expert employs a depthwise separable convolutional structure to perform local temporal modeling within its assigned patch, extracting local dynamic features at different scales layer by layer through multi-layer convolutions, non-linear activations, and residual connections. Ultimately, each expert outputs a local feature representation tailored to its specific time scale. This serves as the foundation for subsequent sparse expert selection and fusion. In this step, a multi-scale expert model is used to extract local temporal features at multiple temporal resolutions. Specifically, this includes:

[0066] 1. Time patch division

[0067] Enhanced feature sequence After entering the expert model, the original sequence is divided according to the patch time granularity corresponding to that expert. Let the size of the current patch be S, and the sequence length be H, then the set of partitioned patches is... .

[0068] 2. Depthwise convolution (the first stage of depthwise separable convolution)

[0069] Within each patch, depthwise convolution is used to independently perform one-dimensional convolution operations on each channel to capture local temporal patterns within the channel. For the ... The first layer Input features of each patch The depthwise convolution process is shown in equation (4). Indicates the first The first in the layer Input features of each patch This represents a depthwise convolution operation, with a corresponding kernel size of . , It is the GELU activation function, and BN represents the batch normalization operation. (4);

[0070] 3. Pointwise convolution (the second stage of depthwise separable convolution)

[0071] Through 1×1 Convolution is performed pointwise on the output of all channels to fuse the feature information of different channels, thereby enhancing the feature expression capability based on patch, as shown in Equation (5). (5).

[0072] 4. Multi-layer stacking and residual connections

[0073] By stacking multiple depthwise separable convolutional layers, the model can progressively improve its temporal modeling capabilities both within and across patches. Furthermore, residual connections are introduced to maintain effective information flow between layers, mitigating the vanishing gradient and feature decay issues in deep structures. The expert model employs a dual-predictor mechanism, addressing linear and nonlinear dynamic modeling tasks respectively. This structure helps capture temporal patterns at multiple levels, from fine-grained to coarse-grained, significantly enhancing the model's ability to express complex dynamic structures.

[0074] S4 Adaptive Sparse Expert Selection and Load Balancing Loss Calculation. To achieve dynamic path scheduling between expert models at different time scales, this invention employs an adaptive sparse routing module, using the low-dimensional features obtained in step S1. The scores and uncertainty estimates of each expert model are calculated as the criteria for judgment. During the training phase, Gaussian noise based on learnable variance is injected into the expert model scores to enhance exploratory power, and sparse gating weights are obtained by combining Softmax and Top-K selection strategies, thereby determining the activated expert models. The load balancing loss is further calculated based on the gating weight distribution and expert usage frequency to ensure the balance of expert model activation during training and improve system stability. In this step, the adaptive sparse routing module consists of a gated expert selection strategy (GESS) and a load balancing loss. Its purpose is to adaptively select a few of the most suitable expert models for computation based on the temporal characteristics of the input sequence, and to ensure that all expert models are used relatively evenly during training.

[0075] 1. Gating Expert Selection Strategy (GESS)

[0076] GESS consists of two parallel parameterized networks, one for estimating the score and uncertainty of each sample on different expert models, thereby determining the expert model path to be activated for the current input. The low-dimensional features obtained in step S1 are then compressed. Based on this, the gating expert selection strategy generates expert model scoring vectors through two fully connected networks. and noise intensity estimation ,in For a learnable parameter matrix, the Softplus activation function is used to ensure that the estimated noise intensity is non-negative; and Gaussian noise is introduced into the expert score L. The expert model score with noise was obtained. As shown in equation (6): (6);

[0077] Subsequently, the expert model was scored. The weights for each expert model are obtained by applying Softmax normalization. As shown in equation (7): (7);

[0078] By retaining the top M largest weights and resetting the remaining weights to 0, we obtain sparse gating weights. This serves as the basis for the expert model path.

[0079] 2. Load balancing losses

[0080] To alleviate the model's excessive reliance on a few expert model paths during training and prevent other expert models from remaining inactive for extended periods, thus limiting modeling ability and reducing generalization performance, a load balancing regularization term is introduced. As an auxiliary loss, the load balancing regularization term is based on the gating score sum of squares. Coefficient of variation of actual load squared The construction is as shown in equation (8): (8);

[0081] Finally, based on the sparse gating weight scores, the input features are distributed to the corresponding expert models to achieve sparse and efficient multi-path modeling.

[0082] S5. Weighted aggregation and residual fusion of multi-scale expert outputs. Based on the sparse gating weights generated by the adaptive sparse routing module, the local feature representations of the expert model outputs at each scale are processed. A unified mapping and weighted fusion are performed. First, the feature representations of all experts are projected onto a consistent feature space. Then, a weighted summation is performed according to the gate weights to obtain the fused multi-scale comprehensive features. Finally, this comprehensive feature is combined with the enhanced feature sequence obtained in step S2. Perform residual connections to obtain the output features of the expert model. This feature simultaneously contains information from the fusion of local fine-grained structure and multi-scale features.

[0083] Step S5, Output Aggregation Mechanism (OAM). Since each expert model is built on different time scales, their output features exhibit inconsistencies in dimensionality. To achieve effective fusion of expert model output features, OAM first introduces a dimensionality transformation function. The outputs of each expert model are processed to obtain a feature representation with a unified dimension.

[0084] S5.2 Introducing the GESS module to calculate the weights of the expert model. The expert output is weighted and aggregated; where an indicator function is defined. ,when When the i-th expert model is activated, then set Conversely, it indicates that it is not activated. (Settings) The fused feature representation is shown in equation (9): (9);

[0085] In equation (9), and Let represent the output and transformation function of the i-th expert model, respectively;

[0086] S5.3, Integrate features With enhanced feature sequences The final fused output features of the expert models are obtained by adding them together using residual connections. As shown in equation (10): (10).

[0087] S6. Global modeling based on channel-weighted closed continuous-time neural networks. In the output features of the expert model... Based on this, a channel feature weighting mechanism is introduced to obtain the feature representation after channel recalibration. Subsequently, this feature is input into a closed continuous-time neural network (CFC). Utilizing an analytical structure consisting of a backbone network and two head networks, the evolution of the feature over time is modeled explicitly in a closed form over the continuous-time domain. The final output is an enhanced representation containing global dynamic features. , which serves as the input to the prediction module.

[0088] Step S6: Channel-weighted Closed Continuous-Time Neural Network Enhancement Module (CWF-CFC). To alleviate the modeling discontinuities caused by sparse expert models and improve the model's ability to model global temporal structure and cross-temporal dependencies, a channel-weighted closed continuous-time neural network enhancement module is introduced based on the expert model output. This module consists of two key parts: a channel-weighted unit and a CFC module.

[0089] 1. Channel weighting

[0090] To enhance the model's ability to model global time series characteristics, this paper proposes a channel feature-weighted CFC enhancement module. This module emphasizes the contribution of key features in time series modeling through an explicit weighting mechanism along the channel dimension. Simultaneously, it leverages the advantages of CfC in continuous-time modeling to improve the model's perception of the overall time series. Specifically, this is achieved by introducing a set of learnable channel weights. ,right The importance of each channel is modeled to guide the model to focus on feature regions with high information density and suppress redundancy or noise interference. The weighting process can be expressed as shown in Equation (11). Weighted feature representation It will be passed as input to the CFC module to enhance the ability to perceive the global temporal structure within the continuous-time modeling framework. (11).

[0091] 2. Modeling of Closed-Form Continuous-Time Neural Networks (CFC)

[0092] The CFC network structure mainly consists of three parts: the backbone network. Head network as well as Gating mechanism The working mechanism of the CFC module is shown in equation (12): where, This represents the state representation generated by the backbone network at time step t; Hyperparameters of a network; backbone network The output determines the activation level of the Sigmoid gate, while the head network... and The output of the network is then gradually integrated with the output of the backbone network, and selective weighting is performed using the Sigmoid function to enhance the network's nonlinear expressive power and long-term time-series dependency modeling ability; the CFC module is used to process the complete time series. The output of CFC is obtained after modeling. As shown in Equation (13), this continuous-time modeling mechanism replaces the traditional recursive model with an analytical expression, effectively avoiding the gradient vanishing problem and significantly improving the model's ability to model long-term dependencies and capture global dynamics. (12); (13).

[0093] S7. Final prediction generation and inverse normalization. The global enhanced features obtained in step S6... Fusion features with the expert model in step S5 Feature fusion is performed using residuals to obtain the final latent space representation. Then, a linear mapping is applied to the output projection layer based on the prediction length to generate the predicted sequence for future time steps. Finally, the inverse normalization operation of RevIN is used to restore the predicted values ​​to their original dimensions, yielding the final multivariate time series prediction result. Specifically:

[0094] 1. Integrate expert model output and CFC output

[0095] First, the global augmented features and the expert model output are summed using residuals to obtain a unified latent space representation that includes local multi-scale structure and global continuous dynamic information, as shown in Equation (14). Then, based on the prediction layer structure, the... The time and feature dimensions are rearranged and flattened, and mapped to the future prediction step size through a linear projection layer to obtain the intermediate prediction result as shown in Equation (15). The parameters of the projection layer can be learned to realize the linear mapping from the fusion latent space to the target prediction sequence. (14); (15);

[0096] 2. RevIN inverse normalization outputs the final prediction result.

[0097] Performing inverse normalization restores the model output to the true dimensions of the original data, as shown in the following expression: (16);

[0098] Ultimately, we will obtain the future. Step-by-step multivariate time series forecast values This enables the joint prediction effect of the multi-scale expert model and the continuous-time modeling module.

[0099] Example 1

[0100] To verify the applicability of the method of this invention in multivariate time series prediction tasks, the following experiments were conducted. This embodiment describes the dataset used in the experiments, the evaluation metrics, the comparison methods, the parameter configurations, and the experimental results.

[0101] 1. Experimental Dataset

[0102] The experiment selected representative multivariate time series datasets, including typical data from the power sector, meteorological monitoring, medical statistics, and renewable energy forecasting. Each dataset consists of multiple channels with sampling periods ranging from minutes to hours, exhibiting varying degrees of trend, seasonality, and noise interference, thus covering a variety of practical application scenarios. All datasets were divided into training, validation, and test sets in a 6:1:1 ratio. The training set was used for model parameter learning, the validation set for hyperparameter selection and early stopping detection, and the test set for performance evaluation.

[0103] 2. Average Indicators

[0104] To objectively measure the error performance of the method of this invention under different prediction scenarios, the following two commonly used regression error indicators are adopted: mean squared error (MSE): used to measure the average of the squared prediction deviations; mean absolute error (MAE): used to measure the average of the absolute values ​​of prediction deviations.

[0105] 3. Comparison Methods

[0106] To evaluate the performance of the method of this invention under different prediction tasks, several representative time series prediction methods in the current research field were selected as comparison objects. The selected comparison methods include linear model-based methods such as DLinear, LightTS, and TiDE, and attention mechanism-based methods such as Autoformer, Fedformer, iTransformer, Pyraformer, and PatchTST. All comparison methods used publicly available implementations or publicly recommended parameter configurations, and were tested under the same dataset and training conditions to ensure the comparability of the experiments.

[0107] 4. Experimental Parameter Settings

[0108] All experiments were conducted using the PyTorch deep learning framework, with an NVIDIA RTX 5090 (32GB GPU) as the training hardware environment. The training process employed the following uniform settings: the optimizer was Adam; the loss function was L1 loss; the learning rate was set to 0.0005; the number of training epochs was 30, with early stopping enabled; and a uniform batch size, input length, and prediction length were used. Furthermore, the key parameters in this invention were set as follows: the number of experts was 4; the patch size was selected from the set {2, 4, 6, 8, 12, 16, 24, 32}; both the channel feature weighting module and the CFC module used default configurations; and the K value of the sparse route in ASRM was dynamically determined by the input features.

[0109] 5. Presentation of Main Experimental Results

[0110] The method of this invention was systematically evaluated against eight comparative methods on ten datasets and multiple prediction length settings, including prediction accuracy and computational efficiency. The prediction accuracy of each task is shown in MSE and MAE. The computational efficiency was evaluated using the torchinfo.summary tool to output model structure information, including the number of model parameters, GPU memory consumption, and inference time on the test set. The results are shown in Tables 1 and 2.

[0111] Table 1. Accuracy Comparison Results

[0112] As shown in Table 1, PatchMoE exhibits the best overall predictive performance. Specifically, in all 80 experiments, PatchMoE-CFC achieved the best results in 75 of them, and ranked second in 4 others. Compared with the baseline model, PatchMoE-CFC demonstrates a significant performance advantage. Compared to the best-performing baseline model, iTransformer, PatchMoE-CFC reduced the average MSE and MAE by 12.1% and 6.6% respectively across 10 datasets; compared to PatchTST, which is also based on patch modeling, PatchMoE-CFC reduced the average MSE and MAE by 16.4% and 8.7% respectively; and compared to DLinear, the best-performing linear baseline model, PatchMoE-CFC reduced the average MSE and MAE by 22.9% and 11.9% respectively. These experimental results fully validate the robustness and broad adaptability of PatchMoE-CFC across different prediction time spans, demonstrating its superior performance and generalization ability in multivariate time series prediction tasks.

[0113] To comprehensively evaluate the computational resource consumption of each model across different prediction time spans, experimental records showing memory overflow under default configurations were retained, and cases of insufficient memory were marked with the symbol "\". The relevant experimental results are shown in Table 2.

[0114] The results in the table show that PatchMoE-CFC exhibits moderate computational efficiency among the benchmark models. Specifically, under the four prediction step sizes, its average ranking in terms of model parameter count, testing time overhead, and GPU memory consumption is 3rd, 4th, and 3rd, respectively. Although it is less computationally efficient than linear models such as DLinear, LightTS, and TiDE, PatchMoE-CFC outperforms most complex attention-based models in terms of efficiency. Overall, PatchMoE-CFC achieves significantly better prediction accuracy than existing methods by sacrificing some computational overhead, demonstrating a superior balance between performance and efficiency.

[0115] Table 2. Computational Efficiency Analysis

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-scale time series forecasting method based on adaptive sparse expert selection policy and closed-form continuous-time neural network, characterized in that, The method comprises the following steps: S1, RevIN normalization and low-dimensional feature mapping: reversible normalization is performed on the original multivariate time series to obtain the normalized input sequence ; All channels in the input sequence are mapped to a unified feature dimension by linear projection to obtain low-dimensional features after feature compression ; S2, multi-scale trend-seasonality structure enhancement: first, the trend item of the input sequence is extracted by a multi-scale smoothing operator, then the seasonal item of the input sequence is extracted from the frequency domain based on the Fourier basis function combination, and finally the trend item and the seasonal item are fused with the input sequence in a residual manner to obtain an enhanced feature sequence ; S3, local temporal feature modeling of multi-scale expert model: according to a plurality of preset slice sizes, the feature sequence is divided into a plurality of time slices of different lengths, each expert model corresponds to data under a specific scale division; each expert model adopts a depth separable convolution structure, performs local time series modeling within the patch belonging to the expert model, and extracts local dynamic features of different scales; each expert model outputs a local feature representation for a specific time scale is divided into a plurality of time slices of different lengths in the time dimension, each expert model corresponds to data under a specific scale division; each expert model adopts a depth separable convolution structure, performs local time series modeling within the patch belonging to the expert model, and extracts local dynamic features of different scales; each expert model outputs a local feature representation for a specific time scale ; S4. Adaptive Sparse Expert Selection and Load Balancing Loss Calculation: An adaptive sparse routing module is used with low-dimensional features. The scores and uncertainty estimates of each expert model are calculated as the criteria for discrimination. During the training phase, Gaussian noise based on learnable variance is injected into the expert model scores to enhance the exploratory nature. Sparse gating weights are obtained by combining Softmax and Top-K selection strategies to determine the activated expert models. The load balancing loss is further calculated based on the distribution of gating weights and the frequency of expert model usage to ensure the balance of expert model activation during training. S5. Weighted aggregation and residual fusion of multi-scale expert outputs: First, represent the local features of all expert models. Projecting the data onto a consistent feature space, and then performing a weighted summation according to the gated weights, yields the fused multi-scale comprehensive features. Finally, the comprehensive features are combined with the enhanced feature sequence. Perform residual connections to obtain the output features of the expert model. ; S6, Global modeling based on channel weighting of closed-form continuous-time neural network: Introduce channel feature weighting mechanism on the basis of expert model output features, obtain channel recalibrated feature representation ; Then, input the features into the closed-form continuous-time neural network, use the analytical structure composed of the backbone network and two head networks to model the evolution process of the features over time in the continuous time domain through the explicit closed-form, and finally output the global enhanced features ; S7, prediction generation and de-normalization: first, the global enhanced features are predicted with the output features of the expert model are fused to obtain the final latent space representation; then, the output projection layer is mapped to the prediction time step, and the reversible normalization operation is performed to obtain the final prediction result.

2. The multi-scale time series prediction method of claim 1, wherein, In the step S2, low-frequency trend components of the input sequence are extracted by a multi-scale smoothing operator composed of a plurality of sliding convolution kernels with different lengths, each of which can generate a trend candidate sequence under a different smoothing scale; the specific steps are as follows: Let the set of pre-defined convolution kernels be , which correspond to trend candidates respectively; then project the input sequence through a learnable linear mapping and generate time-varying fusion coefficients through a Softmax function, so that the contribution ratio of each trend candidate in the final trend term can be adaptively adjusted; where the expression of the final trend term T is: (1)。 3. The multi-scale time series prediction method of claim 2, wherein, The specific steps of extracting the seasonal term based on the Fourier basis function in the step S2 are as follows: performing real number fast Fourier transform on the input sequence in the time dimension to obtain a frequency domain representation, wherein each frequency component contains amplitude and phase information of a corresponding periodic pattern; then selecting a main frequency with the largest amplitude from the full frequency band, and generating three groups of basic Fourier components according to the main frequency, and reconstructing in the time domain to obtain the seasonal term The expression of the seasonal term is as follows:​ (2)； In formula (2), denotes the frequency of the th primary frequency component, and denote the amplitude and phase parameters, respectively, denotes the time index.

4. The multi-scale time series prediction method of claim 3, wherein, In step S2, after the trend item and the seasonal item are extracted, the trend item and the seasonal item are connected with the input sequence in residual error to form a feature sequence after structure enhancement : (3)。 5. The multi-scale time series prediction method of claim 1, wherein, In the step S3, local time features are extracted at a plurality of time resolutions by using multi-scale expert models, comprising: S3.1, Time patch division: enhanced feature sequence After entering the expert model, the original sequence is divided according to the patch time granularity corresponding to the expert model; wherein, assuming that the size corresponding to the current patch is S, and the sequence length is H, then the divided patch set ; S3.2, Deep Convolution: Within each patch, a one-dimensional convolution operation is performed on each channel independently using deep convolution; wherein for the input feature of the i-th Patch of the j-th layer, the deep convolution process is shown in equation (4): (4)​​ (4)； In formula (4), represents the first layer in the input feature of the first Patch, represents a deep convolution operation, and the corresponding convolution kernel size is , is a GELU activation function, and BN represents a batch normalization operation; S3.3, Point-wise convolution: by 1 x 1 The output of the convolution is point-wise for all channels, the process of which is shown in equation (5): (5)； S3.4, the output of the step S3.3 is subjected to multi-layer stacking and residual connection.

6. The multi-scale time series prediction method of claim 1, wherein, In the step S4, the adaptive sparse routing module is composed of a gating expert selection strategy and a load balancing loss, the gating expert selection strategy is composed of two groups of parallel parameterized networks for estimating the scores and uncertainties of each sample on different expert models, so as to determine the activated expert path of the current input; The load balancing loss is used to prevent some experts from being activated for a long time, thereby limiting the modeling ability and reducing the generalization performance; comprising: The low-dimensional features obtained after feature compression in step S1 On the basis of the low-dimensional features obtained after feature compression in step S1 and noise intensity estimation where is a learnable parameter matrix, and a Softplus activation function is used to ensure that the estimated noise intensity is non-negative. and Gaussian noise is introduced on the expert score L , to obtain a noisy expert score as shown in equation (6): (6)； Expert scoring Applying Softmax normalization to obtain the weight corresponding to each expert model As shown in equation (7): (7)； Sparse gating weights are obtained by retaining the first M largest weights and setting the rest to zero as a basis for the expert path.

7. The multi-scale time series prediction method of claim 1, wherein, In the step S4, a load balancing regular term is introduced As an auxiliary loss, the load balancing regular term is based on the gated score square sum The square of the coefficient of variation of the actual load Construction, as shown in equation (8): (8)； Finally, according to the sparse gating weight score, the input feature is distributed to the corresponding expert model, realizing sparse and efficient multi-path modeling.

8. The multi-scale time series prediction method of claim 1, wherein, The specific steps of the step S5 are as follows: S5.1, First introduce dimension transformation function The output of each expert model is processed to obtain a unified dimensional feature representation; S5.2, introduce GESS module to calculate the weight of expert model The weighted aggregation of expert output; wherein, define the indicator function , when , the i-th expert model is activated, set , otherwise, it means not activated, set ; the fused feature representation form is shown in formula (9): (9)； In formula (9), and respectively represent the output of the i-th expert model and a transformation function; S5.3, fusing features with enhanced feature sequence by residual connection form addition, get the final fusion output of the expert model as shown in equation (10): (10)。 9. The multi-scale time series prediction method of claim 1, wherein, In the step S6, the closed-form continuous-time neural network enhancement module comprises a channel weighting unit and a CFC module; the channel weighting unit introduces a set of learnable channel weights , the importance of each channel is modeled, guiding the model to pay attention to the feature area with high information density, and suppressing redundant or noise interference, and the expression of the weighting process is: ​ (11)； The network structure of the CFC module comprises a backbone network , a head network , and a gating mechanism The working mechanism of the CFC module is shown in formula (12): (12)； wherein, denotes the state representation generated by the backbone network at time step t; denotes the hyperparameters of the network; by employing the CFC module on the complete time series modeling results in global enhanced features of the CFC as shown in equation (13): (13)。 10. The multi-scale time series prediction method of claim 1, wherein, The specific steps of the step S7 are as follows: S7.1, the global enhanced features obtained in step S6 are first combined with the output features of the expert model in step S5 Residual addition is performed to obtain a unified hidden space representation containing both local multi-scale structure and global continuous dynamic information, as shown in equation (14): (14)； S7.2, then according to the prediction layer structure, the unified hidden space representation is rearranged and flattened in the time dimension and the feature dimension, and is mapped to the future prediction step through a linear projection layer, to obtain the intermediate result of prediction as shown in formula (15); (15)； S7.3, RevIN denormalizes the output final prediction: performs denormalization to restore the model output to the true dimension of the original data, resulting in future multivariate time series prediction values , expressed as follows: (16)。

Images