Reinforced learning time sequence analysis system and method based on deep mixed experts

Abstract

The invention provides a reinforcement learning sequential sequence analysis system and method based on deep hybrid experts, and the system comprises a preprocessing module which carries out the denoising of a sequential sequence, and carries out the division of the denoised sequential sequence according to a fixed window size, and obtains a preprocessed sequential sequence; the strategy network is used for coding and decoding the time sequence processed by the preprocessing module on the basis of a hybrid expert layer to obtain scoring information of the time sequence; and the weight generator is used for obtaining the weight of each time sequence based on the scoring information of the strategy network. Through the data preprocessing module, the strategy network and the weight generator, better sequential sequence feature extraction and better sequential sequence scoring are realized.

Description

Technical Field

[0001] This invention relates to the field of data processing and analysis technology, and more specifically, to a reinforcement learning time series analysis system and method based on deep hybrid experts. Background Technology

[0002] Time series data are widely used in the real world, such as in weather trends, energy consumption patterns, and stock market fluctuations. The importance of predicting time series data has become a crucial topic. Researchers need to analyze the historical trends of multiple time series to determine their relative importance and make informed decisions. For example, predicting which energy source will experience a significant increase allows for early energy-related planning.

[0003] In recent years, deep learning has been used to predict the future trends of time series to guide decision-making in related downstream tasks. Accurate prediction of time series is extremely difficult due to their high noise levels and susceptibility to various factors. Deep reinforcement learning (DRL), which can interact with the environment to achieve better environmental adaptation, has attracted the attention of researchers across various disciplines, showing promise in solving complex decision-making challenges. In the mixed transportation domain, DRL is used to manage signalized intersections, combining connected and autonomous vehicles (CAVs) with human-driven vehicles (HVs) using a DQN approach. In the financial domain, DeepTrader introduces a causal-based graph convolutional network to capture the connections between stocks, uses a TCN to extract time series features, and finally obtains investment weights through a portfolio generator. Deep reinforcement learning has further enhanced its ability to balance investment returns and risks in the financial field.

[0004] While time series analysis models based on deep reinforcement learning have yielded satisfactory results, limitations remain. Previous reinforcement learning methods often employed a uniform model to analyze different time series, which could lead to suboptimal performance due to the varying characteristics of each time series.

[0005] Different time series are influenced by different factors, so time series analysis is also very important for predicting weather trends, energy consumption patterns, and fluctuations in the financial and stock markets. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a reinforcement learning time sequence analysis system and method based on deep hybrid experts.

[0007] According to one aspect of the present invention, a reinforcement learning time sequence analysis system based on deep hybrid experts is provided, comprising:

[0008] The preprocessing module denoises the time series by dividing the denoised time series into a fixed window size to obtain the preprocessed time series.

[0009] The policy network is a reinforcement learning network that is based on a hybrid expert layer. It encodes and decodes the time series processed by the preprocessing module to obtain the scoring information of the time series.

[0010] The weight generator obtains the weights of each time series based on the scoring information of the policy network.

[0011] Preferably, the preprocessing module includes:

[0012] The decomposition unit receives the time sequence, decomposes it using wavelet basis functions, and obtains wavelet coefficients.

[0013] The filtering unit performs threshold filtering on the wavelet coefficients;

[0014] The reconstruction unit generates a denoised time sequence based on wavelet basis functions from the wavelet coefficients after threshold filtering.

[0015] The time series to be denoised is divided into units with a fixed window size, and the time series divided into equal windows is used as the preprocessing result.

[0016] Preferably, the policy network is a U-shaped encoding / decoding structure, comprising:

[0017] The encoder extracts the encoded temporal features of the preprocessed temporal sequence;

[0018] The decoder decodes the input time sequence based on the encoded time sequence features to obtain the decoded time sequence features.

[0019] The fully connected layer obtains the scoring information of the input time sequence based on the encoded time sequence features and the decoded time sequence features.

[0020] Preferably, the encoder includes a segmentation layer, N hybrid expert layers and N-1 fusion layers, each of the fusion layers is equipped with one of the hybrid expert layers, and the segmentation layer is followed by one of the hybrid expert layers;

[0021] in:

[0022] The segmentation layer divides the input time series sequence into multiple subsequences and concatenates the multiple subsequences.

[0023] The fusion layer merges the sub-sequences output by the segmentation layer or the hybrid expert layer in pairs to reduce the number of sub-sequences.

[0024] The hybrid expert layer extracts encoded temporal features from the sub-sequences output by the segmentation layer or the fusion layer.

[0025] Preferably, the hybrid expert layer includes four expert networks, one routing network, one normalization network, one selection network, and one summing network; wherein,

[0026] A normalization network normalizes the input subsequence;

[0027] The routing network calculates four predicted values ​​for the normalized result, each corresponding to one of the four expert networks;

[0028] Choose the network based on the maximum value of the estimated value, and select the corresponding expert network;

[0029] Expert network: Feature extraction of normalization results based on the selected expert network;

[0030] The summing network fuses the input subsequence, the maximum value, and the features extracted by the expert network to obtain the encoded temporal features of the input subsequence.

[0031] Preferably, the decoder includes N gated multilayer perceptron layers and N-1 separation layers; each separation layer is equipped with one gated multilayer perceptron layer; the gated multilayer perceptron layers are symmetrically distributed with the hybrid expert layers, and the separation layers are symmetrically distributed with the fusion layers;

[0032] in,

[0033] The gated multilayer perceptron layer extracts and decodes temporal features from the input temporal sequence;

[0034] The separation layer separates and decouples the decoding timing features output by the gated multilayer perceptron layer, increasing the number of sub-sequences.

[0035] Preferably, the expert network in the encoder and the gated multilayer perceptron layer structure of the decoder are the same.

[0036] Preferably, the policy network comprises several layers, forming a U-shaped structure, specifically:

[0037] Layer 0 includes a segmentation layer that divides the input time series sequence into multiple subsequences.

[0038] The first layer has a hybrid expert layer, and on the opposite side, there is a gated multilayer perceptron layer;

[0039] From layer 2 to layer N, each layer is configured with a set of hybrid layers and hybrid expert layers; on the opposite side, a set of gated multilayer perceptron layers and separation layers are configured.

[0040] The input of the expert hybrid layer in layer 1 is the output of the segmentation layer in layer 0, and the output is the extracted temporal sequence features, denoted as the layer 1 encoded temporal features.

[0041] From layer 2 to layer N, the input of the fusion layer of layer i is the output of the hybrid expert layer of layer i-1. This input is fused in the fusion layer and output to the hybrid expert layer of layer i for feature extraction to obtain the corresponding i-layer encoded temporal features.

[0042] The input of the gated multilayer perceptron layer of the Nth layer is the output of the hybrid expert layer of the Nth layer, that is, the Nth layer encoded temporal features. After the features are extracted by the gated multilayer perceptron layer, they enter the separation layer and are separated and decoupled. The output of the separation layer of the Nth layer is called the Nth layer decoded temporal features.

[0043] From layer N-1 to layer 1, the input of the gated multilayer perceptron layer of layer i is the sum of the output of the separation layer of layer i+1 and the output of the hybrid expert layer of layer i; the input of the separation layer of layer i is the feature extracted by the gated multilayer perceptron layer, and the output is the decoded temporal feature of layer i after separation and decoupling.

[0044] N fully connected layers are distributed from layer 0 to layer (N-1); the input of the layer 0 connection layer is the output of the layer 0 segmentation and the output of the layer 1 gated multilayer perceptron layer; the output is the sum of the inputs.

[0045] The inputs to the connection layer from layer 1 to layer N-1 are: the output of the i-th hybrid expert layer and the output of the i+1-th separation layer; the output is the sum of all inputs.

[0046] Preferably, the policy network is a reinforcement learning network, whose input is a preprocessed time sequence, i.e., state information, and whose output is a score for each time sequence, i.e., action information;

[0047] Based on the state and actions of the policy network, define the optimization objective:

[0048] r t =y t ·π(a t |s t ,θ)

[0049]

[0050] Where π(a) t |s t ,θ) represents the state s given by the state s t The action a taken by the policy network t y t Represents state s t With s t+1 The rate of return between, rt R(τ) represents the single-step reward obtained after taking an action, and E represents the reward after performing a reinforcement learning trajectory. τ～πθ This represents the mean square error.

[0051] According to a second aspect of the present invention, a reinforcement learning time series analysis method based on deep hybrid experts is provided, comprising:

[0052] The time series is denoised by dividing the denoised time series into a fixed window size to obtain a preprocessed time series.

[0053] Based on hybrid experts, the preprocessed time series is encoded and decoded to obtain the scoring information of the time series;

[0054] Based on the scoring information, the weight of each time series is obtained.

[0055] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0056] The reinforcement learning time series analysis system and method based on deep hybrid experts in this invention achieves better time series feature extraction and better time series scoring through a data preprocessing module, a policy network, and a weight generator.

[0057] The reinforcement learning time series analysis system and method based on deep hybrid expert layers in this invention introduces a U-shaped encoding and decoding structure based on deep hybrid expert layers, which enables the model to extract different time series characteristics and improves the model's feature extraction capability for time series. Attached Figure Description

[0058] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0059] Figure 1 This is a schematic diagram of the policy network of a reinforcement learning temporal sequence analysis system based on deep hybrid experts according to an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the hybrid expert layer of a reinforcement learning time sequence analysis system based on deep hybrid experts, provided as a preferred embodiment of the present invention;

[0061] Figure 3 A schematic diagram of a gated multilayer perceptron for a reinforcement learning time sequence analysis system based on deep hybrid experts, provided as a preferred embodiment of the present invention. Detailed Implementation

[0062] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0063] In one embodiment of the present invention, a reinforcement learning time series analysis system based on a deep hybrid expert model includes: a data preprocessing module, a policy network, and a weight generator. The preprocessing module denoises the time series and divides the denoised time series into segments with a fixed window size to obtain preprocessed time series. The policy network encodes and decodes the time series processed by the preprocessing module based on the hybrid expert layer to obtain scoring information for the time series. The weight generator obtains the weights for each time series based on the scoring information from the policy network.

[0064] The above embodiments, through the data preprocessing module, policy network, and weight generator, achieve better time series feature extraction and better time series scoring.

[0065] In a preferred embodiment of the present invention, the data preprocessing module handles missing values. It primarily utilizes wavelet packet transform to denoise the input time-series signal and includes the following four units: a decomposition unit, which receives the time-series sequence, decomposes it using wavelet basis functions, and obtains wavelet coefficients; a filtering unit, which performs threshold filtering on the wavelet coefficients; a reconstruction unit, which generates a denoised time-series sequence from the threshold-filtered wavelet coefficients based on wavelet basis functions; and a partitioning unit, which divides the denoised time-series sequence into segments of fixed window size, using the equally partitioned time-series sequences as the preprocessing result.

[0066] Specifically, the aforementioned decomposition unit receives the input signal (i.e., the sequential signal) and then decomposes it using wavelet basis functions to obtain the wavelet coefficients. The specific formula can be expressed as: M i =∑ k h k x 2i-k M i Let be the i-th wavelet coefficient, h be the wavelet basis function, and x be the received signal. Each time sequence corresponds to a wavelet coefficient.

[0067] Specifically, the above-mentioned filtering unit performs threshold filtering on the wavelet coefficients obtained by the decomposition unit, which can be done according to the following specific steps:

[0068] 1) Calculate the risk value for each wavelet coefficient:

[0069] 2) Find the wavelet coefficient m that minimizes the risk value;

[0070] 3) Calculate the filtering threshold:

[0071] 4) Use a threshold function to filter wavelet coefficients:

[0072]

[0073] Specifically, the aforementioned reconstruction unit uses wavelet basis functions to generate a denoised time-series signal from the wavelet coefficients filtered by the filtering unit. The specific formula can be expressed as: x i =∑ k h k M k / 2 .

[0074] Specifically, the aforementioned partitioning unit divides the denoised time series into segments with a fixed window size, and uses the time series segments with equal window sizes as the preprocessing result. Since the dataset consists of a long time series, this embodiment can divide the time series by cutting it into smaller segments of time series data, which can be input into the model to effectively and quickly process the time series data in blocks.

[0075] In a preferred embodiment, during the training of the data preprocessing module, for the training of the partitioning unit, several windows can be randomly selected as historical information segments with sixty-four timestamps in each training round; the signal of each extracted window is standardized with a mean of 0 and a variance of 1 to promote the stability of the final training result.

[0076] In this embodiment, threshold filtering and time series information reconstruction are performed based on wavelet basis functions to eliminate outliers and missing values, respectively, for subsequent analysis and modeling. Data standardization can improve the robustness of the model to the data.

[0077] Existing deep reinforcement learning-based time series analysis models possess the ability to capture the trend characteristics of different time series, but there is still room for improvement in the granularity of the extracted features. To further improve the model's ability to extract time series features of different granularities, in a preferred embodiment of the present invention, the policy network employs a U-shaped encoder-decoder, which contains multiple hybrid expert layers.

[0078] Specifically, such as Figure 1 As shown, the policy network includes a U-shaped encoder, a U-shaped decoder, and a fully connected output layer. The U-shaped encoder extracts the timing features of the input timing signal; the U-shaped decoder receives the timing features output by the encoder as input and outputs the final features of the input timing signal; the fully connected output layer receives the timing features output by the decoder as input and outputs the scoring information of the timing features.

[0079] In a preferred embodiment, the encoder of the U-shaped codec structure includes a segmentation layer, a hybrid expert layer, and a fusion layer. Each fusion layer is equipped with a hybrid expert layer, and a hybrid expert layer is connected after the segmentation layer. Specifically: the segmentation layer divides the input time sequence into multiple sub-sequences and concatenates the multiple sub-sequences; the fusion layer merges the sub-sequences output by the segmentation layer or the hybrid expert layer pairwise to reduce the number of sub-sequences; and the hybrid expert layer extracts encoded time sequence features from the sub-sequences output by the segmentation layer or the fusion layer.

[0080] Specifically, assuming the input is data of shape M*T*C, where M represents the number of time series, T represents the length of the time series, and C represents the feature degree. The segmentation layer divides the input time series into multiple subsequences and concatenates these subsequences. Here, segmentation means transforming it into T / t data of shape M*t*C, where t represents the length of the subsequence, and concatenation means concatenating these T / t data into data of shape M*(T / t)*t*C.

[0081] In a preferred embodiment, in order to better extract the characteristics of different time series sequences, such as Figure 2 As shown, the hybrid expert layer includes four expert networks, one routing network, one normalization network, one selection network, and one summing network. The normalization network normalizes the input subsequence. The routing network calculates four predicted values ​​from the normalization result, each corresponding to one of the four expert networks. The selection network selects the corresponding expert network based on the maximum value of the predicted value. The expert networks extract features from the normalization result based on the selected expert networks. The summing network fuses the input subsequence, the maximum value, and the features extracted by the expert networks to obtain the encoded temporal features of the input subsequence.

[0082] The four expert networks have the same structure, which is a gated multilayer perceptron layer, such as... Figure 3 As shown, the gated multilayer perceptron layer consists of a perceptron network with gated units and a LeakyReLU activation function, used to extract time-series features.

[0083] Specifically, in one embodiment, such as Figure 3As shown, the gated multilayer perceptron layer consists of two fully connected projection layers and activation functions and gating units connecting them. First, the first fully connected projection layer concatenates all input temporal features and projects them onto the activation function. Next, the LeakyReLU activation function transforms the linear features projected from the first fully connected projection layer into non-linear features. Then, the non-linear features undergo dual-line processing through the gating unit: one line for segmentation, and the other line for segmentation, normalization, and spatial projection processing in sequence; the outputs of the two lines are fused and used as the final output of the gating unit. Finally, the second fully connected projection layer concatenates the input temporal features and the output of the gating unit and outputs the result, yielding detailed temporal sequence features.

[0084] In a preferred embodiment, the decoder of the U-shaped encoding / decoding structure includes several separation layers and several gated multilayer perceptron layers: wherein, the separation layers decouple the encoded features to increase the number of subsequences and have a symmetrical structure with the fusion layer in the encoder; the gated multilayer perceptron layers consist of a perceptron network with gated units and a LeakyReLU activation function, and are used to extract temporal sequence features.

[0085] In one specific embodiment, the policy network consists of N+1 layers, forming a U-shaped structure. Specifically, the encoder of the U-shaped encoder-decoder structure consists of one segmentation layer, N hybrid expert layers, and N-1 fusion layers; the decoder of the U-shaped encoder-decoder structure consists of N gated multilayer perceptron layers and N-1 separation layers. The policy network also includes N fully connected layers.

[0086] like Figure 1 As shown, layer 0 is a segmentation layer; the time series input by the segmentation layer is divided into multiple subsequences.

[0087] The first layer is a hybrid expert layer, and on the opposite side is a gated multilayer perceptron layer;

[0088] Layers 2 through N are configured with a set of hybrid layers and hybrid expert layers; on the opposite side are gated multilayer perceptron layers and separation layers.

[0089] The input to the expert hybrid layer of layer 1 is the output of the segmentation layer of layer 0, and the output is the extracted temporal sequence features, which can be denoted as the layer 1 encoded temporal features;

[0090] From layer 2 to layer N, the input of the fusion layer of layer i is the output of the hybrid expert layer of layer i-1. They are fused in the fusion layer and output to the hybrid expert layer of layer i for feature extraction to obtain the corresponding i-layer encoded temporal features.

[0091] The input of the gated multilayer perceptron layer of the Nth layer is the output of the hybrid expert layer of the Nth layer, that is, the Nth layer encoded temporal features. After feature extraction, the features are separated and decoupled by the separation layer to increase the number of sub-sequences. The output of the separation layer of the Nth layer is called the Nth layer decoded temporal features.

[0092] From layer N-1 to layer 1, the input of the gated multilayer perceptron layer of layer i is the sum of the output of the separation layer of layer i+1 (decoding timing features of layer i+1) and the output of the hybrid expert layer of layer i (encoding timing features of layer 1); the input of the separation layer of layer i is the features extracted by the gated multilayer perceptron layer, and the output is the decoupled decoding timing features of layer i.

[0093] N fully connected layers are distributed from layer 0 to layer (N-1); the input of the layer 0 connection layer is the output of the layer 0 segmentation and the output of the layer 1 gated multilayer perceptron layer; the output is the sum of the inputs.

[0094] The inputs to the connection layer from layer 1 to layer N-1 are: the output of the i-th hybrid expert layer (i-th layer encoding timing features) and the output of the i+1-th separation layer (i-th layer decoding timing features); the output is the sum of all inputs.

[0095] Finally, all fully connected results are summed and the final score is output using the sigmoid function, with each score corresponding to a time series sequence.

[0096] In the above embodiments, the policy network primarily scores the input historical information. This scoring is equivalent to predicting the probability of events; the higher the score, the greater the probability of the event. The existence of the policy network enables the model to predict future trends based on data. In the implementation of this invention, a U-shaped encoding / decoding structure based on a deep hybrid expert layer is introduced into the policy network. This allows the policy network to extract characteristics of different time series sequences, improving its feature extraction capability and making trend predictions more accurate.

[0097] In a preferred embodiment of the present invention, a weight generator is used to obtain the final time-series sequence weights. Specifically, the scoring vectors generated by the industry policy network are directly subjected to softmax processing.

[0098] In a preferred embodiment, in order for the policy network to generate better actions, the present invention provides an embodiment that updates the parameters of the policy network through policy gradient ascent.

[0099] The policy network takes state information as input and outputs action information. Based on the state and actions of the policy network, the optimization objective is defined as follows:

[0100] r t =y t ·π(a t |s t,θ)

[0101]

[0102] Where π(a) t |s t ,θ) represents the state s given by the state s t The action taken by the policy network, a t y t Represents state s t With s t+1 The rate of return between, r t R(τ) represents the single-step reward obtained after taking an action, and E represents the reward after completing an entire reinforcement learning trajectory. τ～πθ This represents the mean square error.

[0103] Based on the same inventive concept, other embodiments of the present invention also provide a reinforcement learning time sequence analysis method based on deep hybrid experts, including:

[0104] Step 1: Denoise the time series sequence by dividing the denoised time series sequence into a fixed window size to obtain the preprocessed time series sequence.

[0105] Step 2: Based on hybrid experts, the processed time series is encoded and decoded to obtain the scoring information of the time series.

[0106] Step 3: Based on the scoring information, obtain the weight of each time series.

[0107] The specific steps in the above examples of the present invention can be referred to the implementation technology of each module / unit of the reinforcement learning time sequence analysis system based on deep hybrid experts in the above embodiments, and will not be repeated here.

[0108] In some embodiments of the present invention, the weights of different time series obtained by the reinforcement learning time series analysis system or method based on deep hybrid expert models in the above embodiments are used for downstream task processing. Downstream tasks may include: energy consumption decision-making, climate change early warning, financial stock investment, etc.

[0109] In some specific embodiments, stock data is used as an example for analysis as time-series data. Wind and Choic are employed. e Financial software collects historical information on relevant industry indices and stocks.

[0110] The X stocks in a certain industry are arranged into a time series. Each time series is formed by M time points, and each time point is composed of stock time series signals. The historical information of the stocks is the time series signal, including the daily stock closing price, the maximum stock price, the minimum stock price, the stock trading volume, the total market value of the stock, and the stock price-earnings ratio.

[0111] The time series sequences of X stocks are input into the preprocessing module to denoise the time series sequences. The denoised time series sequences are then divided into segments with a fixed window size to obtain the preprocessed time series sequences.

[0112] The preprocessed time series is input into the policy network. Based on deep hybrid experts, the time series processed by the preprocessing module is encoded and decoded to obtain the scoring information of the time series.

[0113] The weight generator obtains the weights for each time series based on the scoring information from the policy network.

[0114] Stock selection is performed based on the weight of each time series sequence, and this has been validated using historical data. Specifically:

[0115] In the US stock market, two sectors were selected: pharmaceuticals and semiconductors, with 12 stocks chosen from each sector. Results on the 2017-2022 dataset: average annual return 72.037%, maximum drawdown 11.69%, and Kalma ratio 6.162.

[0116] In the A-share market, two industries were selected: pharmaceuticals and semiconductors, with 30 stocks chosen from each industry. Results on the 2016-2021 dataset: average annual return 51.257%, maximum drawdown 19.86%, and Kalma ratio 2.581.

[0117] For Hong Kong stocks, we selected two sectors: financials and industrials, with 31 stocks chosen from each sector. Results from the 2018-2022 dataset: average annual return of 62.615%, maximum drawdown of 10.14%, and a Kalma ratio of 6.177.

[0118] Dow Jones Industrial Average. Results on the 2000-2018 dataset: average annual return 21.981%, maximum drawdown 16.47%, Kalma ratio 1.335;

[0119] The Hang Seng Index. Results on the 2007-2019 dataset: average annual return of 50.156%, maximum drawdown of 11.99%, and Kalma ratio of 4.184;

[0120] The CSI 100 Index. Results on the dataset from 2013 to 2019: average annual return of 73.129%, maximum drawdown of 21.02%, and Kalma ratio of 3.479.

[0121] Average annual return represents profitability (the higher the better), maximum drawdown represents risk (the lower the better), and the Kalmar ratio is a measure of the trade-off between profitability and risk (the higher the better).

[0122] This data demonstrates that the method described in this invention achieves good returns and low risk on stock market datasets in the financial sector. This indicates that the method of this invention can effectively extract more detailed features from the time series and accurately predict trends.

[0123] Of course, it can also be applied to other fields, such as climate prediction. First, collect climate data from historical periods, such as daily climate data for the past 10 years, including temperature, wind speed, wind direction, and rainfall. This forms a 365*10 time series. Then, perform weather prediction based on this time series. Specifically: denoise the time series, divide it into preprocessed window segments, and use a deep hybrid expert algorithm to encode and decode the preprocessed time series, obtaining scoring information for each segment (number of preprocessed segments * 10 scoring segments). Based on the scoring information from the policy network, obtain the weight of each time series segment, and determine the magnitude of the climate trend based on the weight.

[0124] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A reinforcement learning-based time series analysis system based on deep hybrid experts, characterized in that, include: The preprocessing module denoises the time series by dividing the denoised time series into a fixed window size to obtain the preprocessed time series. The policy network is a reinforcement learning network that is based on a hybrid expert layer. It encodes and decodes the time series processed by the preprocessing module to obtain the scoring information of the time series. The weight generator obtains the weights of each time series based on the scoring information of the policy network.

2. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 1, characterized in that, The preprocessing module includes: The decomposition unit receives the time sequence, decomposes it using wavelet basis functions, and obtains wavelet coefficients. The filtering unit performs threshold filtering on the wavelet coefficients; The reconstruction unit generates a denoised time sequence based on wavelet basis functions from the wavelet coefficients after threshold filtering. The time series to be denoised is divided into units with a fixed window size, and the time series divided into equal windows is used as the preprocessing result.

3. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 1, characterized in that, The policy network has a U-shaped encoding / decoding structure, including: The encoder extracts the encoded temporal features of the preprocessed temporal sequence; The decoder decodes the input time sequence based on the encoded time sequence features to obtain the decoded time sequence features. The fully connected layer obtains the scoring information of the input time sequence based on the encoded time sequence features and the decoded time sequence features.

4. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 3, characterized in that, The encoder includes a segmentation layer, N hybrid expert layers and N-1 fusion layers, each of the fusion layers is equipped with one of the hybrid expert layers, and the segmentation layer is connected to one of the hybrid expert layers; in: The segmentation layer divides the input time series sequence into multiple subsequences and concatenates the multiple subsequences. The fusion layer merges the sub-sequences output by the segmentation layer or the hybrid expert layer in pairs to reduce the number of sub-sequences. The hybrid expert layer extracts encoded temporal features from the sub-sequences output by the segmentation layer or the fusion layer.

5. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 4, characterized in that, The hybrid expert layer includes four expert networks, one routing network, one normalization network, one selection network, and one summing network; wherein, A normalization network normalizes the input subsequence; The routing network calculates four predicted values ​​for the normalized result, each corresponding to one of the four expert networks; Choose the network based on the maximum value of the estimated value, and select the corresponding expert network; Expert network: Feature extraction of normalization results based on the selected expert network; The summing network fuses the input subsequence, the maximum value, and the features extracted by the expert network to obtain the encoded temporal features of the input subsequence.

6. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 3, characterized in that, The decoder includes N gated multilayer perceptron layers and N-1 separation layers; each separation layer is equipped with one gated multilayer perceptron layer; the gated multilayer perceptron layers are symmetrically distributed with the hybrid expert layers, and the separation layers are symmetrically distributed with the fusion layers; in: The gated multilayer perceptron layer extracts and decodes temporal features from the input temporal sequence; The separation layer separates and decouples the decoding timing features output by the gated multilayer perceptron layer, increasing the number of sub-sequences.

7. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 3, characterized in that, The expert network in the encoder and the gated multilayer perceptron layer structure in the decoder are identical.

8. The reinforcement learning time series analysis system based on deep hybrid experts according to claim 7, characterized in that, The policy network comprises several layers, forming a U-shaped structure, specifically: Layer 0 includes a segmentation layer that divides the input time series sequence into multiple subsequences. The first layer has a hybrid expert layer, and on the opposite side, there is a gated multilayer perceptron layer; From layer 2 to layer N, each layer is configured with a set of hybrid layers and hybrid expert layers; on the opposite side, a set of gated multilayer perceptron layers and separation layers are configured. The input of the expert hybrid layer in layer 1 is the output of the segmentation layer in layer 0, and the output is the extracted temporal sequence features, denoted as the layer 1 encoded temporal features. From layer 2 to layer N, the input of the fusion layer of layer i is the output of the hybrid expert layer of layer i-1. This input is fused in the fusion layer and output to the hybrid expert layer of layer i for feature extraction to obtain the corresponding i-layer encoded temporal features. The input of the gated multilayer perceptron layer of the Nth layer is the output of the hybrid expert layer of the Nth layer, that is, the Nth layer encoded temporal features. After the features are extracted by the gated multilayer perceptron layer, they enter the separation layer and are separated and decoupled. The output of the separation layer of the Nth layer is called the Nth layer decoded temporal features. From layer N-1 to layer 1, the input of the gated multilayer perceptron layer of layer i is the sum of the output of the separation layer of layer i+1 and the output of the hybrid expert layer of layer i; the input of the separation layer of layer i is the feature extracted by the gated multilayer perceptron layer, and the output is the decoded temporal feature of layer i after separation and decoupling. N fully connected layers are distributed from layer 0 to layer (N-1); the input of the layer 0 connection layer is the output of the layer 0 segmentation and the output of the layer 1 gated multilayer perceptron layer; the output is the sum of the inputs. The inputs to the connection layer from layer 1 to layer N-1 are: the output of the i-th hybrid expert layer and the output of the i+1-th separation layer; the output is the sum of all inputs.

9. A reinforcement learning time series analysis system based on deep hybrid experts according to claim 7, characterized in that, The policy network is a reinforcement learning network. Its input is a preprocessed time sequence, which is the state information, and its output is a score for each time sequence, which is the action information. Based on the state and actions of the policy network, define the optimization objective: r t = y t · π(a t |s t , θ) Where π(a) t |s t ,θ) represents the state s given by the state s t The action a taken by the policy network t y t Represents state s t With s t+1 The rate of return between, r t E represents the single-step reward obtained after taking an action, R(τ) represents the reward after performing reinforcement learning on the trajectory, and E represents the reward after performing the action. τ～πθ This represents the mean square error.

10. A reinforcement learning-based temporal sequence analysis method based on deep hybrid experts, characterized in that, include: The time series is denoised by dividing the denoised time series into a fixed window size to obtain a preprocessed time series. Based on hybrid experts, the preprocessed time series is encoded and decoded to obtain the scoring information of the time series; Based on the scoring information, the weight of each time series is obtained.

Images