Robust time sequence diagram prediction method based on information bottleneck theory and applied to multi-layer perceptron

Abstract

The invention relates to the technical field of urban planning and calculation, and discloses a robust time sequence diagram prediction method applied to a multi-layer perceptron based on an information bottleneck theory. A hypothesis X-Z-Y obeyed by a DVIB model is defaulted, so that Z-X-Y is relaxed, and a theoretical guarantee can be provided. After the assumption of loosening Z-X-Y, a new noise information item I (Z; y | X). The method is based on a multi-layer perceptron structure, and can keep more excellent efficiency in some larger data sets.

Description

Technical Field

[0001] The invention relates to the field of urban planning and computing technology, and discloses a robust time series graph prediction method applied to a multi-layer perceptron based on information bottleneck theory. Background Art

[0002] Spatiotemporal forecasting is of great significance in modeling complex dynamic systems. It requires capturing temporal and spatial dependencies to accurately predict important statistics, such as traffic flow status or power consumption. In previous works on spatiotemporal forecasting, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and temporal convolutional networks (TCNs) have been effectively adopted to model spatiotemporal relationships. Recently, spatiotemporal graph neural networks (STGNNs) have attracted increasing interest due to their powerful capabilities. Although STGNNs have achieved excellent performance, STGNN-based methods are insufficient in computational efficiency. To address this issue, some recent studies have adopted multi-layer perceptrons (MLPs) due to their superior efficiency. However, MLP-based models become less effective when faced with spatiotemporal noise interference, which is common in the real world. Due to the presence of noise interference, two time series may become indistinguishable both at the historical input side and the prediction target side, a phenomenon known as "sample indistinguishability". There is no theoretical principle in existing MLP-based models to alleviate this problem. In addition, the present invention can also observe severe feature collapse, that is, feature collapse may become indistinguishable at the historical input end and the predicted target end, which is caused by the learned features.

[0003] To cope with noise interference, methods based on adversarial training, graph information bottleneck (GIB), mathematical tools, frequency domain MLPs, biased TCN, spatiotemporal course dropout, etc. have been proposed to enhance the robustness of representation learning. However, spatiotemporal data are usually preprocessed through a sliding window mechanism, where sequences can be used as inputs or targets in different windows. Noise may affect both ends simultaneously, which is called the "double noise effect". Therefore, the enhancement effect of these methods is often limited because they only consider a single end effect. Although the robust graph information bottleneck (RGIB) effectively combats bilateral edge noise in link prediction, it is difficult to directly generalize RGIB to MLP networks for spatiotemporal prediction because it relies on GNN model architectures and graph data assumptions that are very different from continuous multivariate time series. Therefore, it is meaningful to develop additional guidelines and specific instances to deal with this situation. Summary of the invention

[0004] Based on the current situation and shortcomings described above, the present invention develops a robust time series graph prediction method applied to a multi-layer perceptron based on the information bottleneck theory. The present invention proposes a new principle with theoretical guarantee, namely the Robust Spatial-Temporal Information Bottleneck (RSTIB). RSTIB reduces the double noise effect in spatiotemporal data by extending the previous information bottleneck (IB) method without compromising the essence of IB. This method improves the robustness of the model by explicitly reconstructing and minimizing additional noise information and original redundant information. Under the guidance of RSTIB, the present invention further implements RSTIB-MLP for robust spatiotemporal modeling based on the MLP model.

[0005] The technical solution of the present invention is a robust timing diagram prediction method applied to a multi-layer perceptron based on information bottleneck theory.

[0006] The present invention first follows the following assumptions: Robust Spatial-Temporal Information Bottleneck (RSTIB) theoretical derivation:

[0007] In spatiotemporal data analysis, data usually exhibit inherent noise characteristics. These data can be used as input or target and serve different purposes when located in different windows under the sliding window mechanism. Therefore, noise elements may exist in both input and target areas. In order to simplify the analysis, the present invention assumes that the noise type follows the additive white Gaussian noise (AWGN) model, which is a noise model widely used in practical applications and empirically approximated.

[0008] Therefore, the present invention does not rely on the Markov assumption ZXY generally followed in the field of information bottlenecks, thereby avoiding ignoring the noise information item I(Z; Y|X). The present invention relaxes ZXY and provides theoretical guarantees by using the assumption XZY that the DVIB model follows by default. After relaxing the assumption ZXY, the present invention reconsiders the new noise information item I(Z; Y|X). While considering the original information item I(X; Z|Y), the present invention first gives the following proposition:

[0009] Reconstructing the conditional mutual information, the sum of I(Z; Y|X) and I(X; Z|Y) can be reconstructed as:

[0010] I(Z;Y∣X)+I(Z;X∣Y)=I(Z;X,Y)-I(X;Y;Z)

[0011] The specific reconstruction derivation is as follows:

[0012] First, by definition:

[0013] I(X;Y∣Z)=H(X∣Z)+H(Y∣Z)-H(X,Y∣Z)

[0014] By expanding each conditional entropy, we have the following representation:

[0015] H(X|Z)=H(X,Z)-H(Z)

[0016] H(Y|Z)=H(Y,Z)-H(Z)

[0017] H(X,Y|Z)=H(X,Y,Z)-H(Z)

[0018] therefore:

[0019] I(X;Y∣Z)=H(X,Z)+H(Y,Z)-H(X,Y,Z)-H(Z)

[0020] =[H(X)+H(Y)-H(X,Y)]-[H(Z)+H(X,Y)-H(X,Y,Z)]+[H(Z,Y)+H(X,Y)-H(X,Y,Z)

[0021] -H(Y)]+[H(Z,X)+H(Y,X)-H(X,Y,Z)-H(X)]

[0022] By using the following definition:

[0023] I(X;Y)=H(X)+H(Y)-H(X,Y)

[0024] I(Z;X,Y)=H(Z)+H(X,Y)-H(X,Y,Z)

[0025] I(Z;X∣Y)=H(Z,Y)+H(X,Y)-H(X,Y,Z)-H(Y)

[0026] I(Z;Y∣X)=H(Z,X)+H(Y,X)-H(X,Y,Z)-H(X)

[0027] I(X;Y∣Z)=I(X;Y)-(I(Z;X,Y)-I(Z;X∣Y)-I(Z;Y∣X))

[0028] According to the definition I(X;Y;Z)=I(X;Y)-I(X;Y|Z), the invention can deduce the reconstruction described, namely:

[0029] I(Z;Y∣X)+I(Z;X∣Y)=I(Z;X,Y)-I(X;Y;Z)

[0030] Accordingly, the present invention constructs the corresponding RSTIB principle as follows:

[0031] lm RSTIB =-I(Z,Y)+β 1 ×I(Z;X,Y)-β 2 ×I(X;Y;Z)

[0032] β 1 ,β 2 ≥0 represents the corresponding Lagrange multiplier in order to control the balance within this objective function.

[0033] By applying the above reconstruction formula, the present invention aims to reduce the interference of the noise information Z described by H(Y|X) and H(X|Y) on the analysis result.

[0034] Specific implementation of robust spatiotemporal graph information bottleneck on multi-layer perceptron:

[0035] In order to implement the above-mentioned RSTIB principle, the present invention considers implementing three information items existing in the RSTIB principle, namely I(X; Y; Z), I(Z; X, Y), and I(Z, Y).

[0036] Therefore, the detailed steps are as follows:

[0037] S1 instantiation I(X;Y;Z): specific model implementation

[0038] By definition, the expression I(X;Y;Z) = I(X;Y) - I(X;Y|Z) shows that maximizing I(X;Y;Z) is equivalent to minimizing I(X;Y|Z) while keeping I(X;Y) constant. Therefore, by explicitly minimizing I(X;Y|Z), the goal of the present invention is to learn a sufficient Z while achieving the goal of this step.

[0039] To this end, Z is initially encoded from X. Then, the goal of the present invention is to reduce the relative knowledge between X and Y by observing Z. To achieve this goal, the present invention uses data reparameterization techniques to obtain the reparameterized and At the same time, it is assumed that they have independent and identically distributed (iid) prior distributions, thereby reducing the overlap of conditional information on X, Z, and Y. The present invention directly applies input regularization and target regularization Although the mutual information term is usually difficult to handle, the present invention introduces input regularization in the following introduction and target regularization The upper bound of :

[0040] Assumptions and The prior distribution of and is an independent and identically distributed unit Gaussian distribution N(0,1).

[0041] and The upper bounds of are given by the following formulas:

[0042]

[0043] Where KL represents the Kullback-Leibler divergence, and represents a parameterized distribution.

[0044] According to the above expression, the present invention first uses a simple multi-layer perceptron layer to parameterize the posterior distribution Parameterization here means using MLP to learn the conditional distribution of Z given X. The result of parameterization is a Gaussian distribution, expressed as where μ z and are the mean and variance of the distribution respectively.

[0045] Next, to make the dimensions of X and Y match, two additional fully connected layers are used, one for X and the other for Y. This process parameterizes two distributions, denoted as and Then, build and Each item is:

[0046]

[0047] Subsequently, the present invention adopts data reparameterization to obtain the reparameterized data Each and ∈~N(0,1) is the noise sampled from a standard normal distribution.

[0048] Analytical solution for regularized upper bounds of input and target. Given the mean and variance of two Gaussian distributions, the Kullback-Leibler (KL) divergence between them can be determined analytically. Specifically, in the context of the present invention, the KL divergence of the input and target is calculated as follows:

[0049] The regularized upper bound for the input x is

[0050]

[0051] The regularized upper bound for the target y is Its KL divergence is:

[0052]

[0053] here, represents the upper bound of input regularization, and represents an upper bound on the target regularization. These formulas allow the present invention to compute the difference between two Gaussian distributions, where the mean and variance of one distribution may differ from the standard normal distribution N(0,1).

[0054] by Taking as an example, the derivation of the above analytical solution is as follows:

[0055]

[0056] S2 instantiates I(Z;X,Y).

[0057] According to the mutual information definition between three random variables, the present invention can express I(Z; X, Y) as

[0058] I(Z;X,Y)=H(Z)-H(Z|X,Y)

[0059] Here, H(Z) is the entropy of Z, and H(Z|X,Y) is the conditional entropy of Z given X and Y. Therefore, the goal of the present invention is to minimize the entropy overlap of Z relative to X and Y. This can be achieved by reducing the entropy overlap between Z and X. Under the given Markov chain condition XZY, the present invention can reduce the entropy overlap between Z and X by data reparameterization. Specifically, the reparameterized data is used as the input to the MLP encoder. The encoder maintains the same network structure and parameters as when instantiating I(X;Y;Z). Subsequently, the posterior distribution Parameterized by the encoding process, expressed as Encoding z = μ z +σ z ∈ is obtained by reparameterization, where z belongs to Z and ∈~N(0,1) is the noise sampled from the standard normal distribution.

[0060] In addition, the present invention applies representation regularization The object of the present invention is achieved by assuming that the prior distribution of Z is an independent and identically distributed unit Gaussian distribution N(0,1). The upper bound analytical solution of is given by:

[0061] The upper bound of the regularization can be obtained by the following expression:

[0062]

[0063] Where Q(Z) is an independent and identically distributed unit Gaussian distribution N(0,1), representing the assumed prior Gaussian distribution of Q(Z). At the same time, the present invention can obtain the analytical solution of this upper bound:

[0064]

[0065] here, This formula allows the present invention to calculate the upper bound of the regularization. The Kullback-Leibler divergence between the posterior distribution of Z and the standard normal distribution under the condition of .

[0066] S3 instantiates I(Z;Y). Given an explicit reparameterization of Y, we obtain The object of the present invention is to optimize However, direct calculation Therefore, the present invention introduces the following expression to provide an approximate lower bound:

[0067] The variational lower bound of can be derived and approximated by minimizing the standard regression loss while being free from the Markov assumption ZXY as follows:

[0068]

[0069] The above formula is specifically derived as follows:

[0070]

[0071] let Become difficult to handle Variational approximation of . According to the non-negativity of KL divergence, the present invention has:

[0072]

[0073] Therefore, the inequality simplifies to:

[0074]

[0075] Due to entropy Independently of optimization, the present invention can be implemented through maximization.

[0076]

[0077] Specifically, this step is implemented through a regression layer, which obtains Y based on Z S Here Y SRepresents the predicted value based on Z. In order to maximize the variational lower bound, the present invention adopts a standard regression loss, such as Mean Absolute Error (MAE).

[0078] Beneficial Effects

[0079] The present invention effectively reduces the negative impact of spatiotemporal data noise on prediction input and target by introducing the robust spatiotemporal information bottleneck (RSTIB) principle, thereby improving the accuracy of prediction.

[0080] By reducing the impact of noise on feature variance, the present invention helps to improve the stability of the model to data features, making the performance of the model more consistent on different data sets.

[0081] The RSTIB principle of the present invention allows the model to relax its reliance on the Markov assumption without compromising the nature of the information bottleneck (IB), so that the model can process and optimize information more effectively.

[0082] The present invention provides an instantiation of a model for spatiotemporal graph prediction under theoretical guidance on a multi-layer perceptron, providing a new approach for robust spatiotemporal modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Figure 2 is a diagram of the training information flow of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be further described below in conjunction with the accompanying drawings.

[0085] Specifically, the present invention discloses a method that can maintain robust and efficient spatiotemporal graph prediction in a noisy environment, and the process is strictly theoretically guaranteed. Compared with the traditional method of ensuring a robust time series graph prediction process by building a complex structure, the present invention is based on a multi-layer perceptron structure and can maintain better efficiency in some larger data sets.

[0086] Robust Spatial-Temporal Information Bottleneck (RSTIB) Theoretical Derivation: The present invention first makes an assumption about the noise model, assuming that the noise in the data follows the additive white Gaussian noise (AWGN) model. This helps to simplify the analysis process. Accordingly, the present invention defines X as input data, Y as output data, and Z as the compressed representation of the intermediate layer.

[0087] Specific implementation of the robust spatiotemporal graph information bottleneck on a multi-layer perceptron: First, according to the RSTIB principle derived in S1, the present invention can deduce that the specific instantiation process in S2 is divided into three links, namely, instantiation I(X; Y; Z), instantiation I(Z; X, Y) and instantiation I(Z, Y). In the following specific links, the present invention will specifically explain how to implement the present invention:

[0088] The detailed steps are as follows:

[0089] Instantiation I(X; Y; Z): The present invention first uses a multi-layer perceptron (MLP) to parameterize the posterior distribution. Specifically, the present invention uses MLP to learn the conditional distribution of the intermediate variable Z given the input data X. This distribution is assumed to be a Gaussian distribution, expressed as where μ z and are the mean and variance of the distribution respectively. Subsequently, the present invention needs to reparameterize the data of both X and Y. According to the description of the invention, in order to ensure that the dimensions of the input X and the target Y are consistent, the present invention introduces two fully connected layers to process X and Y respectively. This enables the present invention to parameterize two Gaussian distributions: and The two distributions are parameterized by means of a fully connected layer. and Keep the original data information, each item is:

[0090]

[0091] According to the obtained distribution, the present invention can obtain new data by data reparameterization. Each And ∈~N(0,1) is the noise sampled from the standard normal distribution. According to the content of the invention, the present invention needs to optimize a specific regularization term in a specific loss function to learn the parameters of the re-parameterization process of the present invention. The present invention optimizes the parameter by minimizing the formula of the specific analytical solution. Accordingly, the specific implementation process of instantiating I(X; Y; Z) has been completed.

[0092] Instantiation I(Z;X,Y): reparameterized data is used as the input of the above MLP encoder in S2-1. The encoder maintains the same network structure and parameters as in S2-1. Subsequently, the posterior distribution Parameterized by the encoding process, expressed as Encoding z = μ z +σ z∈ is obtained by reparameterization, where z belongs to Z, and ∈~N(0,1) is the noise sampled from the standard normal distribution. According to the content of the invention, the present invention can also apply a new regularization term to the intermediate layer representation part. The regularization term is also an upper bound of an information term, and the present invention optimizes the loss term of the present invention by minimizing the analytical expression of the upper bound.

[0093] Instantiation I(Z;Y): The specific implementation of this part is mainly carried out through a regression function. Specifically, this step is implemented through a regression layer, which obtains Y based on the intermediate expression Z S Here Y S represents the predicted value based on Z. The present invention uses the standard regression loss and the mean absolute error (MAE). The target value to be fitted in the present invention is obtained by explicitly reparameterizing Y.

[0094] Application Cases: The present invention can be used for downstream scenarios that can be constructed as time series diagram predictions, such as traffic flow prediction, weather prediction, etc. For example, the present invention can be implemented on a simple baseline method based on the MLP model, and the improvement in robustness is evaluated by applying noise of different noise intensities. The robustness evaluation method is based on the mean absolute error (MAE), that is, the MAE performance of the model prediction under different noise conditions. The following considers the example of traffic flow prediction and the example of weather prediction respectively: for the downstream scenario of traffic flow prediction, consider the PEMS04 public dataset; for the downstream scenario of weather prediction, consider the Weather2K-R public dataset:

[0095] PEMS04 dataset:

[0096] Noise intensity = 10% Noise intensity = 30% Noise intensity = 50% Baseline Method 24.83 29.53 29.82 Baseline method + present invention 23.87 27.44 27.55

[0097] Weather2K-R Dataset:

[0098] Noise intensity = 10% Noise intensity = 30% Noise intensity = 50% Baseline Method 4950.03 6301.86 7654.50 Baseline method + present invention 4593.97 6000.52 7200.69

[0099] Spatiotemporal graph prediction plays a key role in urban planning and computing. Although spatiotemporal graph neural networks (STGNNs) have performed well in modeling spatiotemporal dynamics, they are often affected by relatively low computational efficiency. Recently, multilayer perceptrons (MLPs) have received widespread attention in spatiotemporal prediction due to their simplified architecture and better efficiency. However, existing MLP-based models may be susceptible to noise interference, especially when the noise can affect the input and target sequences of noisy data in spatiotemporal prediction. In order to mitigate this effect, the present invention proposes a robust temporal graph prediction method applied to multilayer perceptrons based on information bottleneck theory. The present invention extends the previous information bottleneck (IB) method by relaxing specific Markov assumptions without compromising the essence of IB. Subsequently, by showing that irrelevant noise information is minimized, the representation learning guided by the theory can be more robust against noise interference. In addition, the specific instantiation of the present invention can be implemented on MLP, thereby achieving efficient and robust spatiotemporal modeling. The present invention can be used in noisy environments to provide superior prediction accuracy, and at the same time has good robustness and efficiency.

Claims

1. A robust timing graph prediction method based on information bottleneck theory and applied to multi-layer perceptron, characterized in that: Through the assumption XZY that the DVIB model follows by default, after relaxing the assumption ZXY, the new noise information term I(Z; Y|X) is reconsidered, while considering the information term I(X; Z|Y): Reconstructing the conditional mutual information, the sum of I(Z; Y|X) and I(X; Z|Y) is reconstructed as: I(Z;Y∣X)+I(Z;X∣Y)=I(Z;X,Y)-I(X;Y;Z) According to the definition, I(X;Y;Z)=I(X;Y)-I(X;Y|Z), that is: I(Z;Y∣X)+I(Z;X∣Y)=I(Z;X,Y)-I(X;Y;Z) The corresponding RSTIB principles are constructed as follows: minL RSTIB =-I(Z,Y)+β1×I(Z;X,Y)-β2×I(X;Y;Z) β1,β2≥0 represent the corresponding Lagrange multipliers; By applying the above reconstruction, the interference of the noise information Z described by H(Y|X) and H(X|Y) on the analysis result is reduced.

2. The method according to claim 1, characterized in that The specific implementation of the robust spatiotemporal graph information bottleneck on the multi-layer perceptron is divided into three steps, namely, instantiating I(X; Y; Z), instantiating I(Z; X, Y) and instantiating I(Z; X, Y); The specific steps include: Instantiation I(X; Y; Z): First, a multi-layer perceptron (MLP) is used to parameterize the posterior distribution. MLP is used to learn the conditional distribution of the intermediate variable Z given the input data X. The distribution is assumed to be a Gaussian distribution and is expressed as where μ z and are the mean and variance of the distribution respectively; Reparameterize the data of X and Y, introduce two fully connected layers, process X and Y respectively, and parameterize two Gaussian distributions and The two distributions are parameterized by means of a fully connected layer; By building and Keep the original data information, each item is: According to the obtained distribution, data reparameterization is used to obtain new data Each and ∈~N(0,1) is the noise sampled from a standard normal distribution; Optimize a specific regularization term in a specific loss function, learn the parameters of the re-parameterization process, and optimize the parameters by minimizing the formula of the specific analytical solution; Instantiation I(Z;X,Y): reparameterized data is used as the input to the MLP encoder, which keeps the same network structure and parameters as when instantiating I(X;Y;Z); Then, the posterior distribution Parameterized by the encoding process, expressed as Encoding z = μ z + σ z ∈ is obtained by reparameterization, where z belongs to Z and ∈~N(0,1) is the noise sampled from the standard normal distribution; A new regularization term is applied to the intermediate layer representation part. The regularization term is an upper bound of the information term, and the loss term is optimized by minimizing the analytical expression of the upper bound. Instantiation I(Z;Y): Mainly through the regression function, this step is specifically implemented through a regression layer. This layer is based on the intermediate expression Z to obtain Y S , here Y S represents the predicted value based on Z; Using standard regression loss, mean absolute error, the fitted target value is obtained after explicit reparameterization of Y

Images