Multivariable time sequence completion method based on causal diffusion model

Abstract

The invention discloses a multivariable time sequence completion method based on a causal diffusion model. A normal form which regards a time sequence as a nonlinear noise equation is adopted. By utilizing variation lower bound maximization, the causal structure embedded in the data can be effectively revealed. Then, the learned causal attention features are taken as priori knowledge to be combined with a diffusion network so as to fill missing data. Causal features help determine key determinants that affect data distribution. They also mitigate the adverse effects of false associations caused by confounding factors. In addition, the diffusion model with the accuracy advantage in the generative model is selected, and the problems of collapse, low precision and the like observed in other generative network models are effectively solved. Compared with other seven baseline methods, three data sets in the field of the Internet of Things are subjected to comparative evaluation and result display, and in a data interpolation task, compared with the most advanced method, the method has the advantages that the performance is excellent, and the method can play a role in the problem of data missing.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence of the Internet of Things, and more specifically, to a multivariate time series completion method based on a causal diffusion model. Background Art

[0002] As a numerical expression of the long-term changes in a statistical variable, time series has been widely used in real-world scenarios involving data measurement, such as financial forecasting, meteorological science, healthcare, and wireless communications. However, in the Internet of Things and wireless communication systems, various reasons, including collection terminal failures, unstable deployment environments, and human operational errors, can cause interruptions in the data collection process or missing data values. This not only undermines the interpretability of the data and hinders the effective analysis of time series data, but also poses challenges to higher-level downstream tasks such as prediction, classification, and clustering. For example, in the field of wireless communications, wireless spectrum resources change periodically over time, and spectrum resources need to be dynamically allocated according to time. Therefore, a complete time series is an important condition for spectrum prediction, and it is particularly important to use efficient methods to deal with missing data.

[0003] The time series collected in the application scenarios have their own characteristics and properties. Relatively complete data is not only conducive to fully understanding the trend, periodicity and causality within the time series in practical applications such as medicine and finance, but also of great help in the in-depth analysis of the correlation between sequences, which is very important for improving the performance of downstream tasks such as prediction and classification.

[0004] When collecting data, traditional processing methods for missing data caused by various unstable factors are divided into two categories: one is deletion, that is, deleting some observed samples or features. However, direct deletion will make the data incomplete, and is generally only applicable to a small number of scenarios that do not require high data integrity. In most application scenarios, we need to fill the data to keep the data complete. Direct deletion of data is likely to delete data that is helpful for analysis, which will reduce the expected performance. The other is data interpolation, that is, estimating missing data from observed values, and restoring the original data by analyzing the data before and after the missing point and the overall characteristics to find a certain change pattern. Compared with directly deleting part of the data, interpolation can maintain the integrity of the data, fully consider the characteristics of the original data in specific applications, and meet the completeness and rationality of data analysis requirements. Summary of the invention

[0005] This paper considers time series as a paradigm of nonlinear noise equations. By maximizing the variational lower bound, it focuses on revealing the causal structure embedded in the data. The learned causal attention features are then combined with the diffusion network as prior knowledge to fill in the missing data. It not only uses causal features to determine the key determinants affecting the data distribution, but also mitigates the adverse effects of spurious correlations caused by confounding factors. At the same time, the diffusion model with learning accuracy as an advantage is selected, which effectively solves the problems of collapse and low accuracy observed in other generative network models.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A multivariate time series completion method based on causal diffusion model, such as Figure 1 As shown, the following steps are included:

[0008] S1: Use sensors to collect time series data with missing data;

[0009] S2: Design a mask matrix based on whether each sampling point is missing. The mask matrix is ​​a 0 / 1 matrix. If the data is missing, it is 0, and if it is successfully sampled, it is 1. The matrix dimension is the same as the time series data.

[0010] S3: Consider standardizing or normalizing the sampled data according to the different requirements of the collected data in different scenarios;

[0011] S4: Input the processed data into the causal discovery network to obtain the causal prior matrix;

[0012] S5: The causal prior matrix is ​​used as a weight matrix to represent the causal relationship. It is multiplied by the sliding window with the collected raw data to obtain the causal prior features required for training.

[0013] S6: When the missing data cannot obtain the accurate true value, part of the observed data is also masked as missing points, and the mean square error between the observed data value and its interpolated value will be used as the loss function of the network;

[0014] S7: The causal prior features are used as the conditional knowledge of the diffusion model and are input into the diffusion network together with the collected sequence and mask matrix;

[0015] S8: Use Adam to start training the diffusion network. After the training is completed, the complete data restored by the generation network is obtained, completing the data interpolation task.

[0016] Preferably, the nth sequence of the N multivariate time series data sampled in step S1 can be expressed as:

[0017]

[0018] Among them, 1≤t≤T is the time step and D is the feature. represents the D-dimensional features sampled at time t, where 1≤i≤D.

[0019] Preferably, the mask matrix in step S2 is M (n) , which can be specifically expressed as:

[0020]

[0021] Preferably, the standardization or normalization process in step S3 can directly use conventional processing methods.

[0022] Preferably, the causal discovery method of step S4 is based on a functional causal model and uses Granger causality representation to represent the time series as a nonlinear autoregressive function. The causal relationship between variables and historical observation noise is also considered. The causal model can be expressed by a nonlinear autoregressive equation:

[0023]

[0024] Among them, Pa i represents the parent of node i, f i is a nonlinear differentiable function that captures the causal relationship between variables. Represents historical observation noise, which is generated by combining stationary Gaussian noise and historical observation data.

[0025] The causal matrix required is expressed as G τ , in order to τ In accordance with the causal relationship in i Defined as:

[0026]

[0027] [G τ ] (j,i) express arrive A causal relationship will be established, and i,j∈{1,2,…,D}. In addition, ψ i and Both represent trainable network parameters.

[0028] We use the variational distribution q φ (G) to approximate the true causal matrix distribution p(G|X (1) ,...,X (N) ). Specifically expressed as:

[0029]

[0030] where λ is a trainable network parameter. In the above equation, p(G) represents the prior distribution of the causal structure, subject to the constraints of sparsity, aperiodicity, and empirical knowledge. φ (G)) is q φ The cross entropy of (G). The right side of the equation represents the evidence lower bound (ELBO). By transforming the nonlinear autoregressive equation and replacing the noise data, the approximate distribution of the causal matrix can be obtained by maximizing the ELBO.

[0031] Preferably, in step S5, given that the inherent causal relationship of the data is transitive, we only need to learn the causal relationship of three consecutive time steps.

[0032] like Figure 2 As shown, specifically, we specify the parent node of the feature at time τ as the eigenvalues ​​of the data at time τ-2, τ-1, and τ, taking into account the instantaneous causal relationship (data at time τ). This constraint effectively reduces the redundant information in the causal structure and reduces the computational complexity of causal discovery. Therefore, each multivariate time series produces a fixed three-dimensional causal prior tensor G with a dimension of (3, D, D). Here, "3" represents three time steps τ-2, τ-1, τ. G τ-2 , G τ-1 and G τ The causal matrix between the data features corresponding to the three time points has a dimension of D×D.

[0033] We apply a time sliding window of scale three, moving one unit at a time in the range of (2, T), process the data in each window in turn, and use a simple fully connected layer to assign weights to the three causal matrices. The final causal prior feature is expressed as:

[0034]

[0035] Preferably, the loss function of step S6 is specifically calculated as follows:

[0036]

[0037] Where Ω (n) represents a set of indices corresponding to the missing items that need to be calculated. We use the masking matrix M (n) to distinguish observed values ​​from missing values. When , it means that the data set point is contained in Ω (n) , will participate in the calculation of the loss function.

[0038] The diffusion model that follows consists of two processes: a forward diffusion process, which systematically introduces noise into the initial sequence, gradually distorts it to generate a distorted data sequence. A reverse sampling process, which starts with fully distorted data, systematically samples to remove noise, and finally recovers the original data distribution. To recover the complete data, I will train the network to fit the directional sampling process.

[0039] Preferably, the specific diffusion steps of S7-S8 can be expressed as:

[0040]

[0041] where the diffusion process steps are represented as 1≤s≤S, and Obeys a standard Gaussian distribution. θ is a trainable parameter of the sampling network, This means that the sampled data follows the mean The variance is Gaussian distribution. By chaining inverse sampling from the D step, we finally get The complete sampling data to be restored can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the overall model structure diagram of the present invention.

[0043] Figure 2 Generate a structure diagram for the causal prior features proposed in the present invention.

[0044] Figure 3 The method of the present invention provided in the embodiment is compared with other methods in terms of probability interpolation accuracy on the PhysioNet dataset at a missing rate of 50%.

[0045] Figure 4 The method of the present invention provided in the embodiment is compared with other methods in terms of probability interpolation accuracy when the missing rate is 70% on the PhysioNet dataset.

[0046] Figure 5 The method of the present invention provided in the embodiment is used on the Guangzhou dataset, and a visual comparison chart of the probability interpolation accuracy with other methods when the missing rate is 10%.

[0047] Figure 6 The method of the present invention provided in the embodiment is used on the Guangzhou dataset, and a visual comparison chart of the probability interpolation accuracy with other methods when the missing rate is 30%. DETAILED DESCRIPTION

[0048] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0049] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0050] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] Example 1

[0053] This example provides a multivariate time series completion method based on the causal diffusion model. Figure 1 As shown, the following steps are included:

[0054] S1: Use sensors to collect time series data with missing data;

[0055] S2: Design a mask matrix based on whether each sampling point is missing. The mask matrix is ​​a 0 / 1 matrix. If the data is missing, it is 0, and if it is successfully sampled, it is 1. The matrix dimension is the same as the time series data.

[0056] S3: Consider standardizing or normalizing the sampled data according to the different requirements of the collected data in different scenarios;

[0057] S4: Input the processed data into the causal discovery network to obtain the causal prior matrix;

[0058] S5: The causal prior matrix is ​​used as a weight matrix to represent the causal relationship. It is multiplied by the sliding window with the collected raw data to obtain the causal prior features required for training.

[0059] S6: When the missing data cannot obtain the accurate true value, part of the observed data is also masked as missing points, and the mean square error between the observed data value and its interpolated value will be used as the loss function of the network;

[0060] S7: The causal prior features are used as the conditional knowledge of the diffusion model and are input into the diffusion network together with the collected sequence and mask matrix;

[0061] S8: Use Adam to start training the diffusion network. After the training is completed, the complete data restored by the generation network is obtained, completing the data interpolation task.

[0062] Example 2

[0063] First, data sampling is required. The nth sequence of the N multivariate time series data can be expressed as:

[0064]

[0065] Among them, 1≤t≤T is the time step and D is the feature. represents the D-dimensional features sampled at time t, where 1≤i≤D.

[0066] The mask matrix in step S2 is M (n) , which can be specifically expressed as:

[0067]

[0068] The standardization or normalization process in step S3 can directly use conventional processing methods.

[0069] The causal discovery method in step S4 is based on the functional causal model and uses Granger causality representation to represent the time series as a nonlinear autoregressive function. The causal relationship between variables and historical observation noise is also considered.

[0070] The causal model can be expressed as a nonlinear autoregressive equation:

[0071]

[0072] Among them, Pa i represents the parent of node i, f i is a nonlinear differentiable function that captures the causal relationship between variables. Represents historical observation noise, which is generated by combining stationary Gaussian noise and historical observation data.

[0073] The causal matrix required is expressed as G τ , in order to τ In accordance with the causal relationship in i Defined as:

[0074]

[0075] [G τ ] (j,i) express arrive A causal relationship will be established, and i,j∈{1,2,…,D}. In addition, ψ i and Both represent trainable network parameters.

[0076] We use the variational distribution q φ (G) to approximate the true causal matrix distribution p(G|X (1) ,...,X (N) ). Specifically expressed as:

[0077]

[0078] where λ is a trainable network parameter. In the above equation, p(G) represents the prior distribution of the causal structure, subject to the constraints of sparsity, aperiodicity, and empirical knowledge. φ (G)) is q φ The cross entropy of (G). The right side of the equation represents the evidence lower bound (ELBO). By transforming the nonlinear autoregressive equation and replacing the noise data, the approximate distribution of the causal matrix can be obtained by maximizing the ELBO.

[0079] In step S5, given the transitive nature of the causal relationships inherent in the data, we only need to learn the causal relationships of three consecutive time steps.

[0080] Specifically, we specify the parent node of the feature at time τ as the eigenvalue of the data at time τ-2, τ-1, and τ, taking into account the instantaneous causal relationship (data at time τ). This constraint effectively reduces the redundant information in the causal structure and reduces the computational complexity of causal discovery. Therefore, each multivariate time series produces a fixed three-dimensional causal prior tensor G with a dimension of (3, D, D). Here, "3" represents three time steps τ-2, τ-1, τ. G τ-2 , G τ-1 and G τ The causal matrix between the data features corresponding to the three time points has a dimension of D×D.

[0081] We apply a time sliding window of scale three, moving one unit at a time in the range of (2, T), process the data in each window in turn, and use a simple fully connected layer to assign weights to the three causal matrices. The final causal prior feature is expressed as:

[0082]

[0083] Preferably, the loss function of step S6 is specifically calculated as follows:

[0084]

[0085] Where Ω (n) represents a set of indices corresponding to the missing items that need to be calculated. We use the masking matrix M (n) to distinguish observed values ​​from missing values. When , it means that the data set point is contained in Ω (n) , will participate in the calculation of the loss function.

[0086] The diffusion model that follows consists of two processes: a forward diffusion process, which systematically introduces noise into the initial sequence, gradually distorts it to generate a distorted data sequence. A reverse sampling process, which starts with fully distorted data, systematically samples to remove noise, and finally recovers the original data distribution. To recover the complete data, I will train the network to fit the directional sampling process.

[0087] The specific diffusion steps of S7-S8 can be expressed as:

[0088]

[0089]

[0090] where the diffusion process steps are represented as 1≤s≤S, and Obeys a standard Gaussian distribution. θ is a trainable parameter of the sampling network, This means that the sampled data follows the mean The variance is Gaussian distribution. By chaining inverse sampling from S steps, we finally get The complete sampling data to be restored can be obtained.

[0091] Example 3

[0092] This embodiment provides the following specific embodiments based on Embodiment 1 and Embodiment 2:

[0093] We validated our method on three IoT datasets: PhysioNet, Guangzhou, and Electricity datasets.

[0094] PhysioNet consists of 12,000 multivariate clinical time series, each with 37 variables, all from the monitoring data of patients in the intensive care unit (ICU) for 48 hours, so the dataset is processed into a time series of 48 time steps. The processed dataset contains about 80% missing values. Since there are no real data values ​​for missing data, we randomly select 10% of the observations as the original data to evaluate the model interpolation performance.

[0095] Guangzhou is a traffic speed record for Guangzhou. The data consists of sensor data from 214 anonymous road segments, collected every 10 minutes over a two-month period (from August 1, 2016 to September 30, 2016, a total of 61 days). The original missing rate of this dataset is 1.29%. Compared with the other two datasets, it is considered to be representative due to its small sample size. We divide the data into training set, validation set, and test set in a ratio of 0.7:0.2:0.1, respectively.

[0096] Electricity is a well-known public dataset in UCI. It features electricity consumption data (kWh) collected from 370 customers every 15 minutes. The dataset contains 48 months with no missing data and is split into sequences every 100 time steps. In addition, we divide the first 10 months of data into a test set, the subsequent 10 months of data into a validation set, and the remaining data into a training set. To verify the imputation performance, we introduced artificial missing rates ranging from 10% to 90% in the training, validation, and test sets.

[0097] In the experiments, we set the batch size to 16 for the PhysioNet and Guangzhou datasets and to 8 for the Electricity dataset. The learning rate is fixed to 1e-3. We observed that the loss function of the validation set converged after 100 epochs. Therefore, we implemented an early stopping strategy to terminate the training after 100 epochs. In addition, the model construction was performed using PyTorch and the training process used the Adam optimizer.

[0098] The proposed CDITS is compared with median filling (Median), mean filling (Mean), and five advanced completion methods based on deep learning, namely BRITS, E2GAN, GP-VAE, SAITS and CSDI. The results are shown in Table 1, showing the performance of the proposed method CDITS in completing interpolation. For clarity, the best performance of each dataset is indicated in bold.

[0099] Table 1: Performance comparison of interpolation methods on three datasets. The reported metrics are MAE / RMSE.

[0100]

[0101] As shown in Table 1, the proposed CDITS achieves breakthrough results on three datasets. Among them, the missing rate of the PhysioNet dataset is 80%, which poses a challenge to the interpolation task. CDITS shows a significant performance advantage over SAITS, with both indicators dropping by about 10%. This is a noteworthy breakthrough, confirming the practicality of CDITS in handling interpolation tasks for sequences with high missing probabilities.

[0102] Guangzhou has a significantly reduced sample size compared to the other two datasets. We aggregate data from 50 consecutive time points into a single sample while maintaining the feature dimension at 214 in order to increase the sample size and save resources within limited data. The study found that in training with small sample data, the performance gap between SAITS and CSDI is negligible, and their performance is very similar. In contrast, CDITS achieved a breakthrough of RMSE 4.0, reducing it to around 3.8. In addition, MAE was also improved. This affirms the practicality of CDITS in the field of transportation. In the future, CDITS can be extended to a causal learning model that integrates time and space dimensions to handle longer and more complex traffic datasets.

[0103] Electricity imputation is a very challenging task due to the fact that the temporal and feature dimensions exceed 100. The inference results on the Electricity dataset show that CDITS performs best among the baseline methods when using datasets with dimensions greater than 100. This emphasizes the competitive advantage of the proposed model CDITS in handling long data sequences.

[0104] The confidence of the proposed method was verified on two datasets. The confidence comparison between CDITS and CSDI on the PhysioNet dataset is shown in the figure below. Figure 3-Figure 4 As shown in Figure 2, the confidence comparison on the Guangzhou dataset is as follows: Figure 5-Figure 6 As shown. The median is represented by the line in the picture, and the interpolated 95% confidence interval is represented by the shade. As can be seen from the figure, CDITS excels in achieving more accurate outlier estimates while covering a narrower probability range when outliers exist in the data. At the same time, CDITS not only establishes more reasonable confidence intervals for two specific data points, but also shows higher accuracy in capturing the dynamic trend of data changes.

[0105] The above embodiments show that the proposed metric can produce better processing results when the sampling rate is not high enough, thereby achieving better detection performance.

[0106] The same or similar reference numerals correspond to the same or similar components;

[0107] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0108] Obviously, the above examples of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multivariate time series completion method based on causal diffusion model, characterized in that: The following steps are involved: S1: Use sensors to collect time series data with missing data; S2: Design a mask matrix based on whether each sampling point is missing. The mask matrix is ​​a 0 / 1 matrix. If the data is missing, it is 0, and if it is successfully sampled, it is 1. The matrix dimension is the same as the time series data. S3: Consider standardizing or normalizing the sampled data according to the different requirements of the collected data in different scenarios; S4: Input the processed data into the causal discovery network to obtain the causal prior matrix; S5: The causal prior matrix is ​​used as a weight matrix to represent the causal relationship. It is multiplied by the sliding window with the collected raw data to obtain the causal prior features required for training. S6: When the missing data cannot obtain the accurate true value, part of the observed data is also masked as missing points, and the mean square error between the observed data value and its interpolated value will be used as the loss function of the network; S7: The causal prior features are used as the conditional knowledge of the diffusion model and are input into the diffusion network together with the collected sequence and mask matrix; S8: Use Adam to start training the diffusion network. After the training is completed, the complete data restored by the generation network is obtained, completing the data interpolation task.

2. The multivariate time series completion method based on the causal diffusion model according to claim 1 is characterized in that: In step S1, the nth sequence of the N multivariate time series data sampled can be expressed as: Among them, 1≤t≤T is the time step and D is the feature. represents the D-dimensional features sampled at time t, where 1≤i≤D.

3. The multivariate time series completion method based on the causal diffusion model according to claim 2 is characterized in that: The mask matrix in step S2 is M (n) , which can be specifically expressed as:

4. The multivariate time series completion method based on the causal diffusion model according to claim 3 is characterized in that: The causal discovery method of step S4 is based on the functional causal model, specifically: S4.1: The causal model can be expressed as a nonlinear autoregressive equation: Among them, Pa i represents the parent of node i, f i is a nonlinear differentiable function that captures the causal relationship between variables; Represents historical observation noise, which is the causal matrix required to generate the required causal matrix by combining stationary Gaussian noise and historical observation data; S4.2: for G τ , in order to τ In accordance with the causal relationship in i Defined as: [G τ ] (j,i) express arrive A causal relationship will be established, and i,j∈{1,2,…,D}. In addition, ψ i and All represent trainable network parameters; S4.3: We use the variational distribution q φ (G) to approximate the true causal matrix distribution p(G|X (1) ,...,X (N) ), specifically expressed as: where λ is a trainable network parameter; in the above equation, p(G) represents the prior distribution of the causal structure, subject to the constraints of sparsity, aperiodicity, and empirical knowledge. φ (G)) is q φ The cross entropy of (G); the right side of the equation represents the evidence lower bound (ELBO); by transforming the nonlinear autoregressive equation and replacing the noise data, the approximate distribution of the causal matrix can be obtained by maximizing the ELBO.

5. The multivariate time series completion method based on the causal diffusion model according to claim 4 is characterized in that: In step S5, given the transitive nature of the causal relationships inherent in the data, we only need to learn the causal relationships of three consecutive time steps. Specifically, we specify the parent node of the feature at time τ as the feature value of the data at time τ-2, τ-1, and τ, taking into account the instantaneous causal relationship (data at time τ). This constraint effectively reduces the redundant information in the causal structure and reduces the computational complexity of causal discovery. Therefore, each multivariate time series will produce a fixed three-dimensional causal prior tensor G with a dimension of (3, D, D). Here, "3" represents three time steps τ-2, τ-1, τ. G τ-2 , G τ-1 and G τ The causal matrix between the data features corresponding to the three time points has a dimension of D×D; We apply a time sliding window of scale three, moving one unit at a time in the range of (2, T), process the data in each window in turn, and use a simple fully connected layer to assign weights to the three causal matrices. The final causal prior feature is expressed as:

6. The multivariate time series completion method based on the causal diffusion model according to claim 5 is characterized in that: The loss function of step S6 is specifically calculated as follows: Where Ω (n) Representing a set of indices corresponding to the missing items that need to be calculated, we use the masking matrix M (n) to distinguish observed values ​​from missing values. When , it means that the data set point is contained in Ω (n) , will participate in the calculation of the loss function.

7. The multivariate time series completion method based on the causal diffusion model according to claim 6 is characterized in that: The diffusion model consists of two processes: a forward diffusion process, which systematically introduces noise into the initial sequence, gradually changes it to generate a distorted data sequence. A reverse sampling process, which starts with fully distorted data, systematically samples to remove noise, and finally recovers the original data distribution. To recover the complete data, I will train the network to fit the directional sampling process. The specific diffusion steps of S7-S8 can be expressed as: where the diffusion process steps are represented as 1≤s≤S, and Obeys a standard Gaussian distribution. θ is a trainable parameter of the sampling network, This means that the sampled data follows the mean The variance is Gaussian distribution. By chaining inverse sampling from S steps, we finally get The complete sampling data to be restored can be obtained.

Images