Generative adversarial network-based time sequence continuous missing value interpolation method

Abstract

The invention discloses a time sequence continuous missing value interpolation method based on a generative adversarial network, and belongs to the field of control science and computer application. According to the method, a context prompt information matrix is introduced into a generator of a generative adversarial network, potential data evolution characteristics hidden outside a missing position are mined and captured, a generation result of the generator is restrained to be close to real data probability distribution, and neural network structures of the generator and a discriminator and a loss function calculation strategy are optimized; and constructing a training sample required during training of the generative adversarial network, and constructing a generator model and a discriminator model of the generative adversarial network: inputting the training sample into the generator model and the discriminator model, and carrying out iterative training to complete interpolation work of data. According to the method, local information feature loss is considered, an effective data interpolation method is provided, the method has the advantages of being good in interpolation effect, rapid and simple in operation, high in robustness, wide in application occasion and the like, and the inevitable continuous data loss problem in the time sequence field is solved.

Description

Technical Field

[0001] The present invention belongs to the field of control science and computer application, and relates to a method for interpolating continuous missing values ​​of a time series based on a generative adversarial network. An effective method is proposed for the common problem of missing continuous data in multivariate time series sampling. Background Art

[0002] Multivariate time series are multiple sets of data obtained by interval sampling. The analysis and modeling of multivariate time series can be used to analyze the internal change rules of the system, and have broad application prospects in many fields such as meteorology and economy. During the sampling process of multivariate time series, data are often missing continuously due to sensor failure, human error, equipment maintenance, communication interruption, etc. In this case, all variables of the data may be missing regularly or irregularly, and the length of the missing data is uncertain. The continuous missing data pattern is very common in research and industrial production. In order to ensure information integrity and subsequent analysis, the problem of continuous missing data needs to be solved.

[0003] The existing processing methods can be roughly divided into: interpolation methods based on statistical principles and machine methods based on neural networks. Among them, interpolation methods based on statistical principles rely on existing data distribution or prior knowledge, and estimate missing values ​​through statistical features, represented by multiple interpolation by chained equations (MICE); neural network-related methods are based on data-driven, including using autoencoders to express deep data features (such as VAE), using generative adversarial networks to generate more ideal data (such as GAIN), or using autoregressive models to learn the connection between sampling points (such as RNN and GRU related methods).

[0004] Generative models, such as VAE and GAIN, are widely used in time series data interpolation due to their good unsupervised properties. However, these methods are usually not optimized for the existing continuous missing problem, and are only tested under the ideal completely random missing condition, which lacks practical significance.

[0005] In order to obtain innovative data interpolation solutions that meet the needs, the data completion method needs to keep the original data pure and capture the correlation between data as much as possible, while reducing the computational complexity of the algorithm. In contrast, the GAIN interpolation method assumes the type of missing data. When dealing with continuous missing data interpolation, due to excessive loss of local data features, the generated data is too stable and tends to a constant value that minimizes the model's loss function. It lacks the dynamic change characteristics of the time series, and the generated data distribution deviates seriously from the real data distribution. In addition, since the generator and discriminator loss functions of the GAIN model are the same as those of the GAN model, the GAIN model will still have the same mode collapse problem caused by the loss function defects as the GAN model during the interpolation process. Therefore, it is necessary to establish a simple, effective, fast, lightweight method that can accurately interpolate continuous missing values ​​in multivariate time series. Summary of the invention

[0006] In order to solve the limitation and structural defect problems of GAIN model in processing continuously missing multivariate time series, the present invention proposes a method for interpolating continuous missing values ​​of time series based on generative adversarial network. In view of the problem that the loss function value cannot reach the minimum value and the model training is unstable in the process of interpolating missing data in GAIN model, the present invention proposes a novel unsupervised prompt Wasserstein generative adversarial interpolation network, and adds a gradient penalty method (GP), which is defined as prompt Wasserstein generative adversarial interpolation network (CWGAIN-GP). The present invention constrains the range of interpolation data generated by the generator by adding a prompt information matrix to the generator, thereby improving the generation accuracy of the generator; using W distance to measure the distance between the generated data and the real data, the problem of gradient disappearance can be effectively avoided; at the same time, a gradient penalty term is added to the discriminator, the loss function is limited to a certain range, and the discriminator parameters are prevented from going to the error extreme of the loss function becoming larger and larger, and the modeling ability of the model is enhanced. The experimental results on the real-world meteorological data set show (see the embodiment for details) that the present invention has excellent generalization ability and can greatly improve the accuracy of interpolating continuously missing time series.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for interpolating continuous missing values ​​in time series based on a generative adversarial network. The method first introduces a contextual cue information matrix into the generator of the generative adversarial network, mines and captures potential data evolution features hidden outside the missing position, constrains the generator's generation results to be close to the real data probability distribution, and then optimizes the generator and discriminator neural network structures and loss function calculation strategies, and constructs the training samples required for training the generative adversarial network, the generator model and the discriminator model of the generative adversarial network: the training samples are input into the generator model and the discriminator model, and iterative training is performed to complete the data interpolation work. The method includes the following steps:

[0009] Step 1: First, obtain the multivariate time series that needs to be interpolated, which is generally the data obtained from actual sampling, such as the change data of temperature and pollutant content over time. Then, convert the multivariate time series into a two-dimensional matrix, represented by the data matrix X. The number of rows d and columns s represent the number of sampling times and sampling variables respectively. Each row of data in X is a one-dimensional time series.

[0010] Step 2: Process the data matrix X in step 1 to generate training data. Specifically:

[0011] Before building the model, the data matrix X needs to be preprocessed and training data needs to be prepared. In order to distinguish the missing and non-missing parts of the data matrix X, first, a corresponding mask matrix M is generated based on the data matrix X. The dimension of the mask matrix M is the same as that of the data matrix X. When an element in X is zero or does not exist, the M matrix element corresponding to the position is "0", and the elements at other positions are all "1". Then, a random noise matrix N with the same dimension as the data matrix X is generated. The element n in the random noise matrix ij The value range is [-0.01, +0.01]. Finally, a contextual prompt information matrix T of the generator with the same dimension is generated according to the data matrix X. The T matrix is ​​shown in formula (1):

[0012]

[0013] Among them, t ij represents the i-th row and j-th column element of the contextual hint information matrix T; x ij represents the element in the i-th row and j-th column of the data matrix X; x i±l,j Represents the element in the i±lth row and jth column of the data matrix X; m ij represents the element in the i-th row and j-th column of the mask matrix M; l represents the length of consecutive missing elements.

[0014] In order to avoid the influence of different data scales on the interpolation effect, it is also necessary to perform normalization operation on each row of the data matrix X, that is, all data at the same sampling location, as shown in formula (2):

[0015]

[0016] Record the maximum and minimum values ​​of each row of the data matrix X for inverse normalization of the interpolation results. The normalized data matrix is ​​represented by X′, and the normalized data matrix X′ is used as training data.

[0017] Step 3: Use the training data to train the Generative Adversarial Interpolation Network CWGAIN-GP model, specifically:

[0018] In order to ensure the normal training of the generative adversarial interpolation network CWGAIN-GP model, it is necessary to extract the training data and input it into the generator. Extract the samples x', m, t and n for each processing from X', M, T and N. Perform matrix operations according to formula (3) to generate Then according to formula (4) and M are input into the generator to generate the result.

[0019]

[0020] Where ⊙ represents the multiplication of the elements at the corresponding positions of the matrix. The purpose of adding the random noise matrix N is to prevent the subsequent generator from generating data with large errors due to the existence of elements with values ​​of 0 in the matrix. G(·) is the generation process. The loss function of the generator is shown in formula (5):

[0021] Loss G =L G +αL MSE =▽ G [-(1-M)⊙D(Y)+αM⊙(X′-Y) 2 ] (5)

[0022] Where D(·) is the identification process; α is the generator hyperparameter; L G is the adversarial loss of the generator; L MSE is the mean square error loss between the estimated value Y and the true value X′; G [·] is the gradient of the loss with respect to the generator parameters.

[0023] Step 4: Introduce an additional regularization term, namely gradient penalty, into the discriminator D and discriminate the generated result y in step 3, specifically:

[0024] 4.1) Input the generated result y of step 3 into the discriminator D for discrimination. The discriminator of the CWGAIN-GP model does not only identify the truth or falsity of the normalized data matrix X′, but also makes a probability estimate for each element of X′, discriminates whether the element is real or generated, and finally generates a probability matrix. Each element in the probability matrix corresponds to the element at the same position in the normalized data matrix X′. The discrimination process is shown in formula (6):

[0025]

[0026] Among them, D * (y,m) i is the probability of the discriminator output, and p(y,m) is the joint probability distribution of the matrix (Y,M).

[0027] 4.2) Introducing the gradient penalty term ▽L GP This can avoid serious deviation of model parameters due to weight clipping during training, the disappearance of parameter update gradients, and ultimately the failure of model training. The loss function of the discriminator is shown in formula (7):

[0028]

[0029] in, It is convenient to solve the differential and use the interpolation data synthesized by real and generated data. ε is a random number between 0 and 1. P g and P r are the distributions of generated data samples and real data samples, respectively, λ is the weight of the penalty term (hyperparameter), ▽ is the differential factor of D(*), and χ is The probability distribution of L D is the basic loss of the discriminator; ▽L GP is the gradient penalty term; Denotes the generation distribution P g The output of the generated data Y sampled in the discriminator; D(X′) represents the probability matrix of the discriminator output for the real data X′; D(Y) represents the probability matrix of the discriminator output for the generated data Y; Represents the distribution from the true distribution P r The output of the real data X′ sampled in the discriminator; λ represents the weight coefficient of the gradient penalty term; Represents interpolation data Calculate the expected value of ; Represents interpolation data Compute the gradient of the discriminator output.

[0030] Step 5: Combine the discrimination results obtained in step 4 and Feedback to the generator and discriminator to update the model parameters. The loss function value is calculated using the discriminator's discrimination result to optimize the discriminator D and generator G parameters. The Adam optimizer is used in this invention. The objective function of the CWGAIN-GP model is shown in formula (8):

[0031]

[0032] in, Represents the mutual confrontation between the generator and the discriminator; Represents the generated data Y and interpolated data Calculate the expected value of .

[0033] Step 6: Generator returns the result After the generator generates data and outputs Y, Y is recalculated with the mask matrix M, and only the generated data is integrated into the vacant positions of the normalized data matrix X′ to prevent the real data in the normalized data matrix X′ from being destroyed. The calculation formula is shown in (9):

[0034]

[0035] in, is the final output result. The matrix is ​​inverse normalized to obtain the final multivariate time series interpolation results.

[0036] The beneficial effects of the present invention are:

[0037] (1) The present invention provides a method for interpolating continuous missing time series. Aiming at the problems of unstable training, slow convergence depth and low interpolation accuracy of current methods, a continuous missing time series interpolation model based on generative adversarial network is constructed. Aiming at the common continuous missing problem in time series data sampling, the present invention considers the loss of local information features and proposes an effective data interpolation method with the advantages of good interpolation effect, fast and simple operation, strong robustness, and wide application occasions, so as to solve the inevitable continuous data loss problem in the field of time series.

[0038] (2) Compared with the existing methods, this method fully considers the serious loss of local information characteristics of data caused by continuous missing values, has higher interpolation accuracy and better training stability, and the interpolated values ​​have practical application value. In addition, the present invention adopts GAIN as the deep learning network structure and combines the update strategy of artificial missing position mask to interpolate missing values. This method adopts a self-supervised learning strategy, does not rely on a complete data set, and has higher practicality and application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1This is a flow chart of a method for interpolating continuous missing values ​​in a time series based on a generative adversarial network according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the architecture of a method for interpolating continuous missing values ​​of a time series based on a generative adversarial network according to an embodiment of the present invention;

[0041] Figure 3 This is a heat map of the average Pearson correlation coefficient of data features in Beijing according to an embodiment of the present invention.

[0042] Figure 4 This is the result map for completing the temperature data for Beijing. DETAILED DESCRIPTION

[0043] In order to make the method problems solved by the present invention, the method solutions adopted and the method effects achieved clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It is also necessary to explain that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

[0044] 1. Data and operating environment

[0045] This example uses the UCI public real data set of air quality and meteorology in Beijing to illustrate. It manually sets 10%-50% of the data to be missing continuously, and the length of the data to be missing continuously is 100.

[0046] As an implementable method, in the Pycharm programming environment, the Pytorch deep learning framework is used to input the training samples into CWGAIN-GP for iterative training to complete the data interpolation work.

[0047] 2. Implementation steps

[0048] like Figure 1 As shown, a time series continuous missing value interpolation method based on a generative adversarial network includes the following steps:

[0049] Step 1: In this example, the Beijing area air quality and meteorological UCI public real data set is used, which contains 6 air quality indices, namely PM2.5, PM10, SO2, NO2, CO and O3, and 4 meteorological measurements, namely temperature (T), air pressure (P), dew point temperature (DEWP) and wind speed (WS). The data of all variables are recorded once an hour through the average value of all monitoring stations in the region, ranging from March 2013 to February 2017, with a total of 35,064 sets of data. This example visualizes the data features of the above data set, and illustrates the motivation of the feature correlation mechanism by drawing a heat map of the average Pearson correlation coefficient between any two feature vectors in the data set, such as Figure 3 As shown, it can be seen that the correlation between variables in Beijing is strong. The difference is that PM2.5, PM10, SO2, NO2 and CO are positively correlated, while O3, T, P, DEWP and WS are mostly negatively correlated. In order to facilitate the comparison of the completion effect, the data set selected in this example does not contain missing values. Different missing rates are set manually under the same continuous missing length, and the missing part is set to 0 to obtain the data matrix X, as shown in formula (10):

[0050]

[0051] Step 2: In this example, there are no invalid elements such as NaN in X, so there is no need to use 0 instead. Generate the corresponding mask matrix M based on the data matrix X. The dimensions of the two are the same. If an element in the data matrix X is missing, the element at the corresponding position in the mask matrix M is 0; if the element of the data matrix X is not missing, the element at the corresponding position in the mask matrix M is 1. The mask matrix M generated in this example is shown in formula (11):

[0052]

[0053] Then, each row of the data matrix X is normalized, as shown in formula (2). In this example, the normalized data matrix X′ is shown in formula (12):

[0054]

[0055] Record the maximum and minimum values ​​of each row of the data matrix X before normalization so that the result can be reverse normalized. max and X min Each contains 10 data, as shown in formula (13):

[0056]

[0057] Then, the training samples are extracted from the normalized data matrix X′ and the mask matrix M. The samples for each processing are divided from the normalized data matrix X′ and the mask matrix M. Also generates random small noise samples and random numbers Finally, extract the normalized data matrix X′ context as the prompt information matrix sample bats is the minimum length of batch data. In this example, the minimum length of batch data is 128.

[0058] Step 3: Perform matrix operations on the divided training samples and input them into the generator of the CWGAIN-GP model. In order to facilitate differential solution, the output result y of the generator is processed again, and the overall data is replaced by the synthetic data of the real and generated data, as shown in formula (14):

[0059]

[0060] Step 4: Input the generated result y of step 3 into the discriminator D for discrimination. The discrimination process is shown in formula (6).

[0061] Step 5: Use the Adam optimizer to update the parameters of the generator and discriminator. The optimization of the discriminator and generator is shown in equations (15) and (16):

[0062]

[0063]

[0064] Step 6: Set the number of iterations to 1000. In each iteration, use equations (14), (15), and (16) to input training samples and update the generator and discriminator parameters. After multiple iterations, the model will gradually converge. After the iteration is completed, the completion matrix is ​​reconstructed. This is the completed result. According to the normalization parameters recorded in formula (13), After reverse normalization, we can get the final completion result, as shown in formula (17). The execution process of the overall interpolation algorithm is:

[0065]

[0066]

[0067]

[0068] Finally, it should be noted that the above examples are only used to express the embodiments of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, it is possible for those skilled in the art to make several modifications and improvements without departing from the concept of the present invention. Various equivalent forms of modifications to the present invention all fall within the scope defined by the claims attached to this application and belong to the protection scope of the present invention.

Claims

1. A method for interpolating continuous missing values ​​in time series based on generative adversarial networks, characterized in that: The time series continuous missing value interpolation method first introduces the contextual cue information matrix into the generator of the generative adversarial network to mine and capture the potential data evolution characteristics hidden outside the missing position, and constrains the generation result of the generator to be close to the probability distribution of the real data; Then, optimize the generator and discriminator neural network structures and loss function calculation strategies, construct the training samples required for generative adversarial network training, and construct the generator model and discriminator model of the generative adversarial network. Finally, input the training samples into the generator model and the discriminator model for iterative training to complete the data interpolation.

2. According to claim 1, a method for interpolating continuous missing values ​​in time series based on a generative adversarial network is characterized in that: The time series continuous missing value interpolation method comprises the following steps: Step 1: First, the actual sampled data is used as the multivariate time series to be interpolated, and the data changes over time; then, the multivariate time series is converted into a two-dimensional matrix, represented by the data matrix X. The number of rows d and columns s represent the number of sampling times and sampling variables respectively. Each row of data in X is a one-dimensional time series. Step 2: Process the data matrix X in step 1 to generate training data; specifically: Preprocess the data matrix X and prepare training data. In order to distinguish the missing and non-missing parts of the data matrix X, first, generate the corresponding mask matrix M according to the data matrix X. The dimension of the mask matrix M is the same as that of the data matrix X. When an element in X is zero or does not exist, the M matrix element at the corresponding position is "0", and the elements at other positions are all "1". Then, generate a random noise matrix N with the same dimension as the data matrix X. The element n in the random noise matrix is ij The value range is [-0.01, +0.01]; finally, a contextual prompt information matrix T of the generator with the same dimension is generated according to the data matrix X. The T matrix is ​​shown in formula (1): Among them, t ij represents the i-th row and j-th column element of the contextual hint information matrix T; x ij represents the element in the i-th row and j-th column of the data matrix X; x i±l,j Represents the element in the i±lth row and jth column of the data matrix X; m ij represents the element in the i-th row and j-th column of the mask matrix M; l represents the length of consecutive missing elements; Record the maximum and minimum values ​​of each row of the data matrix X for inverse normalization of the interpolation result; the normalized data matrix is ​​represented by X′, and the normalized data matrix X′ is used as training data; Step 3: Use the training data to train the Generative Adversarial Interpolation Network CWGAIN-GP model, specifically: In order to ensure the normal training of the generative adversarial interpolation network CWGAIN-GP model, the training data is extracted and input into the generator; the samples x′, m, t and n processed each time are extracted from X′, M, T and N; the matrix is ​​generated according to formula (3) According to formula (4), and M are input into the generator to generate the result; Where ⊙ represents the multiplication of the elements at the corresponding positions of the matrix; the purpose of adding the random noise matrix N is to prevent the subsequent generator from generating data with excessive errors due to the existence of elements with values ​​of 0 in the matrix; G(·) is the generation process; Step 4: Introduce an additional regularization term, namely gradient penalty, into the discriminator D and discriminate the generated result y in step 3, specifically: 4.1) Input the generated result y of step 3 into the discriminator D for discrimination; the discriminator of the CWGAIN-GP model not only identifies the truth or falsity of the normalized data matrix X′, but also makes a probability estimate for each element of X′ to determine whether the element is real or generated, and finally generates a probability matrix; each element in the probability matrix corresponds to the element at the same position of the normalized data matrix X′; the discrimination process is shown in formula (6): Among them, D * (y,m) i is the probability of the discriminator output, p(y,m) is the joint probability distribution of the matrix (Y,M); 4.2) Introducing the gradient penalty term It can avoid the serious deviation of model parameters due to weight clipping during the training process, the disappearance of parameter update gradient, and ultimately the failure of model training. The loss function of the discriminator is shown in formula (7): in, It is convenient to solve the differential and use the interpolation data synthesized by real and generated data. ε is a random number between 0 and 1. P g and P r are the distributions of generated data samples and real data samples, respectively, λ is the weight of the penalty term, is the differential factor of D(*), and χ is The probability distribution of L D is the basic loss of the discriminator; is the gradient penalty term; Denotes the generation distribution P g The output of the generated data Y sampled in the discriminator; D(X′) represents the probability matrix of the discriminator output for the real data X′; D(Y) represents the probability matrix of the discriminator output for the generated data Y; Represents the distribution from the true distribution P r The output of the real data X′ sampled in the discriminator; λ represents the weight coefficient of the gradient penalty term; Represents interpolation data Calculate the expected value of ; Represents interpolation data Compute the gradient of the discriminator output; Step 5: Combine the discrimination results obtained in step 4 and Feedback is given to the generator and discriminator to update the model parameters. The loss function value is calculated using the discriminator's discrimination result to optimize the discriminator D and generator G parameters. The objective function of the CWGAIN-GP model is shown in formula (8): in, Represents the mutual confrontation between the generator and the discriminator; Represents the generated data Y and interpolated data Calculate the expected value of ; Step 6: The generator returns the result After the generator generates data and outputs Y, Y is recalculated with the mask matrix M, and the generated data is only integrated into the vacant positions of the normalized data matrix X′ to prevent the real data in the normalized data matrix X′ from being destroyed; the calculation formula is shown in (9): in, is the final output result; The matrix is ​​inverse normalized to obtain the final multivariate time series interpolation results.

3. According to claim 2, a method for interpolating continuous missing values ​​in time series based on a generative adversarial network is characterized in that: In step 2, each row of the data matrix X, that is, all data at the same sampling location, is normalized, as shown in formula (2):

4. According to claim 2, a method for interpolating continuous missing values ​​in time series based on a generative adversarial network is characterized in that: In step 3, the loss function of the generator is shown in formula (5): Where D(·) is the identification process; α is the generator hyperparameter; L G is the adversarial loss of the generator; L MSE is the mean square error loss between the estimated value Y and the true value X′; are the gradients of the loss with respect to the generator parameters.

5. According to claim 2, a method for interpolating continuous missing values ​​in time series based on a generative adversarial network is characterized in that: In the step 5, the Adam optimizer is selected.

Images