A missing data completion method based on sparse topological spatiotemporal attention mechanism

Abstract

The present application relates to a missing data completion method based on a sparse topological spatiotemporal attention mechanism, including: performing random missing processing on time series data and marking the missing values ​​in the time series data; normalizing the processed time series data to construct a data set containing missing values; constructing a sparse topological spatiotemporal attention model, which includes a time attention module and a sparse topological attention module; inputting the data set into the sparse topological spatiotemporal attention model, calculating the loss function, training the sparse topological spatiotemporal attention model based on the loss function, and obtaining a data completion model based on the sparse topological spatiotemporal attention mechanism; inputting the missing data into the data completion model to obtain complete data. This method can capture the complex dependencies between data at different time points and different spatial locations, and can accurately complete the missing data by fusing the topological structure information, thereby improving the accuracy of the completion results.

Description

Technical Field

[0001] The present application relates to the technical field of data completion, and in particular to a missing data completion method based on a sparse topological spatiotemporal attention mechanism. Background Art

[0002] At present, with the rapid development of deep learning technology, many data show the characteristics of high dimensionality, spatiotemporal dependence, and dynamic nonlinearity. In this context, the attention mechanism has been widely used in various tasks due to its powerful feature extraction ability. The attention mechanism can dynamically assign importance weights to different positions, so as to better capture the dependencies between data. Therefore, using the attention mechanism to capture the spatiotemporal dependencies of process data for data completion can avoid the errors caused by separate interpolation, and at the same time use sparse topological relationships to clarify the variables that should be focused on in the completion. However, there are currently few studies exploring the application of sparse topological spatiotemporal attention mechanisms in data completion. Summary of the invention

[0003] Based on this, it is necessary to provide a missing data completion method based on sparse topological spatiotemporal attention mechanism, which includes:

[0004] S1: Collect time series data from different sensors within a period of time, perform random missing processing on the time series data, and mark the missing values ​​in the time series data;

[0005] S2: Normalize the processed time series data and construct a data set with missing values;

[0006] S3: construct a sparse topological spatiotemporal attention model, which includes a time attention module and a sparse topological attention module; the time attention module is used to weight each time step in the data in the data set and output the time series features through a fully connected layer; the sparse topological attention module is used to fill the time series features with adjacent variables in the data to obtain completed data;

[0007] S4: inputting the data set into the sparse topological spatiotemporal attention model, calculating the loss function, training the sparse topological spatiotemporal attention model based on the loss function, and obtaining a data completion model based on the sparse topological spatiotemporal attention mechanism;

[0008] S5: Input the missing data into the data completion model to obtain complete data.

[0009] Beneficial effects: This method can capture the complex dependencies between data at different time points and different spatial locations. By fusing topological structure information, it can accurately complete missing data and improve the accuracy of the completion results. Through sparse processing and attention mechanism, the time and space nodes that should be focused on for completion are selected, which greatly reduces the computational complexity. This method can be applied to processing high-dimensional data with complex time and space dependencies, such as sensor network data, meteorological data, traffic data, etc., and has broad application prospects in multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 This is a flowchart of a missing data completion method based on a sparse topological spatiotemporal attention mechanism according to an embodiment of the present application.

[0012] Figure 2 This is a network architecture diagram of the sparse topological spatiotemporal attention model of an embodiment of the present application.

[0013] Figure 3 This is a flow chart of the hydrocracking process of an embodiment of the present application.

[0014] Figure 4 This is a graph of the loss changes during the training process of an embodiment of the present application.

[0015] Figure 5 This is a graph showing the missing data completion results for variable 28 of an embodiment of the present application.

[0016] Figure 6 This is a graph showing the missing data completion results for variable 48 of an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0018] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0019] This embodiment provides a missing data completion method based on a sparse topological spatiotemporal attention mechanism, which uses the spatiotemporal attention mechanism to capture the complex dependencies between data at different time points and different spatial locations. Among them, the temporal attention mechanism is used to describe the temporal correlation of data to solve the temporal dependency problem in data completion; in terms of space, a sparse topological attention module is proposed to extract the topological relationship between variables. The learned matrix reveals the causal relationship between variables and the variables that should be focused on in the completion, helping the model to more accurately infer the possible values ​​of missing data under limited observation data. Its purpose is to effectively identify and utilize the spatiotemporal features in the data, combined with the sparse topological structure of the data, to accurately and efficiently complete the missing data.

[0020] like Figure 1 As shown, the method includes:

[0021] S1: Collect time series data from different sensors within a period of time, perform random missing processing on the time series data, and mark the missing values ​​in the time series data.

[0022] Specifically, the time series data is two-dimensional tensor data, which is expressed as: ; Indicates i The sensor in j The observed value at time, t Indicates the time period, Indicates the number of sensors;

[0023] Furthermore, in actual monitoring environments, missing values ​​are inevitable in multivariate time series due to sensor failures, communication interruptions, etc. In order to simulate the above data missing situation, random missing values ​​are processed for the collected data.

[0024] The missing values ​​in the marked time series data include:

[0025] ;

[0026] in, Indicates i The sensor in j The missing values ​​of the observations at time instant.

[0027] In this embodiment, the collected time series data includes process data of the hydrocracking process.

[0028] S2: Normalize the processed time series data and construct a data set with missing values.

[0029] S3: Construct a sparse topological spatiotemporal attention model, such as Figure 2 As shown, the sparse topological spatiotemporal attention model includes a time attention module and a sparse topological attention module; the time attention module is used to weight each time step in the data in the data set and output the time series features through a fully connected layer; the sparse topological attention module is used to fill the time series features with adjacent variables in the data to obtain completed data.

[0030] The temporal attention mechanism is a deep learning technology used to capture temporal dependencies in time series data. By assigning different weights to each time step, it can highlight important information in the time series data and suppress redundant or irrelevant information, thereby adaptively extracting key information at different times based on the input sequence data.

[0031] Specifically, in the temporal attention module,

[0032] Step 1: Obtain data in a data set, and flatten the data into a one-dimensional first array in column-major order;

[0033] Step 2: For the first array, map it to a query vector through the first linear layer, map it to a key vector through the second linear layer, and pass the data through the third linear layer to obtain a value vector;

[0034] Step 3: Calculate the attention score based on the query vector and key vector;

[0035] Step 4: In the self-attention model, the model calculates attention scores for different variables in the input sequence, thereby capturing the dependencies between different positions. However, this embodiment only wants to focus on the momentary features of a single variable and does not want to calculate the temporal attention scores between different variables. Therefore, the masked attention mechanism can play a role. Construct a mask matrix to mask the attention scores between different variables;

[0036] Step 5: Calculate the temporal attention feature based on the mask matrix, the value vector and the attention score;

[0037] Step 6: Since the temporal attention feature is the result of weighting the information of each position in the input sequence, it can be regarded as a compressed representation of the input sequence. However, as the length of the input sequence increases, the attention weights of the temporal attention layer may become very concentrated on certain positions, causing other parts of the input sequence to be ignored. Therefore, a fully connected layer is required after the temporal attention layer to increase the model's ability to perform nonlinear transformations. The temporal attention feature is passed through the fully connected layer to obtain the time series feature.

[0038] The calculation formula of the time series characteristics is:

[0039] ;

[0040] ;

[0041] ;

[0042] ; ; ;

[0043] ;

[0044] ;

[0045] in, Represents time series characteristics; represents the activation function of the fully connected layer; W represents the weight matrix of the fully connected layer; b represents the bias of the fully connected layer; Represents temporal attention characteristics; x Representation data; represents the attention score; represents the softmax function; M represents the mask matrix; Represents a value vector Remodeling; represents the first array; represents the weight matrix of the third linear layer; represents the query vector; represents the weight matrix of the first linear layer; represents the key vector; represents the weight matrix of the second linear layer; Represents the dimension of the data; represents transpose; D represents a matrix of all 1s.

[0046] Furthermore, in the spatial dimension, each variable can be regarded as a separate node, and different nodes influence each other. When a variable data is missing, it can be filled with the existing adjacent variable data. Specifically, according to the attention score, the adjacent variable data with a higher weight is selected as the estimated value of the missing data. In this way, filling the missing data by aggregating the data of neighboring nodes can provide an approximate and more complete data representation.

[0047] In the sparse topological attention module,

[0048] Step 1: Obtain the time series features, and flatten the time series features into a one-dimensional second array in a row-first order;

[0049] Step 2: For the second array, map it into a second query vector through the fourth linear layer, map it into a second key vector through the fifth linear layer, and pass the time series feature through the sixth linear layer to obtain a second value vector;

[0050] Step 3: Calculate a second attention score based on the second query vector and the second key vector;

[0051] Step 4: Sum the elements on the main diagonal of each block matrix in the second attention score to obtain multiple block attention scores, and normalize each block attention score to obtain the attention weight matrix;

[0052] Step 5: Construct a sparse topological matrix to highlight the spatial topological relationship between observations in the data through connected data;

[0053] In order to further focus on the variables that should be focused on in the completion, a learnable sparse topological matrix can be defined to describe the topological relationship between variable nodes. The sparse topological matrix is ​​initialized according to prior knowledge, and the variable nodes with connected relationships are assigned 1 in the corresponding positions of the matrix, otherwise they are assigned 0. The sparse topological matrix is ​​multiplied element by element with the attention weight matrix. For the positions where the sparse topological matrix elements are 1, the scores of the corresponding positions of the attention weight matrix remain unchanged after multiplication. For the positions where the sparse topological matrix elements are 0, the scores of the corresponding positions of the attention weight matrix are set to 0 after multiplication. In this way, unnecessary information can be covered up and only the required information is retained. And by training the matrix later, a sparse topological matrix that effectively reveals the causal relationship between variables can be obtained, which helps the model focus on the key variables that should be focused on in data completion.

[0054] Step 6: Obtain a predicted value of the missing position based on the attention weight matrix, the sparse topology matrix and the second value vector;

[0055] The formula for calculating the predicted value is:

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] ; ; ;

[0062] in, Represents the predicted value of the missing position; A represents the attention weight matrix; B represents the sparse topology matrix; represents the normalized block attention score of row 1 and column 1 in the attention weight matrix A; Indicates the first row of the attention weight matrix A. Column-normalized block attention scores; Represents the first Row 1 column normalized block attention score; Represents the first Line Column-normalized block attention scores; R Represents the number of elements on the main diagonal of the first block matrix in the second attention score; represents the first block matrix on the main diagonal of the second attention score r elements; represents the first block matrix on the main diagonal of the second attention score R elements; Represents the second value vector Remodeling; Indicates the operation of taking the main diagonal elements; Represents the block matrix of row 1 and column 1 in the second attention score; Indicates the first row of the second attention score A block matrix of columns; Indicates the second attention score row 1 column block matrix; Indicates the second attention score Line A block matrix of columns; represents the second attention score; represents the weight matrix of the sixth linear layer; Representing time series characteristics Remodeling; represents the second query vector; represents the weight matrix of the fourth linear layer; represents the second array; represents the second key vector; represents the weight matrix of the fifth linear layer; represents transpose; Represents the dimension of the data.

[0063] Step 7: According to the missing values ​​in the data, fill the predicted values ​​at the corresponding missing positions to obtain completed data.

[0064] In this embodiment, the loss function expression is:

[0065] ;

[0066] in, represents the loss function; represents the L2 norm; represents the L1 norm; y Indicates the true value of the missing position; represents the predicted value of the missing position; B represents the sparse topological matrix; Represents a hyperparameter. The entire loss function consists of the error between the predicted value and the true value and the topological matrix sparse constraint term. In addition to the standard loss error, an L1 regularization term is added to the loss function to constrain the size of the model parameters, thereby achieving a sparse effect.

[0067] Based on the back-propagation error of the loss function, the Adam optimization algorithm is used to optimize the network parameters of the sparse topology spatiotemporal attention model until the loss function converges to obtain the data completion model.

[0068] S4: Input the data set into the sparse topological spatiotemporal attention model, calculate the loss function, train the sparse topological spatiotemporal attention model based on the loss function, and obtain a data completion model based on the sparse topological spatiotemporal attention mechanism.

[0069] S5: Input the missing data into the data completion model to obtain complete data.

[0070] The missing data completion method based on the sparse topological spatiotemporal attention mechanism provided in this embodiment can capture the complex dependencies between data at different time points and different spatial locations. By fusing topological structure information, it can accurately complete the missing data, thereby improving the accuracy of the completion results. The sparsification processing and attention mechanism are used to select the spatiotemporal nodes that should be focused on for completion, thereby greatly reducing the computational complexity. The method can be applied to processing high-dimensional data with complex spatiotemporal dependencies, such as sensor network data, meteorological data, traffic data, etc., and has broad application prospects in multiple fields.

[0071] In order to evaluate and verify the performance of the data completion method provided in this example, the hydrocracking process is considered as an example. Hydrocracking is an important part of the oil refining process. It filters sulfur and nitrogen impurities in crude oil and cracks them into light molecular finished oil. Figure 3 The flow chart of a certain country's actual petrochemical hydrocracking process is shown in Figure 1. It mainly consists of two reactors, four separators and a stripping tower.

[0072] In this example, a total of 14,855 samples were collected for missing data completion. After careful selection, 61 process variables were used for training and testing, and an incomplete data set was generated after random processing. ,here n x =61, t Represents the number of samples after random missing. After setting the batch size to 16 and the learning rate to 10-4, the data of each batch is taken to train the data completion model based on the sparse topological spatiotemporal attention mechanism proposed in this paper. The loss function of the model training process is as follows: Figure 4 Due to limited space, the data completion results of variables 28 and 48 are selected for display. The final completion results are as follows Figure 5 , Figure 6 As shown in Figure 2, it can be found that the value completed by the proposed method is almost close to the actual value, and only a very small percentage of the completed value deviates.

[0073] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A missing data completion method based on sparse topological spatiotemporal attention mechanism, characterized in that: include: S1: Collect time series data from different sensors within a period of time, perform random missing processing on the time series data, and mark the missing values ​​in the time series data; S2: Normalize the processed time series data and construct a data set with missing values; S3: construct a sparse topological spatiotemporal attention model, which includes a time attention module and a sparse topological attention module; the time attention module is used to weight each time step in the data in the data set and output the time series features through a fully connected layer; The sparse topological attention module is used to fill the time series features through adjacent variables in the data to obtain completed data; In the sparse topological attention module, Step 1: Obtain the time series features, and flatten the time series features into a one-dimensional second array in a row-first order; Step 2: For the second array, map it into a second query vector through the fourth linear layer, map it into a second key vector through the fifth linear layer, and pass the time series feature through the sixth linear layer to obtain a second value vector; Step 3: Calculate a second attention score based on the second query vector and the second key vector; Step 4: Sum the elements on the main diagonal of each block matrix in the second attention score to obtain multiple block attention scores, and normalize each block attention score to obtain the attention weight matrix; Step 5: Construct a sparse topological matrix to highlight the spatial topological relationship between observations in the data through connected data; Initialize the sparse topology matrix according to prior knowledge, assign 1 to the variable nodes with connected relationships at the corresponding positions in the matrix, otherwise assign 0; multiply the sparse topology matrix and the attention weight matrix element by element, and the score of the position where the sparse topology matrix element is 1 remains unchanged after multiplication of the attention weight matrix, and the score of the position where the sparse topology matrix element is 0 is set to 0 after multiplication of the attention weight matrix; Step 6: Obtain a predicted value of the missing position based on the attention weight matrix, the sparse topology matrix and the second value vector; The formula for calculating the predicted value is: ； ； ； ； ； ； ； ； in, Represents the predicted value of the missing position; A represents the attention weight matrix; B represents the sparse topology matrix; represents the normalized block attention score of row 1 and column 1 in the attention weight matrix A; Indicates the first row of the attention weight matrix A. Column-normalized block attention scores; Represents the first Row 1 column normalized block attention score; Represents the first Line Column-normalized block attention scores; R Represents the number of elements on the main diagonal of the first block matrix in the second attention score; represents the first block matrix on the main diagonal of the second attention score r elements; represents the first block matrix on the main diagonal of the second attention score R elements; Represents the second value vector Remodeling; Indicates the operation of taking the main diagonal elements; Represents the block matrix of row 1 and column 1 in the second attention score; Indicates the first row of the second attention score A block matrix of columns; Indicates the second attention score row 1 column block matrix; Indicates the second attention score Line A block matrix of columns; represents the second attention score; represents the weight matrix of the sixth linear layer; Representing time series characteristics Remodeling; represents the second query vector; represents the weight matrix of the fourth linear layer; represents the second array; represents the second key vector; represents the weight matrix of the fifth linear layer; represents transpose; Represents the dimension of the data; Step 7: Fill the predicted value at the corresponding missing position according to the missing value in the data to obtain the completed data; S4: inputting the data set into the sparse topological spatiotemporal attention model, calculating the loss function, training the sparse topological spatiotemporal attention model based on the loss function, and obtaining a data completion model based on the sparse topological spatiotemporal attention mechanism; S5: Input the missing data into the data completion model to obtain complete data.

2. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 1 is characterized in that: The time series data is two-dimensional tensor data, expressed as: ; Indicates i The sensor in j The observed value at time, t Indicates the time period, Indicates the number of sensors; The missing values ​​in the marked time series data include: ； in, Indicates i The sensor in j The missing values ​​of the observations at time instant.

3. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 1 is characterized in that: In the temporal attention module, Step 1: Obtain data in a data set, and flatten the data into a one-dimensional first array in column-major order; Step 2: For the first array, map it to a query vector through the first linear layer, map it to a key vector through the second linear layer, and pass the data through the third linear layer to obtain a value vector; Step 3: Calculate the attention score based on the query vector and key vector; Step 4: Construct a mask matrix to mask the attention scores between different variables; Step 5: Calculate the temporal attention feature based on the mask matrix, the value vector and the attention score; Step 6: Pass the temporal attention feature through a fully connected layer to obtain the temporal feature.

4. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 3 is characterized in that: The calculation formula of the time series characteristics is: ； ； ； ； ； ； in, Represents time series characteristics; represents the activation function of the fully connected layer; W represents the weight matrix of the fully connected layer; b represents the bias of the fully connected layer; Represents temporal attention characteristics; x Representation data; represents the attention score; represents the softmax function; M represents the mask matrix; Represents a value vector Remodeling; represents the first array; represents the weight matrix of the third linear layer; represents the query vector; represents the weight matrix of the first linear layer; represents the key vector; represents the weight matrix of the second linear layer; Represents the dimension of the data; Indicates transpose.

5. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 3 is characterized in that: The expression of the mask matrix is: ； ； Among them, M represents the mask matrix; D represents the all-1 matrix.

6. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 1 is characterized in that: The loss function expression is: ； in, represents the loss function; represents the L2 norm; represents the L1 norm; y Indicates the true value of the missing position; represents the predicted value of the missing position; B represents the sparse topological matrix; Represents a hyperparameter.

7. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 1 is characterized in that: The training of the sparse topological spatiotemporal attention model based on the loss function comprises: Based on the back-propagation error of the loss function, the Adam optimization algorithm is used to optimize the network parameters of the sparse topology spatiotemporal attention model until the loss function converges to obtain the data completion model.

8. The missing data completion method based on sparse topological spatiotemporal attention mechanism according to claim 1, characterized in that: The collected time series data includes process data of the hydrocracking process.

Images