A graph and attention-based spatiotemporal crack data inference method

Abstract

The application provides a kind of space-time crack data inference method based on graph and attention, comprising the following steps;Step S1: obtain the original sparse matrix containing space-time crack data;Step S2: remove all space-time cracks in the original sparse matrix to obtain a smaller size crack-free sparse matrix;Step S3: infer the crack-free sparse matrix using matrix decomposition techniques;Step S4: put back the space-time cracks removed in step S2 to obtain a preprocessed sparse matrix;Step S5: construct a spatial block based on a graph attention network and a time block based on a Transformer, and combine them into a spatio-temporal neural network structure;Step S6: input the preprocessed sparse matrix into the model of the spatio-temporal neural network structure, obtain the inferred matrix and calculate the loss;Step S7: repeat step S6 to train the model until convergence;The application can effectively infer more ideal complete data given perception data with sparsity and possibly accompanied by space-time crack characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of spatio-temporal data imputation, and in particular to a spatio-temporal crack data imputation method based on graph and attention. BACKGROUND

[0002] Spatio-temporal data imputation aims to learn the underlying correlations in the sparse spatio-temporal data to fill in the missing data. In wireless sensor networks, the data collected from dynamic environment monitoring can usually be represented by a spatio-temporal sensing matrix of size, which records the data from sensors in time slots.

[0003] In the real world, due to limited sensor coverage or other uncertain factors (such as power failure and system failure, etc.), the original spatio-temporal sensing matrix is usually sparse. More seriously, the phenomenon of complete data missing along the row and column directions of the spatio-temporal sensing matrix may occur, which is called spatio-temporal crack.

[0004] The existing technical solutions usually handle the data sparsity and spatio-temporal crack in separate steps when they coexist; or train different models to handle them respectively, which has the problems of high training difficulty and high training cost. In addition, the existing technical solutions are strongly dependent on the distribution of known data, and they usually take known data as the convergence target. However, for spatio-temporal crack data, there is no known data (observation) that can be used as the convergence target, which will lead them to introduce meaningless and chaotic data, and weaken the performance. Moreover, the existing research work usually does not comprehensively consider that the spatio-temporal sensing matrix has complex spatio-temporal correlation.

[0005] Therefore, the present application proposes a spatio-temporal crack data imputation method based on graph and attention, which is divided into two stages. In the first stage, matrix factorization (MF) is used to impute the sparse matrix with removed cracks, and then the removed spatio-temporal cracks are put back to obtain a preprocessed sparse matrix. In the second stage, the proposed spatio-temporal neural network structure based on graph attention network (GAT) and Transformer is used to complete the re-imputation of the preprocessed sparse matrix. The first stage provides more context information for the model to address the problem of insufficient observation information, so as to facilitate the subsequent imputation of spatio-temporal crack data. In the second stage, in order to learn the complex spatio-temporal correlation in the perceived data, GAT is used to capture the spatial correlation, and a Time-Masked Multi-Head Self-Attention (TimeMaskedMHSA) mechanism is proposed in Transformer to more effectively capture the temporal correlation, so as to finally realize the spatio-temporal crack data imputation with smaller error. SUMMARY

[0006] The application provides a space-time crack data inference method based on a graph and attention, which can complete a data inference task with a space-time crack feature through a first stage of pre-inference using matrix decomposition and a second stage of inferring data using a space-time neural network structure based on a graph attention network (GAT) and a Transformer, and can effectively infer ideal complete data given perception data with sparsity and possibly with a space-time crack feature.

[0007] The application adopts the following technical solutions.

[0008] A space-time crack data inference method based on a graph and attention can fill in missing parts of collected data by learning potential correlations in sparse space-time data, and includes the following steps.

[0009] Step S1: obtaining an original sparse matrix containing space-time crack data;

[0010] Step S2: removing all space-time cracks in the original sparse matrix to obtain a smaller size crack-free sparse matrix, i.e., a general sparse matrix;

[0011] Step S3: inferring the crack-free sparse matrix using a matrix decomposition technique;

[0012] Step S4: putting back the space-time cracks removed in step S2 to obtain a preprocessed sparse matrix;

[0013] Step S5: constructing a space block based on a graph attention network and a time block based on a Transformer, and combining them into a space-time neural network structure;

[0014] Step S6: inputting the preprocessed sparse matrix into a model of the space-time neural network structure to obtain an inferred matrix and calculate a loss;

[0015] Step S7: repeating step S6 to train the model until convergence.

[0016] The specific implementation process of step S5 includes the following steps:

[0017] Step S501: constructing a space block based on a graph attention network, including a graph attention network GAT and a linear layer Linear;

[0018] After steps S1 to S4, a preprocessed sparse matrix containing N sensor nodes and T time slots is obtained which is regarded as a node feature matrix Further, a corresponding adjacency matrix A is obtained The adjacent node set of node i can be obtained through A The attention weight between nodes i and j is calculated:

[0019]

[0020] Where LeakReLU(·) is the activation function. It is a shared weight. It is a learnable weight vector. T || represents transpose, and || represents concatenation.

[0021] The normalized attention weights of all nodes are used to compute a linear combination of the corresponding features, which serves as the final output feature for each node:

[0022]

[0023] Where σ(·) is a nonlinear activation function;

[0024] The attention mechanism in GAT can be set up as a multi-head attention with K heads, then each node's features It becomes a combination of K C-dimensional features:

[0025]

[0026] Among them, W k It is the weight matrix of the k-th head;

[0027] After processing N nodes according to the above process, the final output H′ is obtained. Then, H′ is decoded using Linear to obtain the spatial representation as follows:

[0028]

[0029] in, It is the weight matrix, b H′ It is a bias;

[0030] Step S502: Construct a Transformer-based temporal block, which includes two main modules: a temporal mask multi-head self-attention module and a feedforward network module (FFN).

[0031] Given an input spatiotemporal matrix, i.e., the transpose of the matrix to be inferred, Map each of its embedded rows to a dimension of d k The query vector, key vector, and dimension d v The key vector and value vector form matrices Q and V, respectively. The attention score between Q and K is calculated using Scaled Dot-Product Attention, and then the attention weights are obtained using a softmax function. The final output is the attention-weighted V.

[0032]

[0033] To reduce the attention on the far time steps, a time mask, i.e., TimeMask operation TimeMask(·), is applied in self-attention; a thresholded Gaussian kernel function is used to calculate the correlation coefficient of any two time steps (i and j) in the input x, and then it is added to x:

[0034]

[0035] [TimeMask(x)](i,j) = x i，j +a i，j Equation Seven;

[0036] where dist(·,·) is the relative distance operator, β is the shape parameter, and δ is the threshold;

[0037] The self-attention mechanism is modified to a time-masked self-attention TimeMaskedSA:

[0038]

[0039] where Λ represents the attention weight matrix;

[0040] Then, the time-masked multi-head self-attention TimeMaskedMHSA is represented by the following equation:

[0041] TimeMaskedMHSA(x) = Concat(head1, …, head i , …, head h )W O

[0042] Equation Nine;

[0043] where head i = TimeMaskedSA(xW i Q , xW i K , xW i V ), and h is the number of heads;

[0044]

[0045] is a mapping parameter matrix.

[0046] FFN has two linear layers, with a ReLU function activated in between:

[0047] FFN(x) = W2ReLU(W1x + b1) + b2 Equation Ten;

[0048] Here, the position encoding PE is used in the original Transformer to represent the sequence order, that is:

[0049]

[0050]

[0051] where pos is the time step position, i is the dimension; the subsequent formula uses P to represent the position encoding.

[0052] The specific implementation process of step S6 includes the following steps:

[0053] Step S601: input the preprocessed sparse matrix In the input space block, the output spatial representation is obtained after GAT and a Linear layer

[0054] Step S602: in order to preserve the original true observation value, use the replacement operation to obtain the pre-inference matrix:

[0055]

[0056] Step S603: input the pre-inference matrix after transposition into the time block, first use a Linear layer to map the pre-inference data to d model dimension and add the position encoding P to produce its embedding E:

[0057]

[0058] where, is the weight matrix, b E is the bias;

[0059] Step S604: use TimeMaskedMHSA to capture the context dependency of E, and further learn the features of E through FFN; TimeMaskedMHSA and FFN are two sub-layers of the same layer in the Transformer; residual connection and layer normalization are adopted after the two sub-layers of TimeMaskedMHSA and FFN; after stacking L same layers, the output of E is:

[0060] Z st =FFN(TimeMaskedMHSA(E)) L Formula fifteen;

[0061] Step S605: use a Linear layer to map to N dimension, and then use the transposition operation transpose(·) to obtain the spatio-temporal representation:

[0062]

[0063] wherein, is a mapping parameter matrix, is a bias;

[0064] Step S606: using a replacement operation to retain the original true observation value, obtaining an inferred matrix:

[0065]

[0066] Step S607: establishing an observation reconstruction task ORT, the original observation will be reconstructed after model processing; the reconstruction loss is the error between the reconstructed value and the original observation value:

[0067]

[0068] wherein, l(·,·) is an element-level error function;

[0069] The reconstruction loss is calculated using the real matrix X and the spatiotemporal representation matrix obtained after model processing to train the model. At the same time, in order to deal with the problem of insufficient observation information in spatiotemporal crack inference, an auxiliary loss is introduced in the training process. The observation value used to calculate the auxiliary loss is pseudo-real data, so the hyperparameter η is used to control its confidence;

[0070]

[0071]

[0072] The collected data is the data collected by the wireless sensor network from the dynamic environment monitoring, which is represented by a spatiotemporal sensing matrix of size N×T, recording the data from N sensors in T time slots.

[0073] Matrix decomposition, abbreviated as MF;

[0074] Graph attention network, abbreviated as GAT;

[0075] Time-masked multi-head self-attention module, abbreviated as TimeMaskedMHSA.

[0076] Compared with the prior art, the method provided by the application is a graph and attention-based spatiotemporal crack data inference method, which is mainly used for solving data inference problems with spatiotemporal crack characteristics. The problem requires that in the face of sparsity of a spatiotemporal sensing matrix and complete missing of data along the row and column directions of the matrix, complete spatiotemporal sensing data is inferred to minimize the data inference error. The problem is essentially a data inference method. The method can effectively infer ideal complete data under the condition of given sensing data with sparsity and possibly accompanied by spatiotemporal crack characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0077] The application will be further described in detail below in combination with the drawings and specific embodiments:

[0078] FIG. 1 is a flowchart of the application. Figure 1 Specific embodiments

[0079] As shown in the figure, a graph and attention-based spatiotemporal crack data inference method can fill in the missing part of the collected data by learning the potential correlation in the sparse spatiotemporal data, including the following steps:

[0080] Step S1: obtaining an original sparse matrix containing spatiotemporal crack data;

[0081] Step S2: removing all spatiotemporal cracks in the original sparse matrix to obtain a smaller size crack-free sparse matrix, i.e., a general sparse matrix;

[0082] Step S3: inferring the crack-free sparse matrix by using matrix decomposition technology;

[0083] Step S4: putting back the spatiotemporal cracks removed in step S2 to obtain a preprocessed sparse matrix;

[0084] Step S5: constructing a spatial block based on a graph attention network and a time block based on a Transformer, and combining them into a spatiotemporal neural network structure;

[0085] Step S6: inputting the preprocessed sparse matrix into the model of the spatiotemporal neural network structure to obtain an inferred matrix and calculate a loss;

[0086] Step S7: repeating step S6 to train the model until convergence.

[0087] The specific implementation process of step S5 includes the following steps:

[0088] Step S501: constructing a spatial block based on a graph attention network, including a graph attention network GAT and a linear layer Linear;

[0089] ​After steps S1 to S4, a preprocessed sparse matrix containing N sensor nodes and T time slots is obtained. Treat it as a node feature matrix Further access The corresponding adjacency matrix A can be used to obtain the set of neighboring nodes of node i. Calculate the attention weights between nodes i and j:

[0090]

[0091] Where LeakyReLU(·) is the activation function. It is a shared weight. It is a learnable weight vector. T || represents transpose, and || represents concatenation.

[0092] The normalized attention weights of all nodes are used to compute a linear combination of the corresponding features, which serves as the final output feature for each node:

[0093]

[0094] Where σ(·) is a nonlinear activation function;

[0095] The attention mechanism in GAT can be set up as a multi-head attention with K heads, then each node's features It becomes a combination of K C-dimensional features:

[0096]

[0097] Among them, W k It is the weight matrix of the k-th head;

[0098] After processing N nodes according to the above process, the final output H′ is obtained. Then, H′ is decoded using Linear to obtain the spatial representation as follows:

[0099]

[0100] in, It is the weight matrix, b H′ It is a bias;

[0101] Step S502: Construct a Transformer-based temporal block, which includes two main modules: a temporal mask multi-head self-attention module and a feedforward network module (FFN).

[0102] Given an input spatiotemporal matrix, i.e., the transpose of the matrix to be inferred, Map each of its embedded rows to a dimension of d k The query vector, key vector, and dimension dv The value vectors; all query vectors form a matrix Q; the key vectors and the value vectors form matrices K and V, respectively; the attention scores between Q and K are calculated using Scaled Dot-Product Attention, and then a softmax function is used to obtain the attention weights. The final output is the attention-weighted V:

[0103]

[0104] In order to reduce the attention on the far time step, a time mask, TimeMask operation TimeMask(·), is applied in self-attention; a threshold Gaussian kernel function is used to calculate the correlation coefficient of any two time steps (i and j) in the input x, and then it is added to x:

[0105]

[0106] [TimeMask(x)](i, j) = x i，j + a i，j Equation seven;

[0107] where dist(·, ·) is the relative distance operator, β is the shape parameter, and δ is the threshold;

[0108] The self-attention mechanism is modified to TimeMaskedSA:

[0109]

[0110] where Λ represents the attention weight matrix;

[0111] Then, TimeMaskedMHSA is represented by the formula:

[0112] TimeMaskedMHSA(x) = Concat(head1, …, head i , …, head h )W O

[0113] Equation nine;

[0114] where head i = TimeMaskedSA(xW i Q , xW i K , xW i V ), h is the number of heads;

[0115]

[0116] is a mapping parameter matrix.

[0117] FFN has two linear layers, and a ReLU function is activated in the middle:

[0118] FFN(x) = W2ReLU(W1x + b1) + b2 Equation 10

[0119] Here, the original Transformer uses position encoding PE to represent sequence order, that is:

[0120]

[0121]

[0122] where pos is the time step position, and i is the dimension; the subsequent formula uses P to represent the position encoding.

[0123] The specific implementation process of step S6 includes the following steps:

[0124] Step S601: input the preprocessed sparse matrix In the input space block, the output spatial representation is output after GAT and a Linear layer

[0125] Step S602: in order to preserve the original true observation value, use the replacement operation to obtain the pre-inference matrix:

[0126]

[0127] Step S603: input the pre-inference matrix after transposition into the time block, first use a Linear layer to map the pre-inference data to d model dimension and add the position encoding P to produce its embedding E:

[0128]

[0129] where, is a weight matrix, and b E is a bias;

[0130] Step S604: use TimeMaskedMHSA to capture the context dependency of E, and further learn the features of E through FFN; TimeMaskedMHSA and FFN are two sub-layers of the same layer in the Transformer; residual connection and layer normalization are adopted after the two sub-layers of TimeMaskedMHSA and FFN; after stacking L same layers, the output of E is:

[0131] Zst = FFN(TimeMaskedMHSA(E)) L Fifteenth formula;

[0132] Step S605: using a Linear layer to map to N dimensions, and then using the transpose operation transpose(·) to obtain the spatiotemporal representation:

[0133]

[0134] wherein, is a mapping parameter matrix, is a bias;

[0135] Step S606: using the replace operation to retain the original true observation value, to obtain the inferred matrix:

[0136]

[0137] Step S607: establishing an observation reconstruction task ORT, the original observation will be reconstructed after model processing; the reconstruction loss is the error between the reconstructed value and the original observation value:

[0138]

[0139] wherein, l(·,·) is an element-level error function;

[0140] The reconstruction loss is calculated using the real matrix X and the spatiotemporal representation matrix obtained after model processing, to train the model. At the same time, in order to deal with the problem of insufficient observation information in spatiotemporal crack inference, an auxiliary loss is introduced in the training process. The observation value used to calculate the auxiliary loss is pseudo-real data, so the hyperparameter η is used to control its confidence;

[0141]

[0142]

[0143] The collected data is data collected by a wireless sensor network from dynamic environment monitoring, which is represented by a spatiotemporal sensing matrix of size N×T, recording data from N sensors in T time slots.

[0144] Matrix decomposition, abbreviated as MF;

[0145] Graph attention network, abbreviated as GAT;

[0146] Time-Masked Multi-Head Self-Attention, abbreviated as TimeMaskedMHSA.

[0147] The above merely provides the preferred embodiment of the present application, and all the equivalent changes and modifications made according to the scope of the present application should be included in the scope of the present application.

Claims

1. A graph and attention-based spatiotemporal crack data inference method, characterized in that: It fills in the missing parts of the collected data by learning the underlying correlations in the sparse spatio-temporal data; The collected data is the data collected by a wireless sensor network from dynamic environment monitoring, which is represented by a spatio-temporal sensing matrix of size N×T, recording the data from N sensors at T time slots; The spatio-temporal sensing matrix contains data with spatio-temporal crack characteristics, i.e., sparsity and possible data missing along the row and column directions of the matrix; The method completes the data inference task with spatio-temporal crack characteristics through a first stage of pre-inference using matrix decomposition and a second stage of inferring data using a spatio-temporal neural network structure based on GAT and Transformer, including the following steps: Step S1: Obtain the original sparse matrix containing spatio-temporal crack data; Step S2: Remove all spatio-temporal cracks in the original sparse matrix to obtain a smaller size crack-free sparse matrix, i.e., a general sparse matrix; Step S3: Use matrix decomposition techniques to infer the crack-free sparse matrix; Step S4: Put back the spatio-temporal cracks removed in step S2 to obtain a preprocessed sparse matrix; Step S5: Construct a spatial block based on the graph attention network and a time block based on the Transformer, and combine them into a spatio-temporal neural network structure; Step S6: Input the preprocessed sparse matrix into the model of the spatio-temporal neural network structure to obtain the inferred matrix and calculate the loss; Step S7: Repeat step S6 to train the model until convergence.

2. The spatio-temporal crack data inference method based on graph and attention according to claim 1, characterized in that: The specific implementation process of step S5 includes the following steps: Step S501: Construct a spatial block based on the graph attention network, including a graph attention network GAT and a linear layer Linear; After step S1 to step S4, a pre-processed sparse matrix containing N sensor nodes and T time slots is obtained Consider it as a node feature matrix Further obtain The corresponding adjacency matrix A, the adjacent node set of node i can be obtained by A Calculate the attention weight between nodes i and j: where LeakyReLU(·) is an activation function, is a shared weight, is a learnable weight vector, T denotes transpose, and || denotes concatenation operation; Use the normalized attention weight of all nodes to calculate the linear combination of the corresponding features as the final output feature of each node: Where σ(·) is a nonlinear activation function; The attention mechanism in GAT can be set to have multi-head attention with K heads, then each node feature becomes concatenated from K C-dimensional features: wherein W k is the weight matrix of the kth head; After processing N nodes according to the above process, the final output H′ is obtained, and then Linear is used to decode H′ to obtain the spatial representation as: wherein, is a weight matrix, b H′ is a bias; Step S502: Construct a time block based on the Transformer, including two main modules, i.e., a time mask multi-head self-attention module and a feedforward network module FFN; Given an input spatio-temporal matrix, the transpose of the pre-inferred matrix, Each row of which is mapped to a dimension d k Query vectors, key vectors, and value vectors of dimension d v All query vectors form a matrix Q; key and value vectors form matrices K and V, respectively; scaled dot-product attention is used to compute attention scores between Q and K, then a softmax function is used to obtain attention weights; the final output is the attention-weighted V: In order to reduce the attention to distant time steps, a time mask, i.e., TimeMask operation TimeMask(·), is applied in self-attention; a threshold Gaussian kernel function is used to calculate the correlation coefficient of any two time steps i and j in the input x, and then it is added to x: [TimeMask(x)](i, j) = x i，j + a i，j Equation Seven; Where dist(·,·) is a relative distance operator, β is a shape parameter, and δ is a threshold; Modify the self-attention mechanism to TimeMaskedSA: Formula 8; Where Λ represents the attention weight matrix; Then, the TimeMaskedMHSA of the time mask multi-head self-attention is represented by the following formula: TimeMaskedMHSA(x) = Concat(head1,..., head i ,..., head h )W O Formula nine; wherein, H is the number of heads; is a mapping parameter matrix; FFN has two linear layers, with a ReLU function activated in between: FFN(x) = W2ReLU(W1x + b1) + b2 Equation 10 Here, the original Transformer uses position encoding PE to represent the sequence order, that is: Where pos is the time step position, i is the dimension; the following formula uses P to represent the position encoding.

3. The spatio-temporal crack data inference method based on graph and attention according to claim 1, characterized in that: The specific implementation process of step S6 includes the following steps. Step S601: pre-process the sparse matrix The output spatial representation in the input space block after GAT and a Linear layer Step S602: In order to preserve the original true observation value, the replacement operation is used to obtain the pre-inferred matrix: Step S603: The pre-inferred matrix is input into the time block after transposition, first using a Linear layer to map the pre-inferred data to d model dimensions and added to the positional encoding P to produce its embedding E: wherein, is a weight matrix, b E is a bias; Step S604: Use TimeMaskedMHSA to capture the context dependence of E, and further learn the features of E through FFN; TimeMaskedMHSA and FFN are two sub-layers of the same layer in the Transformer; residual connection and layer normalization are adopted after the two sub-layers of TimeMaskedMHSA and FFN; after stacking L identical layers, the output of E is: Z st = FFN(TimeMaskedMHSA(E)) L Equation Fifteen; Step S605: Using a Linear layer to map to N dimensions, and then using the transpose operation transpose(·) to obtain the spatio-temporal representation: wherein is a mapping parameter matrix, is a bias; Step S606: Adopt replacement operation to preserve the original true observation value, and obtain the inferred matrix: Step S607: Establish an observation reconstruction task ORT, and the original observation will be reconstructed after model processing; the reconstruction loss is the error between the reconstruction value and the original observation value: wherein is the element-level error function; using real matrix X and the spatio-temporal representation matrix obtained after model processing to calculate the reconstruction loss to train the model; at the same time, in order to solve the problem of insufficient observation information in spatio-temporal crack inference, an auxiliary loss is introduced in the training process The observation value for calculating the auxiliary loss is pseudo-real data, so the confidence of it is controlled by using the hyperparameter η;

Images