A dynamic spatio-temporal semantic joint embedding-based underground coal mine worker trajectory recognition method

Abstract

The application discloses a kind of dynamic spatio-temporal semantic joint embedding coal mine underground miner track identification method, belong to coal mine underground data analysis technical field.Utilize coal mine underground miner track data design miner track identification model, this model realizes which one miner track data of identification input belongs;Including: time feature extraction module, spatial feature extraction module, semantic feature extraction module and joint representation learning module.Advantages: track feature is extracted from time, space and semantic three dimensions and joint learning feature representation;Dynamic graph embedding method is used to extract the time dimension characteristics of track, obtain the long-term time dependence of variable-length track sequence, and apply multi-head self-attention mechanism to improve the processing efficiency of long sequence;Graph neural network is used to extract the structural topological information of track, and the semantic features of external factors are fused to improve the accuracy of miner track identification, thereby providing technical support for track data mining and intelligent scheduling platform in coal mine underground.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal mine underground data analysis, and particularly relates to a coal mine underground miner track recognition method based on dynamic spatio-temporal semantic joint embedding. BACKGROUND

[0002] In recent years, with the rapid development of wireless communication technology and global positioning technology, the position information of mobile objects (people, animals, vehicles, etc.) can be easily obtained and the mobile track can be stored and managed. Spatio-temporal track data is a sequence of records of the position and time of a mobile object, which contains rich information in the time dimension and the space dimension. In addition, track data provides rich information and value, and track data mining is applied to many fields, such as track prediction, track recommendation, pedestrian-track matching, etc.

[0003] The intelligent production scheduling system in the coal mine is a large production system which is centered on mining and realizes production planning through planning and scheduling of underground miners. Reasonably planning and scheduling the underground miners can not only increase the overall production of the coal mine, but more importantly, can better guarantee the production safety of the underground miners. The track mining of the miners is an important basis for planning and scheduling, and the feature embedding learning of the track of the miners is an important module of the track mining, which makes the spatio-temporal feature joint learning technology applied to the track of the miners in the coal mine a research hotspot.

[0004] At present, most of the traditional track representation methods are based on artificial design features, and usually only the track representation is taken as a part of data preprocessing. With the rise of deep learning, the ability of learning from large-scale data makes the track representation method based on deep learning have a great effect improvement compared with the traditional method, and gives the track representation more possibilities.

[0005] However, the existing methods still have three key limitations.

[0006] 1) Different miners usually visit several fixed work points in hundreds of work points in the underground tunnel, which causes the problem of sparse track data, in addition, the data involves noise and outliers;

[0007] 2) The existing methods only focus on the features in a single dimension of space or time, and do not jointly learn the feature representation of the track from the two dimensions of space and time;

[0008] 3) The existing feature representation learning for the track of the miners in the coal mine only focuses on the features in the time or space dimension, and ignores the rich semantic features of external factors in the track data.

[0009] With the wide application of deep learning in the field of track data, the track feature representation learning based on artificial intelligence method has become a research hotspot. SUMMARY

[0010] Technical problem: The purpose of the present application is to overcome the shortcomings in the prior art, and to provide a dynamic spatio-temporal semantic joint embedding coal mine underground miner trajectory recognition method to solve the problem that the traditional machine learning method cannot fully utilize the context information of the trajectory data, the single trajectory feature dimension, and the sparse trajectory data, which leads to the fact that the existing trajectory feature representation method cannot jointly learn the spatio-temporal and semantic feature representation of the miner trajectory in this specific scene of the coal mine underground.

[0011] Technical scheme: In order to achieve the above-mentioned purpose, the present application provides a dynamic spatio-temporal semantic joint embedding coal mine underground miner trajectory recognition method, which introduces the UWB ultra-wideband positioning data of the underground miner, the underground roadway map data and the surrounding environment monitoring data of the miner; a miner trajectory recognition model is designed using the coal mine underground miner trajectory data; the miner trajectory recognition model comprises a time feature extraction module, a space feature extraction module, a semantic feature extraction module and a joint representation learning module; after the underground miner trajectory data is preprocessed, it is respectively subjected to the time feature extraction module, the space feature extraction module and the semantic feature extraction module, and the time dimension feature representation, the space dimension feature representation and the semantic feature representation of external factors are obtained respectively, and finally the time dimension, the space dimension and the semantic feature representation of external factors are subjected to the joint representation learning module to obtain the spatio-temporal semantic feature representation with spatio-temporal dimension and semantic information, and the probability of the input trajectory data belonging to each miner is calculated, thereby improving the accuracy and efficiency of the miner trajectory recognition and providing technical support for the coal mine underground trajectory data mining and intelligent scheduling platform; the specific steps are as follows:

[0012] Step 1, collect the coal mine underground miner moving trajectory information, and preprocess the collected data to obtain a plurality of sub-trajectory sequences of each miner, wherein the preprocessing is sequentially grouping, cleaning, interpolation, sub-trajectory division and road network matching;

[0013] Step 2, the sub-trajectory sequence preprocessed in step 1 is subjected to the time feature extraction module, the space feature extraction module and the semantic feature extraction module respectively to obtain the trajectory time dimension feature representation, the space structure feature representation and the external factor semantic feature representation;

[0014] Step 3, the trajectory time dimension feature representation, the space structure feature representation and the external factor semantic feature representation obtained in step 2 are subjected to the splicing operation of the joint representation learning module to obtain the spatio-temporal semantic feature representation of the trajectory; then a full connection neural network is used to normalize the spatio-temporal semantic feature representation, and finally a Softmax activation function is used to calculate the probability of the trajectory sequence belonging to each miner;

[0015] Step 4, using cross-entropy loss function to train the miner trajectory recognition model, divide all miner trajectory data into training samples and test samples, and gradually train and optimize parameters to realize the recognition of the trajectory belonging to the miner.

[0016] In step 1,

[0017] The collected coal mine underground miner movement trajectory information includes miner ID, position coordinates at multiple time points based on UWB positioning, and surrounding environment monitoring data when the miner moves;

[0018] The grouping processing: grouping the corresponding trajectory points according to the unique identifier of the miner (miner ID);

[0019] The cleaning processing: removing the trajectory points that frequently fluctuate in long distance within each time slice;

[0020] The interpolation processing: for a missing trajectory point in a time slice, if there are trajectory points before and after the time slice, interpolate according to the adjacent time trajectory points, and the interpolation position is the center of the trajectory points before and after the time slice, to obtain the missing trajectory point;

[0021] The sub-trajectory division processing: dividing the trajectory data of each miner into several sub-trajectories according to the fixed shift interval of 8 hours in the coal mine, TR={tr1, tr2, …, tr n} represents all trajectory data of the miner, and tr i is one of the sub-trajectory sequences;

[0022] The road network matching processing: calculating the cross product of each trajectory point on each sub-trajectory and the end points of each road segment in the underground roadway to determine the underground roadway segment to which each trajectory point belongs, and the calculation formula is as follows:

[0023] (Q.x-A.x)*(B.y-A.y)=(B.x-A.x)*(Q.y-A.y)

[0024] Where Q.x, Q.y are the coordinates of the trajectory point, A.x, A.y are the coordinates of the left end point of the road segment, and B.x, B.y are the coordinates of the right end point of the road segment;

[0025] If the equation is established, and the coordinate Q is between the minimum and maximum values of the coordinates of points A and B, then the trajectory point belongs to the road segment with end points A and B, and the attribute of the belonging road segment is added to the record of the trajectory point.

[0026] If the equation is not established, continue to calculate the cross product of the trajectory point and the end points of the next road segment.

[0027] The specific process of removing the trajectory points that frequently fluctuate in long distance within each time slice is as follows:

[0028] Step 1-1, the trajectory data grouped according to the miner ID is set according to the sampling interval of the trajectory point, and the trajectory point is one-to-one corresponding to the time slice;

[0029] Step 1-2, the center position of all data in each time slice is calculated;

[0030] Step 1-3, the distance between the position information of all data in each time slice and the center position is calculated, and the nearest data to the center position is reserved, and the distance calculation formula is as follows:

[0031]

[0032] In the formula, The distance between points A and B is represented by R, A lat , A lng respectively represent the longitude and latitude coordinates of point A, B lat , B lng respectively represent the longitude and latitude coordinates of point B;

[0033] Step 1-4, the center position of all data reserved after calculation in step 1-3 is recalculated; each group represents the trajectory data of a miner, and only one position data of each miner is reserved at the same time.

[0034] In step 1-1, the time slice is set to 4-6s.

[0035] In step 2, the time feature extraction module generates a dynamic graph using the sub-trajectory sequence tr preprocessed in step 1, representing the time sequence relationship between the trajectory points in the miner trajectory sequence, and adopts a dynamic graph embedding method to obtain the feature representation in the time dimension;

[0036] The dynamic graph generation method is: dynamic graph G temporal ={g 1 ,g 2 ,…,g T} contains T static subgraphs of time slices, which are used to represent the change of the T time slices of the miner trajectory input to the miner trajectory recognition model, g τ =(V,E τ ) is a static subgraph in the τth time slice, which is used to simulate the time sequence relationship of the sub-trajectory sequence tr in the τth time slice to solve the problem of sparse trajectory data in coal mine underground; Each sub-trajectory sequence tr generates a static subgraph of one time slice, and each static subgraph is an undirected weighted graph, in which all road segments in the underground roadway are regarded as nodes of the static subgraph, all static subgraphs share a node set V, and nodes cannot be added or deleted, the edges in the static subgraph represent the movement of the miner from one road segment to another, and each static subgraph has its own edge set Eτ , which changes over time, can increase edges or delete edges, and each static subgraph corresponds to a different adjacency matrix A τ ;

[0037] The dynamic graph embedding method adopts a dynamic graph self-attention model DySAT, which learns structural embeddings on each static subgraph and learns temporal embeddings on multiple static subgraphs. The model mainly consists of two parts: a structural self-attention layer and a temporal self-attention layer, and adopts multi-head attention to improve the ability and stability of feature representation learning to obtain latent node representations. The DySAT is Dynamic Self-Attention, and the specific process includes:

[0038] The structural self-attention layer is a neighbor node information aggregation layer. For any static subgraph g τ of a time slice, the structural self-attention layer calculates attention and then performs weighted summation using the representation of each node in the current time slice to output the static subgraph g τ of the point v v ∈R F , F is the dimension of the new latent node representation set, and V is the node set of the static subgraph. The specific process is as follows:

[0039] Step 2-1-1, the attribute of each node on the static subgraph is represented by a D-dimensional vector x v , where x v is the attribute vector of node v, and x v ∈R D , D is the total number of nodes;

[0040] Step 2-1-2, the neighbor node set of node v is N v ={u∈V:(u,v)∈E}, E is the edge set of the static subgraph, and the formula for calculating the attention value of node v for each neighbor u is as follows:

[0041] e uv =σ(A uv ·a T [W s x u ||W s x v ])

[0042] where e uv is the attention value of node v for each neighbor node u, x v is the attribute vector of node v in the static subgraph g τ , and x uFor static subgraph g τ Attribute vector of middle node u, W s is the weight conversion matrix shared by all nodes, W s ∈R F×D , D is the total number of nodes, F is the dimension of the embedding vector set of all nodes, W s x u represents the embedding vector of the low-dimensional space corresponding to node u, W s x v represents the embedding vector of the low-dimensional space corresponding to node v, a is the weight vector, a ∈R 2D , a T is the transpose of the weight vector a, || is the embedding vector of the low-dimensional space corresponding to node v W s x v and the embedding vector of the low-dimensional space corresponding to node u W s x u , σ(·) is a nonlinear activation function, and the activation function LeakyRELU function is selected, A uv is the weight matrix connecting node v and node u in the current static subgraph.

[0043] Step 2-1-3, after calculating the attention value of point v to all its neighbor nodes, normalize the attention value:

[0044]

[0045] Where a uv is the normalized attention value, e uv is the attention value of node v to each neighbor u, e wv is the attention value of node v to each neighbor node w, w ∈N v , N v is the neighbor node set of node v, and exp(·) is the exponential function with base natural constant e.

[0046] Step 2-1-4, aggregate the embedding vectors of the neighbor nodes of node v according to the attention value to obtain the output vector z v of node v, this neighbor node information aggregation method preserves the local structure information of node v, and the calculation formula is as follows:

[0047]

[0048] Where z v represents the final output vector of node v in the structure self-attention layer, σ(·) is a nonlinear activation function, and the activation function LeakyRELU function is selected, x u is the attribute vector of node u, W sis a weight conversion matrix shared by all nodes, W s ∈R F×D , D is the total number of nodes, F is the dimension of the embedding vector set of all nodes, W s x u represents the embedding vector of the low-dimensional space corresponding to node u, a uv is the normalized attention value of neighbor node u to node v, N v is the neighbor node set of node v, and u is one of the neighbor nodes in the neighbor node set N v of node v.

[0049] The time self-attention layer further captures the time evolution pattern in the dynamic network by learning the correlation between the various representations of the nodes between different time steps; after the structure self-attention layer, the potential representation of any node on the static subgraph at each time step already contains the local information of the node; the input of the time self-attention layer is the representation sequence of a specific node v at different time steps, denoted as wherein, represents the attribute vector of node v at time step t, T' is the number of time steps, and D' is the dimension of the input representation; in the time self-attention layer, the new representation of any node v at each time step is calculated, which saves the dynamic information of the node over time, and the specific process is as follows:

[0050] Step 2-2-1, using the scaled dot-product attention form, calculate the Q, K matrix of each value in the input attribute vector , respectively using linear projection matrices W q , W k convert Q, K to different dimensional spaces, wherein the Q matrix is the Query matrix in English, and the K matrix is the Key matrix in English, and the calculation formula is as follows:

[0051]

[0052]

[0053] wherein Q is the Query matrix obtained after linear transformation of the input attribute vector , K is the Key matrix obtained after linear transformation of the input attribute vector , W q , W k are learning matrices, W q ∈R D'×F' , W k ∈R D'×F'D' is the dimension of the input representation, F' is the dimension of the resulting Q, K matrices, is the attribute vector of node v at time step t, 1≤t≤T', T' is the total number of time steps;

[0054] Step 2-2-2, calculate the attention value of node v at any time step t to all time steps, including t itself, for node v, the attention value of time step i to j is calculated as follows:

[0055]

[0056] wherein, is the attention value of node v at time step i to time step j, Q is the input attribute vector is the Query matrix obtained after linear transformation, K is the input attribute vector is the Key matrix obtained after linear transformation, K T is the transposed Key matrix, is the scaling factor, M ij is the mask matrix from time step i to time step j, M ij is defined as:

[0057]

[0058] wherein, M ij ∈R T'×T' , M ij is the mask matrix from time step i to time step j, T' is the total number of time steps;

[0059] Step 2-2-3, when M ij = -∞, the Softmax function causes the attention weight of time step i to its subsequent time steps to be 0, i.e. This switches the attention from time step j to j, and then normalizes the obtained attention, i.e. normalizes the attention value of time step j to all time steps, the calculation formula is as follows:

[0060]

[0061] wherein, is the normalized attention value of node v at time step i to all time steps, is the attention value of node v at time step i to time step j, is the attention value of node v at time step i to time step k, T' is the total number of time steps, i, j, k are one of the time steps, exp(·) is the exponential function with natural constant e as the base;

[0062] Step 2-2-4, the representation sequence of point v at all time steps is mapped to the output space of F' dimension as the output role, to obtain a matrix V, which is called Value matrix in English;

[0063] V = W v X v

[0064] wherein V is X v Value matrix obtained through linear conversion, W v is a learning matrix, W v ∈R D'×F' , D' is the dimension of the input representation, F' is the dimension of the obtained V matrix, X v is a matrix composed of attribute vectors x v of all time steps;

[0065] Step 2-2-5, taking the attention matrix β v as the weight matrix, aggregating the features of all time steps of node v to obtain the output feature representation of each time step, and weighting the output vector according to the attention value of any time step to all time steps to obtain the final time dimension feature representation of each time step, the calculation formula is as follows:

[0066] Z v = β v V

[0067] wherein Z v is the output vector of the time self-attention layer node v, β v is the attention weight matrix of node v, β v ∈R T'×T' , T' is the total number of time steps, V is the Value matrix obtained through linear conversion of X v ;

[0068] The multi-head attention layer jointly participates in different subspaces of each input, thereby causing a leap in model capacity, and in the structural and time self-attention layer, multiple attention layers are stacked and then connected in series, and the calculation formula is as follows:

[0069]

[0070]

[0071] wherein H is the number of self-attention, is the output vector of the Hth structural self-attention layer node v, is the output vector of the Hth time self-attention layer node v, and CONCAT represents the connection between the output vectors, h v , H v ​The final feature representation is structural multi-head self-attention and time multi-head self-attention, respectively.

[0072] In step 2, the spatial feature extraction module generates a spatial graph using the preprocessed sub-trajectory sequence tr in step 1 to represent the spatial topology structure of the miner moving trajectory segment, and adopts a graph neural network model and a pooling operation to obtain the feature representation of the trajectory spatial structure; the spatial feature extraction module specifically includes:

[0073] The method for generating a spatial graph from the sub-trajectory sequence tr is to construct an undirected graph G spatial τ as a spatial graph to describe the trajectory spatial topology structure of the miner in a fixed time slice, A spatial is the adjacency matrix of the spatial graph G spatial , the segments in the sub-trajectory sequence tr are taken as the nodes V of the spatial graph G spatial , and the edges of the spatial graph G spatial represent the movement of the miner from one segment to another segment; if the miner moves from segment A to segment B for the first time in a time slice, the value corresponding to node A and node B in the adjacency matrix A spatial is set to 1, and if the miner repeatedly moves from segment A to segment B or from segment B to segment A, the adjacency matrix A spatial is not operated;

[0074] The pooling operation: there are a total of L pooling layers, L is a parameter artificially set during model training, in each pooling layer, the embedding vector of each node is obtained using the graph neural network model, and the spatial graph G spatial is coarsened using the pooling model, after passing through all the pooling layers, the global spatial structure feature representation of the spatial graph G spatial is finally learned;

[0075] The graph neural network model is a graph sampling aggregation model GraphSAGE, which learns the embedding representation of each node, the input is the spatial graph G spatial , and the initial feature vector of each node is x v , v∈V, after sampling, aggregation, splicing, and normalization operations, the embedding vector of each node or cluster in each pooling layer is learned, and the GraphSAGE is a Graph Sample and Aggregate;

[0076] ​The pooling model is a differentiable pooling model Diffpool, which is operated by a learning allocation matrix and a pooling allocation matrix, based on the node embedding generated by the GraphSAGE model in the lth layer of the pooling layer and the allocation matrix, and in an end-to-end manner, the nodes are allocated to the cluster nodes in the l+1th layer of the pooling layer, 1≤l≤L, L is the total number of pooling layers, and then the cluster nodes are input to the next layer of the GraphSAGE model, thereby realizing the stacking of multiple GraphSAGE models in a hierarchical manner, and finally using an embedding vector to represent the features of the entire spatial graph, the cluster node is a cluster node obtained by pooling a plurality of nodes after the node pooling operation in each pooling layer, and the Diffpool is Differentiable Pooling.

[0077] The process of learning the allocation matrix includes:

[0078] Step 2-3-1, the lth layer of the pooling layer, obtaining a new embedding matrix Z of all cluster nodes by embedding the GraphSAGE model (l) , the calculation formula is as follows:

[0079] Z (l) =GraphSAGE l,embed (A (l) ,X (l) )

[0080] Wherein, Z (l) is a new embedding matrix calculated, A (l) is a coarse adjacency matrix of the spatial graph G spatial input to the GraphSAGE model in the lth layer of the pooling layer, X (l) is a cluster feature matrix of the spatial graph G spatial input to the GraphSAGE model in the lth layer of the pooling layer, GraphSAGE l,embed (·) represents embedding the GraphSAGE model in the lth layer of the pooling layer, 1≤l≤L, L is the total number of pooling layers;

[0081] Step 2-3-2, pooling GraphSAGE, generating an allocation matrix S (l) from the cluster feature matrix X (l) and the coarse adjacency matrix A (l) , which provides soft assignment of each node to the cluster nodes in the l+1th layer for the lth layer, and the English of soft assignment is SoftAssignment:

[0082] S (l) =Softmax(GraphSAGE l,pool (A (l) ,X (l) ))

[0083] Among them, S (l) S is the calculated allocation matrix. (l) Each row corresponds to n in the l-th pooling layer l One of the nodes, S (l) Each column corresponds to n at position l+1 in the next layer. l+1 One of the clusters, A (l) For spatial graph G spatial The coarsened adjacency matrix X is input to the GraphSAGE model at the l-th pooling layer. (l) For spatial graph G spatial The cluster feature matrix is ​​input into the GraphSAGE model at the l-th pooling layer. l,pool (·) indicates that the GraphSAGE model is pooled in the l-th pooling layer, and the output dimension is the predefined maximum number of clusters in the l-th pooling layer, 1≤l≤L, where L is the total number of pooling layers, and Softmax is the normalized exponential function;

[0084] The pooling allocation matrix process includes:

[0085] Step 2-4-1: In the l-th pooling layer, obtain the allocation matrix S through GraphSAGE. (l) With the new embedding matrix Z of the cluster nodes (l) Subsequently, the Diffpool model pools the cluster nodes of the l-th pooling layer using the following formula, generating a new coarsened adjacency matrix A for each cluster node in the l-th pooling layer. (l+1 ) with the new embedding matrix X (l+1) This leads to the formation of new cluster nodes, which serve as the input to the (l+1)th pooling layer. The calculation formula is as follows:

[0086]

[0087]

[0088] Among them, X (l+1 X represents the new embedding matrix of all cluster nodes obtained by the Diffpool model in the l-th pooling layer. (l+1) The i-th row corresponds to the embedding vector of cluster node i, A (l+1) This represents the new coarsened adjacency matrix for all cluster nodes obtained through the Diffpool model at the l-th pooling layer. The coarsened adjacency matrix A represents the connection strength between cluster node i and cluster node j in the l-th pooling layer. (l+1) The embedding matrix X of cluster nodes (l+1) Together, they can be used as input to the next (l+1)th pooling layer, GraphSAGE. (l)is the assignment matrix obtained by the GraphSAGE model at the l-th layer pooling layer, is the transpose matrix of S (l) is the assignment matrix obtained by the GraphSAGE model at the l-th layer pooling layer, (l) is the new embedding matrix of all cluster nodes obtained by the GraphSAGE model at the l-th layer pooling layer, (l) is the spatial graph G spatial is the coarse adjacency matrix of the l-th layer pooling layer, 1≤l≤L, L is the total number of pooling layers;

[0089] The semantic feature extraction module includes using external factor information in the moving process of the miner, using a word embedding method to obtain semantic feature representation of the external factor information;

[0090] The semantic feature extraction module is: using a word vector embedding method to convert each external factor information into a low-dimensional real vector to obtain semantic feature representation of the external factor of the miner trajectory, and the external factor information includes temperature, humidity, gas concentration, and CO concentration, and the specific process is as follows:

[0091] The word vector embedding method is used to embed the external factor information into a low-dimensional space, and the semantic features of the external factors of the sub-trajectory sequence tr are extracted, and the external factor semantic feature matrix of the t-th time step in the sub-trajectory sequence tr is calculated according to the following formula:

[0092]

[0093] Wherein, O t represents a feature matrix composed of external factor semantic feature vectors at the t-th time step in the sub-trajectory sequence tr, W e represents the weight of the full connection layer, represents an attribute matrix composed of attribute vectors of sensor environment monitoring data at the t-th time step, W e , b e are learnable parameters.

[0094] In step 3,

[0095] The feature representation of the trajectory time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factor are spliced by the joint representation learning module as the CONCAT operation, and the calculation formula is as follows:

[0096]

[0097] Wherein, represents the time dimension feature representation of the node v in the trajectory sequence in the time slice τ, represents the spatial dimension feature representation of the node v in the trajectory sequence in the time slice τ, denotes the semantic feature representation of node v in the trajectory sequence in time slice tau, CONCAT denotes the connection between each feature representation, N tr The spatio-temporal semantic feature representation of the sub-trajectory tr obtained after the joint representation learning module splicing operation on the feature representation of the time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factors;

[0098] The full connection neural network is: a full connection neural network using a normalized exponential function Softmax function to perform classification probability statistics, controlling all neurons in each network layer between 0-1, and the calculation formula is as follows:

[0099]

[0100] Where i is a variable, representing the i-th node, k is a variable, representing the k-th node, x i is the output value of the i-th node, x k is the output value of the k-th node, 1≤i≤n, 1≤k≤n, n is the total number of output nodes, that is, the number of classification categories, and is an exponential function with e as the base, and Softmax is a normalized exponential function.

[0101] In step 4,

[0102] The division method of the training sample and the test sample is: all the collected miner trajectory data is mixed according to the miner ID, and then 70% of the miner trajectory data in all the miner trajectory data is used as the training sample, and 30% of the miner trajectory data is used as the test sample.

[0103] The cross-entropy loss function is used to measure the difference between the identified miner probability and the real miner, and the calculation formula is as follows:

[0104]

[0105] Where M is the number of underground coal miners, c represents the label of the c-th underground coal miner, 1≤c≤M, N represents the total number of input samples, that is, the total number of input miner trajectory sequences, i represents the i-th input miner trajectory sequence, 1≤i≤N, y ic is a sign function, which takes 1 if the true class of sample i is equal to c, otherwise it takes 0, p ic is the predicted probability that input sample i belongs to class c, and Loss is the loss value finally calculated.

[0106] Beneficial effects: As the above scheme is adopted, the time feature, the space feature and the semantic feature are combined to realize the spatio-temporal semantic feature representation learning of the miner track, the problem of sparse track data is solved through the dynamic graph embedding model, and the long-term dependency relationship of the long track sequence is obtained by using the multi-head self-attention mechanism. The miner track recognition model adopted comprises four modules, namely, a time feature extraction module, a space feature extraction module, a semantic feature extraction module and a joint representation learning module. The time feature extraction module adopts the dynamic graph embedding technology to learn the time feature representation of the node, and the representation can save the dynamic structure information of the graph; the space feature extraction module adopts the latest progress of the graph neural network technology to learn the spatial topological structure feature of the track and solve the problem of sparse track data; meanwhile, the influence of external factors on the moving track of the miner is considered to enrich the data information; finally, the feature representation of three dimensions of time, space and semantic information is learned jointly, compared with other miner track recognition models, the method has obvious advantages in the application of miner track recognition in the coal mine underground. The UWB positioning method is adopted to accurately position the target in the coal mine underground, the positioning accuracy of 0.3 meters is realized, and the accurate positioning of the personnel, vehicles and equipment in the coal mine underground is realized, which can vividly and vividly show the distribution and movement of the personnel, vehicles and equipment in the coal mine underground. The use of deep learning to learn the moving mode of the miner and the identification of the type of work according to the track data is an important part of the management task; it has great significance for the intelligent scheduling of the miners in the coal mine underground, avoiding the problem of one person with multiple cards, maintaining the safety of production in the coal mine underground and the like. The problems that the existing classification method cannot fully utilize the context information of the track data, cannot obtain the long-term dependency relationship of the track sequence and cannot accurately and quickly identify the miners in the coal mine underground due to the sparse track data are solved, and the purpose of the present application is achieved. Compared with the prior art, the main advantages are:

[0107] 1) The dynamic graph is constructed to simulate the moving mode of the miner, the problem of sparse data is solved, and the context information can be fully learned;

[0108] 2) When learning the track feature representation, the particularity of the coal mine underground roadway is considered, the rich road network information of the coal mine underground roadway is fully utilized to mine the spatial topological structure feature of the miner moving track;

[0109] 3) The external factor information and the rich spatio-temporal context of the track are used to enhance the track representation learning;

[0110] 4) The multi-head self-attention mechanism is adopted to process the variable long track sequence, and the long-term time dependency of the track sequence can be captured. BRIEF DESCRIPTION OF DRAWINGS

[0111] Fig. 1 The flowchart of the present application.

[0112] Fig. 2The coal mine underground miner track recognition structure diagram of the present application. DETAILED DESCRIPTION

[0113] The present application will be further described below in conjunction with the embodiments in the drawings:

[0114] The dynamic spatio-temporal semantic joint embedding coal mine underground miner track recognition method of the present application introduces the UWB ultra-wideband positioning data of the underground miners, the underground roadway map data and the miner surrounding environment monitoring data; a miner track recognition model is designed using the coal mine underground miner track data; the miner track recognition model comprises a time feature extraction module, a space feature extraction module, a semantic feature extraction module and a joint representation learning module; the UWB English representation is Ultra WideBand;

[0115] After the underground miner track data is preprocessed, it is respectively subjected to the time feature extraction module, the space feature extraction module and the semantic feature extraction module, and the feature representation of the time dimension, the feature representation of the space dimension and the semantic feature representation of the external factors are obtained, respectively; finally, the feature representations of the time dimension, the space dimension and the external factor semantics are subjected to the joint representation learning module, the spatio-temporal semantic feature representation with the spatio-temporal dimension and the semantic information is obtained, the probability of the input track data belonging to each miner is calculated, thereby improving the accuracy and efficiency of the miner track recognition, and providing technical support for the coal mine underground track data mining and intelligent scheduling platform; the specific steps are as follows:

[0116] Step 1, collect the coal mine underground miner moving track information, and preprocess the collected data to obtain a plurality of sub-track sequences of each miner; the preprocessing is sequentially grouping processing, cleaning processing, interpolation processing, sub-track division processing and road network matching processing;

[0117] Step 2, the sub-track sequence preprocessed in step 1 is respectively subjected to the time feature extraction module, the space feature extraction module and the semantic feature extraction module, and the feature representation of the track time dimension, the feature representation of the space structure and the semantic feature representation of the external factors are obtained;

[0118] Step 3, the feature representation of the track time dimension, the feature representation of the space structure and the semantic feature representation of the external factors obtained in step 2 are subjected to the splicing operation of the joint representation learning module, and the spatio-temporal semantic feature representation of the track is obtained; then a fully connected neural network is used to normalize the spatio-temporal semantic feature representation, and finally a Softmax activation function is used to calculate the probability of the track sequence belonging to each miner;

[0119] Step 4, use the cross-entropy loss function to train the miner track recognition model, divide all the miner track data into training samples and test samples, and gradually train and optimize the parameters to realize the recognition of the track belonging to the miner.

[0120] In step 1, the data is preprocessed as follows:

[0121] The collected underground miner moving trajectory information of the coal mine includes: miner ID, position coordinates at multiple time points based on UWB positioning, and surrounding environment monitoring data when the miner moves;

[0122] The grouping processing: grouping the corresponding trajectory points according to the unique identifier of the miner (miner ID);

[0123] The cleaning processing: removing the trajectory points that frequently fluctuate in a long distance within each time slice;

[0124] The interpolation processing: for a missing trajectory point in a time slice, if there are trajectory points before and after the time slice, the trajectory points are interpolated according to the adjacent time trajectory points, the interpolation position is the center of the trajectory points of the previous and next time slices, and the missing trajectory point is obtained;

[0125] The sub-trajectory division processing: dividing the trajectory data of each miner into a plurality of sub-trajectories according to the fixed shift interval of 8 hours in the underground coal mine, TR={tr1, tr2, …, tr n} represents all trajectory data of the miner, and tr i is one of the sub-trajectory sequences;

[0126] The road network matching processing: calculating the cross product of each trajectory point on each sub-trajectory and the end points of each road section in the underground roadway to determine the underground roadway section to which each trajectory point belongs, and the calculation formula is as follows:

[0127] (Q.x-A.x)*(B.y-A.u)=(B.x-A.x)*(Q.y-A.y)

[0128] Where Q.x, Q.y are the coordinates of the trajectory point, A.x, A.y are the coordinates of the left end point of the road section, and B.x, B.y are the coordinates of the right end point of the road section;

[0129] If the equation is established, and the coordinates Q are between the minimum and maximum values of the coordinates of points A and B, then the trajectory point belongs to the road section with end points A and B, and the attribute of the belonging road section is added to the record of the trajectory point;

[0130] If the equation is not established, the cross product calculation of the trajectory point and the end points of the next road section is continued.

[0131] The specific process of removing the trajectory points that frequently fluctuate in a long distance within each time slice is as follows:

[0132] Step 1-1: The trajectory data obtained after grouping according to the miner ID is set according to the sampling interval of the trajectory points, the time slice is set, and the trajectory points are one-to-one corresponding to the time slice, and the time slice is set to 4-6s;

[0133] Step 1-2, calculate the center position of all data in each time slice;

[0134] Step 1-3, calculate the distance between the position information of all data in each time slice and the center position, and keep the data closest to the center position, the distance calculation formula is as follows:

[0135]

[0136] In the formula, denotes the distance between point A and point B, R represents the radius of the earth, A lat , A lng respectively represent the longitude and latitude coordinates of point A, B lat , B lng respectively represent the longitude and latitude coordinates of point B;

[0137] Step 1-4, for all data kept after calculation in step 1-3, recalculate the center position of these data; each group represents the trajectory data of a miner, and only one position data of each miner is kept at the same time.

[0138] In step 2, the time feature extraction module generates a dynamic graph using the sub-trajectory sequence tr preprocessed in step 1, representing the time sequence relationship between the trajectory points in the miner trajectory sequence, and adopts a dynamic graph embedding method to obtain the feature representation in the time dimension;

[0139] The dynamic graph generation method is: the dynamic graph G temporal ={g 1 ,g 2 ,…,g T} contains T static subgraphs of time slices, which are used to represent the change of the T time slices of the miner trajectory input to the miner trajectory recognition model, g τ =(V,E τ ) is a static subgraph in the τth time slice, which is used to simulate the time sequence relationship of the sub-trajectory sequence tr in the τth time slice to solve the problem of sparse trajectory data in the coal mine; Each sub-trajectory sequence tr generates a static subgraph of one time slice, and each static subgraph is an undirected weighted graph, in which all road segments in the underground roadway are regarded as nodes of the static subgraph, all static subgraphs share a node set V, and nodes cannot be added or deleted, the edges in the static subgraph represent the movement of the miner from one road segment to another, each static subgraph has its own edge set E τ , which changes over time and can increase or delete edges, and each static subgraph corresponds to a different adjacency matrix A τ ;

[0140] The dynamic graph embedding method adopts a dynamic graph self-attention model DySAT, the model learns structural embedding on each static subgraph and learns time embedding on multiple static subgraphs, mainly composed of two parts: a structural self-attention layer and a time self-attention layer, and adopts multi-head attention to improve the ability and stability of feature representation learning, to obtain latent node representation, the DySAT is Dynamic Self-Attention, the specific process includes:

[0141] The structural self-attention layer is a neighbor node information aggregation layer, for any static subgraph g τ of a time slice, the structural self-attention layer calculates the attention of each node in the current time slice and then performs weighted summation, and outputs the latent node representation z τ of each node v in the static subgraph g v with local structure characteristics, F , F is the dimension of the new latent node representation set, V is the node set of the static subgraph, and the specific process is as follows:

[0142] Step 2-1-1, the attribute of each node on the static subgraph is represented by a D-dimensional vector x v , wherein x v is the attribute vector of the node v, x v ∈R D , D is the total number of nodes;

[0143] Step 2-1-2, the neighbor node set of the node v is v={u∈V:(N,v)∈E}, E is the edge set of the static subgraph, and the attention value of the node v to each neighbor u is calculated according to the following formula:

[0144] e uv =σ(A uv ·a T [W s x u ||W s x v ])

[0145] Wherein e uv is the attention value of the node v to each neighbor node u, x v is the attribute vector of the node v in the static subgraph g τ , x u is the attribute vector of the node u in the static subgraph g τ , W s is a weight conversion matrix shared by all nodes, W s ∈R F×D, D is the total number of nodes, F is the dimension of the embedding vector set of all nodes, W s x u is the embedding vector of the low-dimensional space corresponding to node u, W s x v is the embedding vector of the low-dimensional space corresponding to node v, a is the weight vector, a ∈ R 2D , a T is the transpose of the weight vector a, || is the embedding vector of the low-dimensional space corresponding to node v W s x v is the embedding vector of the low-dimensional space corresponding to node u W s x u , σ(·) is a nonlinear activation function, the activation function LeakyRELU function is selected, A uv is the weight matrix connecting node v and node u in the current static subgraph;

[0146] Step 2-1-3, after calculating the attention value of node v to all its neighbor nodes, the attention value is normalized:

[0147]

[0148] where a uv is the normalized attention value, e uv is the attention value of node v to each neighbor u, e wv is the attention value of node v to each neighbor node w, w ∈ N v , N v is the neighbor node set of node v, exp(·) is the exponential function with base natural constant e;

[0149] Step 2-1-4, the embedding vectors of the neighbor nodes of node v are aggregated by weighting according to the attention value, and the output vector z v of node v is obtained, this neighbor node information aggregation method preserves the local structure information of node v, and the calculation formula is as follows:

[0150]

[0151] where z v represents the final output vector of node v in the structure self-attention layer, σ(·) is a nonlinear activation function, the activation function LeakyRELU function is selected, x u is the attribute vector of node u, W s is the weight conversion matrix shared by all nodes, W s ∈ R F×D , D is the total number of nodes, F is the dimension of the embedding vector set of all nodes, W s xu is the embedding vector of the low-dimensional space corresponding to node u, a uv is the normalized attention value of neighbor node u for node v, N v is the neighbor node set of node v, u is one of the neighbor nodes in the neighbor node set N v of node v;

[0152] The time self-attention layer is used to further capture the time evolution pattern in the dynamic network, and the correlation between the various representations of the node learned between different time steps; after the structure self-attention layer, the potential representation of any node on the static subgraph at each time step already contains the local information of the node; the input of the time self-attention layer is the representation sequence of a specific node v at different time steps, denoted as wherein, represents the attribute vector of node v at time step t, T' is the number of time steps, and D' is the dimension of the input representation; in the time self-attention layer, the new representation of any node v at each time step is calculated, which saves the dynamic information of the node over time, and the specific process is as follows:

[0153] Step 2-2-1, using the scaled dot-product attention form, the Q, K matrix of each value in the input attribute vector is calculated, respectively using linear projection matrix W q , W k convert Q, K to a space of different dimensions, wherein the Q matrix is the Query matrix in English, and the K matrix is the Key matrix in English, and the calculation formula is as follows:

[0154]

[0155]

[0156] wherein Q is the input attribute vector the Query matrix obtained after linear transformation, and k is the input attribute vector the Key matrix obtained after linear transformation, W q , W k is a learning matrix, W q ∈R D’×F’ , W k ∈R D’×F’ D' is the dimension of the input representation, and F' is the dimension of the obtained Q, K matrix, represents the attribute vector of node v at time step t, 1≤t≤T', T' is the total number of time steps;

[0157] Step 2-2-2, the attention value of all time steps of node v at any time step t, including t itself, for node v, the attention value of time step i to j is calculated as follows:

[0158]

[0159] wherein, is the attention value of time step i to time step j of node v, Q is the input attribute vector Query matrix obtained after linear transformation, K is the input attribute vector Key matrix obtained after linear transformation, K T is the transposed Key matrix, is the scaling factor, M ij is the mask matrix from time step i to time step j, M ij is defined as:

[0160]

[0161] wherein, M ij ∈R T’×T’ , M ij is the mask matrix from time step i to time step j, T' is the total number of time steps;

[0162] Step 2-2-3, when M ij =-∞, the Softmax function causes the attention weight of time step i to the subsequent time steps to be 0, that is This switches the attention from time step i to j, and then normalizes the obtained attention, that is, the attention value of time step i to all time steps of node v is normalized, and the calculation formula is as follows:

[0163]

[0164] wherein, is the normalized attention value of time step i to all time steps of node v, is the attention value of time step i to time step j of node v, is the attention value of time step i to time step k of node v, T' is the total number of time steps, i, j, k are one of the time steps, and exp(·) is the exponential function with natural constant e as the base;

[0165] Step 2-2-4, the representation sequence of point v at all time steps is mapped to the output space of F' dimensions as the output role, and the matrix V is obtained, wherein the full name of the V matrix in English is Value matrix;

[0166] V = W v X v

[0167] where V is X v Value matrix obtained by linear conversion, W v is a learning matrix, W v ∈R D’×F’ , D' is the dimension of the input representation, F' is the dimension of the obtained V matrix, X v is a matrix composed of attribute vectors x v of all time steps;

[0168] Step 2-2-5, taking the attention matrix β v as the weight matrix, aggregating the features of all time steps of node v to obtain the output feature representation of each time step, and calculating the final time-dimension feature representation of each time step according to the attention value of any time step to all time steps, the calculation formula is as follows:

[0169] Z v = β v V

[0170] where Z v is the output vector of the time self-attention layer node v, β v is the attention weight matrix of node v, β v ∈R T’×T’ , T' is the total number of time steps, V is the Value matrix obtained by linear conversion of X v ;

[0171] The multi-head attention layer jointly participates in different subspaces of each input, thereby causing a leap in model capacity, and in the structural and time self-attention layers, multiple attention layers are stacked and then concatenated, the calculation formula is as follows:

[0172]

[0173]

[0174] where H is the number of self-attention, is the output vector of the Hth structural self-attention layer node v, is the output vector of the Hth time self-attention layer node v, CONCAT represents the connection between the output vectors, h v , H v are the final feature representations of the structural multi-head self-attention and the time multi-head self-attention, respectively.

[0175] In step 2, the spatial feature extraction module generates a spatial graph using the sub-trajectory sequence tr preprocessed in step 1 to represent the spatial topology structure of the miner moving trajectory segment, and adopts a graph neural network model and a pooling operation to obtain the feature representation of the trajectory spatial structure; the spatial feature extraction module specifically includes:

[0176] The method for generating a spatial graph from the sub-trajectory sequence tr includes: spatial <V, Eτ> as a spatial graph describing the trajectory spatial topology structure of the miner in a fixed time slice, A spatial is the adjacency matrix of the spatial graph G spatial , and the road segment in the sub-trajectory sequence tr is taken as a node V of the spatial graph G spatial , the edge of the spatial graph G spatial represents the movement of the miner from one road segment to another road segment, if the miner moves from road segment A to road segment B for the first time in a time slice, the value corresponding to node A and node B in the adjacency matrix A spatial is set to 1, if the miner repeatedly moves from road segment A to road segment B or from road segment B to road segment A, the adjacency matrix A spatial is not operated;

[0177] The pooling operation includes L layers of pooling layers, L is a parameter artificially set during model training, the embedding vector of each node is obtained by using the graph neural network model in each pooling layer, and the spatial graph G spatial is coarsened by using a pooling model, after passing through all the pooling layers, the global spatial structure feature representation of the spatial graph G spatial is finally learned;

[0178] The graph neural network model is a graph sampling aggregation model GraphSAGE, which learns the embedding representation of each node, the input is the spatial graph G spatial , and the initial feature vector of each node is x v , v∈V, after sampling, aggregation, splicing and normalization operations, the embedding vector of each node or cluster in each pooling layer is learned, and the GraphSAGE is Graph Sample and Aggregate;

[0179] The pooling model is a differentiable pooling model Diffpool, which is operated by a learning allocation matrix and a pooling allocation matrix, based on the node embedding generated by the GraphSAGE model in the lth layer of the pooling layer and the allocation matrix, and in an end-to-end manner, the nodes are allocated to the cluster nodes in the l+1th layer of the pooling layer, 1≤l≤L, L is the total number of pooling layers, and then the cluster nodes are input to the next layer of the GraphSAGE model, thereby realizing the stacking of multiple GraphSAGE models in a hierarchical manner, and finally using an embedding vector to represent the features of the entire spatial graph, the cluster node is a cluster node obtained by pooling a plurality of nodes in each pooling layer, and the Diffpool is Differentiable Pooling.

[0180] The process of learning the allocation matrix includes:

[0181] Step 2-3-1, the lth layer of the pooling layer, obtaining a new embedding matrix Z of all cluster nodes by embedding the GraphSAGE model (l) , the calculation formula is as follows:

[0182] Z (l) =GraphSAGE l,embed (A (l) ,X (l) )

[0183] Wherein, Z (l) is a new embedding matrix calculated, A (l) is a coarsened adjacency matrix of the spatial graph G spatial input to the GraphSAGE model in the lth layer of the pooling layer, X (l) is a cluster feature matrix of the spatial graph G spatial input to the GraphSAGE model in the lth layer of the pooling layer, GraphSAGE l,embed (·) represents embedding the GraphSAGE model in the lth layer of the pooling layer, 1≤l≤L, L is the total number of pooling layers;

[0184] Step 2-3-2, pooling GraphSAGE, generating an allocation matrix S (l) from the cluster feature matrix X (l) and the coarsened adjacency matrix A (l) , which provides soft assignment of each node to the cluster nodes in the l+1th layer for the lth layer, and the English of soft assignment is SoftAssignment:

[0185] S (l) =SOftmax(GraphSAGE l,pool (A (l) ,X (l) ))

[0186] Among them, S (l) S is the calculated allocation matrix. (l) Each row corresponds to n in the l-th pooling layer l One of the nodes, S (l) Each column corresponds to n at position l+1 in the next layer. l+1 One of the clusters, A (l) For spatial graph G spatial The coarsened adjacency matrix X is input to the GraphSAGE model at the l-th pooling layer. (l) For spatial graph G spatial The cluster feature matrix is ​​input into the GraphSAGE model at the l-th pooling layer. l,pool (·) indicates that the GraphSAGE model is pooled in the l-th pooling layer, and the output dimension is the predefined maximum number of clusters in the l-th pooling layer, 1≤l≤L, where L is the total number of pooling layers, and Softmax is the normalized exponential function;

[0187] The specific process of the pooling allocation matrix is ​​as follows:

[0188] Step 2-4-1: In the l-th pooling layer, obtain the allocation matrix S through GraphSAGE. (l) With the new embedding matrix Z of the cluster nodes (l) Subsequently, the Diffpool model pools the cluster nodes of the l-th pooling layer using the following formula, generating a new coarsened adjacency matrix A for each cluster node in the l-th pooling layer. (l+1) With the new embedding matrix X (l+1) This leads to the formation of new cluster nodes, which serve as the input to the (l+1)th pooling layer. The calculation formula is as follows:

[0189]

[0190]

[0191] Among them, X (l+1) X represents the new embedding matrix of all cluster nodes obtained by the Diffpool model in the l-th pooling layer. (l+1) The i-th row corresponds to the embedding vector of cluster node i, A (l+1) This represents the new coarsened adjacency matrix for all cluster nodes obtained through the Diffpool model at the l-th pooling layer. The coarsened adjacency matrix A represents the connection strength between cluster node i and cluster node j in the l-th pooling layer. (l+1) The embedding matrix X of cluster nodes (l+1) Together, they can be used as input to the next (l+1)th pooling layer, GraphSAGE. (l)is the assignment matrix obtained by the GraphSAGE model at the l-th layer of the pooling layer, is the transpose matrix of S (l) is the new embedding matrix of all cluster nodes obtained by the GraphSAGE model at the l-th layer of the pooling layer, A (l) is the assignment matrix obtained by the GraphSAGE model at the l-th layer of the pooling layer, (l) is a spatial graph G spatial is the coarse adjacency matrix at the l-th layer of the pooling layer, 1≤l≤L, L is the total number of the pooling layers;

[0192] The semantic feature extraction module includes using external factor information in the moving process of the miner, using a word embedding method to obtain semantic feature representation of the external factor information;

[0193] In step 2, the semantic feature extraction module is: using a word vector embedding method to convert each external factor information into a low-dimensional real vector to obtain the semantic feature representation of the external factor of the miner trajectory, and the external factor information includes: temperature, humidity, gas concentration, CO concentration, and the specific process is as follows:

[0194] Step 2-5, using a word vector embedding method to embed these external factor information into a low-dimensional space, and extract the semantic feature of the external factor of the sub-trajectory sequence tr, and the external factor semantic feature matrix of the t-th time step in the sub-trajectory sequence tr is calculated according to the following formula:

[0195]

[0196] Wherein, O t represents a feature matrix composed of external factor semantic feature vectors at the t-th time step in the sub-trajectory sequence tr, W e represents the weight of the full connection layer, represents an attribute matrix composed of attribute vectors of sensor environment monitoring data at the t-th time step, W e , b e are learnable parameters.

[0197] In step 3,

[0198] The feature representation of the trajectory time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factor are spliced by the joint representation learning module to be the CONCAT operation, and the calculation formula is as follows:

[0199]

[0200] Wherein, represents the time dimension feature representation of the node v in the trajectory sequence in the time slice τ, represents the spatial dimension feature representation of the node v in the trajectory sequence in the time slice τ, denotes the semantic feature representation of node v in the trajectory sequence in time slice τ, CONCAT denotes the connection between each feature representation, N tr is the spatio-temporal semantic feature representation of the sub-trajectory tr obtained after the joint representation learning module splicing operation is performed on the feature representation of the time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factor;

[0201] The fully connected neural network is: a fully connected neural network using a normalized exponential function Softmax function to perform classification probability statistics, and all neurons in each layer of the network are controlled between 0-1, and the calculation formula is as follows:

[0202]

[0203] Where i is a variable, representing the i-th node, k is a variable, representing the k-th node, x i is the output value of the i-th node, x k is the output value of the k-th node, 1≤i≤n, 1≤k≤n, n is the total number of output nodes, that is, the number of classification categories, and is an exponential function with e as the base, and Softmax is a normalized exponential function.

[0204] In step 4, the model training specifically includes:

[0205] The division method of the training sample and the test sample is: all the collected miner trajectory data is mixed according to the miner ID, and then 70% of the miner trajectory data in all the miner trajectory data is used as the training sample, and 30% of the miner trajectory data is used as the test sample.

[0206] The cross-entropy loss function is used to measure the difference between the identified miner probability and the real miner, and the calculation formula is as follows:

[0207]

[0208] Where M is the number of underground miners in the coal mine, c represents the label of the c-th underground miner in the coal mine, 1≤c≤M, N represents the total number of input samples, that is, the total number of input miner trajectory sequences, i represents the i-th input miner trajectory sequence, 1≤i≤N, y ic is a sign function, which takes 1 if the true class of sample i is equal to c, otherwise it takes 0, p ic is the predicted probability that input sample i belongs to class c, and Loss is the loss value finally calculated.

[0209] As shown in Figs. 1-2 the instance of the underground miner movement trajectory data UWB data set in a coal mine includes the following steps:

[0210] Step 1, based on the obtained miner UWB trajectory data, pre-process the UWB trajectory data;

[0211] Based on the UWB trajectory data of the miners working in one white shift and night shift in the coal mine, there are a total of 879655 UWB trajectory data, each trajectory data records the miner ID, time, longitude and latitude, work point and sensor environmental monitoring data, such as temperature, humidity, gas concentration, CO concentration and other information;

[0212] Based on the obtained miner unique identification ID and time sequence chaotic miner UWB trajectory data, for each time slice, the corresponding trajectory points are grouped according to the miner ID, the trajectory data of each miner is cleaned and interpolated, the data set is mixed according to the miner ID, and is divided into training samples and test samples according to the proportion of 70% and 30%;

[0213] Step 2, according to the fixed time interval (8 hours), the trajectory data of each miner is divided into multiple sub-trajectories;

[0214] Step 3, match all trajectory points with road network, judge the road section of each trajectory belongs to underground roadway, and add the attribute of the road section in the record of the trajectory point;

[0215] Step 4, input the trajectory sequence into the time feature extraction module, the space feature extraction module and the semantic feature extraction module; In the time feature extraction module, a dynamic graph is constructed to simulate the moving mode of the miner trajectory, DySAT model is adopted, the model learns structure embedding on each static subgraph, learns time embedding on multiple static subgraphs, and adopts multi-head attention to improve the ability and stability of feature representation learning, and gets the latent node representation in time dimension In the space feature extraction module, the space graph is constructed according to the road network and trajectory structure, GraphSAGE model and Diffpool model are adopted to learn the global space structure feature representation of the space graph In the semantic feature extraction module, the word vector embedding method is adopted to convert each external factor information into a low-dimensional real vector, and the external factor semantic feature representation of the miner trajectory is obtained

[0216] Step 5, input the time, space and semantic feature representation obtained in step 4 into the joint representation learning module, connect through a connection layer, get the spatio-temporal semantic feature representation, and input into a single-layer fully connected network with Softmax activation function, calculate the probability of the input trajectory data belonging to each miner;

[0217] Step 6, use cross entropy loss function to train the job type recognition model, and gradually train and optimize the parameters to realize the recognition of the trajectory belonging to the miner;

[0218] Step 7, experimental environment and hyperparameter setting:

[0219] All experiments were implemented on a server with Nvidia GTX 3080Ti GPU using Python, the dependent deep learning framework was PyTorch 1.1.0, the programming language was Python 3.6, Adam was used to optimize training during training, the initial learning rate was 0.001, it was attenuated by 10% every 10 cycles, the batch size was 16, the total number of iterations for training was 160K, the number of K layers in the GNN model was 2, and the output size of the GNN model was 128.

Claims

1. A dynamic spatio-temporal semantic joint embedding method for underground coal mine worker trajectory recognition, characterized in that: The UWB ultra-wideband positioning data of the underground miner, the underground roadway map data and the surrounding environment monitoring data of the miner are introduced; a miner trajectory recognition model is designed by using the trajectory data of the underground miner in the coal mine; the miner trajectory recognition model comprises a time feature extraction module, a space feature extraction module, a semantic feature extraction module and a joint representation learning module; the specific steps are as follows: Step 1, collect the moving trajectory information of the underground miner in the coal mine, and pre-process the collected data to obtain a plurality of sub-trajectory sequences of each miner; the pre-processing is sequentially grouping processing, cleaning processing, interpolation processing, sub-trajectory division processing and road network matching processing; the road network matching processing: the cross product calculation of each trajectory point on each sub-trajectory and the end points of each road section of the underground roadway is carried out, the underground roadway section to which each trajectory point belongs is judged, and the calculation formula is as follows: (Q.x-A.x)*(B.y-A.y)=(B.x-A.x)*(Q.y-A.y) Where Q.x, Q.y are the coordinates of the trajectory point, A.x, A.y are the coordinates of the left end point of the road section, and B.x, B.y are the coordinates of the right end point of the road section; If the equation is established, and the coordinates Q are between the minimum and maximum values of the coordinates of points A and B, then the trajectory point belongs to the road section with end points A and B, and the attribute of the belonging road section is added to the record of the trajectory point; If the equation is not established, the cross product calculation of the trajectory point and the end points of the next road section is continued; Step 2, the sub-trajectory sequences pre-processed in step 1 are respectively subjected to the time feature extraction module, the space feature extraction module and the semantic feature extraction module, and the feature representation of the trajectory time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factors are obtained respectively; the semantic feature extraction module comprises using the external factor information in the moving process of the miner, adopting a word embedding method to obtain the semantic feature representation of the external factor information; The semantic feature extraction module: a word vector embedding method is adopted to convert each external factor information into a low-dimensional real vector to obtain the external factor semantic feature representation of the miner trajectory, and the external factor information comprises temperature, humidity, gas concentration and CO concentration, and the specific process is as follows: The word vector embedding method is used to embed these external factor information into a low-dimensional space to extract the semantic feature of the external factor of the sub-trajectory sequence tr, and the external factor semantic feature matrix calculation formula of the tth time step in the sub-trajectory sequence tr is as follows: wherein O t denotes a feature matrix composed of the external factor semantic feature vectors at the t-th time step in the sub-trajectory sequence tr, W e denotes the weight of the fully connected layer, denotes an attribute matrix composed of the attribute vectors of the sensor environment monitoring data at the t-th time step, W e , b e are learnable parameters; Step 3, the feature representation of the trajectory time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factors obtained in step 2 are subjected to the splicing operation of the joint representation learning module to obtain the spatio-temporal semantic feature representation of the trajectory; then a full connection neural network is used to normalize the spatio-temporal semantic feature representation, and finally a Softmax activation function is used to calculate the probability of the trajectory sequence belonging to each miner; Step 4, a cross entropy loss function is used to train the miner trajectory recognition model, all the miner trajectory data are divided into training samples and test samples, the parameters are gradually trained and optimized to realize the recognition of the trajectory belonging to the miner.

2. The dynamic spatio-temporal semantic joint embedding based underground coal miner trajectory identification method of claim 1, wherein: In step 1, The collected underground miner moving track information of the coal mine comprises a miner ID, position coordinates at multiple time points based on UWB positioning, and surrounding environment monitoring data during miner movement; The grouping processing groups corresponding track points according to the unique identifier of the miner; The cleaning processing removes track points that frequently fluctuate in a long distance within each time slice; The interpolation processing, for a missing track point in a time slice, if there are track points in adjacent times before and after the time slice, interpolates the missing track point according to the adjacent time track points, and the interpolation position is the center of the track points in the adjacent time slices, to obtain the missing track point; The sub-trajectory division processing: dividing the trajectory data of each miner into a plurality of sub-trajectories according to the fixed shift interval of 8 hours underground of the coal mine, TR={tr1, tr2, …, trn} n} represents all trajectory data of the miner, and tr i is a sub-trajectory sequence.

3. The dynamic spatio-temporal semantic joint embedding based underground coal miner trajectory identification method of claim 2, wherein: The specific process of removing the track points that frequently fluctuate in a long distance within each time slice is as follows: Step 1-1, the track data obtained after grouping according to the miner ID is set according to the sampling interval of the track points, a time slice is set, and the track points are corresponded to the time slice one by one; Step 1-2, the center position of all data in each time slice is calculated; Step 1-3, the distance between the position information of all data in each time slice and the center position is calculated, and the nearest data to the center position is retained, and the distance calculation formula is as follows: wherein denotes the distance between points A and B, R represents the earth radius, A lat , A lng denote the longitude and latitude coordinates of point A, respectively, B lat , B lng denote the longitude and latitude coordinates of point B, respectively; Step 1-4, the center position of all data retained after calculation in step 1-3 is recalculated; each group represents the track data of a miner, and only one position data of each miner is retained at the same time.

4. The dynamic spatio-temporal semantic joint embedding based underground coal miner trajectory identification method of claim 3, wherein: In step 1-1, the time slice is set to 4-6s.

5. The dynamic spatio-temporal semantic joint embedding based underground coal miner trajectory identification method of claim 1, wherein: In step 2, the time feature extraction module generates a dynamic graph using the sub-track sequence tr preprocessed in step 1, representing the time sequence relationship between the track points in the miner track sequence, and adopts a dynamic graph embedding method to obtain a feature representation in the time dimension; The dynamic graph generation method is: a dynamic graph G temporal = {g 1 ,g 2 ,…,g T} contains T static subgraphs of time slices, which are used to represent the changes of the miner track in T time slices input to the miner track identification model, g τ =(V, E τ ) is a static subgraph in the τth time slice, which is used to simulate the time sequence relationship of the sub-track sequence tr in the τth time slice, so as to solve the problem of sparse track data in the coal mine; each sub-track sequence tr generates a static subgraph of one time slice, and each static subgraph is an undirected weighted graph, wherein all road segments in the underground roadway are taken as nodes of the static subgraph, all static subgraphs share a node set V, and nodes cannot be added or deleted, the edges in the static subgraph represent the movement of the miner from one road segment to another road segment, each static subgraph has a respective edge set E τ , which changes over time, and the edges can be added or deleted, and each static subgraph corresponds to a different adjacency matrix A τ ; The dynamic graph embedding method adopts a dynamic graph self-attention model DySAT, which learns structure embedding on each static subgraph and learns time embedding on multiple static subgraphs, and is composed of two parts: a structure self-attention layer and a time self-attention layer, and adopts multi-head attention to improve the ability and stability of feature representation learning, to obtain a latent node representation, and the DySAT is Dynamic Self-Attention, and the specific process includes: The structural self-attention layer is a neighbor node information aggregation layer, and for any time slice static subgraph g τ , the attention is calculated using the representation of each node of the current time slice, and then weighted summation is performed to output the static subgraph g τ The potential node representation z of any point v with local structure characteristics v ∈R F , F is the dimension of the new potential node representation set, and V is the node set of the static subgraph, and the specific process is as follows: Step 2-1-1, the attribute of each node on the static subgraph is represented by a D-dimensional vector x v , where x v is the attribute vector of node v, x v ∈R D , D is the total number of nodes; Step 2-1-2, the neighbor node set of node v is N v = {u e V: (u, v) e E}, E is the edge set of the static subgraph, and the attention value of each neighbor u of node v is calculated according to the following formula: e uv = σ(A uv · a T [ W s x u || W s x v ]) where e uv is the attention value of node v to each neighbor node u, x v is the attribute vector of node v in static subgraph g τ , x u is the attribute vector of node u in static subgraph g τ , W s is the weight conversion matrix shared by all nodes, W s ∈R F×D , D is the total number of nodes, F is the dimension of the embedding vector set of all nodes, W s x u represents the embedding vector of the low-dimensional space corresponding to node u, W s x v represents the embedding vector of the low-dimensional space corresponding to node v, a is a weight vector, a ∈ R 2D , a T is the transpose of the weight vector a, || is the concatenation operation between the embedding vector W s x v of the low-dimensional space corresponding to node v and the embedding vector W s x u of the low-dimensional space corresponding to node u, σ(·) is a nonlinear activation function, and the activation function LeakyRELU function is selected, A uv is the weight matrix connecting node v and node u in the current static subgraph; Step 2-1-3, after calculating the attention value of point v to all neighbor nodes, the attention value is normalized: where a uv is the normalized attention value, e uv is the attention value of node v for each neighbor u, e wv is the attention value of node v for each neighbor node w, w ∈ N v , N v is the set of neighbor nodes of node v, exp(·) is the exponential function with base natural constant e; Step 2-1-4, the embedding vectors of the neighbor nodes of node v are aggregated by weighting with attention values to obtain an output vector z of node v v This neighbor node information aggregation method preserves the local structure information of node v, and the calculation formula is as follows: where z v represents the output vector of node v at the end of the structure self-attention layer, σ(·) is a nonlinear activation function, the activation function LeakyRELU function is selected, x u is the attribute vector of node u, W s is the weight conversion matrix shared by all nodes, W s ∈R F×D , D is the total number of nodes, F is the dimension of the embedding vector set of all nodes, X s x u represents the embedding vector of the low-dimensional space corresponding to node u, a uv is the attention value of the normalized neighbor node u of node v, N v is the neighbor node set of node v, and u is one of the neighbor nodes in the neighbor node set N v of node v; The time self-attention layer realizes further capturing the time evolution pattern in the dynamic network to learn the correlation between various representations of nodes between different time steps; after the above structure self-attention layer, the potential representation of any node on the static subgraph at each time step has already contained the local information of the node; the input of the time self-attention layer is the representation sequence of a specific node v at different time steps, denoted as wherein, represents the attribute vector of the node v at the time step t, T′ is the number of time steps, and D′ is the dimension of the input representation; in the time self-attention layer, the new representation of any node v at each time step is to be calculated, and the new representation is to save the dynamic information of the node changing with time, and the specific process is as follows: Step 2-2-1, the attribute vector of the input is calculated using a scaled dot-product attention form Q, K matrices for each value, respectively, using a linear projection matrix W q , W k Convert Q, K to a space of different dimensions, Q matrix in English is called Query matrix, K matrix in English is called Key matrix, the calculation formula is as follows: where Q is the input attribute vector Query matrix after linear transformation, K is the input attribute vector Key matrix after linear transformation, W q , W k is the learning matrix, W q ∈R D′×F′ , W k ∈R D′×F′ , D' is the dimension of the input representation, F' is the dimension of the obtained Q, K matrices, represents the attribute vector of node v at time step t, 1≤t≤T', T' is the total number of time steps; Step 2-2-2, the attention value of node v at any time step t to all time steps is calculated, including t itself, and for node v, the attention value calculation formula of time step i to j is as follows: where, is the attention value from time step i to time step j for node v, Q is the input attribute vector is the Query matrix after linear transformation, K is the input attribute vector is the Key matrix after linear transformation, K T is the transposed Key matrix, is the scaling factor, M ij is the mask matrix from time step i to time step j, M ij is defined as: where M ij ∈ R T′×T′ , M ij is a mask matrix from time step i to time step j, and T' is the total number of time steps. Step 2-2-3, when M ij = -∞, the Softmax function causes the attention weight of time step i to the following time steps to be 0, i.e. This switches the attention from time step i to j, and then normalizes the obtained attention, i.e. the attention value of time step i to all time steps, and the calculation formula is as follows: wherein, is the attention value of time step i of node v to all time steps after normalization, is the attention value of time step i of node v to time step j, is the attention value of time step i of node v to time step k, T' is the total number of time steps, i, j, k are one of the time steps, and exp(·) is the exponential function with the natural constant e as the base. Step 2-2-4, the representation sequence of point v at all time steps is mapped to an output space of F' dimensions as an output role to obtain a matrix V', and the V' matrix is the full name of Value matrix in English; V' = W v X v where V' is X v The Value matrix obtained through linear transformation, W v is the learning matrix, W v ∈ R D′×F′ , D' is the dimension of the input representation, F' is the dimension of the obtained V' matrix, X v is the matrix composed of attribute vectors x v of all time steps; Step 2-2-5, with attention matrix β v is the weight matrix, aggregating the features of all time steps of node v, obtaining the output feature representation of each time step, and the final time-dimension feature representation of each time step is obtained by weighting the output vector according to the attention value of all time steps for any time step, and the calculation formula is as follows: Z v = β v V' wherein Z v is the output vector of the temporal self-attention layer node v, β v is the attention weight matrix of the node v, β v ∈ R T′×T′ , T' is the total number of time steps, V' is X v the Value matrix obtained through linear conversion; The multi-head attention layer jointly participates in different subspaces of each input, thereby causing a leap in model capacity, and in the structure and time self-attention layers, multiple attention layers are stacked and then connected in series, and the calculation formula is as follows: where H is the number of self-attention, is the output vector for the H-th structural self-attention layer node v, is the output vector for the H-th temporal self-attention layer node v, and CONCAT denotes concatenation of the output vectors, h v , H v are the final feature representations for the structural multi-head self-attention and the temporal multi-head self-attention, respectively.

6. The dynamic spatio-temporal semantic joint embedding-based underground coal miner trajectory identification method according to claim 1, characterized in that: In step 2, the spatial feature extraction module generates a spatial graph using the preprocessed sub-trajectory sequence tr in step 1 to represent the spatial topology structure of the miner movement trajectory segment, and adopts a graph neural network model and a pooling operation to obtain the feature representation of the trajectory spatial structure. The spatial feature extraction module specifically comprises: The method for generating a space graph from the sub-trajectory sequence tr: constructing an undirected graph G spatial τ A space graph describes the trajectory space topology of the miner's fixed time slice, A spatial The adjacency matrix of the space graph G spatial The road segments in the sub-trajectory sequence tr are taken as the nodes V of the space graph G spatial The edges of the space graph G spatial represent the movement of the miner from one road segment to another, if the miner moves from road segment A to road segment B for the first time in a time slice, the value corresponding to node A and node B in the adjacency matrix A spatial is set to 1, if the miner repeatedly moves from road segment A to road segment B or from road segment B to road segment A, the adjacency matrix A spatial is not operated.​ The pooling operation: a total of L layers of pooling layers, L is a parameter artificially set during model training, at each pooling layer, an embedding vector of each node is obtained by using a graph neural network model, and a pooling model is used to coarsen the spatial graph G spatial After passing through all the pooling layers, the global spatial structure feature representation of the spatial graph G spatial is finally learned. The graph neural network model is a graph sample aggregation model GraphSAGE, which learns an embedding representation of each node, and an input is a spatial graph G spatial An initial feature vector of each node is x v , v e V, and an embedding vector of each node or cluster in each pooling layer is learned through sampling, aggregation, splicing, and normalization operations. The GraphSAGE is a Graph Sample and Aggregate. The pooling model is a differentiable pooling model Diffpool, which is composed of a learning assignment matrix and a pooling assignment matrix. Based on the node embedding generated by the GraphSAGE model in the lth pooling layer and the assignment matrix, the nodes are assigned to cluster nodes in the l+1th pooling layer in an end-to-end manner, 1≤l≤L, L is the total number of pooling layers. Then these cluster nodes are input into the next layer of GraphSAGE model, and then a plurality of GraphSAGE models are stacked in a hierarchical manner, and finally an embedding vector is used to represent the features of the entire spatial graph. The cluster node is a cluster node obtained by pooling a plurality of nodes in each pooling layer. The Diffpool is a differentiable pooling.

7. The coal mine underground miner trajectory recognition method of claim 6, wherein: The process of learning the assignment matrix comprises: Step 2-3-1, the first layer of the pooling layer, obtain the new embedding matrix Z of all cluster nodes by embedding the GraphSAGE model (l) The calculation formula is as follows: Z (l) = GraphSAGE l,embed (A (l) , X (l) ) wherein Z (l) is a newly calculated embedding matrix, A (l) is a spatial graph G spatial is a coarse adjacency matrix input to the GraphSAGE model at the l-th pooling layer, X (l) is a spatial graph G spatial is a cluster feature matrix input to the GraphSAGE model at the l-th pooling layer, GraphSAGE l,embed (·) represents embedding the GraphSAGE model at the l-th pooling layer, 1≤l≤L, L is the total number of pooling layers; Step 2-3-2, Pooling GraphSAGE, the cluster feature matrix X (l) with the coarse adjacency matrix A (l) generating an assignment matrix, S (l) provides a soft assignment of each node to the cluster nodes in the l+1th layer for the lth layer, the soft assignment is S (l) = Softmax(GraphSAGE l,pool (A (l) , X (l) )) where S (l) is a computed assignment matrix, S (l) each row of S l corresponds to one of n (l) nodes in the l-th pooling layer, each column of S l+1 corresponds to one of n (l) clusters at the next layer l+1, A spatial is a spatial graph G (l) is a coarse adjacency matrix input to the GraphSAGE model at the l-th pooling layer, X spatial is a cluster feature matrix input to the GraphSAGE model at the l-th pooling layer, GraphSAGE l,pool (·) represents the GraphSAGE model pooled at the l-th pooling layer, output dimension is a predefined maximum number of clusters in the l-th pooling layer, 1≤l≤L, L is the total number of pooling layers, Softmax is a normalization exponential function. The process of the pooling assignment matrix comprises: Step 2-4-1, in the l-th layer of the pooling layer, the assignment matrix S is obtained by GraphSAGE (l) with the new embedding matrix Z of the cluster node (l) Then, the Diffpool model pools the cluster nodes in the l-th layer of the pooling layer by the following formula, and generates a new coarse adjacency matrix A for each cluster node in the l-th layer of the pooling layer (l+1) with the new embedding matrix X (l+1) , and further forms a new cluster node as the input of the (l+1)-th layer of the pooling layer, and the calculation formula is as follows: where X (l+1) is the new embedding matrix of all cluster nodes after the Diffpool model at the l-th pooling layer, X (l+1) The i-th row of X (l+1) is the embedding vector of cluster node i, A is the new coarse adjacency matrix of all cluster nodes after the Diffpool model at the l-th pooling layer, (l+1) represents the connection strength between cluster node i and cluster node j at the l-th pooling layer, A (l+1) and the embedding matrix X (l) of cluster nodes together can be used as the input of the next pooling layer GraphSAGE, S is the assignment matrix obtained by the GraphSAGE model at the l-th pooling layer, (l) is the transpose matrix of S (l) is the new embedding matrix of all cluster nodes after the GraphSAGE model at the l-th pooling layer, A (l) is the spatial graph G spatial is the coarse adjacency matrix at the l-th pooling layer, 1≤l≤L, and L is the total number of pooling layers.

8. The dynamic spatio-temporal semantic joint embedding based underground coal miner trajectory identification method of claim 1, wherein: In step 3, The feature representation of the trajectory time dimension, the feature representation of the spatial structure and the semantic feature representation of the external factors are spliced by the joint representation learning module as a CONCAT operation, and the calculation formula is as follows: wherein, denotes a temporal dimension feature representation of the trajectory sequence at node v in time slice τ, denotes a spatial dimension feature representation of the trajectory sequence at node v in time slice τ, denotes a semantic feature representation of the trajectory sequence at node v in time slice τ, CONCAT denotes concatenation between each feature representation, N tr is a spatio-temporal semantic feature representation of the sub-trajectory tr obtained after the joint representation learning module splicing operation on the temporal dimension feature representation, the spatial structure feature representation, and the semantic feature representation of the external factor. The fully connected neural network is a fully connected neural network using a normalized exponential function Softmax function to perform classification probability statistics, and all neurons in each layer of the network are controlled between 0 and 1, and the calculation formula is as follows: wherein, l is a variable, representing the lth node, k is a variable, representing the kth node, x i is an output value of the ith node, x k is an output value of the kth node, 1≤i≤n, 1≤k≤n, n is the total number of output nodes, that is, the number of classification categories, and is an exponential function with e as the base, and Softmax is a normalized exponential function.

9. The dynamic spatio-temporal semantic joint embedding based underground coal miner trajectory identification method of claim 1, wherein: In step 4, The division method of the training sample and the test sample is that all the collected miner trajectory data is mixed according to the miner ID, and then 70% of the miner trajectory data in all the miner trajectory data is used as the training sample, and 30% of the miner trajectory data is used as the test sample. The cross-entropy loss function is used to measure the difference between the recognized miner probability and the real miner, and the calculation formula is as follows: The cross-entropy loss function is used to measure the difference between the recognized miner probability and the real miner, and the calculation formula is as follows: wherein M is the number of underground miners in the coal mine, c represents the underground miner label of the cth coal mine, 1≤c≤M, N represents the total number of input samples, that is, the total number of input miner trajectory sequences, i represents the ith input miner trajectory sequence, 1≤i≤N, y ic is a symbol function, which takes 1 if the real class of the sample i is equal to c, and 0 otherwise, p ic is the predicted probability that the input sample i belongs to the class c, and Loss is the loss value finally calculated.

Images