Explanatable abnormal gait detection method and model based on multi-task graph fusion learning

Abstract

The invention discloses an interpretable abnormal gait detection method and model based on multi-task graph fusion learning. The method comprises the steps that gait graphs are constructed, and a plurality of gait graphs are arranged according to time to obtain a time sequence gait graph; an LRP-MGG neural network model is constructed; gait time dependence features are extracted from the time sequence gait image through an accumulative GRU block of an LRP-MGG neural network model; local space features embedded in a time-dependent feature space are obtained from the gait time-dependent features by using a deep GCN module; carrying out gait mode classification based on a graph through a DiffPool module; interpretable abnormal gait information is obtained by learning through a CNN deep convolutional neural network model by using gait time-space significant difference features; the importance score of each feature is calculated by adopting hierarchical correlation propagation (LRP); according to the method, training of a plurality of tasks is combined, the data amount and complexity are reduced, and therefore the generalization ability and prediction ability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an explainable abnormal gait detection method and model based on multi-task graph fusion learning. Background Art

[0002] Gait is the posture characteristic of human walking. It is the external manifestation of the coupling of bones, muscles and joints related to the human motor system under the coordination of the human nervous system. It is closely related to physiological function factors such as human nerve function, motor function, cognitive function, and visual function. If the related physiological function is pathological, the gait will be abnormal. Accurate identification of abnormal gait and its changes is conducive to the early diagnosis, early prevention, diagnosis and treatment, and rehabilitation evaluation of related diseases (such as senile degenerative diseases and falls in the elderly).

[0003] In gait analysis, how to accurately distinguish and explain abnormal gait patterns has always been a challenging problem to be solved. More and more studies are devoted to finding methods to quantitatively analyze gait changes to avoid the low accuracy caused by qualitative assessments (such as visual inspection and professional experience of clinicians). Obtaining accurate, reliable and robust gait data is the first important step to quantify abnormal gait. In most related studies, a three-dimensional gait analysis system based on motion capture is widely used to collect gait data. The system is usually integrated with a motion capture system based on a high-speed camera, a force platform with a sole pressure sensor, and an electromyography (EMG) acquisition system to simultaneously obtain different gait parameters, such as spatiotemporal kinematic data, kinetic data, and EMG data. These parameters mainly include gait cycle, walking speed, gait rhythm, step length, walking time, stance phase, swing phase, joint angle, joint torque, and foot-ground reaction force, which can accurately, effectively, and reliably quantitatively analyze abnormal gait. However, there are some limitations to advanced three-dimensional gait analysis systems, such as high cost and special laboratory environment requirements, which make it impossible to monitor gait changes in outdoor environments in real time. In recent years, relevant research has mostly used wearable collectors based on MEMS sensors (accelerometers, gyroscopes, etc.) because they are inexpensive, only need to be worn on the anatomical structure of the human body, are easy to operate, and can be collected outdoors and in daily home living environments. Currently, most relevant research focuses on identifying abnormal gait conditions based on wearable sensors.

[0004] In addition, advanced graph neural network models are widely used in various abnormal gait detection tasks. Unlike deep learning models based on Euclidean space, graph neural networks can learn the compensation information hidden in gait movements from nodes and edges. In particular, graph neural networks can effectively construct gait dynamics and power chain compensation models from the structure and function of the entire lower limb, obtain significant differences hidden in the gait power chain in non-Euclidean metric space, significantly enhance the generalization performance of the model, and accurately distinguish the subtle differences between different gait patterns. However, although graph neural networks have excellent generalization performance for gait abnormality recognition, they come at the cost of a more complex model architecture, further exacerbating the black box nature of the model, making it difficult to understand how the model predicts and explains gait changes, and new technical solutions need to be found.

[0005] At present, the deep learning model interpretability methods include ex ante interpretability methods and ex post interpretability methods. The ex ante interpretability method aims to design an interpretable model by considering the interpretability of the model at the beginning of the neural network design. The network structure is adjusted before the model is trained, for example, by introducing additional network structures or changing the training strategy. The ex ante interpretability method has strong explanatory performance, but it is often limited to traditional machine learning or shallow neural network models and cannot effectively interpret complex neural networks. Deep neural networks have good fitting expression capabilities and are suitable for complex real-world scenarios. Although traditional shallow machine learning models have good interpretability, they are difficult to apply to complex scenarios. In order to adapt to complex deep learning network models, more related technical solutions focus on ex post interpretability methods. The basic idea is to provide interpretability information related to model decisions by designing additional technical means while ensuring that the deep neural network does not sacrifice the prediction performance. Therefore, most of the abnormal gait recognition models based on inertial sensors currently focus on the ex post interpretability method of deep learning.

[0006] At present, many gait recognition tasks use interpretability (XAI) technology to seek to construct a multi-task learning problem that can classify abnormal gaits and explain the prediction results, which is different from the post-explanation methods used in abnormal gait recognition (such as LIME, SHAP, and LRP). Some research works use LRP to generate heat maps of the importance of features throughout the gait cycle, consistently identifying peak area data and valley area data as important features, and effectively judging the consistency of previous research results. Other related research works have verified the use of SHAP and LRP techniques to judge the importance of analyzing gait parameters in wearable gait recognition. In particular, LRP is suitable for deep learning to analyze the regional importance of the entire gait pattern input. Although these research works can effectively use XAI to explain the output of the model, they do not consider the actual needs of users. At present, most of the research and development of explanatory AI systems is developer-centric rather than user-centric. Compared with the post-explanation provided by the model, medical professionals and patients are more concerned about identifying areas of concern and areas of abnormal gait changes that need treatment. Summary of the invention

[0007] The purpose of the present invention is to provide an interpretable abnormal gait detection method and model based on multi-task graph fusion learning, which combines the training of multiple tasks, reduces the amount and complexity of data, helps the model to better learn the two tasks of gait recognition and interpretability, and makes the model more robust on different tasks, thereby improving the generalization and prediction capabilities of the model.

[0008] The technical solution adopted by the present invention is:

[0009] The interpretable abnormal gait detection method based on multi-task graph fusion learning includes the following steps:

[0010] Step 1, constructing a gait graph describing the topological structure of the human body according to the natural connection mode of the human body, and arranging multiple gait graphs of the same gait pattern in time series to obtain a time series gait graph;

[0011] Furthermore, the gait graph is an unweighted graph G = {v, e} describing the topological structure of the human body, and is an undirected graph constructed according to the natural connection mode of the human body. is a set of joint nodes, n represents the number of lower limb joints, and each lower limb joint is regarded as a node; is a set of joint nodes, n represents the number of lower limb joints, and each lower limb joint is regarded as a node; represents the set of connecting edges between sensor nodes, m represents the number of edges in the gait graph; the gait graph is equivalently represented as G = {X, A}, are all node attributes of the lower limb nodes, A∈{0,1} n×nAn adjacency matrix that records the connection relationship between nodes.

[0012] Furthermore, the time sequence gait diagram is The equivalent expression is The abnormal gait recognition problem is regarded as training the mapping function f, using the sequence-gait-graph G sgg Predict the gait pattern, defined as:

[0013]

[0014] Among them, S i represents the abnormal gait pattern to be identified, T represents the time length of the time series gait graph, t represents the starting time of the time series gait graph, X t Represents the time series gait graph features at time t.

[0015] Step 2: construct the LRP-MGG neural network model, including the cumulative cGRU model, the deep GCN module (deeperGCN), the DiffPool hierarchical network prediction module, the hierarchical correlation propagation explanation module (LRP) for post-model decision explanation, and the CNN deep convolutional neural network prediction module.

[0016] Step 3: Taking the time-series gait graph as input feature, the gait time-dependent features are extracted through the cumulative GRU block of the LRP-MGG neural network model. The specific expression is as follows:

[0017]

[0018] Where H represents the gait time-dependent features extracted by the cumulative GRU block, h t represents X at timestamp t t Each GRU unit of the cumulative GRU block is regarded as a time window, and each hidden state h t is considered as the output of a sliding window at each timestamp t.

[0019] Step 4: Taking the gait time-dependent features as input, the deep GCN module is used to obtain the local spatial features embedded in the time-dependent feature space;

[0020] The cumulative GRU block and the deep GCN module cooperate with each other to obtain the spatiotemporal gait significant difference feature information hidden in the nonlinear dynamic chain of human gait motion;

[0021] Specifically, assuming that there are temporal dependency features H and adjacency matrix A in gait-graphG, the GCN model can use the temporal dependency features H and adjacency matrix A in non-Euclidean space to extract spatial dependency features between different joint nodes.

[0022] Furthermore, the deep GCN module is constructed using residual connection and pre-activation technology, including two GCN blocks using residual connection, and the two GCN blocks perform feature addition; each GCN block includes layer normalization, ReLU function, Dropout and generalized aggregation network (GEN) set in sequence, layer normalization is used to normalize the input data, which greatly improves the training speed of the neural network; ReLU represents the Sigmoid activation function, and Dropout effectively alleviates the overfitting problem; the GEN of the first GCN block is used to pre-activate the time-dependent feature H and is connected to each residual block to ensure deep training. The generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation function to effectively train deep GCN.

[0023] Step 5, graph-based gait pattern classification is performed through the DiffPool module; that is, by learning differentiable node clustering assignments at each layer of the multi-level pooling network, all nodes are mapped into a set of clustered graph embeddings to perform graph-based abnormal gait pattern classification;

[0024] Specifically, the DiffPool module is constructed to classify graph-based abnormal gait patterns. This block mainly consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). Then, the aggregated cluster is used as the input of the fully connected layer for graph classification. Step 5 specifically includes the following steps:

[0025] Step 5-1, by calculating the input clustering features H and the clustering adjacency matrix A in the deep GNN, the allocation matrix M with differentiable clustering can be generated; the specific calculation formula is as follows:

[0026] M=SoftMax(GNN l,pool (A,h G ) (2-10)

[0027] Step 5-2, by learning the distribution matrix M and the aggregation node features of the differentiable clustering of each layer of the standard GNN, the embedding Z of all aggregated joint nodes can be obtained; the specific calculation formula is as follows:

[0028] Z=GNN l,embed (h G ,M) (2-11)

[0029] Step 5-3, all six joint node embeddings (i.e., graph embeddings) are input into the fully connected layer to accurately predict the graph-based gait pattern. Specifically,

[0030]

[0031] Among them, GNN l,pool represents graph pooling, h G Represents the most abnormal representative spatiotemporal gait graph features, GNN l,embed Indicates the abnormal gait features output by graph pooling, Represents the output aggregation weight matrix, i represents the number of rows in the weight matrix, j represents the number of columns in the weight matrix, represents the weight matrix of the activation function, M represents the clustering assignment matrix of the lth layer of the pooling network, Z represents the aggregate embedding matrix of the graph nodes; k represents the number of graph-based gaits to be classified; SoftMax represents the Softmax function, according to GNN l,embed The output dimension of , in each layer to obtain the maximum number of clusters; represents the label of the graph-based gait pattern to be predicted; Relu represents the Sigmoid activation function.

[0032] Step 6: Using the spatiotemporal significant difference features of gait, the CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality, and the abnormal nodes are visualized to learn and obtain interpretable abnormal gait information contained in the spatiotemporal features;

[0033] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to reduce dimension, extract features and extract local connectivity for the given spatiotemporal feature layer; the last convolutional layer outputs y 3 Input to the fully connected layer to obtain the node prediction vector; among them, the CNN deep convolutional neural network model is given with convolutional layers of different kernel sizes, and any q-th layer gait spatiotemporal feature y q Defined as:

[0034]

[0035] in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, b q represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively;

[0036] The last layer of convolution output y 3 Input to the fully connected layer to get the node prediction vector Defined as

[0037]

[0038] Where W fc1 , W fc2 、b fc1 、b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

[0039] Step 7: Use layered relevance propagation (LRP) to calculate the importance score of each input feature for the model output based on the interpretable abnormal gait information in the spatiotemporal features to achieve abnormal gait detection. The specific calculation formula is as follows:

[0040]

[0041] Among them, R j represents the feature importance score of the jth hidden layer node, a j is the activation value of the jth hidden layer node, w jk is the weight connecting the jth hidden layer node and the kth output layer node, R k is the feature importance score of the kth output layer node.

[0042] Furthermore, a small constant ∈ is added to the output of ReLU based on layer-wise relevance propagation (LRP) in step 7 to prevent the gradient from vanishing. This ensures that the importance scores are correctly propagated while maintaining reasonable interpretation.

[0043] Furthermore, the LRP-MGG neural network model performs multi-task learning by combining the gait pattern prediction task and the gait node abnormality prediction task; the cross entropy is used as the loss function for the subtask to evaluate the minimum error between the true gait label and its corresponding label to be predicted during training and testing, which is defined as

[0044]

[0045] Among them, K represents the total number of abnormal gait recognition labels, O represents the total number of abnormal nodes in the gait graph, and Y 1i and Represent the real gait pattern and the corresponding predicted gait pattern respectively; Y 2j and They represent the real abnormal nodes and the corresponding predicted abnormal nodes respectively.

[0046] The present invention also discloses an interpretable abnormal gait detection model based on multi-task graph fusion learning, which includes an additive cGRU model, a deep GCN module (deeperGCN, i.e., a deep graph convolutional network model), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation explanation module (LRP) for post-explanation of model decisions, and a CNN deep convolutional neural network prediction module;

[0047] The cumulative cGRU model includes more than two GRU units that are accumulated in sequence, and each GRU unit is mainly composed of a reset gate rt and an update gate ut; the deep GCN module (deeperGCN) is constructed using residual connection and pre-activation technology, including two GCN blocks using residual connection, and the two GCN blocks perform feature addition (Addition); each GCN block includes layer normalization (Layer normalization), ReLU function, Dropout and generalized aggregation network (GEN) set in sequence, layer normalization is used to normalize the input data, which greatly improves the training speed of the neural network; ReLU represents the Sigmoid activation function, and Dropout effectively alleviates the overfitting problem; the generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation function to effectively train deep GCN, and the GEN of the first GCN block is used to pre-activate the time-dependent feature H and is connected to each residual block to ensure deep training; DiffPool hierarchical network prediction model The block consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). The aggregated cluster is used as the input of the fully connected layer for graph-based gait pattern classification; the CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality from the spatiotemporal significant difference features of gait, and visualize the abnormal nodes so as to learn the interpretable abnormal gait information contained in the spatiotemporal features; the hierarchical relevance propagation interpretation module (LRP) is used to calculate the importance score of each input feature for the model output from the interpretable abnormal gait information in the spatiotemporal features.

[0048] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to perform dimension reduction, feature extraction and local connectivity extraction on the given spatiotemporal feature convolution layer; the CNN deep convolutional neural network model is given convolutional layers with different kernel sizes, and any q-th layer of gait spatiotemporal feature y q Defined as:

[0049]

[0050] in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, b q represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively;

[0051] The last layer of convolution output y 3 Input to the fully connected layer to get the node prediction vector Defined as

[0052]

[0053] Where W fc1 , W fc2 、b fc1 、b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

[0054] Furthermore, the LRP-MGG neural network model performs multi-task learning by combining the gait pattern prediction task and the gait node abnormality prediction task; the cross entropy is used as the loss function for the subtask to evaluate the minimum error between the true gait label and its corresponding label to be predicted during training and testing, which is defined as:

[0055]

[0056] Among them, K represents the total number of abnormal gait recognition labels, O represents the total number of abnormal nodes in the gait graph, and Y 1i and Represent the real gait pattern and the corresponding predicted gait pattern respectively; Y 2j and They represent the real abnormal nodes and the corresponding predicted abnormal nodes respectively.

[0057] The present invention adopts the above technical scheme, uses the developed cGRU-dGCN model as the backbone model, and outputs the abnormal pattern classification features and abnormal node classification features through two different branches of the graph convolution features containing complex compensation information of the human body. The cumulativeGRU module has a powerful abnormal gait time feature extraction capability, and continuously superimposes the hidden features of multiple GRU time correlation calculation units through the cumulative design, enriching the compensatory feature representation of abnormal gait. The complex graph convolution features extracted by the deep GCN module (deeperGCN) use two channel branches to obtain abnormal pattern classification features and abnormal node classification features respectively. The abnormal pattern classification features obtain the gait abnormal pattern classification through the DiffPool hierarchical network prediction module, and then give the gait pattern classification, which can retain the high generalization characteristics of the cGRU-dGCN model; and the abnormal node classification features obtain the gait abnormal node classification through the CNN deep convolutional neural network prediction module, find out the possible abnormal joint nodes of the abnormal gait pattern, and then give the possible diseased node explanation of the abnormal gait for the user. As different branches in multi-task learning, the gait abnormal pattern classification task and the abnormal node classification task can fit two different tasks at the same time. Finally, the layered relevance propagation (LRP) of the post-hoc solvability method is used for post-hoc interpretation, that is, the decision output of the neural network is assigned contribution values ​​based on the rule of layer-by-layer back-propagation. Each neuron in the neural network will have an impact contribution to the model output. The importance score of each neuron is quantified layer by layer until each wavelength interval of the input sensor signal is calculated, and then an explanation for the model decision is obtained.

[0058] The present invention allows the machine learning model to learn multiple tasks at the same time to improve the performance and efficiency of the model. The training of multiple tasks is combined to reduce the amount and complexity of data, helping the model to better learn the two tasks of gait recognition and interpretability, so that the model has better robustness in different tasks, thereby improving the generalization and prediction capabilities of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments;

[0060] Figure 1 Schematic diagram of the model framework of the interpretable abnormal gait detection method based on multi-task graph fusion learning of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0062] like Figure 1As shown, the present invention discloses an interpretable abnormal gait detection method based on multi-task graph fusion learning. The gait abnormal pattern classification task and the abnormal node classification task are used as different branches in multi-task learning, which can obtain the high generalization of the model abnormal gait recognition and calculate the importance of the model prediction results of each signal interval in the gait signal to be interpreted. The model framework of the present invention is mainly composed of an augmented cGRU model, a deeperGCN deep graph convolutional network model, a diffpool hierarchical network prediction module, a hierarchical correlation propagation explanation module (LRP) for post-model decision explanation, and a CNN deep convolutional neural network prediction module. Its structure is as follows Figure 1 As shown in the figure, its technical route is: using the developed cGRU-dGCN model as the backbone model, the graph convolution features containing complex compensation information of the human body are output through two different branches to output abnormal pattern classification features and abnormal node classification features. The cumulativeGRU module has a powerful ability to extract abnormal gait time features. Through the cumulative design, the hidden features of multiple GRU time correlation calculation units are continuously superimposed, enriching the compensatory feature representation of abnormal gait. The complex graph convolution features extracted by the deeperGCN deep graph convolution network model use two channel branches to obtain abnormal pattern classification features and abnormal node classification features respectively. The abnormal pattern classification features obtain the gait abnormal pattern classification through the diffpool hierarchical network prediction module, and then give the gait pattern classification, which can retain the high generalization characteristics of the cGRU-dGCN model; while the abnormal node classification features obtain the gait abnormal node classification through the CNN deep convolutional neural network prediction module, find out the possible abnormal joint nodes of the abnormal gait pattern, and then give the possible diseased node explanation of the abnormal gait for the user. As different branches in multi-task learning, the gait abnormal pattern classification task and the abnormal node classification task can fit two different tasks at the same time. Finally, the layered relevance propagation (LRP) of the post-hoc solvability method is used for post-hoc interpretation, that is, the decision output of the neural network is assigned contribution values ​​based on the rule of layer-by-layer back-propagation. Each neuron in the neural network will have an impact contribution to the model output. The importance score of each neuron is quantified layer by layer until each wavelength interval of the input sensor signal is calculated, and then an explanation for the model decision is obtained.

[0063] It includes the following steps:

[0064] Step 1, constructing a gait graph describing the topological structure of the human body according to the natural connection mode of the human body, and arranging multiple gait graphs of the same gait pattern in time series to obtain a time series gait graph;

[0065] Furthermore, the gait graph is an unweighted graph G = {v, e} describing the topological structure of the human body, and is an undirected graph constructed according to the natural connection mode of the human body. is a set of joint nodes, n represents the number of lower limb joints, and each lower limb joint is regarded as a node; represents the set of connecting edges between sensor nodes, m represents the number of edges in the gait graph; the gait graph is equivalently represented as G = {X, A}, are all node attributes of the lower limb nodes, A∈{0,1} n×n An adjacency matrix that records the connection relationship between nodes.

[0066] Furthermore, the time sequence gait diagram is The equivalent expression is The abnormal gait recognition problem is regarded as training the mapping function f, using sequence-gait-graphG sgg Predict the gait pattern, defined as:

[0067]

[0068] Among them, S i represents the abnormal gait pattern to be identified, T represents the time length of the time series gait graph, t represents the starting time of the time series gait graph, X t Represents the time series gait graph features at time t.

[0069] Step 2: construct the LRP-MGG neural network model, including the cumulative cGRU model, the deep GCN module (deeperGCN), the DiffPool hierarchical network prediction module, the hierarchical correlation propagation explanation module (LRP) for post-model decision explanation, and the CNN deep convolutional neural network prediction module.

[0070] Step 3: Taking the time-series gait graph as input feature, the gait time-dependent features are extracted through the cumulative GRU block of the LRP-MGG neural network model. The specific expression is as follows:

[0071]

[0072] Where H represents the gait time-dependent features extracted by the cumulative GRU block, h t represents X at timestamp t t Each GRU unit of the cumulative GRU block is regarded as a time window, and each hidden state h t is considered as the output of a sliding window at each timestamp t.

[0073] Step 4: Taking the gait time-dependent features as input, the deep GCN module is used to obtain the local spatial features embedded in the time-dependent feature space;

[0074] The cumulative GRU block and the deep GCN module cooperate with each other to obtain the spatiotemporal gait significant difference feature information hidden in the nonlinear dynamic chain of human gait motion;

[0075] Specifically, assuming that there are temporal dependency features H and adjacency matrix A in gait-graphG, the GCN model can use the temporal dependency features H and adjacency matrix A in non-Euclidean space to extract spatial dependency features between different joint nodes.

[0076] Furthermore, the deep GCN module is constructed using residual connection and pre-activation technology, including two GCN blocks using residual connection, and the two GCN blocks perform feature addition; each GCN block includes layer normalization, ReLU function, Dropout and generalized aggregation network (GEN) set in sequence, layer normalization is used to normalize the input data, which greatly improves the training speed of the neural network; ReLU represents the Sigmoid activation function, and Dropout effectively alleviates the overfitting problem; the GEN of the first GCN block is used to pre-activate the time-dependent feature H and is connected to each residual block to ensure deep training. The generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation function to effectively train deep GCN.

[0077] Step 5, graph-based gait pattern classification is performed through the DiffPool module; that is, by learning differentiable node clustering assignments at each layer of the multi-level pooling network, all nodes are mapped into a set of clustered graph embeddings to perform graph-based abnormal gait pattern classification;

[0078] Specifically, the DiffPool module is constructed to classify graph-based abnormal gait patterns. This block mainly consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). Then, the aggregated cluster is used as the input of the fully connected layer for graph classification. Step 5 specifically includes the following steps:

[0079] Step 5-1, by calculating the input clustering features H and the clustering adjacency matrix A in the deep GNN, the allocation matrix M with differentiable clustering can be generated; the specific calculation formula is as follows:

[0080] M = SoftMax(GNN l,pool (A,h G ) (2-10)

[0081] Step 5-2, by learning the distribution matrix M and the aggregation node features of the differentiable clustering of each layer of the standard GNN, the embedding Z of all aggregated joint nodes can be obtained; the specific calculation formula is as follows:

[0082] Z=GNN l,embed (h G ,M) (2-11)

[0083] Step 5-3, all six joint node embeddings (i.e., graph embeddings) are input into the fully connected layer to accurately predict the graph-based gait pattern. Specifically,

[0084]

[0085] Among them, GNN l,pool represents graph pooling, h G Represents the most abnormal representative spatiotemporal gait graph features, GNN l,embed Indicates the abnormal gait features output by graph pooling, Represents the output aggregation weight matrix, i represents the number of rows in the weight matrix, j represents the number of columns in the weight matrix, represents the weight matrix of the activation function, M represents the clustering assignment matrix of the lth layer of the pooling network, Z represents the aggregate embedding matrix of the graph nodes; k represents the number of graph-based gaits to be classified; SoftMax represents the Softmax function, according to GNN l,embed The output dimension of , in each layer to obtain the maximum number of clusters; represents the label of the graph-based gait pattern to be predicted; Relu represents the Sigmoid activation function.

[0086] Step 6: Using the spatiotemporal significant difference features of gait, the CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality, and the abnormal nodes are visualized to learn and obtain interpretable abnormal gait information contained in the spatiotemporal features;

[0087] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to reduce dimension, extract features and extract local connectivity for the given spatiotemporal feature layer; the last convolutional layer outputs y 3 Input to the fully connected layer to obtain the node prediction vector; among them, the CNN deep convolutional neural network model is given with convolutional layers of different kernel sizes, and any q-th layer gait spatiotemporal feature y q Defined as:

[0088]

[0089] in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, b q represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively;

[0090] The last layer of convolution output y 3Input to the fully connected layer to get the node prediction vector Defined as

[0091]

[0092] Where W fc1 , W fc2 , b fc1 , b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

[0093] Step 7: Use layered relevance propagation (LRP) to calculate the importance score of each input feature for the model output based on the interpretable abnormal gait information in the spatiotemporal features to achieve abnormal gait detection. The specific calculation formula is as follows:

[0094]

[0095] Among them, R j represents the feature importance score of the jth hidden layer node, a j is the activation value of the jth hidden layer node, w jk is the weight connecting the jth hidden layer node and the kth output layer node, R k is the feature importance score of the kth output layer node.

[0096] Furthermore, a small constant ∈ is added to the output of ReLU based on layer-wise relevance propagation (LRP) in step 7 to prevent the gradient from vanishing. This ensures that the importance scores are correctly propagated while maintaining reasonable interpretation.

[0097] Furthermore, the LRP-MGG neural network model performs multi-task learning by combining the gait pattern prediction task and the gait node abnormality prediction task; the cross entropy is used as the loss function for the subtask to evaluate the minimum error between the true gait label and its corresponding label to be predicted during training and testing, which is defined as

[0098]

[0099] Among them, K represents the total number of abnormal gait recognition labels, O represents the total number of abnormal nodes in the gait graph, and Y 1i and Represent the real gait pattern and the corresponding predicted gait pattern respectively; Y 2j and They represent the real abnormal nodes and the corresponding predicted abnormal nodes respectively.

[0100] The present invention also discloses an interpretable abnormal gait detection model based on multi-task graph fusion learning, which includes an additive cGRU model, a deep GCN module (deeperGCN, i.e., a deep graph convolutional network model), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation explanation module (LRP) for post-explanation of model decisions, and a CNN deep convolutional neural network prediction module;

[0101] The cumulative cGRU model includes more than two GRU units that are accumulated in sequence, and each GRU unit is mainly composed of a reset gate rt and an update gate ut; the deep GCN module (deeperGCN) is constructed using residual connection and pre-activation technology, including two GCN blocks using residual connection, and the two GCN blocks perform feature addition (Addition); each GCN block includes layer normalization (Layer normalization), ReLU function, Dropout and generalized aggregation network (GEN) set in sequence, layer normalization is used to normalize the input data, which greatly improves the training speed of the neural network; ReLU represents the Sigmoid activation function, and Dropout effectively alleviates the overfitting problem; the generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation function to effectively train deep GCN, and the GEN of the first GCN block is used to pre-activate the time-dependent feature H and is connected to each residual block to ensure deep training; DiffPool hierarchical network prediction model The block consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). The aggregated cluster is used as the input of the fully connected layer for graph-based gait pattern classification; the CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality from the spatiotemporal significant difference features of gait, and visualize the abnormal nodes so as to learn the interpretable abnormal gait information contained in the spatiotemporal features; the hierarchical relevance propagation interpretation module (LRP) is used to calculate the importance score of each input feature for the model output from the interpretable abnormal gait information in the spatiotemporal features.

[0102] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to perform dimension reduction, feature extraction and local connectivity extraction on the given spatiotemporal feature convolution layer; the CNN deep convolutional neural network model is given convolutional layers with different kernel sizes, and any q-th layer of gait spatiotemporal feature y q Defined as:

[0103]

[0104] in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, bq represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively;

[0105] The last layer of convolution output y 3 Input to the fully connected layer to get the node prediction vector Defined as

[0106]

[0107] Where W fc1 , W fc2 、b fc1 、b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

[0108] Furthermore, the LRP-MGG neural network model performs multi-task learning by combining the gait pattern prediction task and the gait node abnormality prediction task; the cross entropy is used as the loss function for the subtask to evaluate the minimum error between the true gait label and its corresponding label to be predicted during training and testing, which is defined as:

[0109]

[0110] Among them, K represents the total number of abnormal gait recognition labels, O represents the total number of abnormal nodes in the gait graph, and Y 1i and Represent the real gait pattern and the corresponding predicted gait pattern respectively; Y 2j and They represent the real abnormal nodes and the corresponding predicted abnormal nodes respectively.

[0111] The specific principle of the present invention is described in detail below:

[0112] Defining abnormal gait patterns:

[0113] Definition 1: Gait graph G: The present invention uses an unweighted graph gait-graphG = {v, e} to describe the topological structure of the human body, and gait-graphG is an undirected graph constructed according to the natural connection mode of the human body. The present invention regards each lower limb joint as a node. is a set of joint nodes, and n represents the number of lower limb joints. represents the set of connecting edges between sensor nodes, where m represents the number of edges in gait-graphG. Based on the nature of human gait motion, the edge relationship of the graph is constructed in a natural connection manner. At the same time, since motion compensation is interactive, gait-graphG is an undirected graph. The accelerometer and gyroscope data collected by each node are used as node v i Attribute characteristics p represents the number of node attribute features, (as a feasible implementation, a single sensor uses a 3-axis accelerometer and a 3-axis gyroscope to collect data, so p = 6). A given gait-graph G can be represented by an adjacency matrix A∈{0,1} n×n Indicates the connection relationship between record nodes. i,j Represents node v i and node v i The connection relationship between j,i =0 indicates node v j and node v i There is no connection between A j,j =1 indicates node v i and node v j There are connections between them. gait-graphG can be equivalently expressed as G = {X, A}, X = These are all the node attributes of the lower limb nodes.

[0114] Definition 2: Sequence-gait-graph G sgg . The present invention uses a multi-sensor network to form a sequence-gait-graphG sgg The gait pattern is represented by a sequence-gait-graph, which consists of multiple gait-graphs of the same gait pattern arranged in time series, represented as The graph structure of human motion is fixed, that is, the adjacency matrix remains unchanged during the motion, and the only part that changes is the attributes of the nodes. Therefore, the sequence-gait-graph can be equivalently expressed as The abnormal gait recognition problem can be regarded as training the mapping function f, using sequence-gait-graphG sgg Predict the gait pattern, defined as:

[0115]

[0116] Among them, S i represents the abnormal gait pattern to be identified, T represents the time length of the time series gait graph, t represents the starting time of the time series gait graph, X t Represents the time series gait graph features at time t.

[0117] Construct the LRP-MGG neural network model, including the cumulative cGRU model, the deep GCN module (deeperGCN), the DiffPool hierarchical network prediction module, the hierarchical relevance propagation explanation module (LRP) for post-explanation of model decisions, and the CNN deep convolutional neural network prediction module.

[0118] Accumulative GRU module: Gated recurrent neural network (GRU) is a powerful tool based on recurrent neural network (RNN) to capture temporal dependency features in time series data. In the GRU model, previous calculations can be used in recursive calculations. GRU can solve problems such as gradient disappearance, gradient explosion, and long-term memory loss. Unlike the long short-term memory (LSTM) unit, which includes an input gate, a forget gate, and an output gate, the GRU unit mainly consists of a reset gate r t and update gate u t The GRU model has a relatively simple structure and fewer parameters, which is helpful for model training and testing. sgg For each GRU unit at the current time t, assume that at the previous time t-1 there is a hidden state h from the previously calculated gait timing dependency feature information t-1 and sequence-gait-graphG sgg Gait characteristics X t Input, reset gate function r t and update gate function u t It is used to obtain relevant temporary information of gait characteristics between the current time t and the previous time t-1 by calculating equations (2-1) and (2-2).

[0119] r t =σ(W r [X t ,h t-1 ]+b r ) (2-2)

[0120] r t =σ(W u [X t ,h t-1 ]+b u ) (2-3)

[0121] Among them, σ represents the sigmoid activation function; b r and b u is the bias vector learned during training; W r and W u is the weight matrix learned during the training process. Then, based on the features obtained from the reset gate function, the candidate hidden states By calculating the following equation (3-4) at the current time t, we can obtain the value from X t Useful gait feature timing information:

[0122]

[0123] Where tanh represents the hyperbolic tangent activation function; ⊙ is the Hadamard product; and are the weight matrix and bias vector learned during training. Next, based on the candidate hidden state and from the update gate u t The obtained features are defined by calculating the following equation (3-5) to define the updated candidate hidden state To capture the valuable gait temporary dependency features between the current time t and the previous time t-1:

[0124]

[0125] It should be noted here that when u t When the value of is close to 1, more previous temporary dependency feature information is remembered; on the contrary, when u t As the value of approaches 0, irrelevant previous temporal feature information disappears. Therefore, by computing the GRU unit at each timestamp within the time series q, the time series multi-sensor gait data can be Intrinsic long-term dependencies among multiple lower limb joints that are associated with abnormal gait changes were uncovered.

[0126] Since the interactions among multiple joints of the lower limbs are synchronized under the control of the central nervous system, the X t The hidden state h t may contain useful temporal dependency features related to abnormal gait changes. In order to find more useful temporal dependencies in the proposed method, we construct an additive GRU block, where each GRU unit is regarded as a time window and each hidden state h t is considered as the output of a sliding window at each timestamp t, and X at all timestamps t t The hidden state h t By definition

[0127]

[0128] Where H represents the gait time-dependent features extracted by the cumulative GRU block, h t represents X at timestamp t t Each GRU unit of the cumulative GRU block is regarded as a time window, and each hidden state h t is considered as the output of a sliding window at each timestamp t.

[0129] deeperGCN module: Graph convolutional network (GCN) is one of the most representative graph neural network models, which can apply the representation learning ability of convolutional neural network to irregular graph data. Its advantage is to generalize the convolution operation to non-Euclidean graph data for spatial feature extraction. In the GCN model, the graph convolution operation generates a richer feature representation for the target vertex by aggregating the features of the target vertex and its adjacent vertices. The definition of the graph convolution operation in the spatial GCN depends on the spatial relationship between the graph nodes. The present invention constructs a GCN model to extract spatial features based on the topological relationship between the joint node and its adjacent nodes, and its features can be extracted by aggregating its own features and the features of the two adjacent nodes in the GCN model. That is, assuming that there are temporal dependency features H and adjacency matrix A in gait-graphG, the GCN model can use the temporal dependency features H and adjacency matrix A in non-Euclidean space to extract spatial dependency features between different joint nodes. In the GCN model constructed by superimposing multiple convolutional layers, the solution to spatial feature extraction can be defined as:

[0130]

[0131] in, is the adjacency matrix with self-connection added, I N is the identity matrix. is the degree matrix, all elements except the diagonal are 0. H is the feature map of each layer. When the GCN layer l = 0, H (l) =H. W is the weight matrix that needs to be learned during the training process. Through the multi-layer graph convolutional network, the spatial features embedded in the time-dependent features H of the joint nodes can be learned. When the GCN model developed during the training process has the best learning ability, more useful spatial features of each joint node can be discovered by aggregating neighboring joint nodes. However, the existing GCN architecture can only train shallow GCNs. The increase in graph convolutional layers can lead to problems such as gradient vanishing, over-smoothing, and overfitting. This may lead to a decrease in the learning ability of exploring the most representative spatial features using shallow GCN models. Therefore, it is necessary to develop deeper GCNs with stronger learning performance.

[0132] To this end, the present invention utilizes the nonlinear spatial feature learning properties of the deep GCN deep graph convolutional network to help the model capture the nonlinear dynamic chain information hidden in the human gait motion. That is, the time-dependent features extracted by the GRU are used as the input of the deeper GCN module to mine the local spatial features embedded in the time-dependent feature space. This can be seen from the sequence-gait-graphG sgg In view of the topological structure of the gait graph, the cumulative GRU converts the sequence-gait-graphGsgg Aggregate into a gait-graphG. Here, the deeper GCN module is constructed with residual connection and pre-activation techniques to achieve excellent learning ability for more useful spatial dependent features in gait-graphG. In this module, the generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation function to effectively train deep GCN. Based on GEN, the deeper GCN consists of a sequence of layer normalization (Layer normalization) → ReLU → GEN → Addition, in which layer normalization is used to normalize the input data, greatly improving the training speed of the neural network. ReLU represents the Sigmoid activation function. Dropout effectively alleviates the overfitting problem. The first GEN is used to pre-activate the time-dependent feature H and is connected to each residual block to ensure deep training. To this end, the spatiotemporal gait significant difference feature information hidden in the nonlinear dynamic chain in the human gait motion can be obtained.

[0133] DiffPool hierarchical network predicts abnormal joint nodes: After extracting the most representative spatiotemporal features from all joint nodes, a graph-based gait classification task is performed. In recent related studies, DiffPool technology has been successfully applied to high-quality graph classification tasks. A DiffPool module is constructed to perform graph-based gait pattern classification. It aims to map all nodes into a set of clusters by learning differentiable node clustering assignments at each layer of a multi-level pooling network. Its advantage is that the feature information of all nodes is aggregated into a graph embedding through a fully connected layer for graph classification. In the model of the present invention, a DiffPool module is constructed to classify abnormal gait patterns based on graphs. The block is mainly composed of a pooling network and a fully connected layer, and the pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). Then, the aggregated clusters are used as input to the fully connected layer for graph classification. The above solution can be obtained by calculating the equation defined below.

[0134] M=SoftMax(GNN l,pool (A,h G ) (2-10)

[0135] Z=GNN l,embed (h G ,M) (2-11)

[0136]

[0137] Among them, GNN l,pool represents graph pooling, h G Represents the most abnormal representative spatiotemporal gait graph features, GNN l,embedIndicates the abnormal gait features output by graph pooling, Represents the output aggregation weight matrix, i represents the number of rows in the weight matrix, j represents the number of columns in the weight matrix, represents the weight matrix of the activation function, M represents the clustering assignment matrix of the lth layer of the pooling network, and Z represents the aggregate embedding matrix of the graph nodes. k represents the number of graph-based gaits to be classified. SoftMax represents the Softmax function. According to GNN l,embed The output dimension of , can obtain the maximum number of clusters in each layer. Represents the label of the graph-based gait pattern to be predicted. Relu represents the Sigmoid activation function. Based on equation (2-11), by calculating the input clustering features H and the clustering adjacency matrix A in the deep GNN, a distribution matrix M with differentiable clustering can be generated. Then, according to equation (2-11), by learning the distribution matrix M and the aggregated node features of the differentiable clustering of each layer of the standard GNN, all aggregated joint node embeddings Z can be obtained. Finally, based on equation (2-12), two fully connected layers are constructed for accurately predicting graph-based gait patterns, and all six joint node embeddings (i.e., graph embeddings) are input into the fully connected layers.

[0138] CNN deep convolutional neural network predicts abnormal joint nodes: Different from the previous sensor-based abnormal gait recognition XAI task, the present invention uses a deep convolutional neural network (CNN) model to obtain the joint nodes that are most likely to be abnormal, and visualizes the abnormal nodes, which is conducive to the model outputting easy-to-understand and explainable abnormal gait information. The present invention uses the significant spatiotemporal difference features of gait obtained by cGRU and dGCN to further learn the explainable information contained in the spatiotemporal features. CNN is an excellent motion data feature extractor with the characteristics of solving sensor data classification problems. The present invention uses a shallow number of layers to facilitate model learning to identify simpler node classification features and reduce the risk of overfitting. The present invention designs three convolutional layers to perform dimensionality reduction, feature extraction, and extraction of local connectivity on a given spatiotemporal feature accumulation layer. That is, given convolutional layers with different kernel sizes, q (q = 1, 2, 3) layers of gait spatiotemporal features y q Defined as:

[0139]

[0140] in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, b q represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively;

[0141] The last layer of convolution output y 3 Input to the fully connected layer to get the node prediction vector Defined as

[0142]

[0143] Where W fc1 , W fc2 、b fc1 、b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

[0144] LRP hierarchical relevance propagation explains model decisions: The present invention uses hierarchical relevance propagation (LRP) to predict explanations as a propagation-based post-hoc method to provide explanations in the input space, which is the space where experts usually interpret signals in clinical practice. The back-propagation process of the correlation is shown in the figure. LRP traverses the hierarchical structure of the LSTM model in reverse to produce an explanation. LRP calculates the importance score of each input feature for the model output by backward propagation. It propagates the importance of the model output layer by layer, from the output layer to the input layer, and assigns a score to each input feature. In this way, LRP can tell the present invention which input signal areas play a positive or negative role in the model's decision. However, the explanation obtained by LRP is too complex, so there may be a lot of noise. In order to ignore those less important neurons, the present invention uses LRP-∈ (Layer-wise Relevance Propagation with epsilon). In LRP-∈, a small constant ∈ is added to the output of ReLU to prevent the gradient from disappearing. This ensures that the importance score is correctly propagated while maintaining the rationality of the explanation, defined as

[0145]

[0146] Among them, R j represents the feature importance score of the jth hidden layer node, a j is the activation value of the jth hidden layer node, w jk is the weight connecting the jth hidden layer node and the kth output layer node, R k is the feature importance score of the kth output layer node.

[0147] Multi-task model loss function: The multi-task learning designed in the present invention allows the machine learning model to learn multiple tasks at the same time to improve the performance and efficiency of the model. Combining the training of multiple tasks reduces the amount and complexity of data, helps the model to better learn the two tasks of gait recognition and interpretability, and makes the model more robust on different tasks, thereby improving the generalization and prediction capabilities of the model. Multi-task learning is performed by combining the gait pattern prediction task and the gait node abnormality prediction task. For the subtask, the cross entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, which is defined as

[0148]

[0149] Among them, K represents the total number of abnormal gait recognition labels, O represents the total number of abnormal nodes in the gait graph, and Y 1i and Represent the real gait pattern and the corresponding predicted gait pattern respectively; Y 2j and They represent the real abnormal nodes and the corresponding predicted abnormal nodes respectively.

[0150] The present invention adopts the above technical scheme, uses the developed cGRU-dGCN model as the backbone model, and outputs the abnormal pattern classification features and abnormal node classification features through two different branches of the graph convolution features containing complex compensation information of the human body. The cumulativeGRU module has a powerful abnormal gait time feature extraction capability, and continuously superimposes the hidden features of multiple GRU time correlation calculation units through the cumulative design, enriching the compensatory feature representation of abnormal gait. The complex graph convolution features extracted by the deep GCN module (deeperGCN) use two channel branches to obtain abnormal pattern classification features and abnormal node classification features respectively. The abnormal pattern classification features obtain the gait abnormal pattern classification through the DiffPool hierarchical network prediction module, and then give the gait pattern classification, which can retain the high generalization characteristics of the cGRU-dGCN model; and the abnormal node classification features obtain the gait abnormal node classification through the CNN deep convolutional neural network prediction module, find out the possible abnormal joint nodes of the abnormal gait pattern, and then give the possible diseased node explanation of the abnormal gait for the user. As different branches in multi-task learning, the gait abnormal pattern classification task and the abnormal node classification task can fit two different tasks at the same time. Finally, the layered relevance propagation (LRP) of the post-hoc solvability method is used for post-hoc interpretation, that is, the decision output of the neural network is assigned contribution values ​​based on the rule of layer-by-layer back-propagation. Each neuron in the neural network will have an impact contribution to the model output. The importance score of each neuron is quantified layer by layer until each wavelength interval of the input sensor signal is calculated, and then an explanation for the model decision is obtained.

[0151] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.

Claims

1. An interpretable abnormal gait detection method based on multi-task graph fusion learning, characterized by: It includes the following steps: Step 1, constructing a gait graph describing the topological structure of the human body according to the natural connection mode of the human body, and arranging multiple gait graphs of the same gait pattern in time series to obtain a time series gait graph; Step 2: construct the LRP-MGG neural network model, including the cumulative cGRU model, the deep GCN module, the DiffPool hierarchical network prediction module, the hierarchical correlation propagation interpretation module LRP, and the CNN deep convolutional neural network prediction module. Step 3, using the time-series gait graph as input features, extracting gait time-dependent features through the cumulative GRU block of the LRP-MGG neural network model; Step 4: Taking the gait time-dependent features as input, the deep GCN module is used to obtain the local spatial features embedded in the time-dependent feature space; Step 5, graph-based gait pattern classification is performed through the DiffPool module; that is, by learning differentiable node clustering assignments at each layer of the multi-level pooling network, all nodes are mapped into a set of clustered graph embeddings to perform graph-based abnormal gait pattern classification; Step 6: Using the spatiotemporal significant difference features of gait, the CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality, and the abnormal nodes are visualized to learn and obtain interpretable abnormal gait information contained in the spatiotemporal features; In step 7, the interpretable abnormal gait information in the spatiotemporal features is used to calculate the importance score of each input feature for the model output using layered correlation propagation (LRP) to achieve abnormal gait detection.

2. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 1 is characterized in that: The gait graph is an unweighted graph G = {v, e} that describes the topological structure of the human body and is an undirected graph constructed according to the natural connection mode of the human body. is a set of joint nodes, n represents the number of lower limb joints, and each lower limb joint is regarded as a node; represents the set of connecting edges between sensor nodes, m represents the number of edges in the gait graph; the gait graph is equivalently represented as G = {X, A}, are all node attributes of the lower limb nodes, A∈{0,1} n×n An adjacency matrix that represents the connection relationship of the record nodes; the time sequence gait graph is The equivalent expression is The abnormal gait recognition problem is viewed as training a mapping function f using the predicted gait pattern, defined as: Among them, S i represents the abnormal gait pattern to be identified, T represents the time length of the time series gait graph, t represents the starting time of the time series gait graph, X t Represents the time series gait graph features at time t.

3. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 2 is characterized in that: The specific expression for extracting the gait time-dependent features in step 3 is as follows: Where H represents the gait time-dependent features extracted by the cumulative GRU block, h t represents X at timestamp t t Each GRU unit of the cumulative GRU block is regarded as a time window, and each hidden state h t is considered as the output of a sliding window at each timestamp t.

4. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 2 is characterized in that: In step 4, the deep GCN module is constructed using residual connection and pre-activation technology, including two GCN blocks using residual connection, and the two GCN blocks perform feature addition; The CN model can use the temporal dependency feature H and the adjacency matrix A in the non-Euclidean space to extract the spatial dependency features between different joint nodes; each GCN block includes layer normalization, ReLU function, Dropout and generalized aggregation network GEN set in sequence. Layer normalization is used to normalize the input data, which greatly improves the training speed of the neural network; ReLU represents the Sigmoid activation function, and Dropout alleviates the overfitting problem; The GEN of the first GCN block is used to pre-activate the time-dependent features H and is connected with each residual block to ensure deep training.

5. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 2 is characterized in that: In step 5, the DiffPool module is constructed to classify graph-based abnormal gait patterns. The DiffPool module consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster. The aggregated cluster is used as the input of the fully connected layer for graph classification. Specifically, the following steps are included: Step 5-1, by calculating the input clustering features H and the clustering adjacency matrix A in the deep GNN, a distribution matrix M with differentiable clustering is generated; the specific calculation formula is as follows: M=SoftMax(GNN l,pool (A,h G ) (2-10) Step 5-2, by learning the distribution matrix M and the aggregation node features of the differentiable clustering of each layer of the standard GNN, the embedding Z of all aggregated joint nodes is obtained; the specific calculation formula is as follows: Z=GNN l,embed (h G ,M) (2-11) In step 5-3, all six joint node embeddings are input into the fully connected layer to accurately predict the graph-based gait pattern. Specifically, Among them, GNN l,pool represents graph pooling, h G Represents the most abnormal representative spatiotemporal gait graph features, GNN l,embed Indicates the abnormal gait features output by graph pooling, Represents the output aggregation weight matrix, i represents the number of rows in the weight matrix, j represents the number of columns in the weight matrix, represents the weight matrix of the activation function, M represents the clustering assignment matrix of the lth layer of the pooling network, Z represents the aggregate embedding matrix of the graph nodes; k represents the number of graph-based gaits to be classified; SoftMax represents the Softmax function, according to GNN l,embed The output dimension of , in each layer to obtain the maximum number of clusters; represents the label of the graph-based gait pattern to be predicted; Relu represents the Sigmoid activation function.

6. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 2 is characterized in that: The CNN deep convolutional neural network model uses three convolutional layers to reduce dimension, extract features and extract local connectivity for a given spatiotemporal feature layer, and converts the last convolutional layer output y 3 Input to the fully connected layer to obtain the node prediction vector; among them, the CNN deep convolutional neural network model is given with convolutional layers of different kernel sizes, and any q-th layer gait spatiotemporal feature y q Defined as: in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, b q represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively; The last layer of convolution output y 3 Input to the fully connected layer to get the node prediction vector Defined as Where W fc1 , W fc2 , b fc1 , b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

7. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 2 is characterized in that: In step 7, a small constant ∈ is added to the output of ReLU based on layer-wise correlation propagation (LRP) to prevent the gradient from disappearing; the specific calculation formula is as follows: Among them, R j represents the feature importance score of the jth hidden layer node, a j is the activation value of the jth hidden layer node, w jk is the weight connecting the jth hidden layer node and the kth output layer node, R k is the feature importance score of the kth output layer node.

8. An interpretable abnormal gait detection model based on multi-task graph fusion learning, according to the interpretable abnormal gait detection method based on multi-task graph fusion learning according to any one of claims 1 to 7, characterized in that: The model includes the cumulative cGRU model, the deep GCN module, the DiffPool hierarchical network prediction module, the hierarchical correlation propagation interpretation module LRP, and the CNN deep convolutional neural network prediction module; The cumulative cGRU model includes two or more GRU units that are accumulated in sequence. Each GRU unit is mainly composed of a reset gate r t and update gate u t Composition: The deep GCN module is constructed using residual connection and pre-activation technology, including two GCN blocks using residual connection, and the two GCN blocks perform feature addition; each GCN block includes layer normalization, ReLU function, Dropout and generalized aggregation network (GEN) set in sequence. Layer normalization is used to normalize the input data, which greatly improves the training speed of the neural network; ReLU represents the Sigmoid activation function, and Dropout effectively alleviates the overfitting problem; The generalized aggregation network GEN layer is a GCN layer with enhanced message aggregation function to effectively train deep GCN. The GEN of the first GCN block is used to pre-activate the time-dependent feature H and is connected with each residual block to ensure deep training; The DiffPool hierarchical network prediction module consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster, and the aggregated cluster is used as the input of the fully connected layer for graph-based gait pattern classification. The CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality from the spatiotemporal significant difference features of gait, and visualize the abnormal nodes in order to learn and obtain interpretable abnormal gait information contained in the spatiotemporal features. The hierarchical relevance propagation explanation module LRP is used to calculate the importance score of each input feature for the model output from the interpretable abnormal gait information in the spatiotemporal features.

9. The interpretable abnormal gait detection model based on multi-task graph fusion learning according to claim 8, characterized in that: The CNN deep convolutional neural network model uses three convolutional layers to reduce dimension, extract features and extract local connectivity for a given spatiotemporal feature layer. The CNN deep convolutional neural network model gives convolutional layers with different kernel sizes, and any q-th layer of gait spatiotemporal feature y q Defined as: in, Represents the input matrix; Relu() represents the activation function, <·> represents the inner product, represents the convolution kernel vector, b q represents the offset matrix; the values ​​of q are 1, 2, and 3 respectively; The last layer of convolution output y 3 Input to the fully connected layer to get the node prediction vector Defined as Where W fc1 , W fc2 , b fc1 , b fc2 They are the weight matrix and kernel offset matrix of the two fully connected layers respectively.

10. The interpretable abnormal gait detection model based on multi-task graph fusion learning according to claim 8, characterized in that: The LRP-MGG neural network model performs multi-task learning by combining the gait pattern prediction task and the gait node abnormality prediction task; For the subtask, cross entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as Among them, K represents the total number of abnormal gait recognition labels, O represents the total number of abnormal nodes in the gait graph, and Y 1i and Represent the real gait pattern and the corresponding predicted gait pattern respectively; Y 2j and They represent the real abnormal nodes and the corresponding predicted abnormal nodes respectively.

Images